Simulation-oriented multi-agent maneuvering behavior decision system and method

Abstract

The invention discloses a simulation-oriented multi-agent maneuvering behavior decision-making system and method, and the system comprises a message transmission module which introduces the signal intensity into the communication among multiple agents, simulates a complex simulation environment with interference in a probabilistic communication mode for the discrete quantification of the communication intensity among the agents, and transmits the signal intensity to a decision-making module; obtaining joint state information influencing a maneuvering behavior decision; the state coding module is used for discretely quantifying the obtained joint state information into a unified index and coding the unified index to obtain a joint state code corresponding to each agent; the reward optimization module is used for determining a target reward of each agent in combination with the task target and the historical trajectory of each agent, and the target reward is used for optimizing the deep reinforcement learning network model of the strategy execution module; and the strategy execution module is used for determining and executing a final decision result of each agent by adopting a deep reinforcement learning network model based on an MADDPG algorithm by each agent. According to the invention, the problem of multi-agent collaborative navigation in a complex high-dimensional environment is solved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent agents, and in particular to a simulation-oriented multi-agent maneuvering behavior decision-making system and method. Background Art

[0002] Generally speaking, the physical maneuvering behavior in simulation mainly refers to the decision-making of the agent according to the task requirements in the simulated environment, which in turn causes the change of its spatial position. The environmental factors affecting this process include: obstacles, terrain, weather conditions and degree of damage. In addition to the need to analyze the environmental factors faced by single-agent decision-making, multi-agents themselves have different state representation methods from single-agent maneuver models. At the same time, multi-agent collaboration also faces communication problems, which include analyzing communication elements and quantifying communication range. In addition, multi-agent collaboration is often to achieve more complex tasks, and it is expected that the control between agents can achieve overall optimization. Therefore, there are many factors to consider when developing the collaborative strategy algorithm between multiple agents, and the requirements for the effectiveness of the algorithm will also be higher.

[0003] The existing technology has problems such as low convergence speed, unstable algorithm effect, and different usage scenarios. For example, the Chinese patent with authorization number CN113218400B and name of a multi-agent navigation algorithm based on deep reinforcement learning integrates the A* algorithm into the PPO algorithm, and uses the designed reward and punishment function to achieve deep fusion of the two algorithms. The agent inputs the original image data of the sensor and makes decisions to plan the best action path. The specific definition of the reward and punishment function of the algorithm is: rt = rdecision + rcondition + rtime. rt is the total score, rdecision is the reward score obtained by the agent's active selection algorithm, rcondition is the collision score, and rtime is the score deducted for the consumed time. This algorithm can control the amplitude of policy updates through methods such as pruning agents and adaptive penalty coefficients, but in high-dimensional environments, such as system simulation environments, it will lead to instability in policy updates and a decrease in convergence speed. In addition, the target application scenarios of this method are simple scenarios such as automated sorting warehouses, which cannot meet the needs of system simulation, because more environmental factors need to be considered in system simulation, and multi-agent system simulation needs to consider group collaboration capabilities more, not just the obstacle avoidance capabilities and shortest path planning that existing technologies focus on. Summary of the invention

[0004] The purpose of the present invention is to provide a simulation-oriented multi-agent maneuvering behavior decision-making system and method, which solves the problem of multi-agent collaborative navigation in a complex high-dimensional environment in the background technology.

[0005] The technical solution to achieve the purpose of the present invention is:

[0006] A simulation-oriented multi-agent maneuver behavior decision system, comprising:

[0007] The message passing module introduces signal strength into the communication between multiple agents, discretely quantifies the communication strength between agents, and adopts probabilistic communication to simulate a complex simulation environment with interference, and obtains joint state information that affects the decision-making of maneuvering behavior;

[0008] The state encoding module discretizes and quantizes the obtained joint state information into a unified indicator and encodes it to obtain the joint state code corresponding to each intelligent agent;

[0009] The reward optimization module combines the task objectives and the historical trajectories of each agent to determine the target reward of each agent, and the target reward is used to optimize the deep reinforcement learning network model of the strategy execution module;

[0010] In the strategy execution module, each agent uses a deep reinforcement learning network model based on the MADDPG algorithm to determine the final decision results of each agent and execute them.

[0011] Furthermore, the joint state information includes the state information of the agent itself, the environmental information of its own perception range, and the state information of other agents.

[0012] Furthermore, the mathematical model of the state information is:

[0013] S t =(S self ,S others ,S env )

[0014] Where S self Represents its own state information, including its current velocity vector and current location information S others Represents the state information of other agents, S env Indicates environmental information.

[0015] Furthermore, the target reward is:

[0016] R=w1·r time +w2·r distance +w3r goal +w4·r collison +w5·r overlap

[0017] Among them, r time represents the execution time reward for the current agent to complete the task, r distance Reward for distance change, r goalrepresents the reward for the first scout of the destination, r collison represents the reward for the collision between the agent and the obstacle, r overlap represents the reward generated by the collision between agents, w i Represents the coefficient of the i-th reward, i=1, 2, 3, 4, 5.

[0018] Furthermore, the deep reinforcement learning network model takes the task value maximization of three types of maneuvering behaviors, namely, transforming target maneuvering, fixed coordinate maneuvering, and targetless maneuvering, as the goal, and the task value is:

[0019] value=-(D×w D +T×w T +C×w C )

[0020] Where D represents the relative distance between the agent and the target, T represents the total time to complete the task, C represents the number of collisions with obstacles during the entire task, and w D 、w T 、w C They represent the weights corresponding to D, T, and C respectively.

[0021] Furthermore, the signal strength is:

[0022] RSSI = Tx Power +Path Loss +Rx Gain +Tx Gain

[0023] Among them, RSSI represents the received power, that is, the signal strength, Tx Power Indicates the transmit power, Path Loss Represents the path loss, Rx Gain Indicates receiving antenna gain; Tx Gain Represents the transmit antenna gain.

[0024] Furthermore, the relationship between the path loss and the distance between the communicating parties is:

[0025]

[0026] Among them, d is the distance between the two communicating parties, G1 is a constant, and λ represents the wavelength of the frequency band used in analog communication.

[0027] Furthermore, the message transmission module establishes a packet loss rate prediction model according to the relationship between signal strength and packet loss rate, so as to estimate the packet loss rate under a specific signal strength.

[0028] Furthermore, the deep reinforcement learning network model includes an Actor network and a Critic network. The Actor network gives decision actions, and the Critic network gives evaluations based on the joint states and actions of all agents.

[0029] A simulation-oriented multi-agent maneuver behavior decision-making method, comprising the steps of:

[0030] Acquire joint state information that affects maneuver behavior decisions through message passing modules;

[0031] The state encoding module discretizes and quantifies the obtained joint state information into a unified indicator and encodes it, obtains the joint state code corresponding to each intelligent agent, and sends the encoded state to the strategy execution module;

[0032] The final decision results of each agent are determined and executed through the strategy execution module;

[0033] The deep reinforcement learning network model of the strategy execution module is optimized and updated through the reward optimization module.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) According to the characteristics of the simulation environment, the present invention analyzes the environmental factors that affect the decision-making of maneuvering behavior, such as obstacles, topography, weather conditions and damage degree, quantifies each factor by level and discretely sets the pass coefficient for each level, so as to reduce the dimension of the state space. At the same time, considering the communication problem faced by multi-agent collaboration, the strategy of probabilistic communication is adopted to exchange information during the simulation process; secondly, based on the mobility of the simulated equipment, a continuous action space is designed; finally, based on the data type and action space transmitted in the state space, the strategy network Actor and the value network Critic are designed to achieve collaborative decision-making with time efficiency, path optimization and collision avoidance;

[0036] (2) Based on the real data provided by OpenStreetMap, this paper constructs a simulation map to provide experiments and verification for the multi-agent reinforcement learning algorithm;

[0037] (3) The present invention can unify multiple environmental factors into an evaluation system by discretely setting the pass coefficient, so that more environmental factors can be considered, solving the problem that the existing technology has relatively simple usage scenarios; by describing the signal strength and packet loss rate during the communication process in segments, it can simulate a complex environment with communication interference, which also helps the model learn better strategies during the training process;

[0038] (4) The present invention significantly improves the collaboration effect between intelligent agents by solving the collaborative navigation problem in complex environments, and provides efficient collaborative navigation algorithm model support for system simulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Construct a flow chart for a multi-agent maneuvering behavior decision system.

[0040] Figure 2 This is a framework diagram of the maneuvering behavior algorithm based on MADDPG.

[0041] Figure 3 This is a graph showing the transformation of the global average reward during training.

[0042] Figure 4 The graph shows the experimental results. DETAILED DESCRIPTION

[0043] This embodiment provides a simulation-oriented multi-agent maneuvering behavior decision system, such as Figure 1 As shown, including:

[0044] The state coding module analyzes the factors that affect the maneuvering behavior decision and quantifies each factor into a unified index;

[0045] The message passing module introduces the concept of communication strength into the communication problem between multiple agents, and adopts the probabilistic communication method to discretely quantify the communication strength between agents and simulate the complex simulation environment with interference;

[0046] The strategy execution module determines and executes the final decision results of each agent based on the output of the decision model;

[0047] The reward optimization module combines the task objectives and the historical trajectories of each agent to determine the target reward for each agent.

[0048] Figure 2 The schematic diagram of the multi-agent maneuver behavior decision algorithm framework provided by the present invention is as follows: Figure 2 As shown in the figure, the agent obtains state information from the simulation environment and discretizes and quantizes the original environment data based on the state encoding module. The agent can be any entity unit that can obtain simulation environment information, such as a virtual armored vehicle in the simulation.

[0049] The joint state of each agent includes the state information of the agent itself, the environmental information of its own perception range, and the state information of other agents. The state information of other agents obtains the corresponding encoding state according to the message transmission module. According to the probabilistic communication strategy of the message transmission module, it is calculated whether the two communicating agents can obtain each other's state information, and the exchanged information is encoded based on the state encoding module to obtain the joint state encoding corresponding to each agent.

[0050] The joint state code corresponding to each agent is input into the decision model to obtain the maneuvering behavior decision result of each agent, including the speed and angular velocity of each agent. Each agent executes the decision result and generates new state information. The neural network parameters of the strategy execution module are updated by deep reinforcement learning.

[0051] The design process of the state encoding module specifically includes:

[0052] Analyze environmental factors that affect the completion of the task and quantify them into information that the computer can understand to realize the input of the algorithm. Environmental factors include obstacles, topography, weather conditions and damage degree.

[0053] Construct atomic actions in simulation tasks. In simulation tasks, the simulation entity needs to adjust its maneuvering direction and speed within each simulation step. These basic actions are called atomic actions. They are the final decision results of the algorithm and the maneuvering behavior of the simulation entity. It is very important to clarify its parameter range for the design of subsequent action space.

[0054] Finally, in order to achieve the fusion of multiple models, common maneuver models were analyzed and divided into three categories, and the corresponding mathematical features were extracted to assist the subsequent reward function design and calculation.

[0055] Table 1 Correspondence between weather conditions and traffic difficulty

[0056]

[0057] Specifically, the problem is described as a mathematical model, where the state S of the agent at time t is t It consists of three parts:

[0058] S t =(S self ,S others ,S env )

[0059] Where S self Indicates its own status information, S others Represents the state information of other agents, S env Represents environmental information. As shown in Table 1, the state space can be reduced by unifying weather conditions as the difficulty of travel instead of inputting actual data in the state space. The action space defines the speed adjustment of the agent in the x and y directions, allowing the agent to move in two-dimensional space.

[0060] The status information is:

[0061]

[0062] The state of the agent includes its velocity vector at the current moment. and current location information

[0063] At the same time, the information that the agent has about other agents, taking location information as an example, can be expressed as:

[0064]

[0065] On the other hand, terrain directly affects the mobility of equipment. Vector map data is used to represent surface information in a simulation environment, including slope and landform type. By quantifying these features, an environmental data description is formed. The greater the pass coefficient, the more difficult the maneuver, and the greater the influencing factor, the higher its importance. Weather conditions such as rainfall and wind will also affect mobility. This patented technical solution lists the relationship between rainfall and wind levels and difficulty of passage. Hostile actions on the battlefield, such as signal interference and fire strikes, can cause varying degrees of damage, affecting the mobility of equipment until it is completely destroyed and loses its ability to move.

[0066] The objective function used by the reward optimization module is:

[0067] R=w1·r time +w2·r distance +w3r goal +w4·r collison +w5·r overlap

[0068] This objective function calculates the total cost based on different factors. time represents the execution time reward for the current agent to complete the task, r distance Reward for distance change, r goal represents the reward for the first scout of the destination, r collison represents the reward for the collision between the agent and the building, r overlap represents the reward generated by the collision between agents, w i Represents the coefficient of the i-th reward.

[0069] In the simulation task, the maneuvering target of the agent is directly given by the simulation plan, and the target information is a spatial coordinate or entity unit. The purpose of maneuvering generally includes: target following, moving toward the target point or target area, avoiding obstacles, etc. Therefore, the above situations can basically be divided into: target transformation maneuvering, fixed coordinate maneuvering and targetless maneuvering.

[0070] When transforming the target type maneuver, the target is regarded as a mass point, and its coordinate transformation is expressed as Represents the change of its spatial position over time. The current coordinates of the agent are (A x, A y ). Therefore, the main goal of this type of task is to keep the relative distance between the agent and the target as small as possible within the task time. The relative distance between the agent and the target is:

[0071]

[0072] Fixed target class maneuvers are different from transform target classes. The coordinates of fixed target classes do not change and are represented as (X, Y), (A x , A y ) represents the current coordinates of the agent, (V x , V y ) represents its current speed. The goal of this task is to complete the task in the shortest total time, and the total time to complete the task is:

[0073]

[0074] There is no target-based maneuver, only the interaction between the agent and the environment. Taking the agent obstacle avoidance navigation model as an example, Represents the spatial coordinates of each obstacle in the environment. (A x , A y ) represents the current coordinates of the agent. The main goal of this type of task is to minimize the number of collisions with obstacles during the entire task. The number of collisions between the agent and obstacles during the entire task is:

[0075]

[0076] In order to build a unified maneuvering behavior decision model based on deep reinforcement learning, the above three types of maneuvering behaviors can be unified into a whole so that the final task target value is maximized. The task target value is:

[0077] value=-(D×w D +T×w T +C×w C )

[0078] Where w represents each coefficient. By maximizing the corresponding D, T, and C, the corresponding reward can be obtained.

[0079] Figure 2 The algorithm flow shown is the MADDPG strategy optimization method, which includes the Actor network and the Critic network. The agent gives a decision action through the Actor network, and the Critic network agent gives a decision result based on the joint state evaluation of the agent. During the iterative training of the model, the Actor network and the Critic network will continuously optimize their parameters based on the results of the reward optimization module. The pseudo code of the MADDPG strategy optimization method is shown in Table 1.

[0080] The MADDPG algorithm used in the strategy execution module is an extension of the DDPG algorithm and is a deep reinforcement learning algorithm designed for multi-agent systems. The algorithm mainly includes two core networks: the strategy network and the value function network. The strategy network consists of the Actor current network and the Actor target network, while the value function network consists of the Critic current network and the Critic target network. Its execution process includes:

[0081] First, initialize the training environment and the action exploration process;

[0082] Then, the agent selects an action based on its own state and exploration process, and all agents perform joint actions and obtain rewards and next states. The state-action-reward next state is stored in the experience pool;

[0083] According to the value function and the target strategy network, the optimal strategy target network is obtained through continuous iteration;

[0084] Then we optimize the rewards. The technique is to record all the actions, state transitions, and environmental feedback (rewards or penalties) of the agent during the experiment. These data should record in detail the position, speed, direction, and other state information of each agent at each moment, as well as the actions it takes and the resulting immediate rewards. These trajectory data are stored in the experience replay pool. The experience replay pool can be a circular buffer. When new data comes in, the old data will be replaced to keep the size of the pool constant. The experience replay pool should be large enough to accommodate a large amount of historical data so that there are enough samples for subsequent analysis.

[0085] The strategy execution module uses a path overlap algorithm when making path planning decisions. The algorithm can calculate the degree of overlap between any two or more paths. The path overlap can be calculated based on the coordinate points that the agent passes through on the path. For example, the path overlap can be defined as the number of common coordinate points on two paths divided by the total length of the path. Subsequently, the trajectory of the agent is divided into several discrete time segments or spatial positions to facilitate subsequent overlap calculations.

[0086] When calculating the path overlap algorithm, in order to ensure that all trajectory data is stored in a unified format for subsequent processing, the data should include but is not limited to timestamps, location coordinates, speed, acceleration, agent status, and other information. Finally, check the integrity of the data, remove outliers or missing values, and ensure data quality.

[0087] For each pair of paths, a path overlap algorithm is used to calculate the degree of overlap between them. This can include counting the number of coordinate points that are the same on the two paths and comparing it to the path length to get the overlap. For multiple paths, related algorithms from graph theory can be used to identify overlapping subpaths or common path patterns.

[0088] The strategy execution module uses statistical methods to analyze path overlap data, which can include calculating the mean value, standard deviation and other statistics of path overlap, and drawing histograms or box plots to display the distribution of overlap, analyzing the trend of overlap over time, and understanding the changes in path selection of the intelligent agent during the learning process.

[0089] The strategy execution module clusters the agent's paths using a clustering algorithm (such as K-means) to identify groups of paths with similar overlapping patterns. Features of each path group, such as the most frequently visited areas and obstacles avoided, are extracted to understand the agent's behavioral preferences.

[0090] Based on the results of path overlap analysis, the behavior patterns of the agents are explained. If some paths are found to be frequently overlapped, it may indicate that these paths are the optimal or commonly used paths for the agents in a specific environment. The impact of environmental factors (such as obstacle layout, interference area, etc.) on the path selection of the agents is also examined.

[0091] Adjust the agent's strategy based on the analysis results, such as introducing a new reward mechanism or adjusting the action selection strategy to reduce unnecessary path overlap and improve the overall task completion efficiency. Fine-tune the parameters of the reinforcement learning algorithm, such as learning rate, discount factor, etc., to optimize the agent's performance.

[0092] Specifically, each agent represents a vehicle. The agent simulates the state and action of the simulation entity. Each agent has a separate Actor network and Critic network. The agent obtains state information S through its own equipment and gives decision actions through the Actor network, while the Critic network gives evaluations based on the joint state and actions of all agents.

[0093] Table 1 Pseudo code of MADDPG's strategy optimization method

[0094]

[0095] The design of the message transmission module specifically includes: when the maximum communication range of our communication equipment that can effectively transmit information is exceeded, the signal transmission between the agents will weaken as the distance increases, causing problems in the information exchange between the two. Therefore, in the path-based loss model, the signal strength is inversely proportional to the distance between the two communicating parties, and the signal strength is:

[0096] RSSI = Tx Power +Path Loss +Rx Gain +Tx Gain

[0097] RSSI represents the received power, Tx Power Indicates transmit power, Path Loss Represents the path loss, Rx Gain Indicates receiving antenna gain; Tx Gain Represents the transmit antenna gain.

[0098]

[0099] Path Loss represents the path loss, which is inversely proportional to the square of the distance.

[0100] In a real environment, when the distance between the two communicating parties exceeds a certain range, the communication capability will be completely lost. Therefore, it is also necessary to quantitatively analyze the impact of signal strength on the communication success rate in a simulation environment.

[0101] In a multi-agent system, agents need to maintain effective communication in a dynamic environment. To this end, a channel model needs to be established to describe the losses in the signal transmission process, such as free space path loss, multipath effect, shadow fading, etc. Through the channel model, the trend of signal strength changing with distance can be predicted, and the probability of successful communication can be estimated accordingly.

[0102] The agent needs to periodically measure the received signal strength. This is achieved through the received signal strength indication (RSSI) technique. The results of the signal strength measurement can be used to evaluate the quality of the current communication link and make decisions based on whether to retransmit data.

[0103] As shown in Table 2, based on the relationship between signal strength and packet loss rate, a prediction model can be established to estimate the packet loss rate under a specific signal strength. This model can be trained based on historical data or based on theoretical analysis. With this model, the agent can estimate the possibility of successful communication.

[0104] Table 2 Correspondence between signal strength and packet loss rate

[0105]

[0106] In the case of unreliable communication links, the agent can dynamically adjust the communication path based on the probability of successful communication. For example, when it detects that the probability of communicating with a certain node is low, it can choose another path or use relay nodes to improve the communication quality.

[0107] In order to improve the success rate of data transmission, the packet loss problem can be overcome by sending the same data packet multiple times. This method can significantly improve the success rate of data transmission in some cases, especially when the signal strength is weak.

[0108] According to the current channel conditions, the agent can select the most suitable modulation and coding scheme to optimize communication efficiency. When the channel conditions are good, a modulation method with a higher data rate can be selected; otherwise, a more robust modulation method can be selected.

[0109] Combining forward error correction (FEC) and automatic retransmission request (ARQ) technology can overcome packet loss to a certain extent and improve communication reliability. This method corrects errors by adding redundant information to the data and requests retransmission of data that was not received correctly when necessary.

[0110] A simulation-oriented multi-agent maneuver behavior decision-making method, comprising the steps of:

[0111] Acquire joint state information that affects maneuver behavior decisions through message passing modules;

[0112] The state encoding module discretizes and quantifies the obtained joint state information into a unified indicator and encodes it, obtains the joint state code corresponding to each intelligent agent, and sends the encoded state to the strategy execution module;

[0113] The final decision results of each agent are determined and executed through the strategy execution module;

[0114] The deep reinforcement learning network model of the strategy execution module is optimized and updated through the reward optimization module.

[0115] Example 1

[0116] The present invention selects a part of a city in an experimental environment based on OpenStreetMap for algorithm verification. The experimental environment configuration is: Ubuntu20.04, 64-bit operating system, 8GB GTX1070 video memory, 32GB memory, Intel(R) Core(TM) i9-8950HK@2.90GHz 2.90GHz CPU, and PyTorch algorithm framework; this embodiment sets the parameter settings of the Actor network and the Critic network. Considering the three intelligent agents and target points set, as well as the environmental state representation method, it can be seen that each state includes the state of the intelligent agent itself, the state of other intelligent agents, and the state of the environment, which means that S tThe size is 12. The action parameters cover the speed parameters in both the x-axis and y-axis directions; the neural network architecture of the Actor network and the Critic network contains 3 layers, one of which is a hidden layer. BatchSize is set to 128 and the learning rate is set to 0.001. The average number of steps per round of the smart car is about 100, so setting the maximum number of steps per round to 300 is sufficient to ensure the model's exploration ability in the early stages. In the experience pool, 10^6 experiences can cover at least 500 rounds of data, which ensures that the sampled data can be effectively utilized and the number of samples is kept consistent with BatchSize.

[0117] Table 3 shows the reward function settings for this experiment, where D represents the initial distance between the smart car and the destination. A positive reward is given when the smart car is far away from the destination, otherwise a negative reward is given; v t Indicates the current speed, v t-1 is the last speed, v = is the vector sum of the x-direction and the y-direction, and the calculation formula is:

[0118]

[0119] To shorten the task time, the greater the speed, the greater the reward; goal represents the reward for the first detection of the target point i. If multiple vehicles reach the target point, a reward is given; D represents the distance between the two vehicles. When the distance between the vehicles is less than the preset safety value, a negative reward r is given. distance When the vehicle collides with an obstacle, r collison To minimize the mission time, the mission is considered completed when one of the vehicles reaches the destination.

[0120] Table 3 Reward function settings

[0121]

[0122] Set the scene to cooperative mode, and the agents share the reward function, so Figure 3 The graph shows the transformation curve of the global average reward during the training process. It can be seen from the figure that the reward of the model rises rapidly in the early stage and converges to a relatively stable value around 30,000 rounds. As the training execution time of the model gradually increases, it starts to decline after 10,000 rounds, tends to be flat around 20,000 rounds, and only changes slightly afterwards.

[0123] There are three intelligent agents and one target point. The vehicles are in a fully cooperative relationship, that is, the three share a unified reward function. When a vehicle reaches one of the destinations, the entire fleet will receive a positive reward to maximize the overall reward. From the results, it can be seen that in order to maximize the overall benefits, when the distance between the two vehicles is close, one of them may go around in circles or take a long detour before approaching. When there are fewer obstacles around, the path with fewer obstacles will be preferred. In the interference area, the vehicle will give priority to leaving the area as soon as possible and continue to the destination, and may cross obstacles during this period. Therefore, compared with the traditional path planning algorithm, this method cannot guarantee that the intelligent agent strictly avoids obstacles and plans the final path, but overall it shows that the MADDPG algorithm can realize multi-agent mobile collaborative planning.

[0124] The rewards of the algorithm model of the present invention increase rapidly in the early stage and converge to a relatively stable value in about 30,000 rounds. Figure 4 The results of two experiments are shown, where the red curve is the path taken by the patrol car for reconnaissance, and the green curve is the final goal. From the experimental results, the agent's path selection shows a high degree of strategicity. In order to maximize the overall benefit, when the two agents are close, they will move away to help expand the search range. When the two cars are close, in order to avoid potential conflicts and ensure that all agents can reach the destination efficiently, one of the agents will choose to turn around or take a detour. This maneuvering behavior shows that the agent not only considers its own immediate interests when maneuvering, but also considers the long-term interests of the entire fleet.

[0125] The present invention designs a discrete action space, unifies environmental factors into pass coefficients, and reduces the state space. The mechanism of probabilistic communication is introduced to simulate a complex environment with communication interference. It also integrates a variety of reward settings, which is helpful for training more complex coordination strategies. Finally, the MADDPG algorithm was used to verify the algorithm, and the method was verified by an experimental scenario built based on OpenStreetMap open source data. The experimental results show that this method can achieve stable performance after about 30,000 iterations, and the agents show effective collaborative strategies. It provides an effective solution to the problem of multi-agent collaborative navigation in complex environments under simulation and deduction.

[0126] The scenario embodiments described above are merely illustrative, wherein the units described as separate modules may or may not be actually separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution can essentially or contribute to the prior art in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a drone, an unmanned vehicle, an intelligent robot or a simulation device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A simulation-oriented multi-agent maneuvering behavior decision system, characterized in that: include: The message passing module introduces signal strength into the communication between multiple agents, discretely quantifies the communication strength between agents, and adopts probabilistic communication to simulate a complex simulation environment with interference, and obtains joint state information that affects the decision-making of maneuvering behavior; The state encoding module discretizes and quantizes the obtained joint state information into a unified indicator and encodes it to obtain the joint state code corresponding to each intelligent agent; The reward optimization module combines the task objectives and the historical trajectories of each agent to determine the target reward of each agent. The target reward is used to optimize the deep reinforcement learning network model of the strategy execution module. In the strategy execution module, each agent uses a deep reinforcement learning network model based on the MADDPG algorithm to determine the final decision results of each agent and execute them.

2. A simulation-oriented multi-agent maneuvering behavior decision system according to claim 1, characterized in that: The joint state information includes the state information of the agent itself, the environment information within its own perception range, and the state information of other agents.

3. A simulation-oriented multi-agent maneuvering behavior decision system according to claim 2, characterized in that: The mathematical model of the state information is: S t =(S self ,S others ,S env ) Where S self Represents its own state information, including its current velocity vector and current location information S others Represents the state information of other agents, S env Indicates environmental information.

4. The simulation-oriented multi-agent maneuvering behavior decision system according to claim 1, characterized in that: The target rewards are: R=w1·r time +w2·r distance +w3r goal +w4·r collison +w5·r overlap Among them, r time represents the execution time reward for the current agent to complete the task, r distance Reward for distance change, r goal represents the reward for the first scout of the destination, r collison represents the reward for the collision between the agent and the obstacle, r overlap represents the reward generated by the collision between agents, w i Represents the coefficient of the rth reward, i=1, 2, 3, 4, 5.

5. The simulation-oriented multi-agent maneuvering behavior decision system according to claim 1, characterized in that: The deep reinforcement learning network model takes the task value maximization of three types of maneuvering behaviors, namely, transforming target maneuvering, fixed coordinate maneuvering, and targetless maneuvering, as the goal, and the task value is: value=-(D×w D +T×w T +C×w C ) Where D represents the relative distance between the agent and the target, T represents the total time to complete the task, C represents the number of collisions with obstacles during the entire task, and w D 、w T 、w C They represent the weights corresponding to D, T, and C respectively.

6. The simulation-oriented multi-agent maneuvering behavior decision system according to claim 1, characterized in that: The signal strength is: RSSI=Tx Power +Path Loss +Rx Gain +Tx Gain Among them, RSSI represents the received power, that is, the signal strength, Tx Power Indicates the transmit power, Path Loss Represents the path loss, Rx Gain Indicates receiving antenna gain; Tx Gain Represents the transmit antenna gain.

7. The simulation-oriented multi-agent maneuvering behavior decision system according to claim 6, characterized in that: The relationship between the path loss and the distance between the two communicating parties is: Among them, d is the distance between the two communicating parties, G1 is a constant, and λ represents the wavelength of the frequency band used in analog communication.

8. The simulation-oriented multi-agent maneuvering behavior decision system according to claim 6, characterized in that: The message transmission module establishes a packet loss rate prediction model according to the relationship between signal strength and packet loss rate, so as to estimate the packet loss rate under a specific signal strength.

9. The simulation-oriented multi-agent maneuvering behavior decision system according to claim 1, characterized in that: The deep reinforcement learning network model includes an Actor network and a Critic network. The Actor network gives decision actions, and the Critic network gives evaluations based on the joint states and actions of all agents.

10. A simulation-oriented multi-agent maneuvering behavior decision-making method based on the system of any one of claims 1 to 9, characterized in that: Includes steps: Acquire joint state information that affects maneuver behavior decisions through message passing modules; The state encoding module discretizes and quantifies the obtained joint state information into a unified indicator and encodes it, obtains the joint state code corresponding to each intelligent agent, and sends the encoded state to the strategy execution module; The final decision results of each agent are determined and executed through the strategy execution module; The deep reinforcement learning network model of the strategy execution module is optimized and updated through the reward optimization module.

Images