Automatic driving path planning method, device, equipment, medium and program product

Abstract

The invention provides an automatic driving path planning method and device, equipment, a medium and a program product, and relates to the technical field of automatic driving, and the method comprises the steps: obtaining an initial state and an initial motion of a to-be-planned intelligent agent, inputting the initial state and the initial motion into a trained traffic flow simulation model, real-time interaction information of a plurality of intelligent agents and an intelligent agent to be planned in a traffic flow simulation model is obtained, the traffic flow simulation model is obtained by dividing the plurality of intelligent agents into a plurality of groups and training through an upper-layer framework and a lower-layer framework, and the lower-layer framework is used for performing intra-group cooperative training on each group; and the upper-layer architecture is used for carrying out inter-group cooperative training on the plurality of groups, deciding real-time actions and real-time states of the intelligent agent to be planned according to the real-time interaction information, and determining an advancing path according to the real-time actions and the real-time states in a preset time period. Through the traffic flow simulation model, the intelligent agent to be planned interacts with multiple intelligent agents, enough interaction information is obtained, information of real traffic road conditions is simulated, and the precision of path planning is improved.

Description

Background Art

[0002] With the rapid development of autonomous driving technology, the intelligence level of intelligent transportation systems continues to improve. Autonomous driving will completely change the transportation system and is widely regarded as a key technology to effectively alleviate traffic congestion and reduce traffic accidents.

[0003] However, for the testing of autonomous driving, the testing methods in related technologies, such as path planning methods, have problems such as single strategy, few considerations, poor interactivity, and inaccurate travel paths for autonomous driving path planning.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide an autonomous driving path planning method, device, equipment, medium and program product, which can at least to some extent solve the problem of inaccurate autonomous driving path planning in the prior art.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, there is provided an automatic driving path planning method, the method comprising:

[0008] Get the initial state and initial action of the agent to be planned;

[0009] The initial state and initial action are input into a trained traffic flow simulation model to obtain real-time interaction information between multiple agents in the traffic flow simulation model and the agent to be planned; the traffic flow simulation model is obtained by dividing multiple agents into multiple groups and training them through an upper-layer architecture and a lower-layer architecture; wherein the lower-layer architecture performs intra-group collaborative training on each group, and the upper-layer architecture performs inter-group collaborative training on multiple groups;

[0010] Determine the real-time action and real-time state of the to-be-planned intelligent agent according to the real-time interaction information;

[0011] The travel path is determined according to the real-time action and the real-time status in a preset time period.

[0012] In a possible embodiment, the intelligent agent to be planned includes an intelligent agent of autonomous driving type;

[0013] The method further comprises:

[0014] Training multiple traffic flow simulation models; wherein, during the training process of each traffic flow simulation model, a different first reward weight is generated for an intelligent agent of the autonomous driving type; the first reward weight represents the reward value of other types of intelligent agents when performing decision-making actions on the intelligent agent of the autonomous driving type in any traffic flow simulation scenario; wherein, the larger the first reward weight, the larger the reward value; the reward value affects the convergence of the loss value of the traffic flow simulation model;

[0015] According to different first reward weights, a trained traffic flow simulation model of an autonomous driving type agent is obtained in each traffic flow simulation scenario.

[0016] In a possible embodiment, the method further includes:

[0017] During the autonomous driving test, the initial state and initial action of the test agent to be tested are obtained;

[0018] Input the initial state and initial action of the test agent to be tested into multiple trained traffic flow simulation models respectively to obtain the real-time action and real-time state under different traffic flow simulation scenarios;

[0019] According to the real-time actions and real-time states in different traffic flow scenarios, the travel path in each traffic flow scenario is determined to obtain multiple groups of travel paths.

[0020] In a possible embodiment, the method further includes:

[0021] During the training process of the traffic flow simulation model, for any one intelligent agent, a plurality of preset ranges are generated with the any one intelligent agent as the center;

[0022] For any of the intelligent agents, when making a decision action to other intelligent agents located in different preset ranges, different second reward weights are generated; the smaller the preset range from any of the intelligent agents, the greater the second reward weight;

[0023] According to different second reward weights, a trained traffic flow simulation model is obtained.

[0024] In a possible embodiment, the training process of the traffic flow simulation model includes:

[0025] Acquire multiple intelligent agents in a traffic flow, wherein the intelligent agents are objects that interact with each other;

[0026] Collect the current observation state set, action set, reward set and next observation state set of the multiple agents, and store them in a database as a training data set; the reward set represents the reward value corresponding to different actions decided by the agents;

[0027] Grouping multiple agents to obtain multiple groups;

[0028] The upper layer architecture and the lower layer architecture of the traffic flow simulation model are trained according to the training data set to obtain a trained traffic flow simulation model.

[0029] In a possible embodiment, the training of the upper structure and the lower structure of the traffic flow simulation model according to the training data set to obtain the trained traffic flow simulation model includes:

[0030] Extract training samples from the training data set, and iteratively train the traffic flow simulation model based on the training samples;

[0031] Determining a first loss value of the lower architecture and a second loss value of the upper architecture;

[0032] If the first loss value of the lower-level architecture does not meet the first preset value, or the second loss value of the upper-level architecture does not meet the second preset value, the network parameters of the traffic flow simulation model are adjusted according to the preset learning rate until the first loss value meets the first preset value and the second loss value meets the second preset value, thereby obtaining a trained traffic flow simulation model.

[0033] In a possible embodiment, an iterative training process of the traffic flow simulation model includes:

[0034] Based on the observed state and action at the current time step in the training sample and the observed state at the next time step, predict the action of each agent at the next time step;

[0035] Determine the predicted group Q function for each group and the predicted local Q function for each agent in each group based on the observed state and action at the current time step, the observed state at the next time step, and the action of each agent at the next time step;

[0036] According to the current set of observed states and actions, determine the true local Q function of each agent in each group and the importance weight parameter of each agent in the group;

[0037] Determine the true group Q function of each group according to the importance weight parameter of each agent in the group, the set of observed states and the set of actions corresponding to the current time;

[0038] The first loss value of the lower architecture and the second loss value of the upper architecture are determined based on the predicted group Q function of each group, the predicted local Q function of each agent in each group, the true group Q function of each group and the true local Q function of each agent in each group.

[0039] According to another aspect of the present disclosure, there is provided an automatic driving path planning device, comprising:

[0040] An acquisition unit, used to acquire the initial state and initial action of the intelligent body to be planned;

[0041] A real-time information acquisition unit, used for inputting the initial state and initial action into a trained traffic flow simulation model, and acquiring real-time interaction information between multiple agents in the traffic flow simulation model and the agent to be planned; the traffic flow simulation model is obtained by dividing multiple agents into multiple groups and training them through an upper-layer architecture and a lower-layer architecture; wherein the lower-layer architecture performs intra-group collaborative training on each group, and the upper-layer architecture performs inter-group collaborative training on multiple groups;

[0042] A decision-making unit, used to decide the real-time action and real-time state of the to-be-planned intelligent agent according to the real-time interaction information;

[0043] A path planning unit is used to determine the travel path according to the real-time action and real-time status in a preset time period.

[0044] In a possible embodiment, the automatic driving path planning device further includes:

[0045] An offline training unit, used to obtain multiple agents in the traffic flow, wherein the agents are objects that interact with each other;

[0046] Collect the current observation state set, action set, reward set and next observation state set of the multiple agents, and store them in a database as a training data set; the reward set represents the reward value corresponding to different actions decided by the agents;

[0047] Group the defined agents to obtain multiple groups;

[0048] The upper layer structure and the lower layer structure of the traffic flow simulation model are trained according to the training data set to obtain a trained traffic flow simulation model.

[0049] In a possible embodiment, the offline training unit is further used for:

[0050] The plurality of agents include agents of autonomous driving type;

[0051] The method further comprises:

[0052] Training multiple traffic flow simulation models; wherein, during the training process of each traffic flow simulation model, a different first reward weight is generated for an intelligent agent of the autonomous driving type; the first reward weight represents the reward value of other types of intelligent agents when performing decision-making actions on the intelligent agent of the autonomous driving type in any traffic flow simulation scenario; wherein, the larger the first reward weight, the larger the reward value; the reward value affects the convergence of the loss value of the traffic flow simulation model;

[0053] According to different first reward weights, a trained traffic flow simulation model of an autonomous driving type agent is obtained in each traffic flow simulation scenario.

[0054] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the methods in the first aspect by executing the executable instructions.

[0055] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the method of any one of the first aspects is implemented.

[0056] According to another aspect of the present disclosure, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method described in the first aspect above.

[0057] The embodiments of the present disclosure provide an automatic driving path planning method, device, equipment, medium and program product, which relate to the field of automatic driving technology. The method includes: obtaining the initial state and initial action of the intelligent body to be planned; inputting the initial state and initial action into the trained traffic flow simulation model, and obtaining the real-time interaction information between multiple intelligent bodies in the traffic flow simulation model and the intelligent body to be planned; the traffic flow simulation model is obtained by dividing multiple intelligent bodies into multiple groups and training them through the upper-layer architecture and the lower-layer architecture; wherein the lower-layer architecture performs intra-group collaborative training for each group, and the upper-layer architecture performs inter-group collaborative training for multiple groups; according to the real-time interaction information, the real-time action and real-time state of the intelligent body to be planned are determined; according to the real-time action and real-time state of the preset time period, the travel path is determined. In the above manner, the interaction between the intelligent body to be planned and multiple intelligent bodies can be realized based on the traffic flow simulation model, and more interactive information can be obtained, which can simulate the information of real traffic conditions, and the accuracy of path planning can be improved.

[0058] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0060] Figure 1 A schematic diagram showing a flow chart of an automatic driving path planning method in an embodiment of the present disclosure;

[0061] Figure 2 A flow chart showing a traffic flow simulation model training process in an embodiment of the present disclosure;

[0062] Figure 3 A schematic diagram showing a flow chart of another traffic flow simulation model training process in an embodiment of the present disclosure;

[0063] Figure 4 A schematic diagram showing a flow chart of an iterative training process of a traffic flow simulation model in an embodiment of the present disclosure;

[0064] Figure 5 A schematic diagram of a process for determining a first loss value of a lower-layer architecture in an embodiment of the present disclosure is shown;

[0065] Figure 6 A schematic diagram of a process for determining a second loss value of an upper-layer architecture in an embodiment of the present disclosure is shown;

[0066] Figure 7 A schematic diagram showing a HMF learning framework in an embodiment of the present disclosure is shown;

[0067] Figure 8 A flow chart showing another traffic flow simulation model training process in an embodiment of the present disclosure is shown;

[0068] Fig. 9 A schematic diagram showing a flow chart of another method for autonomous driving path planning in an embodiment of the present disclosure;

[0069] Fig.10 A flow chart showing another traffic flow simulation model training process in an embodiment of the present disclosure is shown;

[0070] Fig.11 A schematic diagram showing the structure of an automatic driving path planning device according to an embodiment of the present disclosure is shown;

[0071] Fig.12A schematic structural diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0072] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0073] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0074] In related technologies, when testing autonomous driving, the simulation of traffic flow is generally a core function of simulation software that can support point-to-point communications (PNC). However, it is very difficult to realize large-scale traffic vehicle interaction. The usual method is to use a simple path planning algorithm and a fixed speed trajectory to give each vehicle a motion attribute. However, this type of motion attribute has a series of problems such as a single strategy and increased time consumption. Moreover, it is difficult to reproduce the traffic flow in an actual city using a single strategy method. For actual traffic participants, all objects are intelligent and there are interactive attributes between objects, so a more intelligent method is needed to simulate traffic flow.

[0075] In the traditional methods of related technologies, the interaction between intelligent agents is relatively poor. If one wants to achieve collaboration among multiple intelligent agents and apply a large amount of interaction information between multiple intelligent agents and test vehicles in autonomous driving tests, the joint learning method will fail because the joint state space will grow exponentially.

[0076] The automatic driving path planning method disclosed in the present invention can realize collaborative training by using a trained traffic flow simulation model, a model architecture of the upper and lower layers, and a method of grouping multiple agents, which can effectively reduce the difficulty of processing the amount of data in the process of data explosion, so as to obtain a traffic flow simulation model that can simulate the real traffic environment. In the traffic flow simulation model, the interactive information between multiple agents is intelligent and real-time interactive. When it is necessary to plan a path for an agent, it is only necessary to put a new agent to be planned into the model, place it in the traffic environment simulated by the model, and generate information interaction with multiple agents therein, so that the path planning of the agent to be planned can be carried out. This path planning method not only uses more effective information in planning, but also restores the real traffic environment to the path planning of the agent to be planned, so that the travel path is more accurate and more precise. This method can be used in the scene of testing automatic driving. Through the simulation of the real traffic environment by the traffic flow simulation model, the test process of automatic driving is more accurate and more precise.

[0077] The present disclosure describes an automatic driving path planning method, which is specifically described through the following embodiments.

[0078] Figure 1 A flow chart of an automatic driving path planning method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the steps include:

[0079] S102: Obtain the initial state and initial action of the intelligent body to be planned.

[0080] The intelligent agent to be planned may be an intelligent agent of the autonomous driving type.

[0081] S104: Input the initial state and initial action into the trained traffic flow simulation model to obtain real-time interaction information between multiple intelligent agents in the traffic flow simulation model and the intelligent agent to be planned. The traffic flow simulation model is obtained by dividing multiple intelligent agents into multiple groups and training them through an upper-level architecture and a lower-level architecture. The lower-level architecture performs intra-group collaborative training on each group, and the upper-level architecture performs inter-group collaborative training on multiple groups.

[0082] In a possible embodiment, at least one defined intelligent agent to be planned is input into a traffic flow simulation model. The simulation results of the traffic flow simulation model can be understood as traffic flow data simulated by the trained traffic flow simulation model. The traffic flow data includes various information of multiple intelligent agents, including at least: an observation state set, an action set, a reward set, etc. of the current time step, and may also include an observation state set of the next time step according to model requirements.

[0083] Exemplarily, each time step may be preset to 1 second, and the specific setting is not limited.

[0084] Obtain various information of multiple agents, including real-time interaction information between each agent to be planned and multiple agents within the maximum preset range. It is only necessary to obtain the real-time interaction information with multiple agents within the maximum preset range of each agent to be planned for path planning of the autonomous driving vehicle.

[0085] For example, the maximum preset range is set to a range of 20 meters in front and behind the intelligent body to be planned, or a range of 50 meters in front and behind. For any intelligent body A to be planned, real-time interaction information with multiple intelligent bodies within a range of 20 meters in diameter centered on the intelligent body A to be planned is obtained. The real-time interaction information between intelligent bodies in a larger range and the intelligent body A to be planned is less useful for the intelligent body A to be planned in making decisions on real-time actions and real-time states. It can reduce the acquisition of redundant information and reduce the amount of data processing in the path planning process. In this way, the amount of data processing by the computer in the path planning process is reduced.

[0086] In a possible embodiment, path planning may be performed only for one intelligent agent to be planned, and the result of path planning is more accurate and targeted.

[0087] In a possible embodiment, path planning can also be performed simultaneously for multiple intelligent entities to be planned to simulate real traffic conditions. In the traffic flow, there may be multiple autonomous driving vehicles on the road together. Autonomous driving path planning is performed for multiple intelligent entities to be planned, fully considering the road conditions.

[0088] S106: Determine the real-time action and real-time state of the intelligent body to be planned based on the real-time interaction information.

[0089] In a possible embodiment, the corresponding real-time action and real-time state are decided for each intelligent agent to be planned according to the real-time interaction information.

[0090] S108: Determine a travel path according to the real-time actions and real-time status in a preset time period.

[0091] In a possible embodiment, the route is determined by an algorithm for planning the route by the agent to be planned, for example, by a spatiotemporal joint planning method.

[0092] The preset time period may refer to one second or a period of time, without limitation. In the present disclosure, a real-time path of travel may be obtained based on real-time actions and real-time status. The path of travel may also be determined based on the real-time actions and real-time status at the current moment, and the real-time actions and real-time status within the preset time period before the current moment, that is, the previous actions and status are considered. For example, if the preset time period is 3 seconds, the real-time actions and real-time status at the current moment A (the current 1 second) may be considered, and the real-time actions and real-time status of the previous 3 seconds may also be taken into account.

[0093] In a possible embodiment, after the travel path is obtained, an evaluation can be performed based on the travel path to test the autonomous driving process of the planned intelligent agent. This method can be used in the scenario of the autonomous driving test to obtain the test results for judging the effect of the autonomous driving test. The complex traffic environment simulated by the traffic flow simulation model can enable the autonomous driving type intelligent agent to process complex information. Finally, when the autonomous driving type intelligent agent enters the real road and in the real traffic environment, it can improve the judgment accuracy of the road conditions, improve the accuracy of action decisions, and improve the accuracy of the autonomous driving type intelligent agent in processing complex information.

[0094] It should be noted that in the training process, since a larger number of intelligent agents are involved, the amount of interactive information involved in each intelligent agent is also large. Generally, as the number of intelligent agents increases, the amount of data required for training increases exponentially.

[0095] In a possible embodiment, the model can be trained by online training, targeting the test data generated in the actual test process, that is, the information of the autonomous driving type of intelligent agent, or the real-time acquisition of the information of the autonomous driving type of vehicle in the actual traffic environment, as part of the training data set to enter the iterative training process, the HMF learning framework in the present disclosure can handle the collaboration between multiple intelligent agents, which is sufficient to complete the training of the model. The effect of the training is iterated by adding the data from the test process. When the autonomous driving path planning method is applied to the test process, the traffic flow data generated by the model simulation is more accurate and the test accuracy is higher.

[0096] It can also be carried out through offline training, and the trained traffic flow simulation model can be directly used in the autonomous driving path planning process.

[0097] In a possible embodiment, the intelligent body will make decisions on actions and states according to various network parameters during training, thus forming a traffic condition for the intelligent body to be planned, and then the route of the intelligent body to be planned is obtained by planning the route of the intelligent body to be planned. Finally, the intelligent body to be planned is evaluated according to the real-time actions, real-time states, and routes to determine the effect of the test.

[0098] In a possible embodiment, the traffic flow simulation model can be obtained by dividing multiple intelligent agents into multiple groups and training them through an upper-layer architecture and a lower-layer architecture; wherein the lower-layer architecture performs intra-group collaborative training for each group, and the upper-layer architecture performs inter-group collaborative training for multiple groups, and there is information interaction between the upper-layer architecture and the lower-layer architecture.

[0099] In a possible embodiment, a traffic flow simulation model can be trained in a hierarchical mean field deep reinforcement learning (HMF) manner. Through the trained traffic flow simulation model, rich interaction information of intelligent agents can be obtained to form a continuous traffic flow simulation result, and at least one defined intelligent agent to be planned is placed in the traffic flow simulation result. The traffic flow simulation includes a large number of intelligent agents, and each intelligent agent involves more information, which displays the real traffic situation in real time. By obtaining the interaction information between multiple intelligent agents and the intelligent agent to be planned, decisions are made on the state and action of the intelligent agent to be planned.

[0100] By applying the method disclosed in the present invention in a test scenario, the test effect is closer to the effect of placing a real autonomous driving vehicle in a real traffic environment for driving, and the test accuracy is high.

[0101] The training process of the above trained traffic flow simulation model is as follows, Figure 2 A schematic diagram of a traffic flow simulation model training process is shown in FIG. Figure 2 As shown, the following steps are included:

[0102] S202: Acquire multiple intelligent agents in the traffic flow, where the intelligent agents are objects that interact with each other.

[0103] In a possible embodiment, due to the fact that there are more sudden situations in real traffic conditions and various information, such as acceleration and deceleration of vehicles, sudden braking, etc., pedestrians and people driving electric vehicles and automatic vehicles may make unpredictable movements at any time, whether it is a car, behavior or automatic vehicle, or pets, etc., any object that may interact with the test vehicle can be used as an intelligent agent in the embodiment of the present disclosure.

[0104] S204: Collect the current observation state set, action set, reward set and next observation state set of multiple agents, and store them in the database as a training data set. The reward set represents the corresponding reward value when the agent decides different actions.

[0105] In a possible embodiment, the observed state set o=[o k ] k＝1 NxM , action set a = [ak ] k＝1 NxM , reward set r = [r k ] k＝1 NxM , and the next observed state o ′ .

[0106] Among them, k represents the kth intelligent agent; M represents M groups, and N represents that each group includes N intelligent agents.

[0107] The observation state set may include: the state of the intelligent agent itself and the state of the vehicle's surrounding environment.

[0108] The state of the vehicle's surrounding environment corresponds to multiple different environmental categories, such as intersections, three-way intersections, normal driving roads, zebra crossings, and other different environmental states.

[0109] The state of the agent itself can include stationary, moving, accelerating, decelerating, etc.

[0110] The types of intelligent agents may include: human-driven vehicles, pedestrians, human-driven bicycles, electric vehicles, etc., animals that may appear on the road, such as human pets, etc., and a series of other objects that may have information interaction with the test vehicle.

[0111] The collected data is used as a training data set and stored in the experience revisit pool (o, a, r, o ′ ).

[0112] S206: Group the multiple agents to obtain multiple groups.

[0113] S208: Train the upper structure and the lower structure of the traffic flow simulation model according to the training data set to obtain a trained traffic flow simulation model.

[0114] Through the above method, interactive training can be performed between the upper-level architecture and the lower-level architecture. The lower-level architecture performs intra-group collaborative training for each group, and the upper-level architecture performs inter-group collaborative training for multiple groups. During the training process, the problem of a sudden increase in data volume can be solved and the difficulty of processing the data volume can be reduced.

[0115] In a possible embodiment, the specific training process of step S208 can be as follows: Figure 3 As shown, Figure 3 A schematic diagram of another traffic flow simulation model training process is shown, comprising the following steps:

[0116] S302: extracting training samples from the training data set, and iteratively training the traffic flow simulation model according to the training samples.

[0117] S304: Determine a first loss value of the lower architecture and a second loss value of the upper architecture;

[0118] S306: Determine whether the first loss value satisfies the first preset value, and the second loss value satisfies the second preset value; if so, execute S310; if not, execute S308.

[0119] S308: Adjust the network parameters of the traffic flow simulation model and return to step S302.

[0120] S310: Obtain a trained traffic flow simulation model according to current network parameters.

[0121] When the first loss value of the lower-level architecture and the second loss value of the upper-level architecture of the traffic flow simulation model both meet the conditions, the model training is completed.

[0122] It should be noted that for each agent k, the action of each agent is decided by a greedy algorithm. During the training process, since each agent uses a greedy algorithm when choosing actions, different reward weights will be corresponding when making decisions on different actions. Each time an action is decided, the reward value will be maximized.

[0123] Through the intelligent agent defined in the above model, it is possible to expand the original basic information in simulated traffic during autonomous driving testing, enrich the information that the test vehicle can obtain, increase test data, and improve test accuracy.

[0124] The HMF learning framework is adopted to alleviate the coordination problem of large-scale multi-agent reinforcement learning by decomposing the entire group into multiple groups and implementing a two-level collaborative learning process between agents. The bottom layer is used for intra-group collaboration and the top layer is used for inter-group collaboration. This further alleviates the problem that algorithms such as MFVFD have high computational complexity and the collaborative performance may drop sharply with the size of the group. Among them, MFVFD is a multi-agent reinforcement learning algorithm that combines mean-field theory and value function decomposition.

[0125] Through the HMF algorithm, based on the mean field theory, the interaction between individuals in the group is approximated as the interaction between the individual and the group mean center, and the importance of each agent in the group is evaluated based on the attention mechanism. The upper-level architecture is responsible for managing the interaction within the group, and adopts centralized learning and decentralized execution (CTDE) for learning to achieve group-level collaboration.

[0126] In a possible embodiment, the corresponding loss values ​​of the upper architecture and the lower architecture are determined each time an iterative training is performed. Figure 4 A schematic diagram of an iterative training process of a traffic flow simulation model is shown in FIG. Figure 4 As shown, the following steps are included:

[0127] S402: Predict the action of each agent at the next time step based on the observed state and action at the current time step in the training sample and the observed state at the next time step.

[0128] S404: Determine the predicted group Q function of each group and the predicted local Q function of each agent in each group based on the observed state and action at the current time step, the observed state at the next time step, and the action of each agent at the next time step.

[0129] In one possible embodiment, the Q function of each agent is defined, where the true local Q function is expressed as: The predicted local Q function is expressed as: The true group Q function is denoted as Q GMF ; The predicted group Q function is expressed as:

[0130] Each time step can be set to 1 second. Taking an agent in group i as an example, the observed state and action at the current time step can be expressed as: and

[0131] The observed state and action at the next time step can be expressed as: and

[0132] This information is used to calculate the first loss value and the second loss value of an iterative training process.

[0133] The true local Q function is defined based on the local information of each agent, which includes the state set and action set of the agent. This function is used to represent the strategy of a single agent in selecting actions in its group to maximize its expected return, that is, how to select actions to maximize the reward.

[0134] The predicted local Q function is the predicted value of the model.

[0135] The true group Q function is used to represent the Q function of all agents in a group. This function is used to determine how the agents in a group choose actions to maximize the reward of this group, that is, the direction of loss convergence.

[0136] The predicted population Q function is the predicted value of the model.

[0137] Among them, the Q function is a function used to represent the expected rewards obtained by each agent when taking different actions in different states.

[0138] S406: Determine the true local Q function of each agent in each group and the importance weight parameter of each agent in the group according to the current observed state set and action set.

[0139] S408: Determine the true group Q function of each group based on the importance weight parameter of each agent in the group and the corresponding current observation state set and action set.

[0140] S410: Determine the first loss value of the lower-level architecture and the second loss value of the upper-level architecture based on the predicted group Q function of each group, the predicted local Q function of each agent in each group, the true group Q function of each group, and the true local Q function of each agent in each group.

[0141] In a possible embodiment, the first loss value and the second loss value for each iteration of training can be determined as follows: Figure 5 and Figure 6 shown.

[0142] Figure 5 A schematic diagram of determining the first loss value of the lower architecture is shown, such as Figure 5 As shown, the following steps are included:

[0143] S502: For any group, determine the true group Q function according to the true group Q function of any group and the true local Q function of each agent in the group.

[0144] S504: Determine the first target prediction value of the lower-level architecture for the predicted group Q function of any group and the predicted local Q function of each agent in the group.

[0145] S506: Determine a first loss value of the lower architecture according to the true group Q function and the target prediction value.

[0146] In one possible embodiment, the real group Q function is given by Q GRP The specific formula is as follows:

[0147]

[0148] Among them, λ k represents the importance weight parameter of each agent in the group; x k Represents the input of agent k in the model, which can be understood as the training sample of agent k.

[0149] Where μ(x)=∑ k∈N λ k x k . represents the input that determines the true group Q function.

[0150] Q GRP (x) represents the true group Q function; x represents the input that determines the true group Q function, which is the same as the input x of each agent k in the model. k , and μ(x) are associated.

[0151] After determining the true group Q function in the above manner, the first loss value is determined according to the first target prediction value.

[0152] Figure 6 A flow chart of determining a second loss value of an upper architecture is shown, Figure 5 As shown, the following steps are included:

[0153] S602: Determine the true overall Q function according to the true group Q function of each group.

[0154] S604: Determine the second target prediction value of the upper-level architecture according to the predicted group Q function of each group.

[0155] S606: Determine a second loss value of the upper architecture based on the true overall Q function and the second target prediction value.

[0156] In one possible embodiment, the true overall Q function can be obtained by Q TOT The formula is as follows:

[0157]

[0158] Among them, x (1) ,…,x (m) Represents the input in m groups when determining the overall Q function.

[0159] Through the above method, the overall Q function is determined to determine the second loss value.

[0160] Through the model of the HMF learning framework, each group is associated with a virtual upper layer, which is responsible for managing the training process of multiple agents. At the lower layer, based on the mean field theory and attention mechanism, the average effect of the group is approximated by considering the different influences of the agents within the group, while the top layer learns the mean field information of its group through multiple upper layers to achieve system-level regulation.

[0161] Figure 7 A schematic diagram of an HMF learning framework in an embodiment of the present disclosure is shown, Figure 7 shown.

[0162] The upper framework includes multiple managers, which are used to manage each group in the lower framework. Each group in the lower framework obtains o and a, performs mean field processing, and uses a multi-layer perceptron (MLP) mechanism and an attention head mechanism to obtain the true local Q function of each group, and combines the importance weight parameters of each agent in the group to determine the true group Q function for calculating the first loss value.

[0163] The upper layer architecture determines the true group Q function of each group and transmits it to the lower layer architecture. The upper layer architecture determines the overall Q function for calculating the second loss value.

[0164] In the HMF learning framework, each group is associated with a virtual upper layer, which is responsible for managing the training process of multiple agents. At the bottom layer, based on the mean field theory and attention mechanism, the average effect of the group is approximated by considering the different influences of the agents within the group, while the top layer learns the MF information of its group through multiple upper layers to achieve system-level regulation.

[0165] According to the above framework, in the autonomous driving simulated traffic flow, each moving body, such as vehicles, pedestrians, bicycles, etc., can observe the changes of surrounding objects in real time. In the HMF framework, they can all be used as input state quantities of the intelligent body, and the strategic actions are obtained through the network for traffic body planning.

[0166] Furthermore, the advantage of the HMF framework is that the algorithm has good scalability. As the scale of intelligent agents increases, the performance of the algorithm can be maintained at a good level.

[0167] Embodiment 1:

[0168] For example, in the process of traffic flow training, the types of intelligent agents are increased, and the multiple intelligent agents include multiple types of intelligent agents, which may at least include autonomous driving type vehicles.

[0169] Different reward weights are set for autonomous driving vehicles to simulate different traffic flow scenarios.

[0170] Figure 8 A schematic diagram of the training process of another traffic flow simulation model is shown; Figure 8 As shown, the following steps are included:

[0171] S802: Training multiple traffic flow simulation models, wherein during the training process of each traffic flow simulation model, different first reward weights are generated for the autonomous driving type intelligent agent, and the first reward weight represents the reward value of other types of intelligent agents when the autonomous driving type intelligent agent makes a decision action in any traffic flow simulation scenario, wherein the larger the first reward weight, the larger the reward value, and the reward value affects the convergence of the loss value of the traffic flow simulation model.

[0172] For example, assume that there are three types of agents in the training process, namely, type A, type B, and autonomous driving type vehicles. This is just an example, and there are too many types involved to be explained with examples.

[0173] The reward weight when type A takes action against type B agent can be set to 0.4; the reward weight when type B takes action against type A agent can be set to 0.3.

[0174] Based on this, when setting the reward weights for Type A and Type B agents when they take actions on autonomous driving type vehicles, different first reward weights are set for different models.

[0175] Example 1: For the first model, the reward weight for type A when taking actions against autonomous driving vehicles is set to 0.5; the reward weight for type B when taking actions against autonomous driving vehicles is set to 0.4. Based on this, in order to maximize the reward value for each decision-making action, actions that favor autonomous driving vehicles will be selected when making action decisions.

[0176] For example, when agent A needs to avoid agent B and autonomous driving type vehicle, through the reward weights set above, agent A will choose to avoid the autonomous driving type vehicle, so as to obtain a larger reward value and make the loss value meet the conditions more quickly.

[0177] Example 2: For the second model, the reward weight for type A when taking actions against autonomous vehicles is set to 0.3; the reward weight for type B when taking actions against autonomous vehicles is set to 0.2. Based on this, in order to maximize the reward value for each decision-making action, actions that are biased towards other types of agents will be selected when making action decisions.

[0178] For example, when a type A agent needs to avoid a type B agent and an autonomous driving type vehicle, through the reward weights set above, the type A agent will choose to avoid the action of the type B agent, so as to obtain a larger reward value and make the loss value meet the conditions more quickly.

[0179] S804: Obtain a trained traffic flow simulation model of an autonomous driving type intelligent agent in each traffic flow simulation scenario according to different first reward weights.

[0180] In the scenario of Example 1, other agents will deliberately avoid the autonomous driving vehicle or give way to it. During the test, the test vehicle is also an autonomous driving vehicle. This scenario can be understood as a scenario with low test difficulty.

[0181] In the scenario of Example 2, other agents will cut off the autonomous driving vehicle or ignore the autonomous driving vehicle. During the test, the test vehicle is also an autonomous driving vehicle. This scenario can be understood as a scenario with high test difficulty.

[0182] Through the greedy algorithm during action selection, combined with the weight parameters set by different types of intelligent agents when making action decisions for autonomous driving vehicles, models belonging to multiple different traffic flow scenarios can be trained.

[0183] In a possible embodiment, in order to make the simulated traffic flow closer to the actual traffic situation, when setting the reward weights of different types of agents for autonomous driving vehicles, the reward weights set for agents such as pedestrians are larger, and when making decisions for some actions, pedestrians are avoided first. In the test process, safety considerations are improved.

[0184] It should be noted that the above process of setting the reward weight is executed by a computer, and the training process selects offline training. The computer can obtain different traffic flow simulation models by fine-tuning the first reward weight.

[0185] Through the above training process, multiple trained traffic flow simulation models can be obtained. Fig. 9 A schematic diagram of another method for autonomous driving path planning is shown, such as Fig. 9 As shown, the following steps are included:

[0186] S902: During the autonomous driving test, the initial state and initial action of the test agent to be tested are obtained.

[0187] S904: Inputting the initial state and initial action of the test agent to be tested into a plurality of trained traffic flow simulation models respectively, and obtaining real-time actions and real-time states under different traffic flow simulation scenarios.

[0188] S906: Determine a travel path in each traffic flow scenario according to the real-time actions and real-time states in different traffic flow scenarios, and obtain multiple groups of travel paths.

[0189] Embodiment 2:

[0190] During the autonomous driving test, it is generally sufficient to obtain real-time interactive information within the maximum preset range of the test vehicle, without the need to obtain all of it. Based on this, considering the actual traffic driving process, only the interactive information of the intelligent agents in the relatively close range in front and behind is necessary for the autonomous driving type of vehicle. The information in some farther ranges is unnecessary, or it is impossible to make effective targeted actions after obtaining it.

[0191] Fig.10 A schematic diagram of another traffic flow simulation model training process is shown. Fig.10 As shown, the following steps are included:

[0192] S1002: During the training process of the traffic flow simulation model, for any intelligent agent, multiple preset ranges are generated with any intelligent agent as the center.

[0193] S1004: For any intelligent agent, when making a decision action towards other intelligent agents within different preset ranges, different second reward weights are generated. The smaller the preset range from any intelligent agent, the greater the second reward weight.

[0194] Among them, the reward value affects the convergence of the loss value of the traffic flow simulation model.

[0195] S1006: Obtain a trained traffic flow simulation model according to different second reward weights.

[0196] In the process of training a model, multiple preset ranges can be generated with any agent as the center, and different second reward weights can be generated for the decision actions of other agents except any agent within different preset ranges.

[0197] The trained model pays more attention to the real-time interaction information between closer intelligent agents when performing autonomous driving path planning.

[0198] There may or may not be any correlation between the maximum preset range of the intelligent body to be planned and the multiple preset ranges.

[0199] For example, the largest preset range among the multiple preset ranges may be the maximum preset range. That is, the multiple preset ranges may be 3 meters, 5 meters, 10 meters, 20 meters, and 50 meters around the agent. The maximum preset range may be 50 meters around the test vehicle.

[0200] The maximum preset range in the test process is associated with multiple preset ranges in the training process. When the test vehicle makes a decision, other agents also need to make a decision. Action decisions between agents in different ranges are all included in the maximum preset range, with different reward weights. The complexity of the test scene is higher and closer to the real traffic flow. In actual traffic, a vehicle's action decision will also give priority to vehicles or pedestrians within 3 meters before and after, and will not skip vehicles or pedestrians within 3 meters before and after to consider vehicles or pedestrians 50 meters before and after.

[0201] In a possible embodiment, the training process of the model in Embodiment 1 and Embodiment 2 can be combined with training. For a model, the second reward weights in multiple preset ranges are used during training, and the first reward weights when making action decisions between different types of intelligent agents are also used to obtain a trained traffic flow simulation model. Then, for different first reward weights, multiple models are trained for testing. The multiple models trained in the above manner can take into account most traffic flow scenarios, making the simulation of the traffic flow environment more realistic, improving the accuracy of path planning, and improving the accuracy of testing autonomous driving.

[0202] Testing the action decisions of autonomous vehicles in different scenarios will help the autonomous vehicles obtain more valid data. The more different scenarios there are during the test, the more valid data there will be during the test, and the closer the test is to real traffic conditions, the better the test results will be, and the higher the accuracy will be.

[0203] Based on the same inventive concept as the above method embodiment, the embodiment of the present application also provides an autonomous driving path planning device. Fig.11 A schematic diagram of the structure of an automatic driving path planning device provided in an embodiment of the present application is shown.

[0204] The device 110 includes: an acquisition unit 1101, used to acquire the initial state and initial action of the intelligent body to be planned, a real-time information acquisition unit 1102, used to input the initial state and initial action into a trained traffic flow simulation model, and acquire real-time interaction information between multiple intelligent bodies in the traffic flow simulation model and the intelligent body to be planned, wherein the traffic flow simulation model is obtained by dividing multiple intelligent bodies into multiple groups and training them through an upper-layer architecture and a lower-layer architecture, wherein the lower-layer architecture performs intra-group collaborative training on each group, and the upper-layer architecture performs inter-group collaborative training on multiple groups, a decision unit 1103, used to decide the real-time action and real-time state of the intelligent body to be planned according to the real-time interaction information, and a path planning unit 1104, used to determine the travel path according to the real-time action and real-time state in a preset time period.

[0205] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0206] Refer to the following Fig.12 The electronic device 1200 according to this embodiment of the present invention is described. Fig.12 The electronic device 1200 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0207] like Fig.12 As shown, the electronic device 1200 is in the form of a general computing device. The components of the electronic device 1200 may include but are not limited to: at least one processing unit 1210, at least one storage unit 1220, and a bus 1230 connecting different system components (including the storage unit 1220 and the processing unit 1210).

[0208] The storage unit stores program codes, which can be executed by the processing unit 1210, so that the processing unit 1210 executes the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0209] The storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 12201 and / or a cache storage unit 12202 , and may further include a read-only storage unit (ROM) 12203 .

[0210] The storage unit 1220 may also include a program / utility 12204 having a set (at least one) of program modules 12205, such program modules 12205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0211] Bus 1230 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0212] The electronic device 1200 may also communicate with one or more external devices 1240 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1200, and / or communicate with any device that enables the electronic device 1200 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1250. Furthermore, the electronic device 1200 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1260. As shown, the network adapter 1260 communicates with other modules of the electronic device 1200 via a bus 1230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0213] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0214] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0215] A program product for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0216] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0217] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0218] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0219] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0220] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0221] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0222] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0223] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. An automatic driving path planning method, characterized in that: The method comprises: Get the initial state and initial action of the agent to be planned; The initial state and initial action are input into a trained traffic flow simulation model to obtain real-time interaction information between multiple agents in the traffic flow simulation model and the agent to be planned; the traffic flow simulation model is obtained by dividing multiple agents into multiple groups and training them through an upper-layer architecture and a lower-layer architecture; wherein the lower-layer architecture performs intra-group collaborative training on each group, and the upper-layer architecture performs inter-group collaborative training on multiple groups; Determine the real-time action and real-time state of the to-be-planned intelligent agent according to the real-time interaction information; The travel path is determined according to the real-time action and the real-time status in a preset time period.

2. The method according to claim 1, characterized in that The plurality of agents include agents of autonomous driving type; The method further comprises: Training multiple traffic flow simulation models; wherein, during the training process of each traffic flow simulation model, a different first reward weight is generated for an intelligent agent of the autonomous driving type; the first reward weight represents the reward value of other types of intelligent agents when performing decision-making actions on the intelligent agent of the autonomous driving type in any traffic flow simulation scenario; wherein, the larger the first reward weight, the larger the reward value; the reward value affects the convergence of the loss value of the traffic flow simulation model; According to different first reward weights, a trained traffic flow simulation model of an autonomous driving type agent is obtained in each traffic flow simulation scenario.

3. The method according to claim 2, characterized in that The method further comprises: During the autonomous driving test, the initial state and initial action of the test agent to be tested are obtained; Input the initial state and initial action of the test agent to be tested into multiple trained traffic flow simulation models respectively to obtain the real-time action and real-time state under different traffic flow simulation scenarios; According to the real-time actions and real-time states in different traffic flow scenarios, the travel path in each traffic flow scenario is determined to obtain multiple groups of travel paths.

4. The method according to claim 1, characterized in that The method further comprises: During the training process of the traffic flow simulation model, for any one intelligent agent, a plurality of preset ranges are generated with the any one intelligent agent as the center; For any of the intelligent agents, when making a decision action to other intelligent agents located in different preset ranges, different second reward weights are generated; the smaller the preset range from any of the intelligent agents, the greater the second reward weight; According to different second reward weights, a trained traffic flow simulation model is obtained.

5. The method according to claim 1, characterized in that The training process of the traffic flow simulation model includes: Acquire multiple intelligent agents in a traffic flow, wherein the intelligent agents are objects that interact with each other; Collect the current observation state set, action set, reward set and next observation state set of the multiple agents, and store them in a database as a training data set; the reward set represents the reward value corresponding to different actions decided by the agents; Grouping multiple agents to obtain multiple groups; The upper layer architecture and the lower layer architecture of the traffic flow simulation model are trained according to the training data set to obtain a trained traffic flow simulation model.

6. The method according to claim 5, characterized in that The step of training the upper structure and the lower structure of the traffic flow simulation model according to the training data set to obtain the trained traffic flow simulation model comprises: Extract training samples from the training data set, and iteratively train the traffic flow simulation model according to the training samples; Determining a first loss value of the lower architecture and a second loss value of the upper architecture; If the first loss value of the lower-level architecture does not meet the first preset value, or the second loss value of the upper-level architecture does not meet the second preset value, the network parameters of the traffic flow simulation model are adjusted according to the preset learning rate until the first loss value meets the first preset value and the second loss value meets the second preset value, thereby obtaining a trained traffic flow simulation model.

7. The method according to claim 5 or 6, characterized in that: An iterative training process of the traffic flow simulation model includes: Based on the observed state and action at the current time step in the training sample and the observed state at the next time step, predict the action of each agent at the next time step; Determine the predicted group Q function for each group and the predicted local Q function for each agent in each group based on the observed state and action at the current time step, the observed state at the next time step, and the action of each agent at the next time step; According to the current set of observed states and actions, determine the true local Q function of each agent in each group and the importance weight parameter of each agent in the group; Determine the true group Q function of each group according to the importance weight parameter of each agent in the group, the set of observed states and the set of actions corresponding to the current time; The first loss value of the lower architecture and the second loss value of the upper architecture are determined based on the predicted group Q function of each group, the predicted local Q function of each agent in each group, the true group Q function of each group and the true local Q function of each agent in each group.

8. An automatic driving path planning device, characterized in that: include: An acquisition unit, used to acquire the initial state and initial action of the intelligent body to be planned; A real-time information acquisition unit, used for inputting the initial state and initial action into a trained traffic flow simulation model, and acquiring real-time interaction information between multiple agents in the traffic flow simulation model and the agent to be planned; the traffic flow simulation model is obtained by dividing multiple agents into multiple groups and training them through an upper-layer architecture and a lower-layer architecture; wherein the lower-layer architecture performs intra-group collaborative training on each group, and the upper-layer architecture performs inter-group collaborative training on multiple groups; A decision-making unit, used to decide the real-time action and real-time state of the to-be-planned intelligent agent according to the real-time interaction information; A path planning unit is used to determine the travel path according to the real-time action and real-time status in a preset time period.

9. An electronic device, characterized in that: include: processor; and a memory for storing executable instructions for the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

Images