Path planning method and vehicle

Abstract

The embodiment of the application discloses a path planning method and a vehicle, relates to the technical field of intelligent driving, and the method is applied to a central server and comprises the following steps: acquiring local strategy model parameters of a plurality of intelligent agents for planning a path; training global strategy model parameters commonly used by all the intelligent agents by using the local strategy model parameters of the plurality of intelligent agents; and distributing the global strategy model parameters to the plurality of intelligent agents, so that the intelligent agents use the global strategy model parameters to construct a global strategy model, and plan a path by using the global strategy model. The technical scheme of the embodiment of the application can improve the generalization ability of the global strategy model, and is beneficial to improving the accuracy and planning efficiency of a path planning result.

Description

Technical Field

[0001] This application relates to the field of intelligent driving, specifically to a path planning method and a vehicle. Background Technology

[0002] In scenarios such as multi-agent collaborative operations, path planning is a key technology. Currently, a path planning strategy model is typically trained using a large amount of sample data, and this model is then used to achieve automatic path planning.

[0003] In related technologies, local training is commonly used, where each agent is trained independently on its own local machine, and the path planning strategy model obtained from local training is used to plan the path for that agent. However, models trained in isolation have insufficient generalization ability, and the resulting path planning results are prone to local optima, failing to meet the requirements for accurate and efficient path planning in various scenarios. Summary of the Invention

[0004] In view of this, the embodiments of this application aim to provide a path planning method and vehicle to solve the problem that the poor generalization ability of a single agent's locally trained model easily leads to local optima in the path planning results, which is beneficial to improving the accuracy of the path planning results.

[0005] In a first aspect, one embodiment of this application provides a path planning method applied to a central server. The method includes: obtaining local policy model parameters of multiple agents for path planning; using the local policy model parameters of the multiple agents to train globally shared global policy model parameters; and distributing the global policy model parameters to the multiple agents so that the agents can use the global policy model parameters to construct a global policy model and perform path planning through the global policy model.

[0006] This application embodiment trains a globally shared global policy model by aggregating the local policy model parameters obtained from training multiple agents, and then distributes them to each agent for use. This integrates the training experience of multiple agents, improves the generalization ability of the global policy model, and solves the problem that the poor generalization ability of a single agent's locally trained model easily leads to local optima in path planning results. This is beneficial to improving the accuracy and efficiency of path planning results.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, global policy model parameters shared globally are trained using the local policy model parameters of multiple agents. This includes: performing aggregation processing on the currently acquired local policy model parameters of multiple agents to obtain intermediate model parameters; distributing the intermediate model parameters to multiple agents so that the agents can perform local training based on the intermediate model parameters and generate new local policy model parameters; uploading the newly generated local policy model parameters to the central server; repeating the aggregation processing, distribution, local training, and uploading until the target convergence condition is met; and using the intermediate model parameters obtained from the last aggregation as the global policy model parameters.

[0008] This application provides an iterative training process that ensures that the global policy model parameters are not generated by a single aggregation, but are gradually converged through multiple rounds of local training by each agent and multiple rounds of aggregation iteration by the central server. This process can gradually optimize the model parameters, allowing them to continuously absorb the learning experience of each agent in multiple iterations, thereby gradually improving the model's decision-making accuracy and generalization ability. This solves the problem of insufficient training of the global policy model and difficulty in converging to the optimal solution, ultimately resulting in a global policy model parameter with stable performance and adaptability to multi-scenario collaborative operations.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, aggregation processing is performed on the local policy model parameters of the multiple agents currently acquired to obtain intermediate model parameters, including: performing an arithmetic average on the local policy model parameters of the multiple agents currently acquired, and using the obtained arithmetic average as the intermediate model parameters.

[0010] This application embodiment performs an arithmetic mean operation on all local policy model parameters participating in the aggregation, and uses this arithmetic mean as an intermediate model parameter. This achieves simple fusion with equal weights, avoiding complex weighted or encrypted aggregation calculations. As a result, it enables efficient and unbiased calculation of unified intermediate model parameters in multi-agent collaborative training, effectively reducing the computational complexity and communication cost of the aggregation process.

[0011] Secondly, one embodiment of this application provides a path planning method applied to an intelligent agent. The method includes: receiving global policy model parameters distributed by a central server; wherein the global policy model parameters are generated based on a path planning method; obtaining a global policy model by replacing the local policy model parameters of a local policy model with the global policy model parameters; and planning the driving path of the intelligent agent based on the global policy model.

[0012] This application embodiment directly applies a pre-trained global policy model, allowing agents to obtain a unified policy model that integrates the collaborative experience of multiple agents without retraining. This enables more accurate path planning results in different scenarios, solves the problem of poor model generalization, and effectively improves the safety and efficiency of path planning.

[0013] In conjunction with the second aspect, in some implementations of the first aspect, the method further includes: obtaining an experience tuple; wherein the experience tuple includes the agent's state information, actions, rewards, and next state; using the state information and actions as inputs to the value network, updating the value network so that the difference between the output value of the value network and the target value is less than a target threshold; wherein the target value is determined based on the discounted value of the reward and the next state; using the updated value network, determining the evaluation value of the output action of the policy network, and adjusting the parameters of the policy network in the direction that increases the evaluation value to obtain the updated policy network; and using the parameters of the updated policy network as parameters of the local policy model.

[0014] This application embodiment utilizes state and action inputs to update the value network so that its output approaches the target value. Then, the updated value network evaluates the value of the action output by the policy network, and the policy network parameters are adjusted in the direction of increasing the evaluation value. This enables the intelligent agent to learn from decision-making and optimize action selection based on immediate rewards and future expected values. This solves the problem that the policy model cannot self-optimize after deployment, effectively realizes policy optimization, and further improves the path planning performance of the policy model.

[0015] In conjunction with the second aspect, in some implementations of the first aspect, before obtaining the experience tuple, the method further includes: generating state information based on the agent's first state vector; wherein the first state vector is used to reflect the agent's driving state and / or environmental perception results; inputting the state information into a policy network to generate an action to be executed; determining the reward and next state corresponding to the action; and storing the state information, action, reward, and next state as an experience tuple in the agent.

[0016] The embodiments of this application generate experience tuples through a closed-loop operation of the intelligent agent autonomously performing perception of state, decision-making, action execution, and observation results. This provides high-quality training samples for subsequent updates to local policy model parameters, which helps improve training accuracy.

[0017] In conjunction with the second aspect, in some implementations of the first aspect, generating state information based on the agent's first state vector includes: obtaining the agent's second state vector; and fusing the first and second state vectors to generate fused state information.

[0018] In this embodiment, when generating state information based on the first state vector of the intelligent agent itself, the second state vector of the collaborating intelligent agent is obtained, and the two are fused to generate more comprehensive state information that incorporates information from multiple parties. This allows the state information obtained by the intelligent agent to be extended beyond its own sensors and local data to a wider range, thereby solving the problem of insufficient decision-making information due to the limitations of perception. It enables the training of policy models based on more comprehensive state information, allowing the policy models to learn the interaction rules in a multi-agent collaborative environment and improve the decision-making accuracy of the trained policy models in collaborative operation scenarios.

[0019] In conjunction with the second aspect, in some implementations of the first aspect, the first state vector and the second state vector are fused to generate fused state information, including: determining the attention weight of the agent to the collaborative agent based on the first state vector, the second state vector, and the spatial relationship information between the agent and the collaborative agent; and performing a weighted summation of the first state vector and the second state vector based on the attention weight to obtain the state vector.

[0020] This application's embodiments introduce spatial relationships as a key factor to dynamically calculate attention weights. The agent can focus on the information of those cooperative agents that are spatially closer, while automatically ignoring or weakening the interference information of those irrelevant agents, thereby improving the information quality and decision value of the fused state vector.

[0021] In conjunction with the second aspect, in some implementations of the first aspect, the method further includes: incrementally training the global policy model based on the perception data of the agent during actual driving to obtain training results; and updating the global policy model based on the training results.

[0022] In this embodiment, the intelligent agent continuously collects perception data during actual driving. Based on this newly generated data containing information about environmental changes, the current global policy model is incrementally trained, and the training results are used to update the model. This enables the intelligent agent to fine-tune the model using perception data, thereby solving the problem of performance degradation or even failure of pre-trained models when facing new scenarios, and ensuring the path planning performance of the intelligent agent under different environments and long-term operation.

[0023] Thirdly, one embodiment of this application provides a computer-readable storage medium storing a computer program for performing the above-described method.

[0024] Fourthly, one embodiment of this application provides a vehicle, the vehicle including: a processor; a memory for storing processor-executable instructions; the processor being used to perform the above-described method.

[0025] Fifthly, one embodiment of this application provides a computer program product including instructions that, when executed on a vehicle, cause the vehicle to perform the above-described method.

[0026] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 The diagram shown is a schematic flowchart of a path planning method provided in an embodiment of this application.

[0029] Figure 2 The diagram shown is a schematic diagram of the training process of a path planning strategy model provided in an embodiment of this application.

[0030] Figure 3 The diagram shown is a flowchart of a path planning method provided in another embodiment of this application.

[0031] Figure 4 The diagram shown is a schematic diagram of the path planning device provided in an embodiment of this application.

[0032] Figure 5 The diagram shown is a structural schematic of a vehicle provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0034] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0035] It should be understood that the term "and / or" used in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Furthermore, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. Additionally, the term "based on" used in this document is not limited to relying solely on one object. For example, determining B based on A can mean: determining B based solely on A, or determining B partially based on A.

[0036] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security and network security.

[0037] In multi-agent collaborative operation scenarios, each agent needs to cooperate to complete the task. This agent is a mobile computing entity capable of perceiving the environment, making autonomous decisions, and executing actions. Examples include automated container transfer in ports, automated transportation in mining areas, and automated delivery in industrial parks. In these scenarios, each agent's path not only needs to consider its own travel requirements but also needs to adapt to the paths of the collaborating agents to avoid conflicts and improve overall traffic efficiency. Therefore, this places higher demands on the global adaptability of the path planning model.

[0038] In related technologies, the training method for path planning strategy models involves each agent independently training based solely on data collected during its own driving process. Because each agent can only learn from its limited experience data and lacks experience sharing with collaborating agents, the trained model suffers from insufficient generalization ability. Collaborating agents are those that need to work in concert with the collaborating agent. In practical applications, when faced with new scenarios not present in the local training data, the model's path planning results are prone to getting trapped in local optima, making it difficult to maintain accurate and efficient planning performance in various complex dynamic environments. For example, when an agent enters an unfamiliar area it has never traversed, the independently trained model often struggles to output accurate and reasonable path planning results, easily leading to problems such as detours and conflicts with the collaborating agent's path, affecting the overall traffic efficiency and driving safety of the operation scenario.

[0039] To address the aforementioned issues, this application proposes a path planning method. This method utilizes local policy model parameters obtained from training multiple agents through a central server to train global policy model parameters. The trained global policy model parameters are then distributed to each agent for their use. This approach integrates the training experience of multiple agents and enhances the model's generalization ability.

[0040] To more clearly illustrate the embodiments of this application, an implementation environment is first described. The training system for the path planning strategy model includes a central server and multiple agents. The central server can be a backend service or roadside computing unit with strong computing power deployed in the cloud, used to perform model parameter training and distribution. Each agent can be a car, drone, robot, etc., with autonomous driving capabilities, and internally deploys a policy network, a value network, an experience replay buffer, a computing unit (processor), and a memory (for storing processor-executable instructions). Agents establish a bidirectional data connection with the central server through their communication modules (e.g., mobile communication modules or dedicated short-range communication modules). In the data flow direction, each agent uploads its locally trained local policy model parameters to the central server; the central server then trains the received parameters and distributes the generated global policy model parameters to each agent.

[0041] The following is combined Figures 1 to 3 The path planning method provided in the embodiments of this application will be described in detail.

[0042] Figure 1 The diagram shown is a flowchart of a path planning method provided in an embodiment of this application. like Figure 1 As shown, this method is applied to a central server and includes the following steps.

[0043] Step S110: Obtain the local policy model parameters of multiple agents for path planning.

[0044] In this embodiment, policy model parameters define the set of all learnable weights and biases for a policy model. Specifically, the policy model can be a policy network in a deep deterministic policy gradient algorithm, with a state vector as input and actions (such as throttle, steering angle, braking, etc.) as output. The policy network employs a multi-layer fully connected neural network, using linear rectified functions in the intermediate layers and a probability distribution of the action space as the output layer. By adjusting these parameters, the agent's decision-making behavior in different states can be altered. Local policy model parameters are the model parameters that are independently run and trained on each agent for planning its own path.

[0045] In a specific embodiment, the central server can periodically, or upon receiving a specific request, listen for and receive parameter reports from each authenticated agent through a secure communication protocol. For example, in a distributed multi-agent path planning scenario, N autonomous vehicles participate in collaborative work, and these N vehicles can be trained collaboratively. After each vehicle completes a certain number of rounds of policy network training locally, it uses the weight matrices and bias vectors of all layers in its policy network (e.g., a fully connected neural network containing an input layer, three hidden layers, and one output layer) as local policy model parameters, serializes them into a structured data packet, and uploads the data packet through the communication module. The central server then parses the data packet, stores the parameters of each agent in a temporary storage space, and completes model training based on the local policy model parameters.

[0046] Step S120: Use the local policy model parameters of multiple agents to train the globally shared global policy model parameters.

[0047] The global policy model parameters are policy network parameters that are uniformly trained by the central server and shared by all agents participating in the training.

[0048] In practice, the central server first performs an aggregation process on the local policy model parameters of all agents currently received, and uses the aggregated parameters as the global policy model parameters. For example, for each parameter dimension, the median of the local policy model parameters of all agents in that dimension is taken as the global policy model parameter for that dimension.

[0049] Step S130: Distribute the global policy model parameters to multiple agents so that the agents can use the global policy model parameters to construct a global policy model and perform path planning through the global policy model.

[0050] In practice, the distribution method corresponds to the parameter uploading method. The central server can package and send the aggregated global policy model parameters to all participating agents through the same secure communication protocol.

[0051] Upon receiving this parameter, the agent replaces the locally stored local policy model parameters with the received global policy model parameters, thereby updating its local policy model to a unified global policy model that incorporates experience from multiple parties. Subsequently, when performing path planning tasks, the agent can make decisions based on this updated global policy model.

[0052] To further illustrate the implementation of the embodiments of this application, the complete operation process of the embodiments of this application is described below in the context of a scenario in which multiple automated guided vehicles (AGVs) work together to transport materials in a large logistics warehouse.

[0053] In this scenario, dozens of automated guided vehicles (AGVs) are responsible for moving goods between numerous shelves. The shelf layout may be frequently adjusted, and temporarily stacked packages may block aisles. All AGVs (agents) are deployed with an initial, randomly generated policy network.

[0054] In the first phase, each vehicle conducts an initial exploration based on its local stochastic policy, collects some empirical data, and uses the deep deterministic policy gradient algorithm to train the policy network locally, generating a local policy model.

[0055] Subsequently, the central server (deployed on the central computing node of the warehouse) initiated the first round of collaborative training. It received the local policy model parameters from all participating guide vehicles and aggregated these parameters to generate a global policy model parameter. This global policy model parameter incorporated the exploration experience of all vehicles in the first phase, breaking down information silos.

[0056] Finally, the central server distributes this global strategy model parameter to all guided vehicles. Upon receiving it, each guided vehicle immediately replaces its local strategy model parameter with the global strategy model parameter. During one journey, guided vehicle 1 encountered a temporary cargo pile it had never seen before. Because its strategy incorporated the experience of other guided vehicles (who may have encountered similar obstacles in other areas), guided vehicle 1 adjusted its steering angle, bypassed the obstacle, and achieved accurate obstacle avoidance.

[0057] This application embodiment trains a globally shared global policy model by aggregating the local policy model parameters obtained from training multiple agents, and then distributes them to each agent for use. This integrates the training experience of multiple agents, improves the generalization ability of the global policy model, and solves the problem that the poor generalization ability of a single agent's locally trained model easily leads to local optima in path planning results. This is beneficial to improving the accuracy and efficiency of path planning results.

[0058] To further optimize the stability of the generated global policy model parameters in the above embodiments, this application also provides the following preferred solution. Specifically, using the local policy model parameters of multiple agents to train globally shared global policy model parameters includes: performing aggregation processing on the currently acquired local policy model parameters of multiple agents to obtain intermediate model parameters; distributing the intermediate model parameters to multiple agents so that the agents can perform local training based on the intermediate model parameters and generate new local policy model parameters; uploading the newly generated local policy model parameters to the central server; repeating the aggregation processing, distribution, local training, and uploading until the target convergence condition is met; and using the intermediate model parameters obtained from the last aggregation as the global policy model parameters.

[0059] Intermediate model parameters are the model parameters obtained from each round of aggregation processing that have not yet met the convergence condition. The target convergence condition is the condition that stops iterative training; once the target convergence condition is met, it indicates that the model's performance or parameter state has reached a preset stable level.

[0060] In some embodiments, the aggregation process is performed on the local policy model parameters of the currently acquired multiple agents to obtain intermediate model parameters, including: performing an arithmetic mean on the local policy model parameters of the currently acquired multiple agents, and using the obtained arithmetic mean as the intermediate model parameters.

[0061] For example, intermediate model parameters can be determined using a federated averaging algorithm. The specific calculation formula is as follows: Where N is the total number of agents participating in the training. This represents the local policy model parameters of the i-th agent; the federated averaging algorithm performs an arithmetic average of all local policy model parameters, meaning that each agent's parameters contribute equally to the global strategy.

[0062] This application embodiment performs an arithmetic mean operation on all local policy model parameters participating in the aggregation, and uses this arithmetic mean as an intermediate model parameter. This achieves simple fusion with equal weights, avoiding complex weighted or encrypted aggregation calculations. As a result, it enables efficient and unbiased calculation of unified intermediate model parameters in multi-agent collaborative training, effectively reducing the computational complexity and communication cost of the aggregation process.

[0063] Furthermore, these intermediate model parameters are distributed to each agent. Upon receiving these intermediate model parameters, each agent uses them as the initial parameters for its local policy network and, based on pre-collected empirical data, performs several rounds of local training using reinforcement learning algorithms such as deep deterministic policy gradients to generate a new and better set of local policy model parameters.

[0064] Then, the agent uploads this new set of parameters to the central server. This closed-loop iterative process of "aggregation processing, distribution, local training, and uploading" is repeated until a preset convergence condition is met (e.g., the Euclidean distance between the model parameters generated in two consecutive iterations is less than a positive number, or a preset maximum number of iterations is reached). After meeting this convergence condition, the intermediate model parameters generated in the last aggregation are used as the final global policy model parameters.

[0065] This application provides an iterative training process that ensures that the global policy model parameters are not generated by a single aggregation, but are gradually converged through multiple rounds of local training by each agent and multiple rounds of aggregation iteration by the central server. This process can gradually optimize the model parameters, allowing them to continuously absorb the learning experience of each agent in multiple iterations, thereby gradually improving the model's decision-making accuracy and generalization ability. This solves the problem of insufficient training of the global policy model and difficulty in converging to the optimal solution, ultimately resulting in a global policy model parameter with stable performance and adaptability to multi-scenario collaborative operations.

[0066] Figure 2 The diagram shown is a schematic representation of the training process of a path planning strategy model provided in another embodiment of this application; as follows: Figure 2 As shown, this method is applied to an intelligent agent and includes the following steps.

[0067] Step S210: Receive the global policy model parameters distributed by the central server.

[0068] The global strategy model parameters are generated based on the path planning method applied to the central server.

[0069] Step S220: The global strategy model is obtained by replacing the local strategy model parameters of the local strategy model with the global strategy model parameters.

[0070] After receiving the global policy model parameters, the agent will replace the locally stored local policy model parameters with the received global policy model parameters, thereby updating the local policy model to a unified global policy model that integrates the experience of multiple parties.

[0071] Step S230: Based on the global policy model, plan the driving path of the agent.

[0072] For example, the agent perceives the external environment in real time (such as the intersection ahead, the speed and position of vehicles approaching from the left and right), encodes this perception information into a state vector, and inputs it into the updated global policy model. The global policy model directly outputs an action instruction (such as "turn right at a speed of 25 km / h"), and the agent's execution module then executes the action to achieve path planning.

[0073] This application embodiment directly applies a pre-trained global policy model, allowing agents to obtain a unified policy model that integrates the collaborative experience of multiple agents without retraining. This enables more accurate path planning results in different scenarios, solves the problem of poor model generalization, and effectively improves the safety and efficiency of path planning.

[0074] To enable the policy model deployed on the agent to continuously learn and self-optimize, embodiments of this application provide an implementation method for training local policy model parameters. Optionally, the method further includes: obtaining an experience tuple; wherein the experience tuple includes the agent's state information, actions, rewards, and next state; using the state information and actions as inputs to a value network, updating the value network so that the difference between the output value of the value network and the target value is less than a target threshold; wherein the target value is determined based on the discounted value of the reward and the next state; using the updated value network, determining the evaluation value of the output action of the policy network, and adjusting the parameters of the policy network in the direction that increases the evaluation value to obtain the updated policy network; and using the parameters of the updated policy network as local policy model parameters.

[0075] It should be noted that the local policy model trained in this embodiment can be an initially deployed random parameter model or a model whose parameters have been updated by the global policy model. After each path exploration and action execution, the agent collects a new set of experience tuples, stores these experience tuples in the local experience replay pool, and then randomly samples a batch of experience tuples from the experience replay pool. Following the above process, the parameters of the value network and policy network are updated sequentially, ultimately obtaining the trained local policy model parameters and completing this local training process.

[0076] Experience tuples are structured data units generated from the agent's driving interaction records, used for model training. Experience tuples include the agent's state information, actions, rewards, and next state. State information includes the agent's driving state and environmental perception results; for example, state information includes current position coordinates, exploration path records, local environmental observations, and gradients of policy network parameters.

[0077] The action is the control output of the agent, such as the throttle opening, steering angle adjustment, or braking force. The reward is a feedback value determined based on whether the action meets the expected goal of path planning (such as whether obstacle avoidance is safe, whether the path to the target point is shortened, and whether traffic rules are followed). The next state is the new state that the agent enters after executing the action.

[0078] In practice, the agent uses these experience tuples to update its value network. Specifically, the agent takes the state information and actions in the experience tuples as input to the value network and calculates the output value of the value network, which is an evaluation of whether performing this action in the current state is good or bad.

[0079] Simultaneously, a target value is calculated based on the reward in the experience tuple and the discounted value of the next state. For example, according to the Bellman equation, the target value is obtained by adding the current reward to the maximum discounted value of the next state. The agent, through training, continuously adjusts the parameters of the value network to make its output value as close as possible to this target value, until the difference between the two is less than a target threshold.

[0080] The updated value network is used to evaluate the value of the action output by the policy network. Specifically, the policy network takes the current state as input and outputs an action. The value network then evaluates this action and outputs an evaluation value. Following the principle of adjusting the policy network parameters in the direction that increases the evaluation value, the parameters of the policy network are updated using the backpropagation algorithm. Finally, the agent uses the updated policy network parameters as the new parameters for its local policy model.

[0081] This application embodiment utilizes state and action inputs to update the value network so that its output approaches the target value. Then, the updated value network evaluates the value of the action output by the policy network, and the policy network parameters are adjusted in the direction of increasing the evaluation value. This enables the intelligent agent to learn from decision-making and optimize action selection based on immediate rewards and future expected values. This solves the problem that the policy model cannot self-optimize after deployment, effectively realizes policy optimization, and further improves the path planning performance of the policy model.

[0082] Before performing the aforementioned experience-based model update, an experience tuple is required as training data. This application provides a method for determining the experience tuple. In some embodiments, before obtaining the experience tuple, the method further includes: generating state information based on the agent's first state vector; inputting the state information into a policy network to generate an action to be executed; determining the reward and next state corresponding to the action; and storing the state information, action, reward, and next state as an experience tuple in the agent.

[0083] The first state vector is used to reflect the agent's driving state and / or environmental perception results; for example, it can be a multi-dimensional vector generated from the agent's sensor data after feature extraction and fusion, representing the environmental context in which the agent is located at a certain moment. The first state vector can be directly used as state information.

[0084] In some embodiments, state information is input to the agent's local policy network, which calculates and outputs an action to be executed based on the state information. For example, "maintain current speed and turn right 15 degrees." The agent's execution module then executes this action. It should be noted that the above action can be pre-executed in a simulation environment to collect sufficient experience tuples before entering the actual training phase; alternatively, experience tuples can be collected while the task is being executed during actual operation, achieving dynamic incremental data accumulation.

[0085] After an action is performed, the agent needs to determine the reward and the next state to be reached from the environmental feedback. For example, the reward... It can be determined using the following formula.

[0086] in, , , These are weighting coefficients that can be dynamically adjusted based on the application scenario. For example, in autonomous driving scenarios, safety takes priority. It can be set to 0.5; in drone delivery scenarios, efficiency is the priority, so it can be set to 0.5. Set it to 0.6. In warehousing and logistics scenarios, energy consumption is a priority, so it can be... Set it to 0.5. In multi-robot collaborative scenarios, collaboration takes priority, so it can be... Set it to 0.4.

[0087] Efficiency Rewards The formula used to encourage the agent to quickly approach the target is as follows: in, and These represent the distances between the agent and the target at times t-1 and t, respectively. The initial distance between the agent and the target at the start time.

[0088] Safety Rewards This is used to penalize the risk of collision between an intelligent agent and obstacles or collaborative intelligent agents (such as humans, robots, and drones). The calculation formula is as follows: in, Let i be the distance between the agent and the i-th obstacle in the environment; Let be the distance between the agent and the j-th cooperating agent in the collaborative training. and These are all attenuation coefficients, which can be preset according to the needs of the scenario. This formula will give a negative reward based on the distance between the agent and the obstacle or cooperative agent. The closer the distance, the greater the negative reward, thereby encouraging the agent to maintain a safe distance and reduce the probability of collision.

[0089] Energy consumption reward The formula used to encourage the planning of low-energy routes is as follows: in, This represents the energy consumption of the current step. This represents the maximum permissible energy consumption.

[0090] Collaborative Rewards This value is used to encourage information sharing and path complementarity among agents, reflecting the collaborative efficiency between them; a higher value indicates better collaboration. The formula is as follows: Where N is the number of agents trained collaboratively. Similarity is used to measure path similarity, and Overlap measures path overlap rate. This represents the path planning result for agent i. This represents the path planning result for the collaborative intelligent agent j.

[0091] Furthermore, sensor data collected after the action is performed is used to generate a multi-dimensional vector after feature extraction and fusion of the newly collected sensor data, which serves as the next state.

[0092] The state information, action, reward, and next state are stored as a complete experience tuple in the agent's local experience replay buffer for easy access during model training.

[0093] This application embodiment generates experience tuples through a closed-loop operation of the intelligent agent autonomously performing state perception, decision-making, action execution, and observation results. This provides high-quality, readily sampled training samples for subsequent local policy model parameter updates, which helps improve training accuracy.

[0094] To further enhance the richness of state information and improve the decision-making quality of the trained policy model, this application provides a preferred solution. Specifically, generating state information based on the agent's first state vector includes: obtaining the second state vector of the cooperative agent; and fusing the first and second state vectors to generate fused state information.

[0095] The second state vector may include the driving state of the cooperative agent and / or the results of environmental perception.

[0096] In practical implementation, agents can communicate directly with each other or obtain the second state vector broadcast by surrounding cooperative agents through roadside units or a central server. A weighted summation function or a more complex neural network is used to fuse the first and second state vectors to generate state information that simultaneously contains information about the agent and the cooperative agents.

[0097] In some embodiments, to avoid interference from invalid information, the first state vector and the second state vector are fused to generate fused state information, including: determining the attention weight of the agent to the collaborative agent based on the first state vector, the second state vector, and the spatial relationship information between the agent and the collaborative agent; and performing a weighted summation of the first state vector and the second state vector based on the attention weight to obtain the state vector.

[0098] Spatial relationship information is used to reflect the spatial location relationship between an agent and its collaborating agents.

[0099] In practice, the spatial relationship information (e.g., Euclidean distance) between the agent and the cooperative agents trained together is determined, and the attention weights of the agent towards the other cooperative agents are calculated. These attention weights reflect the importance of a particular cooperative agent's information to the agent in the current decision-making scenario. For example, a vehicle about to merge into the lane ahead will have a high attention weight, while a vehicle far behind traveling in the opposite direction will have a low weight.

[0100] Attention weights between agent i and cooperative agent j The calculation formula is as follows: in, Let be the first state vector. This is the second state vector. Let be the Euclidean distance between agent i and cooperative agent j, MLP be a multilayer perceptron, and || denote vector concatenation. This is the activation function.

[0101] Then, the agent performs a weighted sum of the first and second state vectors based on the calculated attention weights to generate the final state vector used for decision-making.

[0102] For example, the state vector is obtained by weighted summation using the following formula. .

[0103] In some embodiments, the agent updates its local policy model based on the fused state vector to achieve cooperative exploration. The information sharing frequency can be adjusted, such as communicating once every K steps, to balance communication cost and cooperative effect.

[0104] This application's embodiments introduce spatial relationships as a key factor to dynamically calculate attention weights. The agent can focus on the information of those cooperative agents that are spatially closer, while automatically ignoring or weakening the interference information of those irrelevant agents, thereby improving the information quality and decision value of the fused state vector.

[0105] In this embodiment, when generating state information based on the first state vector of the intelligent agent itself, the second state vector of the collaborating intelligent agent is obtained, and the two are fused to generate more comprehensive state information that incorporates information from multiple parties. This allows the state information obtained by the intelligent agent to be extended beyond its own sensors and local data to a wider range, thereby solving the problem of insufficient decision-making information due to the limitations of perception. It enables the training of policy models based on more comprehensive state information, allowing the policy models to learn the interaction rules in a multi-agent collaborative environment and improve the decision-making accuracy of the trained policy models in collaborative operation scenarios.

[0106] To enable an agent with a deployed global policy model to adapt to new and unseen dynamic environments, this application provides an incremental training scheme. Optionally, the method further includes: incrementally training the global policy model based on the agent's perception data during actual driving to obtain training results; and updating the global policy model based on the training results.

[0107] Perception data can be the data collected by the intelligent agent through sensors during actual driving.

[0108] In some embodiments, during the actual driving process of the intelligent agent planning the path according to the global policy model, it continuously uses its own sensors (cameras, lidar, etc.) to collect new perception data, which reflects the characteristics of the current environment.

[0109] Furthermore, based on newly generated perception data containing information about environmental changes, the global policy model currently being used by the agent is incrementally trained. Incremental training does not involve retraining the entire model from scratch, but rather fine-tuning it using the new data based on the current model.

[0110] For example, you can adjust only the parameters of the last few layers of the policy network, or use a very small learning rate to update all parameters.

[0111] In practice, the training results obtained from this incremental training (i.e., the updated model parameters) are used to update the global policy model currently running on the agent, so that the agent can plan a path based on the incrementally trained global policy model.

[0112] It should be noted that, in order to ensure the effectiveness of incremental learning, meta-learning can be used to train the local decision model. This ensures that the model (global policy model) after updating the parameters of the global policy model to the local decision model still retains the characteristic of quickly adapting to the new environment. As a result, in subsequent actual operation, only incremental training is needed to achieve efficient online adaptation.

[0113] For example, the meta-learning loss function is: in, Let P(T) be the expectation of the task distribution (representing the average loss across all possible tasks). For a single task, the loss function is... These are the initial parameters of the model (parameters that need to be learned during the meta-learning phase). The learning rate is the learning rate within the task (the step size of gradient descent on a single task). The gradient of the loss for a single task with respect to the model parameters; These are the parameters updated by the model on a single task (parameters after rapid adaptation). Through meta-learning, it is possible to adapt quickly to new environments with only a small number of samples, while maintaining planning performance.

[0114] In this embodiment, the intelligent agent continuously collects perception data during actual driving. Based on this newly generated data containing information about environmental changes, the current global policy model is incrementally trained, and the training results are used to update the model. This enables the intelligent agent to fine-tune the model using perception data, thereby solving the problem of performance degradation or even failure of pre-trained models when facing new scenarios, and ensuring the path planning performance of the intelligent agent under different environments and long-term operation.

[0115] The embodiments of the path planning method have been described in detail above. To enable those skilled in the art to further understand the technical solution of this method, the following will be combined with... Figure 3 Give an example to illustrate this method.

[0116] In a large smart warehouse, three automated guided vehicles (AGVs) need to work together to complete a transport task: moving a batch of goods from sorting area A to packing area B. The warehouse environment is dynamic, with factors such as temporarily stacked goods and the presence of other AGVs. The strategy model for the AGVs is trained as follows in this scenario.

[0117] First, a path planning task request is received, which includes information such as the starting point location, ending point location, guide vehicle identifier, and priority. Based on the task requirements, a strategy model for each guide vehicle is initialized. Each guide vehicle is configured with an independent state-action-reward architecture, including an environment observation module, a decision-making module, an execution module, and a communication module.

[0118] Each guided vehicle's environmental observation module acquires real-time perception data through sensors such as LiDAR, cameras, GPS, and inertial measurement units. This includes, for example, the location of static obstacles (such as buildings and fixed structures), the location and speed of dynamic obstacles (such as pedestrians and vehicles), and environmental status information (such as weather, lighting, and road conditions). A graph neural network is used to encode the perception data into high-dimensional state vectors. The graph neural network abstracts the perception data into a graph structure, where nodes represent passable areas, edges represent adjacency relationships, and node features encode information such as obstacle type and passage cost. Then, distributed exploration is initiated, with each guided vehicle starting from its initial position and exploring the environment in parallel along different paths. Each guided vehicle makes independent decisions and executes actions.

[0119] The guided vehicle exchanges information (such as current location, exploration path, local environment observation, policy network parameter gradient, etc.) through a communication module. The communication frequency can be adjusted, such as once every K steps.

[0120] The guided vehicle employs an attention mechanism to fuse information from other guided vehicles. For example, guided vehicle 1 calculates attention weights for guided vehicles 2 and 3 based on their Euclidean distances. Higher attention weights are assigned to guided vehicle 2, which is closer and will influence its own path, while lower attention weights are assigned to guided vehicle 3, which is farther away. The state vectors of the guided vehicle and other guided vehicles are then weighted and fused based on these attention weights to obtain more comprehensive state information. Furthermore, a dynamic reward is calculated for each action performed. Specifically, rewards are calculated based on multiple dimensions, including efficiency, safety, energy consumption, and collaboration. The weight coefficients of the reward function are adaptively adjusted according to environmental conditions and scenario requirements.

[0121] Experience tuples are generated based on state information, actions, rewards, and the next action. Each guided vehicle trains its local policy network and value network using a deep deterministic policy gradient algorithm based on the experience tuples in its local experience replay buffer.

[0122] Policy Network The update is performed using gradient ascent, with the goal of maximizing the objective function. The gradient formula is: in, The gradient of the objective function corresponding to the policy network parameters represents the direction of change of the policy network parameters. For a policy network, the input state is s, and the output action is... a . For the state in the experience playback buffer D and actions Calculate the expectation, which represents the average over a large amount of experience. This represents the gradient of the action output by the policy network with respect to the policy network parameters. For value networks Action The gradient represents the impact of changes in action on value.

[0123] Value Network By minimizing the time-difference error update, the loss function for: The loss function measures the error between the value network's predicted and actual state-action pairs, and the goal is to minimize this function. The loss function is reduced by updating the parameters of the value network through backpropagation.

[0124] This represents the expectation of the experience tuples in the experience playback buffer D.

[0125] For the value network to perform actions in state s The evaluation value. This indicates that an action is performed in state s. The reward received later. This is a discount factor used to weigh the current reward against the future reward, and its value ranges from [0, 1]. For the value network in the next state The evaluation value of the actions generated by the policy network is then used.

[0126] Each guided vehicle uploads the parameters of its trained policy network as local policy model parameters to the central server. The central server uses a federated averaging algorithm to aggregate the parameters of each local policy model to obtain intermediate model parameters, which are then distributed to each guided vehicle for local training. The trained model parameters are then uploaded back to the central server to determine if training is complete (meeting the target convergence condition). If not, the parameters are aggregated again, and the aggregated parameters are distributed to the guided vehicles for repeated local training. If training is complete, globally shared global policy model parameters are obtained and deployed on each guided vehicle.

[0127] Using the parameters of the trained global strategy model, path planning is performed during the task execution phase of the guided vehicle to generate its own optimal path.

[0128] During operation, new experiences are continuously collected to update the strategy model parameters, enabling online learning and performance improvement. Furthermore, the status of the guided vehicle is monitored in real time to determine if any abnormalities occur (e.g., path infeasibility, guided vehicle malfunction, communication interruption, and planning timeout). If an anomaly is detected, emergency handling is implemented. For example, emergency handling strategies include: triggering the execution of alternative paths, re-initializing the strategy model, and downgrading to a traditional algorithm (e.g., A). This involves planning and recording anomalies. If no anomalies are detected, online learning and status monitoring continue, forming a closed loop.

[0129] This application first employs a graph neural network to encode the perceived data into a high-dimensional state vector to capture the topological structure and spatial relationships of the environment. Multiple agents explore in parallel along different paths, and through information sharing, the model is trained based on the perceived data of multiple agents to address the problems of limited perception range and insufficient experience of a single agent. Furthermore, the training phase adopts a decentralized training and centralized execution framework, enabling the trained global policy model to provide more accurate and reasonable path planning results in a dynamic environment of multi-agent collaborative operation. Through online learning, new experiences are continuously collected and policy network parameters are updated during agent operation, achieving continuous performance improvement. In addition, anomaly detection and fault tolerance mechanisms are integrated, which can maintain services through emergency strategies such as degradation switching in the event of agent failure, communication interruption, or other anomalies, thereby improving the overall efficiency, robustness, and adaptability of path planning.

[0130] The above text combined Figures 1 to 3 The path planning method embodiments of this application are described in detail below, in conjunction with... Figure 4 This application provides a detailed description of embodiments of the path planning apparatus. It should be understood that the descriptions of the path planning method embodiments correspond to the descriptions of the path planning apparatus embodiments; therefore, any parts not described in detail can be found in the preceding method embodiments.

[0131] Figure 4 The diagram shown is a structural schematic of a path planning device provided in an embodiment of this application. Figure 4 As shown, the path planning device provided in this embodiment is configured on a central server, and the path planning device 40 includes: The parameter acquisition module 410 is used to acquire local policy model parameters for multiple agents to plan paths. Training module 420 is used to train globally shared global policy model parameters using local policy model parameters of multiple agents. The parameter distribution module 430 is used to distribute global policy model parameters to multiple agents so that the agents can use the global policy model parameters to build a global policy model and perform path planning through the global policy model.

[0132] In one embodiment of this application, the training module 420 is further configured to: perform aggregation processing on the local policy model parameters of the multiple agents currently acquired to obtain intermediate model parameters; distribute the intermediate model parameters to the multiple agents so that the agents can perform local training based on the intermediate model parameters and generate new local policy model parameters; upload the newly generated local policy model parameters to the central server; repeat the aggregation processing, distribution, local training and uploading until the target convergence condition is met, and use the intermediate model parameters obtained from the last aggregation as the global policy model parameters.

[0133] In one embodiment of this application, the training module 420 is further configured to perform an arithmetic mean on the local policy model parameters of the multiple agents currently acquired, and use the obtained arithmetic mean as intermediate model parameters.

[0134] This application embodiment also provides a path planning device configured in an intelligent agent, the device comprising: The parameter receiving module is used to receive global policy model parameters distributed by the central server; the global policy model parameters are generated based on the path planning method. The parameter replacement module is used to obtain the global strategy model by replacing the local strategy model parameters of the local strategy model with the global strategy model parameters. The path planning module is used to plan the driving path of the agent based on the global policy model.

[0135] In one embodiment of this application, the device further includes: The model generation module is used to acquire experience tuples, which include the agent's state information, actions, rewards, and next state. The state information and actions are used as inputs to the value network to update it, ensuring that the difference between the value network's output value and the target value is less than a target threshold. The target value is determined based on the discounted value of the reward and the next state. Using the updated value network, the evaluation value of the policy network's output action is determined, and the parameters of the policy network are adjusted in the direction that increases the evaluation value to obtain the updated policy network. The parameters of the updated policy network are used as the parameters of the local policy model.

[0136] In one embodiment of this application, the device further includes: The experience tuple generation module is used to generate state information based on the agent's first state vector before acquiring the experience tuple; wherein the first state vector is used to reflect the agent's driving state and / or environmental perception results; the state information is input into the policy network to generate the action to be executed; the reward and next state corresponding to the action are determined; the state information, action, reward and next state are used as experience tuples and stored in the agent.

[0137] In one embodiment of this application, the experience tuple generation module is further configured to: obtain the second state vector of the cooperative agent; and perform fusion processing on the first state vector and the second state vector to generate fused state information.

[0138] In one embodiment of this application, the experience tuple generation module is further configured to: determine the attention weight of the agent to the collaborative agent based on the first state vector, the second state vector, and the spatial relationship information between the agent and the collaborative agent; and, based on the attention weight, perform a weighted summation of the first state vector and the second state vector to obtain a state vector.

[0139] In one embodiment of this application, the device further includes: The model update module is used to incrementally train the global policy model based on the perception data of the agent during actual driving, and obtain the training results; and update the global policy model based on the training results.

[0140] It is worth noting that in the embodiments of the above-mentioned path planning device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0141] Below, for reference Figure 5 To describe the vehicle according to embodiments of this application. Figure 5 The diagram shown is a structural schematic of a vehicle provided in an exemplary embodiment of this application.

[0142] like Figure 5 As shown, vehicle 50 includes one or more processors 501 and memory 502.

[0143] The processor 501 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the vehicle 50 to perform desired functions.

[0144] Memory 502 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and processor 501 may execute the program instructions to implement the various embodiments of this application described above and / or other desired functions.

[0145] In one example, vehicle 50 may also include input device 503 and output device 504, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0146] The input device 503 may include, for example, a keyboard, a mouse, etc.

[0147] The output device 504 can output various information to the outside. The output device 504 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0148] Of course, for the sake of simplicity, Figure 5 Only some of the components of the vehicle 50 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the vehicle 50 may include any other suitable components depending on the specific application.

[0149] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the path planning methods according to various embodiments of this application described above.

[0150] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0151] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the path planning methods according to the various embodiments of this application described above.

[0152] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0153] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details of the above application are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0154] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0155] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0156] The above description of the claimed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be applied within the widest scope consistent with the principles and novel features of this application.

[0157] The above description has been given for illustrative and descriptive purposes. Furthermore, this description is not intended to limit the embodiments of this application to the forms described herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A path planning method, characterized in that, Applied to a central server, the method includes: Obtain local policy model parameters for multiple agents to plan paths; Using the local policy model parameters of the multiple agents, a globally shared global policy model parameter is trained; The global policy model parameters are distributed to multiple agents so that the agents can use the global policy model parameters to construct a global policy model and perform path planning through the global policy model.

2. The method according to claim 1, characterized in that, The step of training globally shared global policy model parameters using the local policy model parameters of multiple agents includes: Aggregation processing is performed on the local policy model parameters of the multiple agents currently acquired to obtain intermediate model parameters; The intermediate model parameters are distributed to multiple agents, so that the agents perform local training based on the intermediate model parameters and generate new local policy model parameters, and the newly generated local policy model parameters are uploaded to the central server. Repeat the aggregation process, distribution, local training, and uploading until the target convergence condition is met, and use the intermediate model parameters obtained from the last aggregation as the global policy model parameters.

3. The method according to claim 2, characterized in that, The aggregation process performed on the local policy model parameters of the multiple agents currently acquired to obtain intermediate model parameters includes: The arithmetic mean of the local policy model parameters of the multiple agents currently acquired is used as the intermediate model parameters.

4. A path planning method, characterized in that, Applied to intelligent agents, the method includes: The global strategy model parameters are distributed by the receiving center server; wherein the global strategy model parameters are generated based on the path planning method described in any one of claims 1 to 3. The global strategy model is obtained by replacing the local strategy model parameters with the global strategy model parameters. Based on the global policy model, the agent's driving path is planned.

5. The method according to claim 4, characterized in that, Also includes: Obtain an experience tuple; wherein the experience tuple includes the agent's state information, actions, rewards, and next state; The state information and the action are used as inputs to the value network, which is then updated so that the difference between the output value of the value network and the target value is less than a target threshold; wherein the target value is determined based on the reward and the discount value of the next state. The updated value network is used to determine the evaluation value of the output action of the policy network, and the parameters of the policy network are adjusted in the direction that increases the evaluation value to obtain the updated policy network. The parameters of the updated policy network are used as the parameters of the local policy model.

6. The method according to claim 5, characterized in that, Before obtaining the experience tuple, the following is also included: Based on the first state vector of the intelligent agent, state information is generated; wherein, the first state vector is used to reflect the driving state and / or environmental perception results of the intelligent agent; The state information is input into the policy network to generate actions to be executed. Determine the reward and next state corresponding to the action; The state information, the action, the reward, and the next state are used as the experience tuple, and the experience tuple is stored in the agent.

7. The method according to claim 6, characterized in that, The generation of state information based on the first state vector of the agent includes: Obtain the second state vector of the cooperative agent; The first state vector and the second state vector are fused to generate fused state information.

8. The method according to claim 7, characterized in that, The process of fusing the first state vector and the second state vector to generate fused state information includes: Based on the first state vector, the second state vector, and the spatial relationship information between the agent and the collaborating agent, the attention weight of the agent to the collaborating agent is determined. Based on the attention weights, the first state vector and the second state vector are weighted and summed to obtain the state vector.

9. The method according to claim 4, characterized in that, Also includes: Based on the perception data of the intelligent agent during actual driving, the global policy model is incrementally trained to obtain the training results; The global policy model is updated based on the training results.

10. A vehicle, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is used to execute the path planning method according to any one of claims 4 to 9.

Images