Automatic driving traffic flow simulation method and device based on closed-loop reinforcement learning

Abstract

The invention relates to the technical field of traffic simulation, in particular to an automatic driving traffic flow simulation method and device based on closed-loop reinforcement learning, and the method comprises the steps: obtaining a real driving data set based on a pre-constructed traffic flow simulation frame with an automatic driving vehicle as the center; performing imitation learning pre-training according to the real driving data set to obtain an initial trajectory generation model; and constructing a closed-loop reinforcement learning fine-tuning target function according to the initial trajectory generation model, and updating the initial trajectory generation model by using the closed-loop reinforcement learning fine-tuning target function to obtain a final trajectory generation model. Therefore, the problems that in the prior art, authenticity and controllability cannot be considered at the same time, or covariant offset, mode collapse, unstable training and the like are faced in closed-loop deployment are solved.

Description

Technical Field

[0001] This invention relates to the field of traffic simulation technology, and in particular to a realistic and controllable autonomous driving traffic flow simulation method and apparatus based on closed-loop reinforcement learning. Background Technology

[0002] In the context of the rapid development of autonomous driving technology, high-quality traffic simulation systems have become an indispensable key infrastructure for algorithm training and evaluation. To improve the robustness and safety of autonomous driving systems in complex road environments, researchers widely employ simulators for large-scale data generation, behavioral policy learning, and closed-loop deployment testing. Compared to expensive and risky real-vehicle testing, simulation environments offer advantages such as high efficiency, strong controllability, and low cost, making them particularly suitable for exploring extreme situations, edge scenarios, and rare interaction patterns. Furthermore, the behavior in real traffic systems exhibits strong diversity and highly complex coupling, placing higher demands on the trajectory realism, interaction rationality, and behavioral controllability of simulation systems. Therefore, building a traffic simulator that can both preserve human behavioral styles and support large-scale closed-loop training and evaluation is one of the core elements driving the development and practical application of autonomous driving.

[0003] Existing traffic simulation technologies can be broadly categorized into two types: The first is open-loop imitation learning schemes based on data-driven simulators. This approach directly trains trajectory generation models on real road data, learning multimodal behavioral patterns to reproduce large-scale traffic flows. However, because the training phase does not consider closed-loop coupling with the environment, covariate shift is highly likely during deployment—that is, the model's state distribution during testing differs from that during training, causing errors to accumulate over time. Furthermore, these simulators typically use a uniform and simplified dynamics model for all vehicles, ignoring the complex coupling relationship between vehicles, roads, and the environment, resulting in less realistic and stable closed-loop interactions. The second type is reinforcement learning or rule-based control schemes based on physics engine simulators. Physics engines can finely control scene elements, allowing researchers to generate controllable closed-loop interaction scenarios online using reinforcement learning or rule-based methods. However, due to the lack of large-scale expert demonstrations, these methods often struggle to reproduce trajectories that conform to human driving habits; if relying solely on reinforcement learning, its reward function often prioritizes completing specific tasks, further sacrificing behavioral realism. In addition, some hybrid improvement attempts (combining reinforcement learning and imitation learning) have alleviated covariate bias, but generally only optimize the "optimal" trajectory or action, ignoring the other candidate trajectories, thus disrupting the multimodal distribution; at the same time, it is very difficult to design rewards that are consistent with humans, and the training process is prone to unstable oscillations.

[0004] In summary, there is an urgent need for an autonomous driving traffic flow simulation method that can address the problems of existing technologies, such as the inability to balance realism and controllability, or the covariate shift, pattern collapse, and training instability encountered in closed-loop deployment. Summary of the Invention

[0005] This invention provides an autonomous driving traffic flow simulation method and device based on closed-loop reinforcement learning, which solves the problems of existing technologies, such as the inability to balance realism and controllability, or the covariate shift, pattern collapse and training instability in closed-loop deployment.

[0006] The first aspect of this invention provides an autonomous driving traffic flow simulation method based on closed-loop reinforcement learning, comprising the following steps: based on a pre-constructed traffic flow simulation framework centered on autonomous vehicles, performing imitation learning pre-training based on a pre-acquired real driving dataset to obtain an initial trajectory generation model; constructing a closed-loop reinforcement learning fine-tuning objective function based on the initial trajectory generation model, and updating the initial trajectory generation model using the closed-loop reinforcement learning fine-tuning objective function to obtain a final trajectory generation model.

[0007] Optionally, the traffic flow simulation framework centered on the autonomous vehicle includes an autonomous vehicle traveling along a preset global route and multiple background vehicles based on preset rules, wherein the multiple background vehicles based on preset rules include multiple initial key background vehicles that interact with the autonomous vehicle.

[0008] Optionally, the step of performing imitation learning pre-training based on a pre-acquired real driving dataset to obtain an initial trajectory generation model includes:

[0009] Extract the current features, historical features of neighboring vehicles, and vectorized map features of each initial key background vehicle from the real driving dataset; concatenate and aggregate the current features, the historical features of neighboring vehicles, and the vectorized map features to obtain the key background vehicle scene code; construct a high-level lateral query using preset reference line information and introduce a learnable vertical query; concatenate and project the high-level lateral query and the learnable vertical query to generate a multimodal navigation query; pre-train the key background vehicle scene code and the multimodal navigation query on a pre-built N-layer Transformer decoder using imitation learning to obtain the initial trajectory generation model.

[0010] Optionally, the specific expression for the multimodal navigation query is:

[0011] Q nav =Projection(concat(Q) lat Q lon))

[0012]

[0013] Among them, Q nav For multimodal navigation queries, Q lat For horizontal queries, Q lon For vertical queries, N ref N is the reference line number. lon Where is the number of vertical anchor points, and D is the embedding dimension.

[0014] Optionally, the step of constructing a closed-loop reinforcement learning fine-tuning objective function based on the initial trajectory generation model, and updating the initial trajectory generation model using the closed-loop reinforcement learning fine-tuning objective function to obtain the final trajectory generation model, includes:

[0015] Estimate the path-level interaction probability between the global preset path of the autonomous vehicle and each background vehicle based on preset rules, and select the background vehicle with the highest path-level interaction probability as the new key background vehicle; input the multiple initial key background vehicles and the new key background vehicle into the initial trajectory generation model to generate multiple candidate trajectories for each key background vehicle as its corresponding candidate trajectory set; calculate the trajectory-level group relative reward and group relative advantage for each key background vehicle's candidate trajectory set; construct a group relative advantage optimization objective function based on the trajectory-level group relative reward and the group relative advantage; introduce a dual pruning mechanism into the group relative advantage optimization objective function to obtain a closed-loop reinforcement learning fine-tuning objective function; update the initial trajectory generation model using the closed-loop reinforcement learning fine-tuning objective function to generate the final trajectory generation model.

[0016] Optionally, the specific expression for the closed-loop reinforcement learning fine-tuning objective function is as follows:

[0017]

[0018] Among them, J RIFT (θ) is the closed-loop reinforcement learning fine-tuning objective function. Let G be the candidate trajectory set, and G = N ref ×N lon N ref N is the reference line number. lon θ represents the number of vertical anchor points. old For the old model parameters, ρ i (θ) represents the importance sampling ratio at the trajectory level. The relative advantage of the group is defined by clip(), where ∈ represents the clipping range and c is the lower bound of the constant.

[0019] A second aspect of the present invention provides an autonomous driving traffic flow simulation device based on closed-loop reinforcement learning, comprising:

[0020] The pre-training module is used to perform imitation learning pre-training based on a pre-built traffic flow simulation framework centered on autonomous vehicles and a pre-acquired real driving dataset to obtain an initial trajectory generation model; the closed-loop fine-tuning module is used to construct a closed-loop reinforcement learning fine-tuning objective function based on the initial trajectory generation model and use the closed-loop reinforcement learning fine-tuning objective function to update the initial trajectory generation model to obtain the final trajectory generation model.

[0021] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the autonomous driving traffic flow simulation method based on closed-loop reinforcement learning as described in the above embodiments.

[0022] A fourth aspect of the present invention provides a computer program product that, when executed by a processor, implements the above-described method for simulating autonomous driving traffic flow based on closed-loop reinforcement learning.

[0023] A fifth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described autonomous driving traffic flow simulation method based on closed-loop reinforcement learning.

[0024] The autonomous driving traffic flow simulation method and device based on closed-loop reinforcement learning proposed in this invention adopts a two-stage structure design of "imitation learning open-loop pre-training + reinforcement learning closed-loop fine-tuning". It combines key background vehicle screening, multimodal trajectory generation, group-relative advantage optimization and dual-clip mechanism to propose a complete traffic simulation behavior modeling method. It can effectively balance trajectory realism, interaction controllability and training stability, and provide technical support for large-scale closed-loop simulation, testing and deployment of autonomous driving systems.

[0025] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0026] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0027] Figure 1A flowchart illustrating an autonomous driving traffic flow simulation method based on closed-loop reinforcement learning, provided in an embodiment of the present invention;

[0028] Figure 2 This is a macroscopic flowchart of an open-loop pre-training process for imitation learning based on a real dataset, provided in an embodiment of the present invention.

[0029] Figure 3 This invention provides a macroscopic flowchart of reinforcement learning closed-loop fine-tuning based on a high-fidelity simulator, as provided in an embodiment of the invention.

[0030] Figure 4 A flowchart for calculating the relative advantage of a group is provided in an embodiment of the present invention;

[0031] Figure 5 The flowchart illustrates the specific execution of a closed-loop fine-tuning method based on reinforcement learning in a high-fidelity simulator, as provided in this embodiment of the invention.

[0032] Figure 6 A block diagram illustrating an automated driving traffic flow simulation device based on closed-loop reinforcement learning, provided in an embodiment of the present invention.

[0033] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0034] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0035] The following describes an embodiment of the autonomous driving traffic flow simulation method and apparatus based on closed-loop reinforcement learning according to the accompanying drawings. Addressing the issues mentioned in the background section, existing simulation methods based on imitation learning rely on offline expert data during training, neglecting environmental feedback. After deployment, errors often accumulate due to state distribution shifts, leading to unreasonable or even dangerous behaviors (i.e., closed-loop performance degradation caused by covariate shifts). Existing physics engine-based simulators can achieve highly controllable interactive scenarios, but lack human demonstration support, easily generating behaviors inconsistent with human driving styles. While reinforcement learning methods possess closed-loop adaptability, realistic reward design is difficult, often prioritizing task completion at the expense of behavioral realism (i.e., trajectory realism and controllability are difficult to achieve simultaneously). Some existing hybrid methods attempt to combine imitation learning and reinforcement learning, but typically only optimize a single optimal trajectory, ignoring other candidates, leading to mode collapse. Furthermore, combining the optimization objectives of both reinforcement learning and imitation learning easily leads to training instability (i.e., difficulty in unifying multimodal preservation and training stability). Therefore, this invention provides an autonomous driving traffic flow simulation method based on closed-loop reinforcement learning. In this method, firstly, during the pre-training phase, such as... Figure 2 As shown, a trajectory generation model with multimodal output capability is trained through imitation learning using large-scale real driving data to capture the characteristics of human driving behavior and ensure that the generated trajectories have a realistic driving style and rich behavioral diversity; subsequently, in the closed-loop fine-tuning stage, such as... Figure 3 As shown, by combining a high-fidelity physics engine environment, a reinforcement learning fine-tuning method based on Group-Relative Policy Optimization (GRPO) is introduced to perform closed-loop fine-tuning of the trajectory generation model, enabling it to possess rationality, responsiveness, and scene adaptability in interactions with autonomous vehicles. The entire system also incorporates a Critical Background Vehicles (CBVs) screening mechanism, a Group-Relative Advantage evaluation method, and a Dual-Clip stabilization optimization strategy to ensure the effective utilization of multimodal trajectories and stable convergence of the training process. This addresses the problems of existing technologies, such as the inability to balance realism and controllability, or the covariate shift, mode collapse, and training instability encountered in closed-loop deployment.

[0036] Specifically, Figure 1 This is a flowchart illustrating an autonomous driving traffic flow simulation method based on closed-loop reinforcement learning, provided in an embodiment of the present invention.

[0037] like Figure 1 As shown, this autonomous driving traffic flow simulation method based on closed-loop reinforcement learning includes the following steps:

[0038] In step S101, based on a pre-built traffic flow simulation framework centered on autonomous vehicles, an initial trajectory generation model is obtained by imitation learning pre-training using a pre-acquired real driving dataset.

[0039] In some embodiments, an initial trajectory generation model is obtained by imitation learning pre-training based on a pre-acquired real driving dataset, including:

[0040] Extract the current features, historical features of neighboring vehicles, and vectorized map features of each initial key background vehicle from the real driving dataset;

[0041] The current features, historical features of nearby vehicles, and vectorized map features are stitched together and aggregated to obtain the key background vehicle scene code;

[0042] High-level horizontal queries are constructed using preset reference line information, and learnable vertical queries are introduced.

[0043] High-level horizontal queries and learnable vertical queries are concatenated and projected to generate multimodal navigation queries;

[0044] The key background vehicle scene encoding and multimodal navigation query are used to perform imitation learning pre-training on a pre-built N-layer Transformer decoder to obtain an initial trajectory generation model.

[0045] In practical implementation, this embodiment of the invention, based on the closed-loop evaluation and training paradigm widely used in autonomous driving, pre-constructs a traffic flow simulation framework centered on the autonomous vehicle. This framework includes an autonomous vehicle (AV) traveling along a predefined global route, accompanied by multiple background vehicles (BVs) based on preset rules, collectively forming a closed-loop simulation environment centered on the autonomous vehicle (AV). These background vehicles (BVs) either provide diverse interaction data for training or are used to evaluate the robustness of the autonomous vehicle (AV). Furthermore, this embodiment of the invention identifies a subset of initial key background vehicles (CBVs), i.e., those BVs more likely to interact with the autonomous vehicle (AV). For these initial key background vehicles (CBVs), this embodiment of the invention replaces the original rule-based controller with a well-trained trajectory generation model, thereby synthesizing more realistic and controllable behavior in the interactive closed-loop scenario.

[0046] Based on a traffic flow simulation framework centered on autonomous vehicles, in a fully observable simulation environment, existing autonomous driving trajectory generation methods such as Pluto can generate more reliable trajectories by utilizing real-world state information. Their modular encoder-decoder structure also facilitates subsequent fine-tuning. Therefore, this embodiment of the invention uses Pluto as the trajectory generation model to synthesize realistic, multimodal initial key background vehicle CBV trajectories, such as... Figure 2 As shown, its specific implementation is as follows:

[0047] First, based on Pluto, for each initial key background vehicle (CBV) in the scene, this embodiment of the invention extracts its current feature F. cbv Historical characteristics of its neighboring vehicles F neighbor and vectorized map features F map These features are encoded as follows: and Where, N neighbor and N map represents the number of neighboring vehicles and map elements, respectively, and D is the embedding dimension. To establish interaction between these embeddings, this embodiment of the invention concatenates them and adds a global location embedding (PE) to form a unified scene embedding:

[0048] E s =concat(E cnv E neighbor E map )+PE

[0049] This scene embeds E s The input is then fed into an N-layer Transformer encoder for feature aggregation, yielding the final key background vehicle scene code E. enc Each encoder layer follows the standard Transformer architecture.

[0050]

[0051] Here, MHA stands for Multi-Head Attention Mechanism, and FFN is a feedforward neural network layer.

[0052] Furthermore, to capture the multimodality of real-world driving behavior, this embodiment of the invention employs the longitudinal-lateral decoupling mechanism proposed by Pluto. This longitudinal-lateral decoupling mechanism utilizes reference line information to construct a high-level lateral query. And introduce learnable vertical queries The two data points are then combined and projected to obtain a multimodal navigation query.

[0053] Q nav =Projection(concat(Q)lat Q lon ))

[0054]

[0055] Among them, Q nav For multimodal navigation queries, Q lat For horizontal queries, Q lon For vertical queries, N ref N is the reference line number. lon Where is the number of vertical anchor points, and D is the embedding dimension.

[0056] Furthermore, the multimodal navigation query Q nav Key background vehicle scene coding E enc These are input together into an N-layer Transformer decoder for imitation learning pre-training to establish lateral, longitudinal, and cross-modal interactions, resulting in an initial trajectory generation model. The structure of each decoder layer is as follows: as follows:

[0057]

[0058] Here, SelfAttn represents the self-attention mechanism, and CrossAttn represents the cross-attention mechanism.

[0059] The model generates the final output Q using the initial trajectory. dec Predict the multimodal trajectory of CBV using two MLP heads (Multi-Layer Perceptron Heads). and its confidence score in:

[0060] T = MLP(Q) dec S = MLP(Q) dec )

[0061] Where each trajectory point Q dec Including [p x ,p y ,cosθ,sinθ,v x ,v y [] indicates position, orientation, and speed.

[0062] In step S102, a closed-loop reinforcement learning fine-tuning objective function is constructed based on the initial trajectory generation model, and the initial trajectory generation model is updated using the closed-loop reinforcement learning fine-tuning objective function to obtain the final trajectory generation model.

[0063] In some embodiments, a closed-loop reinforcement learning fine-tuning objective function is constructed based on the initial trajectory generation model, and the initial trajectory generation model is updated using the closed-loop reinforcement learning fine-tuning objective function to obtain the final trajectory generation model, including:

[0064] Estimate the path-level interaction probability between the global preset path of the autonomous vehicle and each background vehicle based on preset rules, and select the background vehicle with the highest path-level interaction probability as the new key background vehicle;

[0065] Multiple initial key background vehicles and new key background vehicles are input into the initial trajectory generation model to generate multiple candidate trajectories for each key background vehicle, which serve as its corresponding candidate trajectory set.

[0066] For each key background vehicle, calculate the corresponding trajectory-level group relative reward and group relative advantage for the candidate trajectory set;

[0067] Construct a group relative advantage optimization objective function based on trajectory-level group relative reward and group relative advantage;

[0068] A dual pruning mechanism is introduced into the objective function for optimizing the relative advantage of the group to obtain a closed-loop reinforcement learning fine-tuning objective function;

[0069] Closed-loop reinforcement learning is used to fine-tune the objective function and update the initial trajectory generation model to generate the final trajectory generation model.

[0070] In actual implementation, based on the trajectory generation model introduced in step S101, this embodiment of the invention can generate realistic and multimodal trajectories in diverse scenarios. However, strategies under the open training paradigm are susceptible to covariate shift, meaning that the distribution of expert states observed during training is inconsistent with the distribution of states encountered during actual deployment. Even with the introduction of techniques such as contrastive learning or data augmentation, this problem is difficult to completely solve.

[0071] Therefore, this invention proposes a simple yet effective closed-loop reinforcement learning fine-tuning objective function, aiming to enhance the controllability of behavior, mitigate covariate shifts, and simultaneously maintain trajectory-level realism and multimodality, such as... Figure 3 As shown, the specific implementation process is as follows:

[0072] This invention addresses the "curse of rarity" by intervening only in a small number of critical background vehicle CBVs at critical moments, while still using a rule controller for non-critical background vehicles to reduce computational overhead.

[0073] To enhance the interactivity between autonomous vehicles (AVs) and surrounding vehicles, this invention decouples "path-level interactivity" from "behavioral-level realism." Specifically, this invention first estimates the path-level interaction probability between the AV's globally preset path and candidate paths of other background vehicles. The vehicle with the highest probability is selected as the new key background vehicle (CBV), and its future trajectory is synthesized based on the conflict points for subsequent closed-loop simulation centered on the AV.

[0074] This invention further proposes a strategy for identifying initial key background vehicle CBVs, avoiding modeling and controlling all background vehicles, improving simulation efficiency, and obtaining new key background vehicle CBVs. The criteria for determining the new key background vehicle CBVs are as follows:

[0075]

[0076] in, Indicates based on A * The algorithm calculates the arrival path distance, where δ is an adjustable threshold (e.g., 15 meters), and p k Let V be the k-th discrete navigation point on the global navigation path of the autonomous vehicle. i With V AV Let i represent the i-th background vehicle and the autonomous vehicle, respectively. If the two are close to each other, it indicates that there may be an interaction relationship. In this case, the background vehicle is regarded as a new key background vehicle (CBV), and its subsequent behavior trajectory will be controlled by the generator mentioned above and participate in the closed-loop reinforcement learning fine-tuning stage.

[0077] For the identified initial key background vehicle CBVs, this embodiment of the invention no longer uses a rule controller, but instead uses the initial trajectory generation model generated in step S101 to generate N for each key background vehicle CBV. ref ×N lon There are several candidate trajectories. Typically, the trajectory with the highest score is selected for closed-loop execution. However, due to the distribution differences between training and deployment, this approach often accumulates errors over time, leading to unrealistic long-term behavior.

[0078] Existing research has attempted to use reinforcement learning or IL-RL hybrid methods for closed-loop fine-tuning, but these typically only calculate the loss for the executed action or a single optimal trajectory, ignoring other candidate trajectories and thus losing behavioral diversity. Therefore, as... Figure 4 As shown, embodiments of the present invention will include all candidate trajectories. (where G = N) ref ×N lon Treat them as a whole and calculate the corresponding trajectory-level group relative reward. and group relative advantage As shown below:

[0079]

[0080] Where StateWiseRM is a user-defined state-by-state reward function, and γ is a discount factor.

[0081] Based on this, embodiments of the present invention construct an objective function J for optimizing GRPO based on a group relative strategy. GRPO (θ):

[0082]

[0083] in, Let G be the candidate trajectory set, and G = N ref ×N lon N ref N is the reference line number. lon θ represents the number of vertical anchor points. old For the old model parameters, It is the importance sampling ratio at the trajectory level. For the relative advantage of the group, D KL Let β be the KL divergence, β be the regularization coefficient, ∈ be the clipping range, and π be the distance between the two sides. ref This is a strategy for reference.

[0084] This optimization objective addresses the problem of traditional methods optimizing only a single trajectory by preserving trajectory-level multimodality. However, in the task setting of this embodiment, the reference strategy π... ref Because the pre-training distribution differs significantly from that of imitation learning, the KL regularization term forces the model to approach the old policy, hindering alignment with user preferences. Therefore, this embodiment of the invention directly removes this term. However, removing the KL regularization term can easily lead to training instability. Although PPO-Clip (Proximal Policy Optimization with Clipping, a policy gradient method based on reinforcement learning) proposes to limit the policy update magnitude by clipping the sampling ratio (trust domain learning), this is not applicable in group-relative settings. Especially in multi-trajectory evaluation, when... And ρ i When the product is greater than 1, the product may produce a very large negative value, leading to gradient explosion.

[0085] Therefore, such as Figure 5 As shown, this embodiment of the invention introduces a dual-clip mechanism, using a constant c>1 as a lower bound for the negative dominance function to stabilize training. This ultimately yields the RIFT reinforcement learning framework, which combines group relative policy optimization (GRPO) with dual-clip. This objective function preserves behavioral diversity while significantly improving training stability and controllability through the dual-clip mechanism. The expression for the RIFT objective function is:

[0086]

[0087] Among them, J RIFT (θ) is the closed-loop reinforcement learning fine-tuning objective function. Let G be the candidate trajectory set, and G = N ref ×N lon N ref N is the reference line number. lon θ represents the number of vertical anchor points. old For the old model parameters, ρ i (θ) represents the importance sampling ratio at the trajectory level. The relative advantage of the group is defined by clip(), where ∈ represents the clipping range and c is the lower bound of the constant.

[0088] In summary, the autonomous driving traffic flow simulation method based on closed-loop reinforcement learning proposed in this invention has the following beneficial effects:

[0089] (1) Traditional simulation methods based on imitation learning use open-loop training, which leads to a significant difference between the state distribution faced by the model during deployment and the training phase, resulting in covariate offset problems. This can easily cause unreasonable behaviors such as trajectory deviation, abnormal interaction, or even collision. In this embodiment of the invention, reinforcement learning is used to fine-tune the trajectory in a real physics engine, thereby achieving direct optimization of the trajectory in a closed-loop environment. This effectively makes up for the defects of open-loop training and greatly improves the interaction robustness and system stability during deployment.

[0090] (2) Existing reinforcement learning simulation methods often only optimize the optimal action or a single path, resulting in shrinkage of trajectory distribution after training, single behavior pattern, and difficulty in reproducing the rich strategies of human driving; the embodiments of the present invention retain the multimodal candidate trajectory of each key background vehicle in the trajectory generation stage, and introduce the group relative advantage strategy in the fine-tuning process, so that the contribution of all trajectories is fully utilized, thereby effectively avoiding the pattern collapse problem and maintaining the diversity and authenticity of trajectory distribution.

[0091] (3) In order to avoid gradient explosion or violent oscillation during reinforcement learning, this embodiment of the invention introduces a double pruning mechanism on the basis of the GRPO objective function, sets a lower bound for the truncation of the negative dominant trajectory, limits the update amplitude, ensures smooth convergence of the training process, and improves training stability.

[0092] (4) By freezing the pre-trained trajectory generator structure and fine-tuning the scoring head part only during the reinforcement learning stage, the realism of the trajectory obtained from the original imitation learning is preserved. At the same time, the optimization of the scoring function enables flexible control of behavioral styles (such as conservative / aggressive). With this "realism-based and task-oriented" structural design, the organic unity of behavioral naturalness and strategy controllability is achieved.

[0093] (5) In view of the fact that existing simulation systems usually include all background vehicles in the modeling, which increases the training cost and introduces a large number of noisy trajectories that do not directly interact with autonomous vehicles, the present invention designs a key background vehicle identification mechanism based on global path distance, which only controls vehicles that may affect the decision of autonomous vehicles, effectively improving the interaction density, improving training efficiency, and strengthening the rationality of interaction logic.

[0094] (6) It can be widely applied to traffic behavior modeling tasks in various complex scenarios, and has the advantages of high realism, high controllability and high versatility. It is especially suitable for autonomous driving simulation platforms for closed-loop evaluation and adversarial testing.

[0095] Next, referring to the accompanying drawings, an autonomous driving traffic flow simulation device based on closed-loop reinforcement learning proposed according to an embodiment of the present invention is described.

[0096] Figure 6 This is a block diagram of an autonomous driving traffic flow simulation device based on closed-loop reinforcement learning, provided as an embodiment of the present invention.

[0097] like Figure 6 As shown, the autonomous driving traffic flow simulation device 60 based on closed-loop reinforcement learning includes: a pre-training module 601 and a closed-loop fine-tuning module 602.

[0098] The pre-training module 601 is used to perform imitation learning pre-training based on a pre-built traffic flow simulation framework centered on autonomous vehicles, using a pre-acquired real driving dataset to obtain an initial trajectory generation model. The closed-loop fine-tuning module 602 is used to construct a closed-loop reinforcement learning fine-tuning objective function based on the initial trajectory generation model, and use the closed-loop reinforcement learning fine-tuning objective function to update the initial trajectory generation model to obtain the final trajectory generation model.

[0099] In some embodiments, the traffic flow simulation framework centered on autonomous vehicles includes an autonomous vehicle traveling along a preset global route and multiple background vehicles based on preset rules, wherein the multiple background vehicles based on preset rules include multiple initial key background vehicles that interact with the autonomous vehicle.

[0100] In some embodiments, the pre-training module 601 includes:

[0101] The extraction unit is used to extract the current features, historical features of neighboring vehicles, and vectorized map features of each initial key background vehicle in the real driving dataset;

[0102] The splicing and aggregation unit is used to splice and aggregate the current features, the historical features of neighboring vehicles, and the vectorized map features to obtain the key background vehicle scene encoding;

[0103] The construction and introduction unit is used to construct a high-level horizontal query using preset reference line information and introduce a learnable vertical query.

[0104] The splicing and projection unit is used to splice and project high-level horizontal queries and learnable vertical queries to generate multimodal navigation queries.

[0105] The pre-training unit is used to perform imitation learning pre-training on a pre-built N-layer Transformer decoder by encoding key background vehicle scenes and multimodal navigation queries to obtain an initial trajectory generation model.

[0106] In some embodiments, the specific expression for a multimodal navigation query is:

[0107] Q nav =Projection(concat(Q) lat Q lon ))

[0108]

[0109] Among them, Q nav For multimodal navigation queries, Q lat For horizontal queries, Q lon For vertical queries, N ref N is the reference line number. lon Where is the number of vertical anchor points, and D is the embedding dimension.

[0110] In some embodiments, the closed-loop fine-tuning module 602 includes:

[0111] The selected unit is used to estimate the path-level interaction probability between the global preset path of the autonomous vehicle and each background vehicle based on preset rules, and selects the background vehicle with the highest path-level interaction probability as the new key background vehicle.

[0112] The trajectory generation unit is used to input multiple initial key background vehicles and new key background vehicles into the initial trajectory generation model, and generate multiple candidate trajectories for each key background vehicle as its corresponding candidate trajectory set;

[0113] The calculation unit is used to calculate the corresponding trajectory-level group relative reward and group relative advantage for each candidate trajectory set of key background vehicles;

[0114] A building unit is used to construct a group relative advantage optimization objective function based on trajectory-level group relative reward and group relative advantage.

[0115] An introduction unit is used to introduce a dual pruning mechanism into the group relative advantage optimization objective function to obtain a closed-loop reinforcement learning fine-tuning objective function;

[0116] The update unit is used to fine-tune the objective function using closed-loop reinforcement learning to update the initial trajectory generation model in order to generate the final trajectory generation model.

[0117] In some embodiments, the specific expression for the closed-loop reinforcement learning fine-tuning objective function is:

[0118]

[0119] Among them, J RIFT (θ) is the closed-loop reinforcement learning fine-tuning objective function. Let G be the candidate trajectory set, and G = N ref ×N lon N ref N is the reference line number. lon θ represents the number of vertical anchor points. old For the old model parameters, ρ i (θ) represents the importance sampling ratio at the trajectory level. The relative advantage of the group is defined by clip(), where ∈ represents the clipping range and c is the lower bound of the constant.

[0120] It should be noted that the foregoing explanation of the embodiment of the autonomous driving traffic flow simulation method based on closed-loop reinforcement learning also applies to the autonomous driving traffic flow simulation device based on closed-loop reinforcement learning in this embodiment, and will not be repeated here.

[0121] The autonomous driving traffic flow simulation device based on closed-loop reinforcement learning proposed in the embodiments of the present invention has the following beneficial effects:

[0122] (1) Traditional simulation methods based on imitation learning use open-loop training, which leads to a significant difference between the state distribution faced by the model during deployment and the training phase, resulting in covariate offset problems. This can easily cause unreasonable behaviors such as trajectory deviation, abnormal interaction, or even collision. In this embodiment of the invention, reinforcement learning is used to fine-tune the trajectory in a real physics engine, thereby achieving direct optimization of the trajectory in a closed-loop environment. This effectively makes up for the defects of open-loop training and greatly improves the interaction robustness and system stability during deployment.

[0123] (2) Existing reinforcement learning simulation methods often only optimize the optimal action or a single path, resulting in shrinkage of trajectory distribution after training, single behavior pattern, and difficulty in reproducing the rich strategies of human driving; the embodiments of the present invention retain the multimodal candidate trajectory of each key background vehicle in the trajectory generation stage, and introduce the group relative advantage strategy in the fine-tuning process, so that the contribution of all trajectories is fully utilized, thereby effectively avoiding the pattern collapse problem and maintaining the diversity and authenticity of trajectory distribution.

[0124] (3) In order to avoid gradient explosion or violent oscillation during reinforcement learning, this embodiment of the invention introduces a double pruning mechanism on the basis of the GRPO objective function, sets a lower bound for the truncation of the negative dominant trajectory, limits the update amplitude, ensures smooth convergence of the training process, and improves training stability.

[0125] (4) By freezing the pre-trained trajectory generator structure and fine-tuning the scoring head part only during the reinforcement learning stage, the realism of the trajectory obtained from the original imitation learning is preserved. At the same time, the optimization of the scoring function enables flexible control of behavioral styles (such as conservative / aggressive). With this "realism-based and task-oriented" structural design, the organic unity of behavioral naturalness and strategy controllability is achieved.

[0126] (5) In view of the fact that existing simulation systems usually include all background vehicles in the modeling, which increases the training cost and introduces a large number of noisy trajectories that do not directly interact with autonomous vehicles, the present invention designs a key background vehicle identification mechanism based on global path distance, which only controls vehicles that may affect the decision of autonomous vehicles, effectively improving the interaction density, improving training efficiency, and strengthening the rationality of interaction logic.

[0127] (6) It can be widely applied to traffic behavior modeling tasks in various complex scenarios, and has the advantages of high realism, high controllability and high versatility. It is especially suitable for autonomous driving simulation platforms for closed-loop evaluation and adversarial testing.

[0128] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0129] The electronic device may include:

[0130] The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0131] When the processor 702 executes the program, it implements the autonomous driving traffic flow simulation method based on closed-loop reinforcement learning provided in the above embodiments.

[0132] Furthermore, electronic devices also include:

[0133] Communication interface 703 is used for communication between memory 701 and processor 702.

[0134] The memory 701 is used to store computer programs that can run on the processor 702.

[0135] The memory 701 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0136] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0137] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0138] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0139] This invention also provides a computer program product, which, when executed by a processor, implements the above-described autonomous driving traffic flow simulation method based on closed-loop reinforcement learning.

[0140] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described autonomous driving traffic flow simulation method based on closed-loop reinforcement learning.

[0141] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0142] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0143] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0144] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0145] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0146] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0147] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0148] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for simulating traffic flow in autonomous driving based on closed-loop reinforcement learning, characterized in that, Includes the following steps: Based on a pre-built traffic flow simulation framework centered on autonomous vehicles, an initial trajectory generation model is obtained by imitation learning pre-training using a pre-acquired real driving dataset. Based on the initial trajectory generation model, a closed-loop reinforcement learning fine-tuning objective function is constructed, and the initial trajectory generation model is updated using the closed-loop reinforcement learning fine-tuning objective function to obtain the final trajectory generation model.

2. The autonomous driving traffic flow simulation method based on closed-loop reinforcement learning according to claim 1, characterized in that, The traffic flow simulation framework centered on autonomous vehicles includes an autonomous vehicle traveling along a preset global route and multiple background vehicles based on preset rules. Among the multiple background vehicles based on preset rules, there are several initial key background vehicles that interact with the autonomous vehicle.

3. The autonomous driving traffic flow simulation method based on closed-loop reinforcement learning according to claim 2, characterized in that, The step of performing imitation learning pre-training based on a pre-acquired real driving dataset to obtain an initial trajectory generation model includes: Extract the current features, historical features of neighboring vehicles, and vectorized map features of each initial key background vehicle from the real driving dataset; The current features, the historical features of neighboring vehicles, and the vectorized map features are concatenated and aggregated to obtain the key background vehicle scene code; High-level horizontal queries are constructed using preset reference line information, and learnable vertical queries are introduced. The high-level horizontal query and the learnable vertical query are concatenated and projected to generate a multimodal navigation query; The key background vehicle scene encoding and the multimodal navigation query are used to perform imitation learning pre-training on a pre-built N-layer Transformer decoder to obtain the initial trajectory generation model.

4. The autonomous driving traffic flow simulation method based on closed-loop reinforcement learning according to claim 3, characterized in that, The specific expression for the multimodal navigation query is: Q nav =Projection(concat(Q lat ,Q lon )) Among them, Q nav For multimodal navigation queries, Q lat For horizontal queries, Q lon For vertical queries, N ref N is the reference line number. lon Where is the number of vertical anchor points, and D is the embedding dimension.

5. The autonomous driving traffic flow simulation method based on closed-loop reinforcement learning according to claim 2, characterized in that, The step of constructing a closed-loop reinforcement learning fine-tuning objective function based on the initial trajectory generation model, and updating the initial trajectory generation model using the closed-loop reinforcement learning fine-tuning objective function to obtain the final trajectory generation model, includes: Estimate the path-level interaction probability between the global preset path of the autonomous vehicle and each background vehicle based on preset rules, and select the background vehicle with the highest path-level interaction probability as the new key background vehicle; The multiple initial key background vehicles and the new key background vehicles are input into the initial trajectory generation model to generate multiple candidate trajectories for each key background vehicle, which serve as its corresponding candidate trajectory set. For each key background vehicle, calculate its corresponding trajectory-level group relative reward and group relative advantage for the candidate trajectory set; Construct a group relative advantage optimization objective function based on the trajectory-level group relative reward and the group relative advantage; A dual pruning mechanism is introduced into the objective function for optimizing the relative advantage of the group to obtain a closed-loop reinforcement learning fine-tuning objective function; The initial trajectory generation model is updated by fine-tuning the objective function using closed-loop reinforcement learning to generate the final trajectory generation model.

6. The autonomous driving traffic flow simulation method based on closed-loop reinforcement learning according to claim 5, characterized in that, The specific expression for the closed-loop reinforcement learning fine-tuning objective function is as follows: Among them, J RIFT (θ) is the closed-loop reinforcement learning fine-tuning objective function. Let G be the candidate trajectory set, and G = N ref ×N lon N ref N is the reference line number. lon θ represents the number of vertical anchor points. old For the old model parameters, ρ i (θ) represents the importance sampling ratio at the trajectory level. The relative advantage of the group is defined by clip(), where ∈ represents the clipping range and c is the lower bound of the constant.

7. An autonomous driving traffic flow simulation device based on closed-loop reinforcement learning, characterized in that, include: The pre-training module is used to perform imitation learning pre-training based on a pre-built traffic flow simulation framework centered on autonomous vehicles, and to obtain an initial trajectory generation model based on a pre-acquired real driving dataset. The closed-loop fine-tuning module is used to construct a closed-loop reinforcement learning fine-tuning objective function based on the initial trajectory generation model, and to update the initial trajectory generation model using the closed-loop reinforcement learning fine-tuning objective function to obtain the final trajectory generation model.

8. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the autonomous driving traffic flow simulation method based on closed-loop reinforcement learning as described in any one of claims 1-6.

9. A computer program product, characterized in that, When the computer program / instruction is executed by the processor, it implements the autonomous driving traffic flow simulation method based on closed-loop reinforcement learning as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the autonomous driving traffic flow simulation method based on closed-loop reinforcement learning as described in any one of claims 1-6.

Images