Method, device and product for training a driving trajectory generation model

Abstract

The present disclosure relates to a method for training a driving trajectory generation model, a method for generating a driving trajectory, an apparatus and a product. The method for training a driving trajectory generation model comprises generating training scene data and a training agent driving trajectory based on observation scene data; and determining a training sample based on the training scene data and the training agent driving trajectory. The method further comprises determining a reward score of the training agent driving trajectory based on the training sample; and updating a parameter of the model based on the reward score. In this way, the robustness of the trained driving trajectory generation model can be enhanced, and the safety and adaptability of the generated trajectory to the environment can be increased.

Description

Technical Field

[0001] This disclosure relates to the fields of robotics and autonomous systems, and more specifically to training methods for driving trajectory generation models, methods for generating driving trajectories, devices, and products. Background Technology

[0002] With the acceleration of urbanization and the increasing saturation of road resources, traditional transportation systems are facing challenges in safety and efficiency. Against this backdrop, computer-aided intelligent driving technology has emerged. In recent years, advancements in generative modeling technology have injected new vitality into intelligent driving. For example, in the field of intelligent driving decision-making and planning, and in applications such as embodied intelligence, AGVs (Automated Guided Vehicles), and AMRs (Autonomous Mobile Robots), generative models can generate multiple candidate driving trajectories based on real-time scene observation data, and select the optimal trajectory (e.g., based on a generative-discriminative framework), or directly select an optimal trajectory from a trajectory vocabulary (e.g., based on a fixed trajectory vocabulary), for use by intelligent agents such as AGVs and AMRs. Summary of the Invention

[0003] In a first aspect of the embodiments of this disclosure, a method for training a driving trajectory generation model is provided. The method includes generating training scene data and a driving trajectory of a training agent based on observed scene data. The method further includes determining training samples based on the training scene data and the driving trajectory of the training agent. The method also includes determining a reward score for the driving trajectory of the training agent based on the training samples. Finally, the method includes updating the parameters of the model based on the reward score.

[0004] In a second aspect of the embodiments of this disclosure, a method for generating a driving trajectory is provided. The method includes processing scene data collected by an agent based on a driving trajectory generation model to generate a candidate driving trajectory set for the agent. The method further includes processing the candidate driving trajectory set and the collected scene data based on the driving trajectory generation model to determine a trajectory score set of the candidate driving trajectory set. The method also includes determining a target driving trajectory from the candidate driving trajectory set based on the trajectory score set.

[0005] In a third aspect of the embodiments of this disclosure, an electronic device is provided. The electronic device includes a processor; and a memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method of the first aspect or the second aspect.

[0006] In a fourth aspect of the embodiments of this disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to implement the method of the first or second aspect.

[0007] In a fifth aspect of the embodiments of this disclosure, a non-transitory computer-readable medium is provided having machine-executable instructions stored thereon, which, when executed, cause a machine to implement the method of the first aspect or the second aspect.

[0008] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an exemplary system in which several embodiments of the present disclosure may be implemented is shown;

[0011] Figure 2 A schematic diagram of candidate driving trajectories according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram showing a vehicle traveling along a driving trajectory according to some embodiments of the present disclosure is shown;

[0013] Figure 4A and Figure 4B A block diagram illustrating a closed-loop data collection process according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A block diagram illustrating a process for training a discriminator using closed-loop data according to some embodiments of the present disclosure is shown;

[0015] Figure 6 A schematic diagram of sub-training scenario data according to some embodiments of the present disclosure is shown;

[0016] Figure 7 A block diagram illustrating the diversity of training scenario data according to some embodiments of the present disclosure is shown;

[0017] Figure 8 A block diagram illustrating the process of generating a target driving trajectory according to some embodiments of the present disclosure is shown;

[0018] Figure 9-21 A flowchart illustrating a training method for a driving trajectory generation model according to some embodiments of the present disclosure is shown;

[0019] Figure 22-25 A flowchart of a method for generating a driving trajectory according to some embodiments of the present disclosure is shown; and

[0020] Figure 26 A block diagram of a device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0021] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0022] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0023] As mentioned above, for intelligent driving decision-making and planning schemes, if an open-loop training scheme based on a generative-discriminative framework is chosen, while this scheme can dynamically generate candidate trajectories, its discriminator training relies on offline datasets and cannot evaluate the long-term consequences of trajectories in a closed-loop environment, resulting in short-sighted decision-making. If an open-loop scoring scheme based on a fixed trajectory planning vocabulary is chosen, this scheme relies on a predefined, finite set of trajectories, using static rules or models to score and filter candidate trajectories in a single pass, lacking the ability to adapt to dynamic scenarios online (in real-time).

[0024] It is evident that existing trajectory generation schemes lack robustness and security, and are not well-suited to real-time, dynamic scenarios. Therefore, an improved trajectory generation scheme is needed to address at least one of these technical problems. For example, how to generate a set of multimodal candidate trajectories that both conform to traffic rules (mimicking expert behavior) and can handle various future possibilities, and accurately and reliably select the optimal (e.g., safe and efficient) trajectory for execution within a long-term temporal field of view, thereby enhancing the decision-making robustness and adaptability of the intelligent driving system in dynamic environments and improving the safety of the generated trajectory.

[0025] To this end, in a first aspect, this disclosure proposes a training method for a driving trajectory generation model. In this training method, a model is constructed in which a generator and a discriminator work collaboratively. The generator is responsible for generating multiple candidate driving trajectories based on real-time road conditions, while the discriminator, in a simulation environment, evaluates the safety and efficiency performance of these long-sequence trajectories through a closed-loop process and outputs associated reward scores for training the driving trajectory generation model. In some embodiments, this training method also utilizes the discriminator's evaluation results to fine-tune and optimize the generator online, enabling it to produce better trajectories. Thus, the generator and discriminator mutually promote and integrate with each other, achieving collaborative optimization of the trajectory generator and discriminator through a closed-loop simulation environment, jointly driving continuous improvement in decision robustness, safety, and adaptability.

[0026] In a second aspect, this disclosure proposes a method for generating a driving trajectory, which is performed by a driving trajectory generation model. This driving trajectory generation model is trained according to the training method of the first aspect. Therefore, it is possible to achieve similar technical effects to the first aspect.

[0027] In this paper, "autonomous vehicle" can refer to the controlled intelligent driving vehicle. In the perception, localization, and decision-making processes of intelligent driving, a coordinate system is typically established with the autonomous vehicle at the center. For example, the positions of targets detected by multiple sensors are relative to the autonomous vehicle; multimodal behavioral decisions (e.g., acceleration, braking, steering) are made for the autonomous vehicle; and in the event of an accident or when determining liability, the question usually revolves around why the autonomous vehicle made a certain decision. In this paper, "dynamic obstacles," "dynamic objects," and "dynamic entities" can all refer to objects in the environment that can move, including objects that are moving or have a tendency to move (e.g., other vehicles, cyclists, pedestrians, animals, etc.), which are the main objects that the autonomous vehicle needs to predict and interact with in real time. The system needs to predict the future trajectory of dynamic obstacles and make safe responses. In this paper, "static obstacles," "static objects," and "static entities" can all refer to objects in the environment that are stationary, such as road guardrails, curbs, trees, traffic sign poles, construction cones, etc. Static obstacles can refer to objects that the autonomous vehicle needs to avoid, mainly used to define the boundaries of the drivable area. In this article, "drivable area" can refer to the physical area where a vehicle can drive safely, usually the road surface, which is the basis for route planning and decision-making.

[0028] It should be noted that although this article mainly uses intelligent driving scenarios as examples to describe the various embodiments of this disclosure, the methods, devices, products, etc. involved in the various embodiments of this disclosure can also be applied to any kind of intelligent agent such as robots and automated equipment in other scenarios such as embodied intelligence, AGV, AMR, etc.

[0029] refer to Figure 1 It illustrates a schematic diagram of an exemplary system 100 in which various embodiments of the present disclosure may be implemented. For example... Figure 1 As shown, the exemplary system 100 may include a computing device 102. The computing device 102 may be any electronic device or processor with processing capabilities. For example, the computing device 102 may be an NPU (Neural Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), ASIC (Application-Specific Integrated Circuit), system-on-chip (SOC), embedded and special-purpose devices, or other types of accelerator units capable of executing predetermined computational logic, embedded systems, microcontrollers, motherboards, etc. The computing device 102 may run a driving trajectory generation model 104. In some embodiments, the computing device 102 may also execute a training method for training the driving trajectory generation model 104.

[0030] In environment 100, intelligent agents (such as vehicles, not shown, but described hereinafter as an example of a vehicle, but not as a limitation) may be equipped with sensors (e.g., cameras, lidar, millimeter-wave radar, etc.). These sensors can perceive two-dimensional image data or three-dimensional point cloud data, referred to as raw sensor data. Raw sensor data can be directly used as input to the driving trajectory generation model 104, i.e., observation scene data 110. Furthermore, in some embodiments, observation features (e.g., representations, vectors, or embeddings) can be determined based on the raw sensor data. In this case, the observation features can also be referred to as observation scene data 110. It is understood that observation scene data 110 may also include both.

[0031] The driving trajectory generation model 104 can process the observed scene data 110 and output a target driving trajectory 116. The vehicle's intelligent driving system can output final vehicle control commands (e.g., steering wheel angle, throttle opening, braking, etc.) according to the target driving trajectory 116. In some embodiments, the driving trajectory generation model 104 may include a generator 106 and a discriminator 108. The generator 106 can receive the observed scene data 110 and can generate multiple candidate driving trajectories, i.e., a candidate driving trajectory set 112, based on the observed scene data 110.

[0032] The discriminator 108 can acquire a candidate driving trajectory set 112 and score each candidate driving trajectory in the candidate driving trajectory set 112. The trajectory scores of each candidate driving trajectory obtained from the scoring can form a trajectory score set 114. In the trajectory score set 114, the candidate driving trajectory with the highest score can generally be considered the optimal trajectory. Therefore, the candidate driving trajectory with the highest score can be determined as the target driving trajectory 116. The target driving trajectory 116 can be used by the vehicle's intelligent driving system to generate control commands. In some embodiments, the observed scene data 110 can have a fixed or variable time length (or number of frames). This time length and the number of frames can be correlated; for example, when the frame rate is 0.1 seconds / frame, 8 frames can represent a time length of 0.8 seconds. Both the candidate driving trajectory set 112 and the target driving trajectory 116 can be updated periodically with this time length to meet the real-time requirements of autonomous driving.

[0033] refer to Figure 2 This illustrates a schematic diagram 200 of candidate driving trajectories according to some embodiments of the present disclosure. For example... Figure 2As shown, vehicle 202 is traveling on a road with three lanes and initially in the middle lane. Candidate driving trajectories may include trajectory 204, trajectory 206, and trajectory 208. Trajectories 204, 206, and 208 may have different directions and lengths; that is, these trajectories can be a smooth set of candidate trajectories covering multimodal intentions. Multimodality can be understood as driving intentions such as following, overtaking, and decelerating. In other words, modality in this paper refers to different driving behaviors or intentions. Therefore, "multimodal" means that these candidate trajectories can represent different strategy choices.

[0034] If vehicle 202 travels along trajectory 204, then the end point of trajectory 204 will be located at the end point 202-2 of the left lane. If vehicle 202 travels along trajectory 206, then the end point of trajectory 206 will be located at the end point 202-1 of the middle lane. If vehicle 202 travels along trajectory 208, then the end point of trajectory 208 will be located at the end point 202-3 of the right lane.

[0035] It's understandable that the view of vehicle 202 is different at the start and end of its trajectory. Therefore, from the vehicle's coordinate system, the poses of other objects relative to the vehicle may change with the trajectory, especially for static objects. Thus, considering these changes during model training is beneficial for generating better candidate trajectories and selecting better target trajectories.

[0036] refer to Figure 3 This illustration shows a schematic diagram 300 of a vehicle traveling along a trajectory according to some embodiments of the present disclosure. In scenario 330, vehicle 302 is traveling on a road with three lanes and is initially located in the middle lane. Vehicle 310 is also traveling in the same lane ahead of vehicle 302. A pedestrian 312 is walking on the edge of the right lane to the right front of vehicle 302. The relative poses of vehicle 302 and vehicle 310 can be represented by vector 320. The relative poses of vehicle 302 and pedestrian 312 can be represented by vector 322. In scenario 330, vehicle 302 chooses to travel along trajectory 308.

[0037] In this paper, the orientation will be described using the Vehicle Coordinate System (VSC) of vehicle 302. The origin is the center of the rear axle of vehicle 302 (or the center of gravity of vehicle 302 or the center of the front axle). The X-axis is directly in front of vehicle 302, the Y-axis is to the left of vehicle 302, and the Z-axis is determined by the right-hand rule and points upwards (not shown). For example, in... Figure 3In scenario 340, vehicle 302 travels along trajectory 308 to its endpoint. At this point, vehicle 310 remains in the middle lane and is still ahead of vehicle 302 on the X-axis, but the distance between them has decreased. Pedestrian 312 walks at a slower speed, and their distance from vehicle 302 has also decreased. It can be seen that in scenario 340, the relative pose between vehicle 302 and vehicle 310 has changed to vector 324, whose magnitude and direction are different from vector 320. Similarly, the relative pose between vehicle 302 and pedestrian 312 has changed to vector 326, whose magnitude and direction are different from vector 322. Therefore, during the training of the driving trajectory generation model, considering the dynamically changing pose information along the driving trajectory is helpful for optimizing the trajectory generation strategy and more effectively scoring candidate trajectories; that is, generating closed-loop training environment data is advantageous.

[0038] refer to Figure 4A and Figure 4B It illustrates block diagrams of closed-loop data collection processes 400A and 400B according to some embodiments of the present disclosure. For training such... Figure 1 The present disclosure proposes a training method for the driving trajectory generation model 104. This method utilizes a high-fidelity closed-loop simulation environment to simulate the real-time interaction between the vehicle and a dynamic scene, enabling trajectory selection to generate actual environmental state changes and reward feedback based on indicators such as safety and efficiency. This provides a crucial dynamic data foundation for the discriminator to learn reward scores related to performance such as safety and efficiency for long sequences. In some embodiments, the high-fidelity closed-loop simulation environment can generate training data in image space, for example, using 3D Gaussian Splatting (3DGS) technology and / or a world model. In some embodiments, the high-fidelity closed-loop simulation environment can generate training data in feature space, for example, using Bird's-eye View Warp (BEV-Warp) technology, image feature transformation (which may include depth information), and other techniques.

[0039] In process 400A, the observed scene data 402 of the vehicle can be processed by perception processing 404 to generate the pose of the object (agent) in the vehicle coordinate system. The observed scene data 402, after perception processing 404, can also generate map information, such as lane lines and guardrails. The data obtained after perception processing 404 of the observed scene data 402 can also be referred to as scene features 420 (e.g., scene representations such as BEV features). Scene features 420 can be input to generator 406. Generator 406 can generate multiple trajectories 408 based on scene features and noise. The multiple trajectories 408 can be a set of trajectory candidates covering multimodal intents.

[0040] Unlike process 400A, in process 400B, the observed scene data 402 can be directly input into generator 406. That is, the only difference between process 400A and process 400B is that in process 400A, the observed scene data 402 can be processed by perception processing 404 to obtain scene features 420, which can then be input into generator 406. In process 400B, the observed scene data 402 can be directly input into generator 406.

[0041] In some embodiments, generator 406 may include an Autonomous Driving (AD) strategy module. Observed scene data 402 (or scene features 420) may be input into the AD strategy module. Observed scene data 402 (or scene features 420) may be denoted as... ,Right now This can be used as input to generator 406. The AD policy module can include a diffusion-based Imitation Learning (IL) generator. For example, a diffusion-based IL generator can include a diffusion-based end-to-end trajectory generator that can directly use the currently observed scene data. Predicting continuous future trajectories ,in, Indicates the current moment. Indicates the forecast time range. Indicates a future moment. Indicates from the current moment To the future The predicted trajectory point sequence, which consists of multiple future time points. Predicted two-dimensional location point Composition, in which and They represent future moments. The predicted X-axis and Y-axis position coordinates are obtained.

[0042] In some embodiments, the AD strategy module may include a feature encoder. Observations (whether feature representations in feature space or image frames in image space) can be processed by a unified feature encoder to construct a tokenized scene representation, i.e., generating tokens to represent dynamic and / or static element features. Tokenization can also be referred to as vectorization. Taking the observations as BEV feature representations in feature space as an example, the feature encoder can extract a BEV feature map, which consists of BEV tokens. Based on the BEV tokens, a set of static tokens can be generated to represent immutable scene elements, such as lane geometry, road boundaries, and surrounding structures. Furthermore, a set of dynamic tokens can be generated based on the BEV tokens, which can be used to represent the state of nearby traffic participants, including their location, orientation, and motion cues. All tokens can be projected into a shared latent space and collectively form the input to a diffusion-based IL generator.

[0043] In some embodiments, the tokenized scene representation may include BEV information, dynamic object information, static map element information, and navigation information. The BEV information, dynamic object information, static map element information, and navigation information can be tokenized to obtain a BEV token, a dynamic token, a static token, and a navigation token. In some embodiments, the BEV information, dynamic object information, and static map element information can be obtained based on the perception processing 404 in process 400A, or based on the observed scene data 402 in process 400B. In some embodiments, feature fusion can be performed on the BEV token, dynamic token, static token, and navigation token to obtain scene features, such as scene feature 420.

[0044] In some embodiments, the AD strategy module may include a planning head. The planning head can predict multimodal future trajectories (also known as candidate trajectories) based on a unified scene representation. In some embodiments, it can be set to... and These represent features associated with static map elements and features associated with dynamic objects, respectively. These elements and features can be converted into a set of tokens using a dedicated encoder, for example, through formula (A-1): (A-1)

[0045] in, and This represents a lightweight encoder that maps static and dynamic components to token embeddings. The resulting token is then compared with the BEV token. and navigation token Through learnable fusion modules The scene features are fused to generate scene features (or vectors), which can be considered as factors in generating multiple candidate trajectories, for example, through formula (A-2): (A-2)

[0046] in, Indicates feature fusion operation; Indicate scene characteristics; Indicates a static token; This indicates a dynamic token.

[0047] Then, scene features can be injected into a DiffusionTransformer (DiT)-based trajectory generator using a cross-attention mechanism to generate multimodal trajectories. In some embodiments, to generate multimodal trajectories, M trajectory modes can be instantiated, i.e., M different noises are sampled, and for each mode... m It can draw an independent, initial trajectory with noise. And based on scene characteristics As a denoising condition, the application conditional denoising network ,conduct K Step-by-step noise reduction, for example, using formula (A-3): (A-3)

[0048] in and , This represents a conditional denoising network that takes noisy trajectory samples as input. Scene characteristics and diffusion step size index k As input.

[0049] The final candidate trajectory set obtained after denoising the trajectory using formula (A-3) can be represented by formula (A-4): (A-4)

[0050] in, In The trajectory is represented by the superscript 0, which indicates complete denoising, and the superscript m, which indicates the m-th mode; where for each future trajectory Within the planning scope H There is a continuous line inside. Trajectory. This process explicitly models the multimodal distribution of future motion under the same scene conditions. This allows for the generation of multimodal candidate driving trajectories.

[0051] In process 400A or 400B, generator 406 can generate multiple trajectories 408 based on scene features 420 or observed scene data 402. In process 412, which generates training scene data, a trajectory can be randomly selected from the multiple trajectories 408. The selected trajectory can be input into closed-loop simulation platform 410. Closed-loop simulation platform 410 can be pre-built. Closed-loop simulation platform 410 can reconstruct a scene representation model based on observed scene data 402 or scene features 420, so after the selected trajectory is input into simulation platform 410, the surrounding environment can be re-observed at any trajectory point (i.e., the scene representation model can be reconstructed). Therefore, closed-loop simulation platform 410 can acquire a randomly selected trajectory and, based on the acquired trajectory and the reconstructed scene representation model, generate observation information for each trajectory point in that trajectory. The generated new observation information for each trajectory point can be used to generate training observation scene data. In some embodiments, closed-loop simulation platform 410 can perform simulation using 3DGS technology. Observation scene data 402 can include a sequence of multiple image frames. The closed-loop simulation platform 410 can reconstruct a scene representation model composed of Gaussian spheres based on an image frame sequence, and render an image frame sequence that matches the selected trajectory based on the input selected trajectory and the scene representation model composed of Gaussian spheres.

[0052] In the closed-loop simulation platform 410, a vehicle can be simulated to travel along a selected trajectory. While traveling along the selected trajectory, at each trajectory point, the vehicle can observe the surrounding environment based on a reconstructed scene representation model, and the closed-loop simulation platform 410 can generate simulation observation data for that point. This simulation observation data can represent the surrounding environment that the vehicle can perceive ("see") at that trajectory point, such as the poses of other objects relative to the vehicle. In some embodiments, the surrounding environment can be observed based on the reconstructed scene representation model every few trajectory points, and for the endpoint of each set of trajectory points, the closed-loop simulation platform 410 can generate simulation observation data for that specific moment.

[0053] In some embodiments, a predetermined number of trajectory generation processes (via generator 406) and a process 412 for generating training scene data can be performed for the same observation scene data 402 or scene feature 420 to obtain the final training scene data. In some embodiments, different observation scene data 402 or scene feature 420 can be acquired, and different trajectories can be generated for different observation scene data 402 or scene feature 420, and the process 412 for generating training scene data can be performed to obtain the final training scene data.

[0054] In some embodiments, when generating training scene data for the same observation scene data 402 or scene feature 420, the training scene data can be generated directly, or some sub-training scene data can be generated first, and these sub-training scene data can be combined into the training scene data. These detailed processes will be discussed in conjunction with... Figure 6 That will not be described here.

[0055] In some embodiments, simulation observation data can be rendered as image frames, for example using 3DGS technology. In some embodiments, simulation observation data can be represented as BEV features. In some embodiments, the positional and / or motion relationships between the vehicle and other objects, i.e., the poses of other objects relative to the vehicle, can be determined using the vehicle's pose, because the sample observation data is pre-collected, for example, by a video capture vehicle.

[0056] It can be understood that in process 400A or process 400B, the same or different scene features 420 or observed scene data 402 can be input into the generator 406 to generate multiple trajectories 408, and subsequent processes can be executed to generate corresponding simulation observation data. By analogy, a large amount of simulation observation data can be generated as training scene data. Since this training scene data takes into account the changes in environmental state as the vehicle moves along the trajectory, it can be called closed-loop (training) data.

[0057] In some embodiments, the range of the generated simulated observation data may not exceed the observation scene corresponding to the observation scene data 402 or scene feature 420. In some embodiments, if the range of the generated simulated observation data exceeds the observation scene corresponding to the observation scene data 402 or scene feature 420, the reliability of the generated simulated observation data will decrease. In some embodiments, simulated observation data exceeding the observation scene corresponding to the observation scene data 402 or scene feature 420 will be deleted when training the model used to generate the trajectory.

[0058] In some embodiments, a BEV-Warp-based simulation system can be constructed for the BEV space environment. Since the input to the perception processing 404 is only the BEV features extracted from the input multi-view images (e.g., observation scene data 402), the input to the system can be directly controlled by transforming the BEV features, thereby avoiding the complex image generation process and thus constructing a low-cost, high-efficiency closed-loop simulation environment.

[0059] For example, in a closed-loop simulation of a BEV-Warp environment, reference BEV features can be obtained by inputting recorded multi-view images into a feature encoder. At each time step... t The current BEV characteristics can be denoted as B t In some embodiments, in inference mode, it can be based on B t The system predicts the trajectory and updates the vehicle's future pose by tracking that trajectory with a controller. The rigid body transformation between the vehicle's current and future poses can be calculated. This transformation can indicate the movement of the vehicle. It can be converted to the BEV plane using the function GridSample[]. Given the current B t Distorted future features B t+1 It can be calculated using formula (B-1): (B-1) in, Represents the space warp operator; This represents the transformation to BEV (Browser Electric Vehicle). Spatial warping of the original BEV features is equivalent to transforming the visual observation of the input. The warped future BEV features. B t+1 It is then input into the AD strategy model for the next time step, thus forming a closed-loop simulation.

[0060] In some embodiments, for a 3DGS environment, a realistic simulated world can be constructed. For example, at each moment... t The position of the bicycle The system can update the output of the AD strategy and render the scene based on the current pose to generate a representation of the vehicle's pose. The current observation scene data that can be seen below For example, using formula (B-2): (B-2) in Gaussian representation of the sampling scenario, This represents the rendering operator.

[0061] In some embodiments, for 3DGS and / or BEV-Warp environments, an iLQR (Iterative Linear Quadratic Regulator)-based controller can be used to optimize the continuous prediction of AD policy. Trajectory. For example, let... This represents the vehicle's state at a future time s. This represents the control input. The AD strategy predicts the future reference trajectory. This trajectory serves as the target of iLQR. Optimal control sequence. It can be calculated using formula (B-3), where the control sequence can be used to iteratively update the vehicle's pose in the environment: (B-3) in, This represents a function that finds the minimum value of the independent variable. Indicates the current moment. Indicates the forecast time range; s represents a future time. and They represent future moments. The predicted X-axis and Y-axis position coordinates; Indicates a future moment The position coordinates of the reference X-axis and reference Y-axis; This represents the trajectory tracking error term. For each future time s, the vehicle's current position can be calculated. Corresponding reference position The difference between the two is measured using a weighted quadratic form. The weighting matrix Q can be a positive semi-definite matrix, used to assign different importance weights to errors in different directions. For example, in some embodiments, a higher weight can be assigned to the Y-axis deviation to improve the Y-axis stability of the vehicle during path tracking. As the actual trajectory of the vehicle gradually approaches the reference trajectory, the value of the above error term gradually decreases, and when the vehicle completely tracks the reference trajectory, the error term will approach zero. This represents the control cost term, used to characterize the magnitude and degree of change of the control input. For example, for each future time s, the control input can be... A weighted secondary penalty is applied, where the weighting matrix R is a positive definite matrix used to adjust the penalty intensity for different control variables. For example, in vehicle control scenarios, higher weights can be assigned to the rate of change of steering angle or acceleration to suppress excessive maneuvering. As the amplitude of the control input decreases, the value of this control cost term decreases accordingly; when the control input tends to be smooth and the amplitude is small, the control cost term decreases further. By introducing a control cost term, it is possible to ensure trajectory tracking accuracy while avoiding excessively abrupt or discontinuous control commands, thereby improving the stability and comfort of vehicle operation. By simultaneously setting the trajectory tracking error term and the control cost term, a trade-off between tracking accuracy and control smoothness can be established.

[0062] Furthermore, considering vehicle dynamics, it is possible to make each control input Update vehicle state based on vehicle dynamics model Initial control Used to update vehicle status This leads to a new self-positioning posture. Then, using the updated pose, the next observation is calculated, thereby closing the loop between the policy prediction trajectory, underlying control, and vehicle state. This generates training scenario data for the closed loop.

[0063] refer to Figure 5 This illustrates a block diagram of a process 500 for training a discriminator using closed-loop data according to some embodiments of the present disclosure. Figure 5 As shown, large-scale grouped closed-loop driving data 502 can be referenced for example... Figure 4A or Figure 4B The data is generated and collected in a manner that allows for the perception of dynamic objects 510 and static elements 512 (as shown in box 504) in closed-loop driving data 502. Dynamic objects 510 may include other vehicles and vulnerable road users (VRUs), whether they are stationary or moving. Static elements 512 may be determined based on static objects.

[0064] In process 500, navigation information 514 may also be acquired. Navigation information 514 may come from, for example, navigation planning information from an external map application. In some embodiments, navigation information 514 may include global navigation from the origin to the destination, for example in the form of waypoints (e.g., one waypoint every 5 meters).

[0065] In process 500, taking one training step (corresponding to one time point) as an example, as shown in box 506, the generator can generate multiple candidate driving trajectories based on closed-loop driving data 502, and select one driving trajectory 520 from the multiple candidate driving trajectories. The discriminator 508 can then analyze the selected driving trajectory 520 (e.g., driving trajectory 520 may include...). Figure 4A or Figure 4B A reward score of 522 is generated by randomly selecting a driving trajectory from box 412. The reward score 522 reflects whether the driving trajectory 520 is a "good" trajectory or a "bad" trajectory. A "good" trajectory can be a safe and efficient trajectory. A "bad" trajectory has the opposite characteristics to a "good" trajectory, or is less safe or less efficient than a "good" trajectory.

[0066] In process 500, the smallest granularity of closed-loop driving data 502 can be a collected observation scene data or its corresponding scene feature. The training scene data corresponding to this smallest granularity of observation scene data or scene feature can be called a set of training scene data. For example, for the same observation scene data or its corresponding scene feature, the above steps can be executed iteratively, for example, 4 times, to obtain 4 sub-training scene data. The set of these 4 sub-training scene data can be called a set of training scene data or a set of closed-loop driving data.

[0067] For the same set of closed-loop driving data, the above steps can be performed iteratively, for example, four times, to obtain four reward scores for four trajectory sequences. The parameters of the discriminator 508 can be adjusted based on these reward scores, enabling it to distinguish between "good" and "bad" trajectories in the same scenario. For example, for a "good" trajectory, the discriminator 508 can increase its trajectory score. For a "bad" trajectory, the discriminator 508 can decrease its trajectory score. In other words, through such training, the discriminator 508 can learn which trajectory is a better choice relative to other trajectories in a given scenario.

[0068] In process 500, the above steps can be performed iteratively for closed-loop driving data for each scenario (e.g., a time period or a number of frames). For example, for closed-loop driving data of scenarios 1, 2, and 3, corresponding candidate trajectories can be generated respectively, driving trajectories can be iteratively selected to generate corresponding reward scores, and the parameters of discriminator 508 can be adjusted using the corresponding reward scores. In this way, discriminator 508 can know which trajectories in scenarios 1, 2, and 3 are preferred.

[0069] In some embodiments, the discriminator 508 can evaluate candidate trajectories based on their compatibility with the current scene. Each consecutive trajectory It can first be converted into action embeddings (also known as action features) using a multilayer perceptron (MLP), for example, through formula (C-1): (C-1) in This indicates action embedding, and MLP stands for Multilayer Perceptron.

[0070] Action embeddings can be used as queries for cross-attention mechanisms with the fused scene token. The scene representation is constructed from the following components: BEV token. Static token and dynamic tokens For example, through formula (C-2): (C-2)

[0071] in and It shares the same network architecture as the corresponding encoder in the planning head, but has independent parameters.

[0072] Action embeddings can be progressively integrated with scene tokens through a series of cross-attention operations. In some embodiments, action embeddings can interact with BEV tokens, static tokens, and dynamic tokens in a hierarchical manner, for example, through formula group (C-3): (C-3) Each attention interaction network It can have the same network structure as the planning head in the AD policy module, and load the pre-trained parameters of the planning head as initialization. For example, the pre-trained parameters can include the parameters of the feed-forward network (FFN) and / or MLP in the cross-attention network. This indicates a bird's-eye view fusion feature, or the first fusion feature; This represents a static fusion feature, or a second fusion feature; This indicates a fusion feature, or a fusion result, or a third fusion feature.

[0073] In formula (C-3), It can be used as a query vector. It can be used as both a key vector and a value vector; It can be used as a query vector. It can be used as both a key vector and a value vector; It can be used as a query vector. It can be used as both a key vector and a value vector.

[0074] In some embodiments, the fused embedding vector can be The input is fed into a scoring multilayer perceptron with a sigmoid activation function to generate trajectory scores. For example, through formula group (C-4): (C-4) in, This represents the sigmoid activation function; It represents a trajectory.

[0075] The trajectory score reflects the compatibility of the candidate trajectory with the scene geometry and the vehicle's behavior, i.e., how "good" or "bad" the trajectory is in that scene.

[0076] In some embodiments, to guide AD policymakers in exploring safe and efficient driving strategies, a reward score may be determined using one or a weighted average of three reward functions. The three reward functions may include a collision risk reward (also known as a first reward score), a braking quality reward (also known as a second reward score), and a route completion reward (also known as a third reward score).

[0077] Collision Risk Reward. The collision risk reward is used to characterize the collision risk level of the vehicle in the current driving scenario. The collision risk reward is calculated based on Time-To-Collision (TTC). Specifically, based on the vehicle's current speed, the X-axis and Y-axis position coordinates of the actual driving trajectory obtained from the simulation in the closed-loop environment are interpolated according to the cumulative driving distance to construct a vehicle trajectory under the assumption that the current speed is the subsequent driving speed of the vehicle (hereinafter referred to as constant speed). Based on the positional relationship between this constant speed trajectory and surrounding traffic participants, the collision time between the vehicle and surrounding traffic participants is calculated, where the collision time is the time interval from the current moment to the earliest collision. At each time t, the collision risk reward can be expressed as: (D-1) in, Indicates at time The constant speed trajectory of the vehicle is constructed at the location. This represents the set of positions of all traffic participants at time t. The maximum threshold for collision time is set to 3 seconds in some embodiments. This represents a function that calculates the collision time based on the vehicle's trajectory and information about other traffic participants. minThis indicates the search for the minimum value. When the calculated collision time is less than the maximum threshold, the corresponding collision risk reward is negative, used to penalize high-collision-risk driving situations.

[0078] Braking quality bonus. Order This represents the ideal distance, which is the distance between the vehicle and the vehicle in front when the vehicle comes to a complete stop, such as the distance between the vehicle and an obstacle in front of it when it stops. When the actual stopping distance deviates from the ideal distance... At the same time, regardless of whether the deviation is due to premature or delayed braking, it will lead to a reduction in braking quality reward. In this way, both overly aggressive and overly conservative braking behaviors can be penalized simultaneously, thereby forming a continuous feedback signal to guide the vehicle to achieve safe and precise braking control.

[0079] Route completion reward. Route completion can be measured using ego progress (EP), where ego progress is defined as the vehicle's progress along the travel path. As the vehicle continues to move forward along the reference route, its ego progress gradually increases, thereby correspondingly increasing the route completion reward. When the vehicle reaches or exceeds the predetermined target position on the reference route, the ego progress reaches its upper limit and no longer increases, while the corresponding route completion reward remains unchanged. By introducing a route completion reward, the vehicle is encouraged to continue moving towards the target direction while adhering to the constraints of the reference route.

[0080] In some embodiments, the EP can also be measured using a fixed distance the vehicle needs to travel. For example, a fixed target distance is preset, and a timer is started when the vehicle begins its task to record the time required to complete the fixed distance. The shorter the completion time, the higher the vehicle's traffic efficiency under safe and constrained conditions, and the higher the corresponding route completion reward. Conversely, if the completion time is long, the route completion reward is correspondingly reduced. In some embodiments, a certain length of real-world scene data can be collected, for example... Figure 4A or Figure 4B The observation scenario 402 is described as lasting 30 seconds. One or more segments of varying lengths, such as 10-second intervals, are extracted from the actual scene. These segments are used as a standard, and during training, the vehicle travels along the actual trajectory within each segment. At the end of the 10-second interval, the distance traveled by the vehicle during training is compared to the actual distance traveled in the segment. The route completion reward is calculated in this way. Similarly, the vehicle can be trained to travel along the actual trajectory to the destination, and the time taken to reach the destination can be recorded. This time is then compared to the actual travel time in the segment to calculate the route completion reward.

[0081] To train the discriminator, you can let Let represent the training sample data within a set of multimodal trajectory hypotheses (i.e., multiple closed-loop simulations for the same observation scenario), where ... The included execution traces (e.g., as shown below) Figure 6 The combination of trajectories 620, 622, and 624 described in the text, also referred to as the training agent's driving trajectory or at least a portion thereof, is [number missing]. ,make For sequence The reward. At any moment. t Advantage value at the location The calculation formula (E-1) is as follows: (E-1) Here, mean() means to calculate the average; std means to calculate the standard deviation.

[0082] The cutoff target is defined by formula (E-2). (E-2)

[0083] in express The importance sampling ratio at a given moment can reflect the ratio of the current action token's relative probability to that of the previous discriminator under the updated discriminator; This represents the truncation function. This represents a constant used to determine the upper and lower limits of the cutoff function.

[0084] In some embodiments, an entropy regularization term can be used for stable token-level training, for example, through formula (E-3): (E-3)

[0085] Entropy regularization encourages exploration, prevents premature convergence to deterministic token-level decisions, and alleviates optimization difficulties caused by saturation of the sigmoid function when probabilities are close to 0 or 1—a phenomenon known as entropy collapse. For example, suppose there are only two trajectories, and the scorer assigns 1 point to a good trajectory and 0 points to a bad one. Entropy regularization can encourage trajectory scores to avoid being too biased towards 0 or 1, ideally leaning towards 0.5.

[0086] In some embodiments, the final optimization objective function (including entropy regularization) is Equation (E-4): (E-4)

[0087] in Let represent the entropy regularization term, where β represents the strength of the entropy regularization.

[0088] In some embodiments, the planning head can generate a set of multimodal trajectory hypotheses at each time step and select one trajectory to execute. However, regenerating a new set of trajectories at each step during reinforcement learning training can disrupt the continuity of the vehicle's short-term motion intentions. Such frequent switching can prevent the policy from maintaining a stable exploration direction and reduce the correlation between each selected trajectory and the final driving outcome, resulting in inefficient exploration and hindering the steady improvement of the policy.

[0089] To alleviate these problems, a trajectory reuse mechanism can be introduced. For example, from time 0:00... t Select a trajectory from the generated trajectory set. Then, it can be transformed into the optimal control sequence by using an iLQR controller. For example, through formula (F-1): (F-1)

[0090] Trajectory reuse mechanisms can be applied at the control level, operating within a fixed time range. In each step The corresponding control is executed, for example, through formula (F-2): (F-2) Where H represents the predicted trajectory range.

[0091] In some embodiments, the discriminator in reinforcement learning can only explore relatively good driving trajectories provided by the generator. If the generator cannot provide sufficiently reliable and feasible solutions, the performance of the entire system will be fundamentally limited. To address the problem that the trajector-generated trajectories are unreliable, the training mechanism can be improved: that is, in addition to the planning pre-training phase based on imitation learning, an on-policy generator optimization (OGO) phase can be introduced, which can improve the quality of generated trajectories through an iterative self-correction mechanism.

[0092] For example, multiple driving trajectories are first obtained through closed-loop simulation using the current strategy. Safety and efficiency metrics are calculated for each trajectory. Trajectories with collision risks and inefficiencies are identified and corrected. The corrected trajectories are then used as supervisory data for generator optimization.

[0093] For safety reasons, when the TTC value is less than the safe TTC threshold, the trajectory to be corrected will be shortened by a fixed proportion to obtain the corrected trajectory. This encourages the model to drive cautiously at lower speeds, thereby reducing the risk of collision. To improve efficiency, if the current speed of the vehicle is lower than the expert trajectory and its position is behind the expert trajectory, but there is sufficient safe acceleration distance, the trajectory can be corrected by lengthening it. This encourages the model to drive more efficiently at higher speeds, for example, through the loss function in formula (G-1). Implement optimizations for the generator. (G-1) in This indicates the corrected trajectory. This represents the trajectory predicted by the generator. This represents the set of identified and collected trajectory samples that require correction. This indicates the predicted trajectory range.

[0094] In some embodiments, the updated AD strategy can be used in the next round of generator optimization. Using self-generated closed-loop trajectories as supervision information can better conform to the distribution pattern of the AD strategy, thereby making the optimization process and the prioritized results more targeted and stable.

[0095] refer to Figure 6 This illustrates a schematic diagram 600 of sub-training scenario data according to some embodiments of the present disclosure. Figure 6 In this model, the training scene data can be divided into three sub-training scene data: sub-training scene data 632, sub-training scene data 634, and sub-training scene data 636. Sub-training scene data 632 represents the scene data between frames 602 and 604. Sub-training scene data 634 represents the scene data between frames 604 and 606. Sub-training scene data 636 represents the scene data between frames 606 and 608. In some embodiments that do not involve image frames, frames 602 to 608 can also be represented using timestamps. Furthermore, the scene data between frames 602 and 604 does not represent only two frames, but rather the frame data within that interval; the specific number of frames can be set according to requirements.

[0096] At frame 602, which serves as the starting point for training, the generator can generate some candidate driving trajectories based on frame 602 (e.g., candidate driving trajectories 620, 640, 642, and 644, also referred to as the first driving trajectory set or the second driving trajectory set), and the vehicle randomly selects one of these trajectories as its target driving trajectory, for example, in... Figure 6The vehicle chose to drive along trajectory 620 (also known as the training agent driving trajectory or the second training agent driving trajectory). Frame 602 can generate multiple candidate trajectories using data from the original sensors or data from one of the observation representations determined based on the original sensor data (e.g., the earliest time can be selected).

[0097] After selecting trajectory 620, the vehicle begins traveling along trajectory 620 at point 610 and reaches the end point 612 of trajectory 620 at frame 604. In some embodiments, the surrounding environment is observed at each trajectory point from frame 602 to frame 604, and simulation observation data for that moment is generated using a closed-loop simulation platform (e.g., closed-loop simulation platform 410 in Figure 4) for each trajectory point. In some embodiments, the surrounding environment is observed at every few trajectory points from frame 602 to frame 604, and simulation observation data for that moment is generated using a closed-loop simulation platform for the end points of every few trajectory points. Some or all of the multiple simulation observation data corresponding to these different trajectory points can be used to form sub-training scene data 632 (also known as first sub-training scene data).

[0098] At frame 604, the vehicle has reached the endpoint 612 of trajectory 620. The generator can then regenerate some candidate driving trajectories based on frame 604 (also known as the first endpoint scene data). These candidate driving trajectories can be referred to as the third driving trajectory set. The vehicle randomly selects a trajectory to drive on, for example, following trajectory 622 (also known as the third training agent driving trajectory). After selecting trajectory 622, the vehicle starts driving along trajectory 622 at 612 and reaches the endpoint 614 of trajectory 622 at frame 606. In some embodiments, for each trajectory point from frame 604 to frame 606, the surrounding environment is observed, and simulation observation data for that moment is generated using a closed-loop simulation platform for each trajectory point. In some embodiments, from frame 604 to frame 606, the surrounding environment is observed at every few trajectory points, and simulation observation data for that moment is generated using a closed-loop simulation platform for the endpoints of every few trajectory points. Some or all of these simulation observation data corresponding to different trajectory points can be used to form sub-training scene data 634 (also known as the second sub-training scene data).

[0099] At endpoint 614, the generator can regenerate some candidate driving trajectories based on frame 606, and the vehicle randomly selects a trajectory to drive on, for example, according to trajectory 624. After selecting trajectory 624, the vehicle starts driving along trajectory 624 at 614 and reaches the endpoint 616 of trajectory 624 at frame 608. In some embodiments, the surrounding environment is observed at each trajectory point from frame 606 to frame 608, and simulation observation data at that moment is generated using a closed-loop simulation platform for each trajectory point. In some embodiments, the surrounding environment is observed at every few trajectory points from frame 606 to frame 608, and simulation observation data at that moment is generated using a closed-loop simulation platform for the endpoints of every few trajectory points. Some or all of the multiple simulation observation data corresponding to these different trajectory points can be used to form sub-training scene data 636. It is understood that, for the sake of simplicity, Figure 6 The process of generating three candidate trajectories is shown, but this is not intended as a limitation.

[0100] In some embodiments, the combination of sub-training scene data 632, 634, and 636 can be referred to as training scene data. In some embodiments, sub-training scene data 632, 634, and 636 can be referred to as training scene data individually. It is understood that if another trajectory, such as trajectory 640, is randomly selected at frame 602, the subsequent sub-training scene data and subsequent trajectories will also be different from those when trajectory 620 was selected. In some embodiments, the sub-training scene data generated for trajectories 620, 640, 642, and 644 can be referred to as training scene data, or can constitute at least a portion of the training scene data.

[0101] In some embodiments, the combination of trajectories 620, 622, and 624 can form a complete trajectory (also known as the driving trajectory of the training agent or at least a part of the driving trajectory of the training agent), and the combination of sub-training scene data 632, sub-training scene data 634, and sub-training scene data 636 can form a complete scene data. The complete trajectory and the complete scene data can together form a training subsample. In some embodiments, the above process can be repeated multiple times starting from frame 602 to form a training sample. In some embodiments, data collected from multiple real vehicles can generate multiple training samples.

[0102] refer to Figure 7The diagram illustrates a block diagram of diversity 700 for training scenario data according to some embodiments of the present disclosure. Diversity 700 demonstrates that multiple candidate driving trajectories, such as candidate driving trajectories 720, 722, 724, and 726, can be generated for observed scenario data 710. It is understood that the number of candidate driving trajectories presented here is merely exemplary and not limiting.

[0103] In diversity 700, since a candidate driving trajectory is randomly selected for closed-loop simulation to generate training scenario data, the training scenario data corresponding to the randomly selected trajectory is also different due to randomness; that is, training scenario data 730, 732, 734, and 736 are all different. Therefore, using some or all of these training scenario data 730, 732, 734, and 736 in combination 738 to train the driving trajectory generation model can satisfy the diversity of the vehicle's driving process in real-world environments.

[0104] refer to Figure 8 The diagram illustrates a block diagram of a process 800 for generating a target driving trajectory according to some embodiments of the present disclosure. Process 800 can be performed by a trained driving trajectory generation model, for example via... Figures 4A-6 The training process described yields a driving trajectory generation model. The observation scene data 802 collected by the vehicle during its driving process can be input into the AD policy module 804 (e.g., Figure 4A or Figure 4B The generator 406 is described. The AD strategy module 804 can generate multiple candidate driving trajectories, such as trajectory 808, 810 and other trajectories.

[0105] Multiple candidate driving trajectories can be input into the discriminator 806 (e.g., Figure 5 The generator 508 is described. The discriminator 806 can score each candidate driving trajectory, as shown in box 812. The discriminator 806 can output the candidate driving trajectory with the highest score (e.g., trajectory 808) as the target driving trajectory.

[0106] The driving trajectory generation model obtained using the training method disclosed herein can be run on a test dataset to test the model's performance. During training, the pre-training of the diffusion generator utilizes real-world driving data, providing vehicle trajectories covering various traffic scenarios. These trajectories are used for motion planning in the pre-trained generator, enabling it to capture the multimodal distribution of human driving behavior. For closed-loop reinforcement learning, a test dataset of real-world driving scenarios is constructed.

[0107] The training dataset consists of 8,000 cut-in camera clips and 8,000 emergency braking clips. Each clip is a continuous driving sequence, ranging from 9 to 13 seconds in length, focusing on safety-critical interactions. All clips were used for reinforcement learning training and discriminator optimization. For the test dataset used in the inference phase, 512 challenging cut-in scenarios, 512 braking scenarios, and 512 cruise clips were retained for closed-loop evaluation. Test results are shown in Table 1.

[0108] Wherein, CR (Collision Ratio) represents the average collision ratio of all collisions in the test set, and EP represents the average percentage of collisions in the closed-loop test set (the percentage of vehicles that can reach the expert trajectory endpoint). The first row, IL, represents the evaluation results of a generator trained only using traditional imitation learning; the trajectory used in the evaluation is a randomly selected trajectory from multiple trajectories generated by the generator. The second row, IL+OGO, represents the results of training the generator using IL and then combining it with OGO. Figure 5 The evaluation results described are based on the fine-tuning and optimization of the end-to-end model of the multi-mode trajectory according to the closed-loop results. The trajectory used in the evaluation is a trajectory randomly selected from multiple trajectories generated by the generator. The third line, IL+RL+OGO, indicates that after the generator is fine-tuned and optimized by combining IL and OGO in the second line, a discriminator is added and trained using the RL closed-loop training method proposed in this disclosure. The final evaluation result is calculated by the discriminator trained with RL closed-loop from the multi-mode trajectories generated by the generator according to the score.

[0109] As can be seen, the training method disclosed herein constructs a decision core in which a generator and a discriminator work together. In some embodiments, the generator can generate online a smooth trajectory candidate set covering multimodal intentions, conditioned on scene representations such as bird's-eye view features extracted by the perception system. This set is based on pre-training through imitation learning to inherit expert driving behavior. Simultaneously, the discriminator, trained through reinforcement learning, can quantitatively score these candidate trajectories, focusing on evaluating their value (quality) in long-sequence trajectories executed in future dynamic environments, rather than making static, short-sighted judgments.

[0110] Furthermore, in some embodiments, the training method of this disclosure also employs a collaborative optimization mechanism, which enables the generator and discriminator to form a virtuous cycle of mutual promotion. By employing methods such as grouped relative strategy optimization, the discriminator is trained using closed-loop simulation data, aligning its scoring criteria with long-term driving rewards, thereby enabling it to distinguish the merits of long-sequence trajectories. In addition, in some embodiments, an online fine-tuning technique for the strategy generator is introduced, specifically correcting trajectories judged as suboptimal by the discriminator in the simulation, and using this as a supervisory signal to fine-tune the generator, enabling it to continuously produce candidate trajectories that better meet higher-order evaluation criteria, ultimately driving the continuous self-improvement of the system's decision-making performance.

[0111] refer to Figure 9 The diagram illustrates a flowchart of a training method 900 for a driving trajectory generation model according to some embodiments of the present disclosure. Method 900 can be executed by an apparatus for training the driving trajectory generation model, which may be, for example, a standalone device or system. This apparatus can be implemented in software and / or hardware. The following description will illustrately illustrate method 900 using an apparatus for training the driving trajectory generation model as an example. Method 900 includes blocks 902, 904, 906, and 908.

[0112] At box 902, training scene data and the driving trajectory of the training agent are generated based on the observed scene data. For example, it can be based on... Figure 7 The described observation scene data 710 generates training scene data 738 and training agent driving trajectories 720 to 726. For the sake of brevity, this disclosure will not elaborate on this process, and references will be made below. Figures 1-8 The process that has already been described will not be repeated here.

[0113] At box 904, training samples are determined based on the training scenario data and the driving trajectory of the training agent. For example, a combination of the training scenario data and the driving trajectory of the training agent can be used to determine the training samples. At box 906, the reward score for the driving trajectory of the training agent is determined based on the training samples. For example, it can be based on... Figure 5 The process described in section 500 includes boxes 506, formulas (B-1) through (B-3). At box 908, the model parameters are updated based on the reward score. For example, this can be based on... Figure 5 The process described is reinforcement learning training in 500.

[0114] Thus, the trajectory generation model trained using method 900 can enhance decision robustness in dynamic environments, improve adaptability to dynamic scenes, and enhance the safety of generated trajectories.

[0115] refer to Figure 10It illustrates some embodiments according to this disclosure. Figure 9 The flowchart of the detailed process 1000 is as follows. Box 902 may include boxes 1002 and 1004. At box 1002, training scene data is generated based on observed scene data. At box 1004, training scene data is generated based on the driving trajectory of the training agent and observed scene data. For example, it can be based on... Figure 4A The described process 400A or as Figure 4B In process 400B, box 412, or based on Figure 7 The process described is 700.

[0116] refer to Figure 11 It illustrates some embodiments according to this disclosure. Figure 9 The flowchart of the detailed process 1100 is as follows. Box 902 may include boxes 1102, 1104, and 1106. At box 1102, a first set of driving trajectories is generated based on the observed scene data. At box 1104, the driving trajectory of the training agent is determined based on the first set of driving trajectories. At box 1106, training scene data is generated based on the observed scene data and the driving trajectory of the training agent. For example, it can be based on... Figure 4A The described process 400A, Figure 4B The described process 400B, or Figure 6 The described process 600 or Figure 5 The formulas described are (F-1) to (F-2).

[0117] refer to Figure 12 It illustrates some embodiments according to this disclosure. Figure 9 The flowchart of the detailed process 1200. Box 902 may include boxes 1202, 1204, 1206, 1208, 1210, 1212, 1214, 1216 and 1218.

[0118] At box 1202, a second set of driving trajectories is generated based on the observed scene data. At box 1204, the driving trajectory of the second training agent is determined based on the second set of driving trajectories. At box 1206, the first sub-training scene data is determined based on the observed scene data and the driving trajectory of the second training agent. At box 1208, the first endpoint scene data at the endpoint of the second training agent's driving trajectory is determined based on the driving trajectory of the second training agent and the first sub-training scene data.

[0119] At box 1210, a third set of driving trajectories is generated based on the first endpoint scene data. At box 1212, the driving trajectory of the third training agent is determined based on the third set of driving trajectories. At box 1214, the second sub-training scene data is determined based on the observed scene data and the third training agent's driving trajectory. At box 1216, the driving trajectory of the training agent is determined based on the second and third training agent's driving trajectories. At box 1218, the training scene data is determined based on the first and second sub-training scene data. For example, it can be based on... Figure 4A The described process 400A, Figure 4B The described process 400B, or Figure 6 The process described is 600.

[0120] refer to Figure 13 It illustrates some embodiments according to this disclosure. Figure 9 The flowchart of the detailed process 1300 is as follows. Box 906 may include boxes 1302, 1304, 1306, and 1308. At box 1302, a first reward score associated with collision risk is determined. At box 1304, a second reward score associated with braking quality is determined. At box 1306, a third reward score associated with path completion is determined. At box 1308, a reward score is determined based on at least one of the first, second, or third reward scores. For example, it may be based on... Figure 5 The process described in 500 is a weighted sum of one or more of the following: the collision risk reward, the braking quality reward, or the route completion reward, calculated by formula (D-1).

[0121] refer to Figure 14 It illustrates some embodiments according to this disclosure. Figure 13 The flowchart for the detailed process 1400 is shown. Box 1302 may include boxes 1402 and 1404. At box 1402, based on training samples, the estimated time of collision between the agent and other objects in the scene is determined. At box 1404, a first reward score is determined based on the estimated time and a threshold time. For example, it can be based on... Figure 5 The formula (D-1) in process 500 is described.

[0122] refer to Figure 15 It illustrates some embodiments according to this disclosure. Figure 13The flowchart of the detailed process 1500 is as follows. Box 1304 may include boxes 1502, 1504, and 1506. At box 1502, braking of the agent is determined based on training samples. At box 1504, the braking distance or braking time of the agent is determined. At box 1506, a second reward score is determined based on the braking distance or braking time. For example, it can be based on... Figure 5 The braking quality reward in the described process 500.

[0123] refer to Figure 16 It illustrates some embodiments according to this disclosure. Figure 13 The flowchart of the detailed process 1600. Box 1302 may include boxes 1602 and 1604. At box 1602, based on training samples, the driving time required for the agent to complete at least a portion of the training agent's driving trajectory is determined, or the driving distance of the agent on the training agent's driving trajectory within a predetermined time period is determined. At box 1604, a third reward score is determined based on the driving time or driving distance. For example, it can be based on... Figure 5 The described process includes a route completion reward.

[0124] refer to Figure 17 It illustrates some embodiments according to this disclosure. Figure 9 The detailed process flow chart for process 1700. In process 1700, Figure 9 The driving trajectory of the training agent can include the driving trajectory of the first training agent and the driving trajectory of the second training agent. Figure 9 The reward score can include the first reward score of the first training agent's driving trajectory and the second reward score of the second training agent's driving trajectory.

[0125] Box 908 may include boxes 1702, 1704, and 1706. In box 1702, in response to a first reward score being higher than a second reward score, the model parameters are adjusted to increase the first trajectory score of the trajectory driven by the first training agent. In box 1704, in response to a first reward score being less than a second reward score, the model parameters are adjusted to increase the second trajectory score of the trajectory driven by the second training agent. In box 1706, the model parameters are updated using a loss function, wherein the loss function includes an entropy regularization term for making the first trajectory score and the second trajectory score optimal with a first value, where the first value is the midpoint of the trajectory score interval. For example, it can be based on... Figure 5 The formula described is (E-4).

[0126] refer to Figure 18 It illustrates some embodiments according to this disclosure. Figure 17The detailed process flow chart for process 1800. In process 1800, Figure 17 The first trajectory score of the first training agent's driving trajectory can be determined based on boxes 1802, 1804, 1806, and 1808. At box 1802, trajectory features are determined as a query vector based on the first training agent's driving trajectory. At box 1804, key and value vectors are determined based on scene features associated with the training scene data. At box 1806, attention-based feature fusion is performed on the query vector, key vector, and value vector. At box 1808, the trajectory score of the first training driving trajectory is determined based on the fused features. For example, it can be based on... Figure 5 The formulas described are (C-1) to (C-4).

[0127] refer to Figure 19 It illustrates some embodiments according to this disclosure. Figure 18 The detailed process flow of process 1900. In process 1900, Figure 18 Scene features associated with the training scenario data can be determined based on boxes 1902 and 1904. In box 1902, bird's-eye view features, dynamic features, and static features associated with the agent are acquired, where static features represent the state of immovable objects in the scene, and dynamic features represent the state of other movable objects besides the agent. In box 1904, scene features are determined based on the bird's-eye view features, dynamic features, static features, and navigation features associated with the agent's destination. For example, they can be based on... Figure 4A or Figure 4B The formulas described are (A-1) to (A-2).

[0128] refer to Figure 20 It illustrates some embodiments according to this disclosure. Figure 18 The flowchart of the detailed process 2000 is as follows. Box 1806 may include boxes 2002, 2004, 2006, 2008, and 2010. At box 2002, action features are determined based on the driving trajectory of the first training agent. At box 2004, a first fused feature is determined using a cross-attention mechanism based on the action features and bird's-eye view features. At box 2006, a second fused feature is determined using a cross-attention mechanism based on the first fused feature and static features. At box 2008, a third fused feature is determined using a cross-attention mechanism based on the second fused feature and dynamic features. At box 2010, the third fused feature is determined as the fused feature. For example, it can be based on... Figure 5 The formula described is (C-3).

[0129] refer to Figure 21 It illustrates some embodiments and according to this disclosure. Figure 9and Figure 11 The detailed process is shown in the flowchart of 2100. Figure 11 Box 1102 in the diagram may include box 2102. Figure 9 Box 908 in the diagram can include boxes 2104 and 2106. At box 2102, a first set of driving trajectories is generated using the model's generative network, based on scene features and noise. For example, it can be based on... Figure 4A or Figure 4B The formulas described are (A-3) and (A-4).

[0130] At box 2104, in response to the training agent's driving trajectory not meeting the training requirements, a training trajectory that meets the training requirements is obtained. At box 2106, the parameters of the generator network are updated so that the generator network can generate a training trajectory that meets the training requirements. For example, it can be based on... Figure 5 The formula described is (G-1).

[0131] Thus, by integrating a trajectory discriminator trained using reinforcement learning and a trajectory generator trained using imitation learning, at least one of the problems in existing intelligent driving decision-making and planning—such as insufficient long-term robustness, poor trajectory feasibility, and low exploration efficiency—can be solved. The scheme disclosed herein, supported by closed-loop training, generates multimodal candidate trajectories, which are then scored and selected by a discriminator trained in a closed-loop manner to find suitable trajectories. Combined with closed-loop simulation and optimization mechanisms, performance is iteratively improved, ultimately enhancing the system's safety, efficiency, and robustness in dynamic scenarios. This approach is suitable for end-to-end autonomous driving decision-making and planning.

[0132] refer to Figure 22 The document illustrates a flowchart of a method 2200 for generating a driving trajectory according to some embodiments of the present disclosure. Method 2200 can be performed via... Figures 9-22 The method described above trains a driving trajectory generation model, which can be executed on a standalone device or system. This device can be implemented in software and / or hardware. The following description uses a device running the driving trajectory generation model as an example to illustrate method 2200. Method 2200 includes blocks 2202, 2204, and 2206.

[0133] At box 2202, the scene data collected by the agent is processed based on the driving trajectory generation model to generate a set of candidate driving trajectories for the agent. For example, through... Figure 1 The observation scene data described is 110.

[0134] At box 2204, the candidate driving trajectory set and the collected scene data are processed based on the driving trajectory generation model to determine the trajectory score set of the candidate driving trajectory set. For example, through methods such as... Figure 1 The trajectory generation model 104 is described.

[0135] At box 2206, the target driving trajectory is determined from the candidate driving trajectory set based on the trajectory score set. For example, the result is as follows: Figure 1 The described target trajectory is 116.

[0136] In some embodiments, the model parameters of the driving trajectory generation model can be trained based on the reward scores of the driving trajectories of the training agent in the training samples. In some embodiments, the training samples may include training scene data and the driving trajectories of the training agent, which can be determined based on the sample observed scene data.

[0137] Thus, by using method 2200, the decision robustness of the intelligent driving system in dynamic environments and its adaptability to dynamic scenarios can be enhanced, and the safety of the generated trajectory can be improved.

[0138] refer to Figure 23 It illustrates some embodiments and according to this disclosure. Figure 22 The detailed process is shown in the flowchart of 2300. Figure 22 Box 2204 in the diagram may include boxes 2302, 2304, 2306, and 2308. At box 2302, trajectory features are determined based on the first candidate driving trajectory in the candidate trajectory set. At box 2304, scene features are determined based on observed scene data. At box 2306, feature fusion is performed on the trajectory features and scene features using an attention mechanism to determine the fused features. At box 2308, a first trajectory score is determined based on the fused features. For example, it can be based on... Figure 5 The formulas described are (C-1) to (C-4).

[0139] refer to Figure 24 It illustrates some embodiments and according to this disclosure. Figure 23 The detailed process is shown in the flowchart of 2400. Figure 23 The scene features in box 2304 can be determined by boxes 2402 and 2404. In box 2402, bird's-eye view features, dynamic features, and static features associated with the agent are acquired, where static features represent the state of immovable objects in the scene, and dynamic features represent the state of other movable objects besides the agent. In box 2404, scene features are determined based on the bird's-eye view features, dynamic features, static features, and navigation features associated with the agent's destination. For example, they can be based on... Figure 4A or Figure 4B The formulas described are (A-1) to (A-2).

[0140] refer to Figure 25 It illustrates some embodiments and according to this disclosure. Figure 23 The detailed process is shown in the flowchart of 2500. Figure 23 The fusion feature in box 2306 can be determined by boxes 2502, 2504, 2506 and 2508.

[0141] At box 2502, action features are determined based on the first candidate driving trajectory. At box 2504, bird's-eye view fusion features are determined using a cross-attention mechanism based on action features and bird's-eye view features. At box 2506, static fusion features are determined using a cross-attention mechanism based on bird's-eye view fusion features and static features. At box 2508, fusion features are determined based on static fusion features and dynamic features. For example, it can be based on... Figure 5 The formula described is (C-3).

[0142] Figure 26 A block diagram of a device 2600 capable of implementing various embodiments of the present disclosure is shown. (See diagram for example.) Figure 26 As shown, device 2600 includes a processing unit 2601 such as a central processing unit (CPU) and / or a graphics processing unit (GPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 2602 or loaded from storage unit 2608 into random access memory (RAM) 2603. Various programs and data required for the operation of device 2600 can also be stored in RAM 2603. The processing unit 2601, ROM 2602, and RAM 2603 are interconnected via bus 2604. Input / output (I / O) interface 2605 is also connected to bus 2604. Although not shown in... Figure 26 As shown, device 2600 may also include a coprocessor.

[0143] Multiple components in device 2600 are connected to I / O interface 2605, including: input unit 2606, such as keyboard, mouse, etc.; output unit 2607, such as various types of monitors, speakers, etc.; storage unit 2608, such as disk, optical disk, etc.; and communication unit 2609, such as network card, modem, wireless transceiver, etc. Communication unit 2609 allows device 2600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0144] The various methods or processes described above can be executed by processing unit 2601. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 2608. In some embodiments, part or all of the computer program can be loaded and / or installed on device 2600 via ROM 2602 and / or communication unit 2609. When the computer program is loaded into RAM 2603 and executed by processing unit 2601, one or more steps or actions in the methods or processes described above can be performed.

[0145] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0146] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0147] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0148] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to execute the computer-readable program instructions, thereby implementing various aspects of this disclosure.

[0149] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0150] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0152] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0153] The following are some example implementations of this disclosure.

[0154] Example 1. A training method for a driving trajectory generation model, comprising: Based on the observed scene data, training scene data and the driving trajectory of the training agent are generated; Training samples are determined based on training scenario data and the driving trajectory of the training agent. Based on training samples, determine the reward score for the trajectory of the trained agent; and The model parameters are updated based on the reward scores.

[0155] Example 2. According to the method described in Example 1, generating training scene data based on observed scene data includes: Training scenario data is generated based on the driving trajectory of the training agent and the observation scene data.

[0156] Example 3. According to the method described in Example 1 or 2, generating training scene data and the driving trajectory of the training agent based on observed scene data includes: Based on the observed scene data, a first set of driving trajectories is generated; Based on the first set of driving trajectories, determine the driving trajectory of the training agent; and Training scenario data is generated based on the observed scenario data and the driving trajectory of the training agent.

[0157] Example 4. According to any one of Examples 1-3, generating training scene data and the driving trajectory of the training agent based on observed scene data includes: A second set of driving trajectories is generated based on the observed scene data; Based on the second set of driving trajectories, determine the driving trajectory of the second training agent; based on the observed scene data and the driving trajectory of the second training agent, determine the first sub-training scene data. Based on the driving trajectory of the second training agent and the first sub-training scenario data, determine the first endpoint scenario data at the endpoint of the driving trajectory of the second training agent; Based on the first destination scene data, a third set of driving trajectories is generated; Based on the third set of driving trajectories, determine the driving trajectory of the third training agent; Based on the observed scene data and the driving trajectory of the third training agent, the second sub-training scene data is determined. Based on the driving trajectories of the second and third training agents, the driving trajectory of the training agent is determined; and The training scenario data is determined based on the first sub-training scenario data and the second sub-training scenario data.

[0158] Example 5. The method according to any one of Examples 1-4, wherein the observation scene data includes: raw sensor data or observation features determined based on the raw sensor data.

[0159] Example 6. The method according to any one of Examples 1-5, wherein the reward score for determining the trajectory of the training agent based on training samples includes at least one of the following: Determine the first reward score associated with collision risk; Determine a second bonus score associated with braking quality; or Determine the third reward score associated with path completion.

[0160] Example 7. The method according to any one of Examples 1-6, wherein determining the first reward score associated with collision risk includes: Based on training samples, the estimated time of collision between the agent and other objects in the scene is determined; and The first reward score is determined based on the estimated time and the threshold time.

[0161] Example 8. The method according to any one of Examples 1-7, wherein determining the second reward score associated with braking quality includes: Based on the training samples, determine the braking action to be taken against the agent; Determine the braking distance or braking time of the agent; and The second bonus score is determined based on braking distance or braking time.

[0162] Example 9. The method according to any one of Examples 1-8, wherein determining the third reward score associated with path completion includes: Based on training samples, determine the travel time required for the agent to complete at least a portion of the training agent's travel trajectory, or determine the travel distance of the agent on the training agent's travel trajectory within a predetermined time period; and The third reward score is determined based on the driving time or driving distance.

[0163] Example 10. The method according to any one of Examples 1-9, wherein determining the reward score for the trajectory of the training agent based on training samples includes: The reward score is determined based on at least one of the first reward score, the second reward score, or the third reward score.

[0164] Example 11. The method according to any one of Examples 1-10, wherein the driving trajectory of the training agent includes a first driving trajectory of the training agent and a second driving trajectory of the training agent, the reward score includes a first reward score of the first driving trajectory of the training agent and a second reward score of the second driving trajectory of the training agent, and wherein the parameters of the model are updated based on the reward score, including: In response to the first reward score being higher than the second reward score, the model parameters are adjusted to increase the first trajectory score of the first training agent's driving trajectory; or In response to the first reward score being less than the second reward score, the model parameters are adjusted to increase the second trajectory score of the second training agent's driving trajectory.

[0165] Example 12. The method according to any one of Examples 1-11, wherein updating the model parameters based on the reward score further includes: The model parameters are updated using a loss function, which includes an entropy regularization term to make the scores of the first and second trajectories optimal with a first value, where the first value is the midpoint of the trajectory score interval.

[0166] Example 13. The method according to any one of Examples 1-12, wherein a first trajectory score of the trajectory of the first training agent is determined based on the following: Based on the driving trajectory of the first trained agent, determine the trajectory features to serve as the query vector; Based on scene features associated with the training scene data, determine the key vector and value vector; Perform attention-based feature fusion on the query vector, key vector, and value vector; and Based on the fused features, the trajectory score of the first training driving trajectory is determined.

[0167] Example 14. The method according to any one of Examples 1-13, wherein scene features are determined based on the following: Acquire bird's-eye view features, dynamic features, and static features associated with the agent, where static features represent the state of immovable objects in the scene, and dynamic features represent the state of other movable objects besides the agent; and Scene features are determined based on bird's-eye view features, dynamic features, static features, and navigation features associated with the agent's destination.

[0168] Example 15. The method according to any one of Examples 1-14, wherein performing attention-based feature fusion on the query vector, key vector, and value vector includes: Based on the driving trajectory of the first trained agent, determine the action characteristics; Based on action features and bird's-eye view features, the first fusion feature is determined using a cross-attention mechanism; Based on the first fusion feature and the static feature, the second fusion feature is determined using a cross-attention mechanism; Based on the second fusion feature and dynamic features, a cross-attention mechanism is used to determine the third fusion feature; and The third fusion feature is determined as the fusion feature.

[0169] Example 16. The method according to any one of Examples 1-15, wherein generating the first set of driving trajectories based on observed scene data includes: Based on scene features and noise, the model's generative network is used to generate the first set of driving trajectories. Example 17. The method according to any one of Examples 1-16, wherein updating the model parameters based on the reward score includes: In response to the fact that the driving trajectory of the training agent does not meet the training requirements, a training trajectory that meets the training requirements is obtained; and Update the parameters of the generator network so that it can generate training trajectories that meet the training requirements.

[0170] Example 18. A method for generating a driving trajectory, comprising: The scene data collected by the intelligent agent is processed based on the driving trajectory generation model to generate a set of candidate driving trajectories for the intelligent agent. The candidate driving trajectory set and the collected scene data are processed based on the driving trajectory generation model to determine the trajectory score set of the candidate driving trajectory set; and The target driving trajectory is determined from the candidate driving trajectory set based on the trajectory score set.

[0171] Example 19. The method described in Example 18, wherein the model parameters of the driving trajectory generation model are obtained by training based on the reward scores of the driving trajectories of the training agent, and the driving trajectories of the training agent are determined based on sample observation scene data.

[0172] Example 20. The method according to Example 18 or 19, wherein the acquired scene data includes: raw sensor data or observation features determined based on the raw sensor data.

[0173] Example 21. The method according to any one of Examples 18-20, wherein processing the candidate driving trajectory set and the collected scene data based on the driving trajectory generation model to determine the trajectory score set of the candidate driving trajectory set includes: Based on the first candidate driving trajectory in the candidate driving trajectory set, determine the trajectory features; Based on the collected scene data, the scene characteristics are determined; Feature fusion is performed on trajectory features and scene features based on an attention mechanism to determine the fused features; and Based on the fusion features, the first trajectory score of the first candidate driving trajectory is determined.

[0174] Example 22. The method according to any one of Examples 18-21, wherein scene features are determined based on the following: Acquire bird's-eye view features, dynamic features, and static features associated with the agent, where static features represent the state of immovable objects in the scene, and dynamic features represent the state of other movable objects besides the agent; and Scene features are determined based on bird's-eye view features, dynamic features, static features, and navigation features associated with the agent's destination.

[0175] Example 23. The method according to any one of Examples 18-22, wherein feature fusion is performed on trajectory features and scene features based on an attention mechanism, and the fused features are determined by: Based on the first candidate driving trajectory, determine the motion characteristics; Based on action features and bird's-eye view features, a cross-attention mechanism is used to determine the bird's-eye view fusion features. Based on bird's-eye view fusion features and static features, a cross-attention mechanism is used to determine static fusion features; Based on static and dynamic fusion features, the fusion features are determined.

[0176] Example 24. The method according to any one of Examples 18-23, wherein processing the collected scene data based on the driving trajectory generation model to generate a set of candidate driving trajectories for the agent includes: A set of candidate driving trajectories is generated based on scene features and noise.

[0177] Example 25. The method according to any one of Examples 18-24, wherein the driving trajectory generation model is trained according to any one of Examples 1-17.

[0178] 26. An electronic device, comprising: Processor; and A memory coupled to a processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of Examples 1 to 17 or 18 to 25.

[0179] 27. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of Examples 1 to 17 or 18 to 25.

[0180] 28. A non-transitory computer-readable medium having stored thereon machine-executable instructions that, when executed, cause a machine to perform the method according to any one of Examples 1 to 17 or 18 to 25.

[0181] Although this disclosure has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A training method for a driving trajectory generation model, comprising: Based on the observed scene data, training scene data and the driving trajectory of the training agent are generated; Based on the training scenario data and the driving trajectory of the training agent, training samples are determined; Based on the training samples, the reward score of the driving trajectory of the training agent is determined; as well as The parameters of the model are updated based on the reward score.

2. The method according to claim 1, wherein generating training scene data based on observed scene data includes: The training scenario data is generated based on the driving trajectory of the training agent and the observation scenario data.

3. The method according to claim 2, wherein generating training scene data and the driving trajectory of the training agent based on the observed scene data includes: Based on the observed scene data, a first set of driving trajectories is generated; Based on the first set of driving trajectories, the driving trajectory of the training agent is determined; as well as The training scenario data is generated based on the observed scenario data and the driving trajectory of the training agent.

4. The method according to claim 2, wherein generating training scene data and the driving trajectory of the training agent based on the observed scene data includes: Based on the observed scene data, a second set of driving trajectories is generated; Based on the second set of driving trajectories, determine the driving trajectory of the second training agent; Based on the observed scene data and the driving trajectory of the second training agent, the first sub-training scene data is determined; Based on the driving trajectory of the second training agent and the first sub-training scenario data, determine the first endpoint scenario data at the endpoint of the driving trajectory of the second training agent; Based on the first destination scene data, a third set of driving trajectories is generated; Based on the third set of driving trajectories, the driving trajectory of the third training agent is determined; Based on the observed scene data and the driving trajectory of the third training agent, the second sub-training scene data is determined; The driving trajectory of the training agent is determined based on the driving trajectory of the second training agent and the driving trajectory of the third training agent. as well as The training scenario data is determined based on the first sub-training scenario data and the second sub-training scenario data.

5. The method according to claim 1, wherein determining the reward score of the training agent's driving trajectory based on the training samples includes at least one of the following: Determine the first reward score associated with collision risk; Determine a second bonus score associated with braking quality; or Determine the third reward score associated with path completion.

6. The method of claim 5, wherein determining the first reward score associated with collision risk comprises: Based on the training samples, the estimated time of collision between the agent and other objects in the scene is determined; as well as The first reward score is determined based on the estimated time and the threshold time.

7. The method of claim 5, wherein determining the second reward score associated with braking quality comprises: Based on the training samples, it is determined that braking should be applied to the agent; Determine the braking distance or braking time of the intelligent agent; as well as The second bonus score is determined based on the braking distance or the braking time.

8. The method of claim 5, wherein determining the third reward score associated with path completion comprises: Based on the training samples, determine the travel time required for the agent to complete travel on at least a portion of the travel trajectory of the training agent, or determine the travel distance of the agent on the travel trajectory of the training agent within a predetermined time period. as well as The third reward score is determined based on the driving time or the driving distance.

9. The method according to claim 5, wherein determining the reward score for the training agent's trajectory based on the training samples includes: The reward score is determined based on at least one of the first reward score, the second reward score, or the third reward score.

10. The method of claim 1, wherein the driving trajectory of the training agent includes a first driving trajectory of the training agent and a second driving trajectory of the training agent, the reward score includes a first reward score of the first driving trajectory of the training agent and a second reward score of the second driving trajectory of the training agent, and wherein updating the parameters of the model based on the reward score includes: In response to the first reward score being higher than the second reward score, the parameters of the model are adjusted so that the first trajectory score of the first training agent's driving trajectory increases; or In response to the first reward score being less than the second reward score, the parameters of the model are adjusted so that the second trajectory score of the second training agent's driving trajectory increases.

11. The method of claim 10, wherein updating the parameters of the model based on the reward score further comprises: The parameters of the model are updated using a loss function, wherein the loss function includes an entropy regularization term for making the first trajectory score and the second trajectory score optimal with a first value, wherein the first value is the midpoint of the trajectory score interval.

12. The method of claim 11, wherein the first trajectory score of the first training agent's driving trajectory is determined based on the following: Based on the driving trajectory of the first training agent, determine the trajectory features as a query vector; Based on the scene features associated with the training scene data, determine the key vector and value vector; Perform attention-based feature fusion on the query vector, the key vector, and the value vector; as well as Based on the fused features, the trajectory score of the first training driving trajectory is determined.

13. The method of claim 12, wherein the scene features are determined based on the following: Acquire bird's-eye view features, dynamic features, and static features associated with the agent, wherein the static features represent the state of immovable objects in the scene, and the dynamic features represent the state of other movable objects besides the agent; and The scene features are determined based on the bird's-eye view features, the dynamic features, the static features, and the navigation features associated with the destination of the agent.

14. The method of claim 13, wherein performing attention-based feature fusion on the query vector, the key vector, and the value vector comprises: Based on the driving trajectory of the first training agent, determine the action characteristics; Based on the action features and the bird's-eye view features, a first fusion feature is determined using a cross-attention mechanism; Based on the first fusion feature and the static feature, the second fusion feature is determined using the cross-attention mechanism; Based on the second fusion feature and the dynamic feature, the third fusion feature is determined using the cross-attention mechanism; as well as The third fusion feature is determined as the fused feature.

15. The method according to claim 3, wherein generating the first set of driving trajectories based on the observed scene data comprises: Based on scene features and noise, the generative network of the model is used to generate the first set of driving trajectories.

16. The method of claim 15, wherein updating the parameters of the model based on the reward score comprises: In response to the fact that the driving trajectory of the training agent does not meet the training requirements, a training driving trajectory that meets the training requirements is obtained; as well as Update the parameters of the generator network so that it can generate the training trajectory that meets the training requirements.

17. A method for generating a driving trajectory, comprising: The scene data collected by the intelligent agent is processed based on the driving trajectory generation model to generate a set of candidate driving trajectories for the intelligent agent. The candidate driving trajectory set and the collected scene data are processed based on the driving trajectory generation model to determine the trajectory score set of the candidate driving trajectory set. as well as Based on the trajectory score set, the target driving trajectory is determined from the candidate driving trajectory set.

18. The method according to claim 17, wherein, The model parameters of the driving trajectory generation model are obtained by training based on the reward scores of the driving trajectory of the training agent, and the driving trajectory of the training agent is determined based on the sample observation scene data.

19. The method according to claim 17, wherein processing the candidate driving trajectory set and the collected scene data based on the driving trajectory generation model to determine the trajectory score set of the candidate driving trajectory set includes: Based on the first candidate driving trajectory in the set of candidate driving trajectories, determine the trajectory features; Based on the collected scene data, scene characteristics are determined; Based on an attention mechanism, feature fusion is performed on the trajectory features and the scene features to determine the fused features; as well as Based on the fusion features, a first trajectory score is determined for the first candidate driving trajectory.

20. The method of claim 19, wherein the scene features are determined based on the following: Acquire bird's-eye view features, dynamic features, and static features associated with the agent, wherein the static features represent the state of immovable objects in the scene, and the dynamic features represent the state of other movable objects besides the agent; and The scene features are determined based on the bird's-eye view features, the dynamic features, the static features, and the navigation features associated with the destination of the agent.

21. The method of claim 20, wherein performing feature fusion on the trajectory features and the scene features based on an attention mechanism to determine the fused features includes: Based on the first candidate driving trajectory, determine the motion characteristics; Based on the action features and the bird's-eye view features, a cross-attention mechanism is used to determine the bird's-eye view fusion features; Based on the bird's-eye view fusion features and the static features, the static fusion features are determined using a cross-attention mechanism; The fusion features are determined based on the static fusion features and the dynamic features.

22. The method according to claim 19, wherein processing the scene data collected by the agent based on the driving trajectory generation model to generate a candidate driving trajectory set for the agent includes: The candidate driving trajectory set is generated based on scene features and noise.

23. The method according to any one of claims 17-22, wherein the driving trajectory generation model is trained according to the method according to any one of claims 1-16.

24. An electronic device comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 16 or 17 to 23.

25. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 16 or 17 to 23.

Images