A demand-driven method for generating real-world autonomous driving scenarios

By generating real-world autonomous driving scenarios using multimodal large language models and optimal control theory, the problem of insufficient coverage of extreme scenarios in existing technologies is solved, achieving efficient and controllable scenario generation and simulation testing. The generated scenarios meet user needs at both the visual and physical levels.

CN123069928APending Publication Date: 2026-11-10BEIJING JIAOTONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611584910.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-14
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing autonomous driving technologies struggle to cover all possibilities when generating extreme driving scenarios, especially since extreme scenarios occur infrequently, resulting in insufficient system robustness. Furthermore, existing text-to-scene generation methods are unable to fully and accurately express user needs.

Method used

By employing a multimodal large language model combined with optimal control theory, a realistic autonomous driving scenario is generated from user demand information, initial images, and the initial pose of the agent. The agent's trajectory is then kinematically optimized. The generation process revolves around the user-specified scenario type and traffic participant behavior, and supports multiple input formats, including text, images, and poses.

Benefits of technology

The generated scenarios are highly matched with user needs, are visually realistic, and the vehicle behavior is executable at the physical and kinematic levels, making them suitable for autonomous driving simulation testing. This improves the controllability and design efficiency of the generated scenarios and can eliminate unreasonable motion problems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123069928A_ABST
    Figure CN123069928A_ABST
Patent Text Reader

Abstract

This invention relates to the field of autonomous driving technology, specifically to a demand-driven method for generating realistic autonomous driving scenarios. The method involves acquiring user demand information and a pre-processed map. The user demand information includes a scene text description, an initial scene image, and the initial pose of the agent within the scene. The user demand information is input into a pre-trained multimodal large language model to obtain integer-discrete agent behavior codes. The agent's initial pose, behavior codes, and pre-processed map are then input into a pre-trained scene generation model to obtain a realistic autonomous driving scenario matching the user demand information. Based on optimal control theory, the agent's trajectory in the realistic autonomous driving scenario is kinematically optimized. This method integrates demand understanding, behavior generation, scene generation, and trajectory optimization to form a relatively complete automated process. The resulting scenario is not only visually realistic but also exhibits physically and kinematically executable vehicle behavior, making it suitable for autonomous driving simulation testing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This specification relates to the field of autonomous driving technology, and in particular to a demand-driven method for generating real-world autonomous driving scenarios. Background Technology

[0002] In recent years, autonomous driving technology has made significant progress, enabling systems to operate with minimal or no human intervention within designated areas. Simulation testing, as a key means of evaluating the safety and performance of autonomous driving systems, provides an indispensable foundational environment for the development and verification of core algorithms such as perception, decision-making, and planning.

[0003] The performance of autonomous driving systems is highly dependent on massive amounts of high-quality real-world driving scenario data. However, the real-world driving environment is virtually limitless, and the collected data can hardly cover all possibilities. Especially for extreme scenarios that are crucial to safety (such as sudden accidents and dangerous driving behaviors), their natural occurrence frequency is extremely low, but their absence will seriously affect the robustness of the system.

[0004] With the rise of large language models, some work has attempted to apply them to data-driven text-to-scenario generation. However, textual forms are difficult to fully and accurately express all of a user's needs.

[0005] Therefore, this specification provides a demand-driven method for generating real-world autonomous driving scenarios. Summary of the Invention

[0006] This specification provides a demand-driven method for generating real-world autonomous driving scenarios, which partially addresses the aforementioned problems in existing technologies.

[0007] The following technical solution is adopted in this specification: This specification provides a demand-driven method for generating real-world autonomous driving scenarios, including: S1. Obtain user demand information and preprocessed map, wherein the user demand information includes scene text description, initial scene image and initial pose of the agent in the scene; S2. Input the user demand information into a pre-trained multimodal large language model to obtain integer-type discrete agent behavior codes; S3. Input the initial pose of the agent, the behavior encoding of the agent, and the preprocessed map into the pre-trained scene generation model to obtain a real autonomous driving scene that matches the user demand information, and perform kinematic optimization on the agent trajectory in the real autonomous driving scene based on optimal control theory.

[0008] Based on the aforementioned technical means, this solution uses the user's scene text description, initial image, and initial pose of the agent as input. The generation process is no longer completely random but closely follows the scene type, traffic participant behavior, and environmental layout specified by the user. The generated scene is highly matched with user needs, and controllability is significantly improved. The solution supports multiple input formats such as text, image, and pose, and can express complex scene semantics and initial conditions. Users can describe the desired traffic scene in natural language or constrain the spatial layout through initial images and poses, lowering the threshold for scene construction and improving design efficiency. After scene generation, further kinematic optimization of the agent's trajectory can eliminate unreasonable movements that may occur during the generation process, such as sudden speed changes, excessive acceleration, uneven trajectory, or violations of vehicle kinematic constraints. The final generated scene is not only visually realistic, but the vehicle behavior is also executable at the physical and kinematic levels, making it suitable for autonomous driving simulation testing. The solution links requirement understanding, behavior generation, scene generation, and trajectory optimization into a relatively complete automated process. Each module uses a pre-trained model, which has good generalization ability and substitutability, making it easy to introduce more complex demand types, more traffic participants, or more refined dynamic constraints in the future.

[0009] Furthermore, in S3, based on optimal control theory, kinematic optimization is performed on the agent's trajectory in the real-world autonomous driving scenario, specifically including: Based on optimal control theory, a nonlinear programming problem is constructed to optimize the kinematics of the agent trajectory in the real-world autonomous driving scenario. Solve the nonlinear programming problem to perform kinematic optimization on the agent trajectory in the real-world autonomous driving scenario.

[0010] Furthermore, the expression for the nonlinear programming problem is:

[0011]

[0012]

[0013]

[0014]

[0015] in, For the first in the real-world autonomous driving scenario The state vector of the agent at each time step; For optimization of the first The state vector of the agent at each time step. These are the global position coordinates of the agent. The heading angle of the agent. The longitudinal velocity of the agent. For the longitudinal acceleration of the intelligent agent, The yaw rate of the agent; For optimization of the first Control vector of the time-step agent To accelerate the longitudinal direction of the intelligent agent, The yaw acceleration of the intelligent agent; For the first Missing frame mask for time steps, The Pseudo-Huber loss function is used. To control the smoothing weight matrix, The total number of time steps for the agent's trajectory. For time step.

[0016] Furthermore, the agent behavior encoding in S2 is a 13-dimensional integer vector. The agent behavior encoding includes agent type encoding, speed interval index, size interval index, action trend encoding within five consecutive time segments, and action type encoding within the five consecutive time segments. The action trend encoding includes start encoding, hold encoding, and end encoding. The action type encoding includes stop encoding, U-turn encoding, left turn encoding, left lane change encoding, deceleration encoding, constant speed encoding, acceleration encoding, right lane change encoding, and right turn encoding.

[0017] Furthermore, the action trend encoding within each time segment is determined based on the action type encoding of the adjacent time segments.

[0018] Furthermore, the rule expression for the determination is:

[0019] in, This represents the action trend encoding, with start encoding, hold encoding, and end encoding being 2, 1, and 0 respectively; for a given time segment, This indicates the action type encoding of the time segment preceding this time segment. This indicates the action type encoding for the time segment following this time segment; This indicates the preset priority of each action type encoding.

[0020] Furthermore, the pre-trained scene generation model includes a multi-context gating block, a position encoding module, a multi-layer perceptron module, a multi-layer Transformer module, and a trajectory generation module; each Transformer module includes a multi-head self-attention sub-module and a multi-head cross-attention sub-module; the trajectory generation module includes an initial state attribute distribution prediction sub-module and a future multimodal trajectory prediction sub-module; In S3, the initial pose of the agent, the behavior encoding of the agent, and the preprocessed map are input into a pre-trained scene generation model to obtain a real-world autonomous driving scene that matches the user's needs information, specifically including: The initial pose of the agent, the behavior encoding of the agent, and the preprocessed map are input into the pre-trained scene generation model. The preprocessed map is encoded by the multi-context gating block to obtain the map feature vector. The initial pose of the agent and the behavior code of the agent are concatenated, and the concatenation result is sequentially passed through the position encoding module and the multilayer perceptron module to obtain the agent query feature vector; The agent query feature vector and the map feature vector are sequentially processed through multiple Transformer modules to obtain the agent comprehensive feature vector; The agent's comprehensive feature vector is input into the initial state attribute distribution prediction submodule and the future multimodal trajectory prediction submodule, respectively, to obtain the agent's initial state attribute distribution and agent trajectory.

[0021] Furthermore, the agent query feature vector and the map feature vector are sequentially processed through multiple Transformer modules to obtain the agent comprehensive feature vector, specifically including: For the nth layer Transformer module, the feature vector output by the (n-1)th layer Transformer module is input into the multi-head self-attention submodule of the nth layer Transformer module to obtain the feature vector output by the multi-head self-attention submodule of the nth layer Transformer module; The feature vector output by the multi-head self-attention submodule of the nth layer Transformer module and the map feature vector are input into the multi-head cross-attention submodule to obtain the feature vector input to the (n+1)th layer Transformer; where... n is a positive integer; when n equals 1, the agent queries the feature vector and inputs it into the multi-head self-attention submodule of the first-layer Transformer module; when n equals N, the multi-head cross-attention submodule of the Nth-layer Transformer module outputs the agent's comprehensive feature vector.

[0022] Furthermore, S1 obtains user demand information, specifically including: Obtain the user-inputted scenario text description; Based on the scene text description, retrieve a map that matches the scene text description from a preset map database; The matched map is rendered into a user-operable graphical interface; In response to the user's operation on the graphical interface, determine the initial pose of the intelligent agent set by the user; The scene text description, the image interface after the operation is completed, and the initial pose of the intelligent agent are used as user requirement information.

[0023] Furthermore, S1 obtains user demand information and preprocesses maps, specifically including: Obtain the agent density level and road topology type input by the user; Based on the agent density level and the road topology type, match scene templates in the preprocessed scene template dataset; Based on the matched scene template, determine the user's needs and preprocess the map.

[0024] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects: This solution uses the user's scene description, initial image, and agent's initial pose as inputs. The generation process is no longer completely random but closely follows the user-specified scene type, traffic participant behavior, and environmental layout. The generated scene highly matches user needs, significantly improving controllability. The solution supports multiple input formats, including text, images, and poses, enabling the expression of complex scene semantics and initial conditions. Users can describe desired traffic scenes using natural language or constrain spatial layout with initial images and poses, lowering the barrier to scene construction and improving design efficiency. After scene generation, further kinematic optimization of the agent's trajectory eliminates unreasonable movements that may occur during generation, such as sudden speed changes, excessive acceleration, uneven trajectories, or violations of vehicle kinematic constraints. The final generated scene is not only visually realistic but also exhibits physically and kinematically executable vehicle behavior, making it suitable for autonomous driving simulation testing. The solution integrates requirement understanding, behavior generation, scene generation, and trajectory optimization into a relatively complete automated process. Each module uses pre-trained models with good generalization ability and replaceability, facilitating the subsequent introduction of more complex requirement types, more traffic participants, or more refined dynamic constraints. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings: Figure 1 A flowchart illustrating a demand-driven method for generating real-world autonomous driving scenarios, as provided in an embodiment of this specification. Figure 2 This specification provides a corresponding Figure 1 A schematic diagram of the structure of an electronic device. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0027] In embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0028] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0029] Figure 1 A flowchart illustrating a demand-driven autonomous driving real-world scenario generation method provided in this specification includes the following steps: S1: Obtain user requirement information and preprocessed map, wherein the user requirement information includes scene text description, initial scene image and initial pose of agent in scene.

[0030] S2: Input the user demand information into a pre-trained multimodal large language model to obtain integer-type discrete agent behavior codes.

[0031] In this specification, the process of generating demand-guided real-world autonomous driving scenarios is described. In the embodiments described herein, this process can be executed by a server. However, this specification does not limit the type of device or platform used to perform this demand-guided real-world autonomous driving scenario generation process; for example, personal computers, mobile terminals, and other devices or platforms can also be used. For ease of description, the following explanation uses a server as the executing entity.

[0032] In one or more embodiments of this specification, the server may obtain user requirement information and a preprocessed map, the user requirement information including scene text description. Initial scene image and the initial pose of the agent in the scene Then, the user information is input into a pre-trained multimodal large language model to obtain integer-type discretized agent behavior codes.

[0033] In this specification, agent behavior encoding is a 13-dimensional integer vector. Agent behavior encoding includes agent type encoding, velocity interval index, size interval index, action trend encoding over five consecutive time segments, and action type encoding over five consecutive time segments. Action trend encoding includes start encoding, hold encoding, and end encoding. Action type encoding includes stop encoding, U-turn encoding, left turn encoding, left lane change encoding, deceleration encoding, constant speed encoding, acceleration encoding, right lane change encoding, and right turn encoding. Agent behavior encoding can be represented as... , This represents the number of agents (dynamic objects, such as vehicles, pedestrians, etc.), and 13 represents the behavioral encoding dimension of each agent. Represent a The matrix consists of 13 rows and 13 columns of integers. Each row corresponds to the behavior code of an agent. Each dimension of the behavior code is a discrete integer, not a continuous floating-point number.

[0034] Among them, the agent type encoding describes which type of dynamic target the agent belongs to, for example: 0: sedan; 1: SUV; 2: truck; 3: bus; 4: pedestrian; 5: bicycle; 6: motorcycle, etc.

[0035] The velocity interval index describes the approximate velocity range of the agent, discretizing continuous velocity into several intervals, for example: 0: Stationary (0-1 m / s, i.e., speed range [0,1]); 1: Low speed (1-5 m / s, i.e., speed range (1,5]); 2: Medium speed (5-15 m / s, i.e., speed range (5,15]); 3: High speed (15-30 m / s, i.e., speed range (15,30]); 4: Ultra-high speed (>30 m / s, i.e., speed range (30,∞]).

[0036] Size range indexes describe the physical size range of an agent, similar to speed ranges, discretizing the dimensions (such as length and width) of vehicles / pedestrians. For example: 0: Small (length < 4 m, i.e., length range is (0, 4]); 1: Medium (4 to 6 m, i.e., length range is (4, 6]); 2: Large (6 to 10 m, i.e., length range is (6, 10]); 3: Extra Large (> 10 m, i.e., length range is (10, ∞]).

[0037] The agent's behavior timeline in the scene is divided into five consecutive time segments. For example, the entire scene duration can be evenly divided into five segments. Each time segment corresponds to an action trend code, representing the stage state of that time segment in the entire action process. The action trend code includes a start code, a hold code, and an end code. In this specification, the start code is 2, the hold code is 1, and the end code is 0, representing the start, hold, and end of the action trend, respectively.

[0038] Each time segment, in addition to its action trend, also has an action type code, representing the specific driving action performed within that time segment. Action type codes include stop code, U-turn code, left turn code, left lane change code, deceleration code, constant speed code, acceleration code, right lane change code, and right turn code. The values ​​for each action type code can be defined as follows: 1: Stop 2: U-turn 3: Turn left 4: Change lanes to the left 5: Decelerate 6: Constant speed 7: Accelerate 8: Right lane change 9: Turn right It's worth noting that the action trend encoding within each time segment is determined based on the action type encoding of adjacent time segments. The rule expression for this determination is:

[0039] in, This represents the action trend encoding, with start encoding, hold encoding, and end encoding being 2, 1, and 0 respectively. For a given time segment, This indicates the action type encoding of the time segment preceding this time segment. This indicates the action type encoding for the time segment following this time segment. This indicates the preset priority of each action type code. For example, the priorities of stop code, U-turn code, left turn code, left lane change code, deceleration code, constant speed code, acceleration code, right lane change code, and right turn code are 5, 8, 7, 6, 3, 1, 2, 4, and 7, respectively. The higher the value, the higher the priority.

[0040] Pre-trained multimodal large language models, through context learning and thought chain prompts, interpret the input user demand information. Parsing into a structured description in YAML format (YAML stands for YAML Ain't Markup Language, a recursive abbreviation, a human-readable data serialization format), and then mapping it to... .

[0041] Specifically, contextual learning: Several examples (user requirement information - YAML structure pairs) can be provided to the multimodal large language model, allowing the multimodal large language model to learn the task format and mapping rules so that the multimodal large language model can imitate this transformation.

[0042] Mind chain prompts: Instead of directly outputting the final result, guide the multimodal large language model through step-by-step reasoning using prompts. For example, first have the multimodal large language model identify which agents are involved, then determine the behavior of each agent, and finally organize this into a YAML file. After obtaining the YAML description, map the discrete labels within it to integer codes using rules or code. Agent type → Integer category; Speed ​​range → Index value; Size range → Index value; The action trend for each segment → 0 / 1 / 2; Action type for each segment → 1~9.

[0043] This yields the final agent behavior code. .

[0044] Of course, pre-trained multimodal large language models can also be trained in the following ways: Historical user demand information is used as training samples, and the agent behavior encoding of this historical user demand information is used as training labels. Then, the training samples are input into the multimodal large language model to be trained, yielding the predicted behavior encoding output by the multimodal large language model. Based on the difference between the predicted behavior encoding and the training labels, a loss value is determined, and the multimodal large language model is trained with the minimum loss value as the optimization objective.

[0045] S3: Input the initial pose of the agent, the behavior encoding of the agent, and the preprocessed map into the pre-trained scene generation model to obtain a real autonomous driving scene that matches the user demand information, and perform kinematic optimization on the agent trajectory in the real autonomous driving scene based on optimal control theory.

[0046] In one or more embodiments of this specification, the server can input the agent's initial pose, agent behavior encoding, and preprocessed map into a pre-trained scene generation model to obtain a real-world autonomous driving scene that matches the user's needs.

[0047] Specifically, the pre-trained scene generation model includes a multi-context gating block, a position encoding module, a multi-layer perceptron module, a multi-layer Transformer module, and a trajectory generation module. Each Transformer module includes a multi-head self-attention sub-module and a multi-head cross-attention sub-module. The trajectory generation module includes an initial state attribute distribution prediction sub-module and a future multimodal trajectory prediction sub-module.

[0048] The server can determine the agent's initial pose. Agent behavior coding and preprocessed maps Input a pre-trained scene generation model and preprocess the map. Map feature vectors are obtained through multi-context gating block encoding. .

[0049]

[0050]

[0051] in, This indicates the number of map elements in the preprocessed map, such as lane lines, curbs, traffic signs, intersection nodes, lane center lines, etc. This represents the attribute dimensions of each map element, which may include location coordinates, length, curvature, type, connectivity, etc. This represents the feature dimension of each encoded map element. Preprocessed map. It is a map representation that has already been cleaned, vectorized, or rasterized. Multi-Context Gated Blocks (MCGs) are a type of neural network module specifically designed for map encoding.

[0052] Next, the agent's initial pose and the agent's behavior codes are concatenated, and the concatenation result ( The query feature vector of the intelligent agent is obtained by sequentially passing through the position encoding module and the multilayer perceptron module. .

[0053]

[0054] in, This indicates positional encoding. To define the dimension of the feature vector for the intelligent agent, It is a multilayer perceptron.

[0055] Then, the agent query feature vector and map feature vector are sequentially passed through multiple Transformer modules to obtain the agent's comprehensive feature vector.

[0056] Specifically, for the nth layer Transformer module, the feature vector output by the (n-1)th layer Transformer module is input into the multi-head self-attention submodule of the nth layer Transformer module to obtain the feature vector output by the multi-head self-attention submodule of the nth layer Transformer module. Then, the feature vector output by the multi-head self-attention submodule of the nth layer Transformer module and the map feature vector are input into the multi-head cross-attention submodule to obtain the feature vector input to the (n+1)th layer Transformer. n is a positive integer. When n equals 1, the agent's query feature vector is input into the multi-head self-attention submodule of the first-layer Transformer module. When n equals N, the multi-head cross-attention submodule of the Nth-layer Transformer module outputs the agent's comprehensive feature vector.

[0057] The operations for each Transformer module are as follows:

[0058] In the formula, MHSA represents multi-head self-attention, and MHCA represents multi-head cross-attention (with...). (as key value) . This is the comprehensive feature vector of the intelligent agent.

[0059] Finally, the agent's comprehensive feature vector is input into the initial state attribute distribution prediction submodule and the future multimodal trajectory prediction submodule, respectively, to obtain the agent's initial state attribute distribution and agent trajectory.

[0060] In the initial state attribute distribution prediction submodule, for each agent... Predict the distribution of initial state attributes of the agent :

[0061] In the formula, This represents the distribution of the initial state attributes of the predictive agent. For the first A smart agent passes through The comprehensive feature vector of the agent after the Transformer module. Let be the mean vector of each component in the Gaussian mixture model. Represents the covariance matrix. This represents the mixing coefficient, i.e., the weight of each component. The number of each component in the Gaussian mixture model. The initial state of an agent (such as initial position, velocity, etc.) may not be a deterministic value, but rather follows a multimodal distribution. Gaussian mixture models can be used to describe the uncertainty or multimodality of the initial state.

[0062] In the future multimodal trajectory prediction submodule, for each agent Predict its future multimodal trajectory:

[0063]

[0064] In the formula, This indicates the prediction of the agent's future multimodal trajectory. For indexing intelligent agents, For pattern index, , The number of future trajectory patterns. For the first The displacement sequence of the pattern represents the time step from the second time step to the first time step. The displacement increment at each time step (relative to the previous moment). This represents the scene duration (total time steps). For the first The probability of each pattern. Represents intelligent agents The initial position can be derived from the agent's initial pose. Or it could be a sampling of the initial state attribute distribution of the agent. Indicates the first The pattern in the first The displacement increment of the step. Indicates the first Agent in each mode At time step Location, based on Then the first one can be determined Agents in each mode Trajectory.

[0065] Therefore, the final output is a real-world autonomous driving scenario, which includes the possible trajectories and probabilities of all agents, satisfies the action types and action trends specified by the agent behavior encoding, and is aligned with the map structure.

[0066] Furthermore, the server can also perform kinematic optimization of the agent's trajectory in real-world autonomous driving scenarios based on optimal control theory.

[0067] Specifically, based on optimal control theory, a nonlinear programming problem is constructed to optimize the kinematics of an agent's trajectory in a real-world autonomous driving scenario. The nonlinear programming problem is then solved to optimize the kinematics of the agent's trajectory in the real-world autonomous driving scenario.

[0068] The expression for the nonlinear programming problem is:

[0069]

[0070]

[0071]

[0072]

[0073] in, For the first in real-world autonomous driving scenarios The state vector of the agent at each time step. For optimization of the first The state vector of the agent at each time step. These are the global position coordinates of the agent. The heading angle of the agent. The longitudinal velocity of the agent. For the longitudinal acceleration of the intelligent agent, denoted as the yaw rate of the agent. For optimization of the first Control vector of the time-step agent To accelerate the longitudinal direction of the intelligent agent, Let be the yaw acceleration of the intelligent agent. For the first Missing frame mask for time steps: When the original trajectory of a certain frame is missing in a real-world autonomous driving scenario, the optimized trajectory is not forced to match the observation of that frame. The Pseudo-Huber loss function is used. The preset control smoothing weight matrix, The total number of time steps for the agent's trajectory. For time step.

[0074] The Pseudo-Huber loss function is defined as follows:

[0075] in, , For the first in real-world autonomous driving scenarios The initial position of the time-step agent. For the first in real-world autonomous driving scenarios The initial course of the time-step agent. For Huber threshold, These are the weighting coefficients. Physical boundary constraints are then applied. , .

[0076] This nonlinear programming problem can be modeled using the CasADi automatic differentiation framework and solved numerically using the Ipopt (InteriorPoint OPTimizer) interior-point method solver.

[0077] Furthermore, in one or more embodiments of this specification, the server can obtain a scene text description input by the user. Based on the scene text description, a map matching the scene text description is retrieved from a preset map database. This matching map is then rendered into an interactive image, i.e., a user-operable graphical interface. In response to user actions on the graphical interface, such as placing an agent and setting its initial pose by dragging and dropping with a mouse, the initial pose of the agent set by the user is determined. Finally, the scene text description, the completed graphical interface, and the initial pose of the agent are used as user requirement information to determine the agent's behavior encoding using a multimodal large language model.

[0078] In this specification, the server can also obtain the agent density level and road topology type input by the user. Based on the agent density level and road topology type, a scene template is matched in the preprocessed scene template dataset. Finally, based on the matched scene template, user requirement information and a preprocessed map are determined.

[0079] Among them, agent density level (Low / Medium / High) 0: Low density (few dynamic objects in the scene, such as sparse traffic); 1: Medium density (normal traffic flow); 2: High density (congested or busy traffic).

[0080] Road topology type (Road section / Three-way intersection / Four-way intersection / Other) 0: Road segment (straight or curved, no intersections); 1: Three-way intersection (T-shaped intersection or Y-shaped intersection); 2: Four-way intersection (crossroads); 3: Other (such as roundabouts, ramps, complex topology, etc.).

[0081] It's worth noting that each scene template in the preprocessed scene template dataset includes a preprocessed map and the agent's initial pose. Based on the user-input agent density level and road topology type, a matching subset of templates is selected from the preprocessed scene template dataset, and then one or more scene templates are randomly selected. For each selected scene template, the preprocessed map and the agent's initial pose are directly read.

[0082] Furthermore, the scene templates in the preprocessed scene template dataset can include not only the preprocessed map and the agent's initial pose, but also the corresponding agent behavior encoding.

[0083] In one or more embodiments of this specification, the training process of the scene generation model can acquire real-world driving scene data. The system performs kinematic optimization based on optimal control theory on the agent's trajectory in real-world driving scenario data. The server can then analyze the kinematically optimized driving scenario data... In the process, predefined extraction rules are used. Determine the training samples:

[0084] Real-world scenario data after kinematic optimization In the process, through predefined extraction rules Extract the initial pose of the agent for each scene. and behavioral coding Data collected from real driving scenarios In the middle, through preprocessing functions Extracting preprocessed maps Using training samples Input to the model for scene generation, using real-world scenes. The scene generation model is trained using the supervisory signal.

[0085] The loss function used for training is:

[0086] The initial state loss is the negative log-likelihood:

[0087] The future trajectory loss is the minimum mean square error:

[0088] in, These are the weighting coefficients. This represents the number of intelligent agents in the scene. The initial state attributes represent the number of components in the Gaussian mixture model. For the first The first intelligent agent The weights of each Gaussian component, The mean is The covariance matrix is Gaussian probability density function, For intelligent agents The initial property truth value. The number of future trajectory patterns predicted for each agent. For intelligent agents No. The pattern at time step The position is obtained by adding the displacement to the initial position:

[0089] For intelligent agents Time step in kinematically optimized driving scenario data The true location This represents the scene duration (total time steps). For all Choose one of the following patterns.

[0090] During training, by minimizing Perform backpropagation to update the parameters of the following modules: multi-context gating block, position encoding module, multi-layer perceptron module, multi-layer Transformer module (including multi-head self-attention sub-module and multi-head cross-attention sub-module), and trajectory generation module (including initial state attribute distribution prediction sub-module and future multimodal trajectory prediction sub-module).

[0091] Referring to the above content on the demand-driven autonomous driving real-world scenario generation method, the following is an embodiment of the demand-driven autonomous driving real-world scenario generation method, which includes two stages: offline processing and online inference.

[0092] In the offline processing phase, kinematic optimization is first performed: physical consistency restoration is performed on the original acquired agent trajectory data. State vectors are defined. Control vector The kinematic differential equation is Discretization is performed using the first-order Euler method: The objective function to be optimized is:

[0093] in, This is the mask for missing frames (0 indicates that the frame is missing, and 1 indicates that it exists). Weight Preset based on agent type. Physical boundary constraints include: speed limit (e.g., for passenger vehicles). ), acceleration upper limit ( ), accelerometer limit ( (etc.). A sparse nonlinear programming problem is constructed using the CasaADi framework and solved using the Ipopt solver. The optimality tolerance is relaxed to [value missing] during the solution process. To improve efficiency.

[0094] During the model training phase, the initial pose of the agent is automatically extracted from the kinematically optimized scene data using predefined rules. and agent behavior encoding The training loss function is: Using the Adam optimizer, the learning rate is... .

[0095] During the online inference phase, two working modes are supported.

[0096] a. Key Scenario-Oriented Generation Mode: The user inputs a scenario text description: "A test vehicle approaches an intersection at a target speed V1. A target vehicle is placed in the oncoming lane, turning left at a speed V2. The time required for the test vehicle to reach the collision point is the same as the time required for the target vehicle to reach the collision point." Based on the map description near the user's vehicle in the text (e.g., "three lanes," "straight lane," etc.), the system retrieves a matching map from a pre-set map database and renders it as an interactive image. The user clicks and drags on the image with the mouse to place the user's vehicle and the target vehicle, setting their initial poses. Subsequently, the scenario text description, the initial scenario snapshot (the interactive image after setting the initial poses), and the user-defined initial poses are fed into a multimodal large language model, transforming the user's requirements into a 13-dimensional integer vector. For example, in an intersection scenario, the behavior encoding of the autonomous vehicle is: Type = 4 (autonomous vehicle), speed range = 4 (corresponding to 7.5-10.0 m / s), size range = 4 (corresponding to 4-5 meters vehicle length), first 5 action trends = [2,2,2,2,2] (hold), last 5 action types = [6,6,6,6,6] (go straight); the behavior encoding of the target vehicle is: Type = 3 (human-driven vehicle), speed range = 4 (corresponding to 7.5-10.0 m / s), size range = 4 (corresponding to 4-5 meters vehicle length), first 5 action trends = [2,2,2,2,2] (hold), last 5 action types = [3,3,3,3,3] (turn left). After obtaining the behavior encoding, it is compared with the initial pose and preprocessed location. Figure 1 The same input scene is used to generate a model, which outputs a complete driving scene (such as a trajectory for the next 50 steps, with a step length of 0.1 seconds). Finally, motion optimization is performed again to smooth the generated trajectory and output the final high-realism scene.

[0097] b. Batch generation mode for regular scenarios: The user specifies the agent density as medium (approximately 20 agents / 10,000 square meters) and the road topology type as a three-way intersection. Ten matching scenario templates are randomly selected from the preprocessed scenario dataset, and their initial poses and behavior codes are directly extracted to generate ten different but statistically distributed regular driving scenarios in batches for generalization testing of the autonomous driving model.

[0098] Figure 1 The method for generating realistic autonomous driving scenarios, as shown, is based on optimal control theory and uses the Pseudo-Huber robust loss function to perform bidirectional physical correction on the original acquired data and the generated results. This effectively eliminates kinematic distortions caused by sensor noise, missing frames, etc., and significantly improves the realism of the generated scenarios.

[0099] A 13-dimensional fine-grained intelligent agent behavior encoding was designed, and an action trend determination formula based on action type priority was introduced, which enables users to precisely control the complex movement patterns of the intelligent agent and solves the problems of ambiguous instructions and insufficient granularity of behavior description in the existing technology.

[0100] It supports interactive and targeted generation of key scenarios as well as batch generation of regular scenarios, which can flexibly meet the diverse scenario requirements of autonomous driving system testing and effectively alleviate the problem of long-tail data scarcity.

[0101] By using the user's scene description, initial image, and agent's initial pose as input, the generation process is no longer completely random, but closely revolves around the user-specified scene type, traffic participant behavior, and environmental layout. The generated scene highly matches user needs, significantly improving controllability. The solution supports multiple input formats, including text, images, and poses, enabling the expression of complex scene semantics and initial conditions. Users can describe desired traffic scenes using natural language or constrain spatial layout through initial images and poses, lowering the threshold for scene construction and improving design efficiency. After scene generation, further kinematic optimization of the agent's trajectory eliminates unreasonable movements that may occur during generation, such as sudden speed changes, excessive acceleration, uneven trajectories, or violations of vehicle kinematic constraints. The final generated scene is not only visually realistic, but the vehicle behavior is also physically and kinematically executable, making it suitable for autonomous driving simulation testing. The solution links requirement understanding, behavior generation, scene generation, and trajectory optimization into a relatively complete automated process. Each module uses pre-trained models, possessing good generalization ability and replaceability, facilitating the subsequent introduction of more complex requirement types, more traffic participants, or more refined dynamic constraints.

[0102] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 This paper presents a demand-driven method for generating real-world autonomous driving scenarios.

[0103] This instruction manual also provides Figure 2 The diagram shows a schematic structural representation of the electronic device. Figure 2 As shown, at the hardware level, this electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above. Figure 1 This paper presents a demand-driven method for generating real-world autonomous driving scenarios.

[0104] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0105] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0106] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0107] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0108] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0109] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0114] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0115] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic or disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0116] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0117] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0119] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0120] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A demand-driven method for generating real-world autonomous driving scenarios, characterized in that, include: S1. Obtain user demand information and preprocessed map, wherein the user demand information includes scene text description, initial scene image and initial pose of the agent in the scene; S2. Input the user demand information into a pre-trained multimodal large language model to obtain integer-type discrete agent behavior codes; S3. Input the initial pose of the agent, the behavior encoding of the agent, and the preprocessed map into the pre-trained scene generation model to obtain a real autonomous driving scene that matches the user demand information, and perform kinematic optimization on the agent trajectory in the real autonomous driving scene based on optimal control theory.

2. The demand-driven autonomous driving real-world scenario generation method as described in claim 1, characterized in that, Based on optimal control theory, S3 performs kinematic optimization on the agent trajectory in the real-world autonomous driving scenario, specifically including: Based on optimal control theory, a nonlinear programming problem is constructed to optimize the kinematics of the agent trajectory in the real-world autonomous driving scenario. Solve the nonlinear programming problem to perform kinematic optimization on the agent trajectory in the real-world autonomous driving scenario.

3. The demand-driven autonomous driving real-world scenario generation method as described in claim 2, characterized in that, The expression for the nonlinear programming problem is: in, For the first in the real-world autonomous driving scenario The state vector of the agent at each time step; For optimization of the first The state vector of the agent at each time step. These are the global position coordinates of the agent. The heading angle of the agent. The longitudinal velocity of the agent. For the longitudinal acceleration of the intelligent agent, The yaw rate of the agent; For optimization of the first Control vector of the time-step agent To accelerate the longitudinal direction of the intelligent agent, The yaw acceleration of the intelligent agent; For the first Missing frame mask for time steps, The Pseudo-Huber loss function is used. To control the smoothing weight matrix, The total number of time steps for the agent's trajectory. For time step.

4. The demand-driven autonomous driving real-world scenario generation method as described in claim 1, characterized in that, The agent behavior encoding described in S2 is a 13-dimensional integer vector. The agent behavior encoding includes agent type encoding, speed interval index, size interval index, action trend encoding within five consecutive time segments, and action type encoding within the five consecutive time segments. The action trend encoding includes start encoding, hold encoding, and end encoding. The action type encoding includes stop encoding, U-turn encoding, left turn encoding, left lane change encoding, deceleration encoding, constant speed encoding, acceleration encoding, right lane change encoding, and right turn encoding.

5. The demand-driven autonomous driving real-world scenario generation method as described in claim 4, characterized in that, The action trend encoding within each time segment is determined based on the action type encoding of the adjacent time segments.

6. The demand-driven autonomous driving real-world scenario generation method as described in claim 5, characterized in that, The rule expression for the determination is: in, This represents the action trend encoding, with start encoding, hold encoding, and end encoding being 2, 1, and 0 respectively; for a given time segment, This indicates the action type encoding of the time segment preceding this time segment. This indicates the action type encoding for the time segment following this time segment; This indicates the preset priority of each action type encoding.

7. The demand-driven autonomous driving real-world scenario generation method as described in claim 4, characterized in that, The pre-trained scene generation model includes a multi-context gating block, a position encoding module, a multi-layer perceptron module, a multi-layer Transformer module, and a trajectory generation module; each Transformer module includes a multi-head self-attention sub-module and a multi-head cross-attention sub-module; the trajectory generation module includes an initial state attribute distribution prediction sub-module and a future multimodal trajectory prediction sub-module; In S3, the initial pose of the agent, the behavior encoding of the agent, and the preprocessed map are input into a pre-trained scene generation model to obtain a real-world autonomous driving scene that matches the user's needs information, specifically including: The initial pose of the agent, the behavior encoding of the agent, and the preprocessed map are input into the pre-trained scene generation model. The preprocessed map is encoded by the multi-context gating block to obtain the map feature vector. The initial pose of the agent and the behavior code of the agent are concatenated, and the concatenation result is sequentially passed through the position encoding module and the multilayer perceptron module to obtain the agent query feature vector; The agent query feature vector and the map feature vector are sequentially processed through multiple Transformer modules to obtain the agent comprehensive feature vector; The agent's comprehensive feature vector is input into the initial state attribute distribution prediction submodule and the future multimodal trajectory prediction submodule, respectively, to obtain the agent's initial state attribute distribution and agent trajectory.

8. The demand-driven autonomous driving real-world scenario generation method as described in claim 7, characterized in that, The agent query feature vector and the map feature vector are sequentially processed through multiple Transformer modules to obtain the agent's comprehensive feature vector, specifically including: For the nth layer Transformer module, the feature vector output by the (n-1)th layer Transformer module is input into the multi-head self-attention submodule of the nth layer Transformer module to obtain the feature vector output by the multi-head self-attention submodule of the nth layer Transformer module; The feature vector output by the multi-head self-attention submodule of the nth layer Transformer module and the map feature vector are input into the multi-head cross-attention submodule to obtain the feature vector input to the (n+1)th layer Transformer; where... n is a positive integer; when n equals 1, the agent queries the feature vector and inputs it into the multi-head self-attention submodule of the first-layer Transformer module; when n equals N, the multi-head cross-attention submodule of the Nth-layer Transformer module outputs the agent's comprehensive feature vector.

9. The demand-driven method for generating real-world autonomous driving scenarios as described in claim 1, characterized in that, S1 obtains user demand information, specifically including: Obtain the user-inputted scenario text description; Based on the scene text description, retrieve a map that matches the scene text description from a preset map database; The matched map is rendered into a user-operable graphical interface; In response to the user's operation on the graphical interface, determine the initial pose of the intelligent agent set by the user; The scene text description, the image interface after the operation is completed, and the initial pose of the intelligent agent are used as user requirement information.

10. The demand-driven autonomous driving real-world scenario generation method as described in claim 1, characterized in that, S1 obtains user demand information and preprocessed maps, specifically including: Obtain the agent density level and road topology type input by the user; Based on the agent density level and the road topology type, match scene templates in the preprocessed scene template dataset; Based on the matched scene template, determine the user's needs and preprocess the map.