Interactive modal scene generation method based on space-time optimization controllable diffusion model

Abstract

The invention relates to an interaction mode scene generation method based on a space-time optimization controllable diffusion model, and the method comprises the steps: carrying out the decoupling of a multi-agent interaction event based on conflict points, enumerating the possible passing order of the conflict points, and obtaining a complete semantic layer interaction mode set; constructing a space-time network model, optimizing the passing time of a vehicle at a conflict point, and generating an anchor point sequence which accords with a specific interaction mode and also meets dynamic constraints; taking the optimized anchor point as a guide signal to be integrated into a sampling process of a diffusion model, and generating a track of a specified interaction mode in a controllable mode; and pruning the generated scene based on a multi-dimensional rationality pruning mechanism, and eliminating unreasonable scenes. Compared with the prior art, the method has the advantages that completeness and coverage of interaction scenes are remarkably improved, physical feasibility and space-time consistency of generated results are ensured, and result credibility is high.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving scene generation, and in particular to a method for generating interactive modal scenes based on a spatiotemporally optimized and controllable diffusion model. Background Technology

[0002] Faced with highly heterogeneous interactive environments, autonomous driving systems learn from real-world driving data to understand and respond to various interactive scenarios, thereby achieving safe and efficient passage. However, natural driving data, as a single sample of potential outcomes of driving interaction events, only records one modality of event evolution and cannot reflect all physically feasible directions of interaction evolution, i.e., it lacks modal completeness. Autonomous driving algorithms trained and validated on such data can effectively handle high-frequency modalities in interactive scenarios, but it is difficult to guarantee safety in low-frequency modalities. In relatively complex interactive scenarios, potential low-frequency modalities may not even have been collected by the natural driving dataset, leading to the omission of potentially high-risk interactive scenarios. To ensure the safety of autonomous driving algorithms in high-risk low-frequency modalities, it is necessary to perform potential modal inference and completion based on the direct collection results of natural driving data, thereby generating all physically feasible but unobserved interactive scenario evolution scenarios for training and validation of autonomous driving models.

[0003] To alleviate this problem, existing research mainly follows two paths: one is the knowledge-driven approach, which exhaustively enumerates scenarios through artificial rules or physical models. While it can achieve formal completeness, it lacks realism and scalability. The other is the data-driven approach, which learns real distributions through deep generative models. While it can generate realistic trajectory patterns, it struggles to capture the diverse interactions under long-tail distributions. Even with multimodal prediction, the models tend to distinguish coarse-grained semantic categories (such as going straight versus turning left) and cannot characterize more refined differences in interaction strategies (such as yielding versus cutting in). Therefore, existing technologies generally face the fundamental contradiction of being unable to balance realism and completeness, and cannot systematically generate a full spectrum of interaction modalities that are both physically feasible and semantically reasonable.

[0004] Existing autonomous driving interaction modeling and scene generation technologies face an irreconcilable contradiction between diversity and realism. Furthermore, the complexity of multi-agent interactions necessitates that scene generation not only satisfy individual dynamic constraints but also maintain global causal consistency and social rationality.

[0005] Chinese invention patent CN119380308A discloses a method for scene extraction, generation, and evaluation in vehicle safety testing. This method employs a semantic segmentation network to retain scene element features from original natural driving data to the greatest extent possible. It uses a high-level decision-making deep reinforcement learning algorithm to extract association generation rules for typical dangerous scenarios using rough sets, and can automatically generate typical dangerous scenarios in conjunction with simulation software. The method evaluates the overall intelligence of autonomous vehicles from two dimensions: natural driving evaluation and test scenario complexity evaluation. This invention automatically generates typical dangerous scenarios using simulation software, and improves the multidimensionality of autonomous vehicle evaluation by combining the advantages of different key indicators; it significantly improves the efficiency and intelligence level of scene generation. However, it still has problems such as pattern loss due to limited sample distribution, sudden speed changes and path intersections in the generated trajectory, and unreasonable or physically inconsistent output results.

[0006] In summary, there is currently a lack of an interactive modal scene generation method based on a spatiotemporally optimized and controllable diffusion model to solve or partially solve the aforementioned problems. Summary of the Invention

[0007] The purpose of this invention is to overcome the defects of the existing technology and provide an interactive modal scene generation method based on a spatiotemporally optimized controllable diffusion model, so as to solve or partially solve the problems of mode loss caused by limited sample distribution, velocity abrupt changes and path intersections in the generated trajectory, and unreasonable output results or violations of physical laws.

[0008] The objective of this invention can be achieved through the following technical solutions: This invention provides a method for generating interactive modal scenes based on a spatiotemporally optimized and controllable diffusion model, comprising: S1. Acquire natural driving interaction data, identify and extract key conflict points based on the natural driving interaction data, and construct a structured conflict information set using the extracted key conflict points; S2. Construct a complete interactive modality semantic generation model. Based on the structured conflict information set, use the complete interactive modality semantic generation model to exhaustively enumerate semantic interactive modality combinations and generate a scene semantic layer interactive modality set. S3. Construct a spatiotemporal network optimization model, and optimize the semantic interaction modality set under the constraints to transform it into a complete set of interaction modality anchor point sequences; S4. Construct an anchor-controlled diffusion sampling model, using the complete set of interactive modal anchor sequence as a condition guide to control the diffusion sampling process and generate vehicle trajectories in interactive modal scenes; S5. Construct a scene rationality pruning index to evaluate the rationality of the generated interactive modal scene vehicle trajectories, eliminate unreasonable interactive modal scene vehicle trajectories, and realize the generation of interactive modal scenes.

[0009] As a preferred technical solution, S1 specifically includes: S1.1. Select scene segments that meet the interaction intensity from the acquired natural driving interaction data and extract the interactive vehicle trajectory and the corresponding scene map information; S1.2. Based on the extracted vehicle trajectory, identify key locations in time and space where there is potential mutual interference as key conflict points, and construct a structured conflict information set based on the key conflict points and associated vehicle dynamic features.

[0010] As a preferred technical solution, the scene semantic layer interaction modality set includes a local semantic modality set and a global interaction modality set, and its acquisition process includes the following steps; Based on the structured conflict information set, the set of vehicles participating in the interaction is identified at the key conflict point. The vehicles are enumerated and combined in sequence, and the local interaction results at the key conflict point are captured to generate a local semantic modality set. By globally combining all the local semantic modalities, the feasibility of each key conflict point is combined and mapped at the scene level through sequences, thereby constructing a global interaction modal set.

[0011] As a preferred technical solution, the spatiotemporal network optimization model constructs a joint graph of key conflict points and time. Using semantic interaction modalities as input, graph optimization is used to obtain the set of anchor points corresponding to complete interaction modalities that satisfy dynamic constraints. Represents a set of spatiotemporal nodes. This is the set of directed edges that characterize node transitions.

[0012] As a preferred technical solution, the objective function of the spatiotemporal network optimization model is to minimize the time deviation of each vehicle at the key conflict point, as expressed in the formula: In the formula, This represents the set of all vehicles participating in the interaction within the scene. Represents the set of key conflict points. Indicates vehicle Passing through key conflict points The optimized time Indicates vehicle Actual transit time observed in natural driving data This represents the global objective function value, used to measure the consistency between the optimization results and the real data over time.

[0013] As a preferred technical solution, the constraints include: Spatiotemporal consistency constraint: Vehicles maintain path continuity in the spatiotemporal network; Traffic order constraint: The order in which vehicles pass through key conflict points conforms to the expected interaction semantics requirements of the interaction logic; Dynamic constraints: The vehicle's speed, acceleration, and time intervals satisfy physical feasibility.

[0014] As a preferred technical solution, the anchor point controllable diffusion sampling model is expressed as follows: In the formula, For the diffusion model, unguided adjustment at time 10:00 For the sample The basic forecast mean, It is the variance matrix. This is a guiding objective function based on anchor points, used to constrain the generated result to approximate the spatiotemporal position of the target anchor point. This is a sample representing the current diffusion state during the generation process. To calculate the gradient for this sample, Given a sample of target interaction modalities, This refers to the number of vehicles participating in the scene. The generated vehicle trajectory sequence; For the target anchor point sequence; The smoothed L1 loss function is used to measure the spatiotemporal deviation between the generated trajectory and the target anchor point. The diffusion model was adjusted at time [time]. For the sample The predicted mean.

[0015] As a preferred technical solution, the sampling process of the anchor point controllable diffusion sampling model adopts a denoising guidance mechanism based on specific interaction modal anchor point conditions. The set of different interaction modal anchor points output by the spatiotemporal optimization model is used as the condition input of the diffusion sampling process. In the reverse denoising process, the sampling direction is dynamically adjusted according to the anchor point constraints to generate vehicle trajectories of the interaction modality scene that meet the requirements of the target interaction modality at the semantic level.

[0016] As a preferred technical solution, the scenario rationality pruning indicators include: Social consistency index: detects whether there is envelope overlap among various entities at the same time; Behavioral compliance indicators: detect whether there are any out-of-bounds or reverse-flow behaviors; Kinematic feasibility indicators: Detect whether the vehicle's acceleration and steering curvature exceed the set thresholds; If a scenario violates any metric, it is deemed unreasonable and removed.

[0017] As a preferred technical solution, the method further includes integrating the complete interactive modal semantic generation model, the spatiotemporal network optimization model, and the anchor point controllable diffusion sampling model to form an interactive scene generation framework.

[0018] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) Significantly improves the completeness and coverage of interactive scenarios: This invention constructs a complete interactive modal semantic generation model semantic exhaustive mechanism to exhaustively output semantic interactive modal combinations, generate a set of all dynamically feasible interactive modalities in the scenario, overcomes the mode missing problem caused by the limitation of sample distribution in traditional data-driven models, realizes full modal coverage of interactive results, and provides more comprehensive testing and training scenarios for autonomous driving systems.

[0019] (2) Ensure the physical feasibility and spatiotemporal consistency of the generated results: This invention introduces a spatiotemporal network optimization model to optimize the dynamic constraint solution of the generated semantic interaction modal set, transforms the abstract semantics into a physically realizable anchor sequence, generates a complete set of interaction modal anchor sequences, solves problems such as speed mutation and path intersection in the generated trajectory, and ensures the continuity and realizability of the scene evolution process.

[0020] (3) Improve the credibility of the results: This invention establishes a multi-dimensional rationality pruning mechanism that includes three types of indicators: social consistency, behavioral compliance and kinematic feasibility. It automatically evaluates and filters the vehicle trajectories of the generated interactive modal scenarios, solves the problem that the output results are unreasonable or violate physical laws, and ensures that the final output interactive scenarios are real and credible in terms of social logic and physical constraints, thereby improving the credibility of the output results.

[0021] (4) It has good scalability and engineering adaptability: The present invention integrates the interactive modal semantic generation model, the spatiotemporal network optimization model and the anchor point controllable diffusion sampling model into a modular system to form an interactive scene generation framework. This solves the problem of existing systems being closed, rigid and difficult to extend. It achieves the technical effect of being able to flexibly adapt to different natural driving datasets and simulation platforms and support large-scale, multi-scenario systematic expansion, providing a complete and reliable test scenario set for the training and safety verification of autonomous driving algorithms. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall framework of the interactive modal scene generation method based on a spatiotemporally constrained controllable diffusion model provided by the present invention. Figure 2 This is a schematic diagram of the complete interactive modal semantic evolution process in an embodiment of the present invention; Figure 3 This is a schematic diagram of the statistical analysis results of the generated interactive scene in an embodiment of the present invention; Figure 4 This is a schematic diagram of the dual-vehicle interaction mode generation result in an embodiment of the present invention; Figure 5 This is a schematic diagram of the multi-vehicle interaction modal generation result in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] To address the problems existing in the prior art, this embodiment provides a method for generating interactive modal scenarios based on a spatiotemporally optimized controllable diffusion model, which is used to complete the unobserved potential interactive modalities in natural driving data.

[0025] The core idea of ​​this method is to integrate the completeness advantage of knowledge-driven methods in modal exhaustion with the ability of data-driven methods in generating real-world scenarios, thereby achieving the generation of all physically feasible interaction modalities in an interactive scenario. Specifically, firstly, based on a knowledge-driven semantic modeling method, conflict points in a given interaction scenario are systematically decomposed to exhaustively obtain a complete set of semantic interaction modalities. Secondly, a spatiotemporal network optimization model is constructed, transforming the abstract set of semantic interaction modalities into a concrete, physically feasible set of anchor point sequences under dynamic and temporal constraints. Finally, based on a controllable diffusion model, the scenario distribution is learned from real driving data, and sampling is performed under the guidance of anchor point constraints to generate modally complete interaction scenarios that conform to physical laws and social rationality. Through the above process, this method achieves unified modeling from semantic-level exhaustion to physical-level generation, taking into account the completeness, realism, and controllability of the scenario.

[0026] Specifically, the following steps are included: S1: Interaction Scene Analysis and Conflict Information Extraction. Extract effective fragments of multi-agent interaction from natural driving data, perform scene analysis, identify and extract conflict points, participating subjects and their spatiotemporal relationships, and form a structured conflict information set describing the core structure of the interaction, providing an input basis for subsequent pattern reasoning.

[0027] Furthermore, step S1 specifically includes the following sub-steps: S11: Selection of Interaction Events. Based on the intensity index of interaction events in the driving interaction dataset, scene segments with high interaction intensity are selected to ensure that the selected events can fully represent typical interaction behaviors and provide high-quality samples for subsequent modeling.

[0028] S12: Interaction Events and Key Information Extraction. Specifically, this includes: Interaction event extraction. Based on the start and end times of interaction events and the participating vehicle numbers recorded in the driving interaction dataset, corresponding valid time segments are extracted from the natural driving data, and all vehicle and status data related to the interaction behavior are retained.

[0029] Key information extraction. Extract the trajectory sequences of each participating vehicle and their corresponding scene map information, and represent them uniformly as a vectorized polyline composed of a series of coordinate points.

[0030] S13: Conflict Point Identification. Based on vectorized vehicle trajectories, the spatial intersections between the trajectories of participating vehicles are calculated, identifying key locations where potential mutual interference exists in space and time, i.e., conflict points. Each conflict point is associated with the participating vehicle's number, location coordinates, passage time, and relative priority order, used to describe the spatial-temporal structure of the interaction.

[0031] S14: Construction of Structured Conflict Information. The identified conflict points and their associated vehicle dynamic characteristics (such as passage time, arrival order, and distance from start time) are structurally integrated and organized and stored in the form of a dictionary to form a set of structured conflict information for subsequent modeling.

[0032] Through the above processing, while maintaining data simplicity and indexability, a unified input interface can be provided for subsequent semantic modality exhaustive search and spatiotemporal optimization models.

[0033] S2: Construct a complete interactive modality semantic generation model. Based on a structured conflict information set, infer and exhaustively enumerate all dynamically feasible semantic interaction modal combinations in the scene, thereby obtaining a complete semantic layer interaction modality set and realizing the expansion from observed data to the potential interaction semantic space. The complete interactive modality semantic generation model adopts a semantic exhaustive mechanism based on key conflict point decomposition. Its function is to perform structured parsing of multi-agent interaction events in the scene, and to construct a globally comprehensive semantic layer complete interactive modality set by systematically enumerating all potential interaction modalities at the semantic level. The structured parsing is based on the decomposition of multi-agent interaction relationships in the scene based on conflict points. Each conflict point and its corresponding two passing objects are abstracted into a conflict unit in the form of "conflict point – passing vehicle 1 – passing vehicle 2". The dynamic information of vehicles and conflict points is extracted to form a structured conflict information representation that can be used for subsequent semantic exhaustiveness and spatiotemporal optimization.

[0034] Furthermore, step S2 specifically includes the following sub-steps: S21: Local Interaction Decomposition. Based on the structured conflict information set obtained in step S1, the complex multi-vehicle interaction problem is locally divided according to conflict points. At each conflict point, the set of vehicles involved in the interaction is identified, and it is assumed that they need to complete the negotiation without collision. For local conflicts, all possible vehicle passage combinations (i.e., different passage priorities) are enumerated to capture all possible local interaction results at that conflict point, forming a local semantic modality set.

[0035] S22: Global Modal Composition. After enumerating the local semantic modalities for each conflict point, all local modalities are globally combined. By mapping the feasibility of each conflict point through sequences at the scene level, the system constructs a set of all possible global interaction modalities for the entire interaction scene.

[0036] The combination process employs full permutations to ensure that no dynamically possible interaction sequences are overlooked in scenarios with multiple conflict points and multiple vehicles. The final result is a complete set of semantic layer interaction modalities, providing semantic prior input for subsequent spatiotemporal network optimization and anchor point solution, thus realizing a systematic expansion from observed data to the semantic space of potential interaction modalities.

[0037] S3: Construct a spatiotemporal network optimization model. Based on the semantic interaction modalities, establish a graph structure optimization model that considers time and space constraints. Through dynamic constraint solving, the abstract semantic interaction modalities are transformed into a concrete and physically feasible set of anchor point sequences, ensuring that each mode is kinematically feasible.

[0038] Furthermore, step S3 specifically includes the following sub-steps: S31: Construction of the Spatiotemporal Network. Based on the semantic interaction modality, a spatiotemporally discretized network model is established for each scenario to describe the dynamic relationships between multiple vehicles in the temporal and spatial dimensions. The spatiotemporal network optimization model is achieved by constructing a conflict point-time joint graph. Using semantic interaction modalities as input, graph optimization is used to obtain the set of anchor points corresponding to complete interaction modalities that satisfy dynamic constraints. Represents a set of spatiotemporal nodes. This is the set of directed edges representing node transitions. The network consists of a set of nodes and a set of directed edges. The node set includes: Starting point node set This indicates the initial state of the vehicle's trajectory; Conflict point node set This indicates the vehicle's passage status within the conflict zone; Endpoint node set This indicates the end of the scene where the vehicle leaves.

[0039] Directed edge set Each edge represents the path a vehicle takes between adjacent nodes. Records the vehicle from the node At the time step Departure, in time steps Arrival Node The process.

[0040] By simultaneously discretizing the time and space dimensions, a basic network structure was constructed that can be used to constrain vehicle traffic order and spatiotemporal consistency.

[0041] S32: Definition of the optimization objective function. To maintain scene realism while satisfying semantic constraints, an anchor point optimization objective function is defined to minimize the time deviation of each vehicle at the conflict point. Its objective function can be expressed as: in, This represents the set of all vehicles participating in the interaction within the scene; Represents the set of conflict points; Indicates vehicle Passing through the point of conflict The optimized time (i.e., the time obtained by solving the solution); Indicates vehicle Actual transit time observed in natural driving data; This represents the global objective function value, used to measure the consistency between the optimization result and the real data in the time dimension. The physical meaning of this objective function is: to minimize the average deviation between the optimized time and the observation time across all vehicles and all conflict points, thereby ensuring that the generated anchor point sequence both follows the semantic interaction modality and retains the temporal characteristics of the real scene to the greatest extent possible. The optimization process uses the spatiotemporal traffic status of vehicles at each conflict point as the decision variable. ,in ( , , , indicating the interactive subject At the time step From node Departure, in time steps Reaching the node .

[0042] S33: Definition of constraints. In practical optimization, The path chosen by the vehicle in the spatiotemporal network and its speed are determined, and the following constraints must be met simultaneously: Spatiotemporal consistency constraint: When a vehicle enters or leaves a conflict point, it must maintain path continuity and there must be no temporal or spatial breaks. Traffic order constraint: Within the same conflict point, the order in which vehicles pass should conform to predefined interaction semantics (such as yielding, cutting in, etc.). Dynamic constraints: The vehicle's acceleration, speed, etc., should be kept within the physically permissible range to prevent unrealistic motion (such as negative speed or excessive acceleration).

[0043] Through the above optimization process, the set of physically feasible and semantically consistent passage times for each vehicle at each conflict point can be obtained, thereby generating the corresponding anchor point sequence, providing accurate spatiotemporal guidance for subsequent controllable diffusion sampling.

[0044] S4: Construct an anchor-point controllable diffusion sampling model, using the optimized anchor-point sequence as a condition guide to control the sampling process of the diffusion model, perform denoising generation in the real driving data distribution, and obtain a multi-agent trajectory sequence consistent with the target interaction modality, thus achieving controllable generation from semantics to trajectory level.

[0045] Furthermore, step S4 specifically includes the following sub-steps: S41: Construct an anchor-guided diffusion generation model. Design a controllable diffusion model structure, using the anchor sequence obtained in step S3 as conditional input to guide the diffusion sampling process. Its core generation function can be expressed as: in, For the diffusion model, unguided adjustment at time 10:00 For the sample The basic forecast mean, It is the variance matrix. This is a guiding objective function based on anchor points, used to constrain the generated result to approximate the spatiotemporal position of the target anchor point. This is a sample representing the current diffusion state during the generation process. To calculate the gradient for this sample, Given a sample of target interaction modalities, The diffusion model was adjusted at time [time]. For the sample The predicted mean.

[0046] The anchor-point controlled diffusion model sampling process employs a denoising guidance mechanism based on specific interaction modal anchor point conditions. Its function is to integrate the different interaction modal anchor point sets output by the spatiotemporal optimization model. As a conditional input to the diffusion sampling process, the sampling direction is dynamically adjusted according to the anchor point constraint during the reverse denoising process, so as to ensure that the generated trajectory meets the target interaction modality requirements at the semantic level and maintains compatibility with the real driving distribution at the dynamic level.

[0047] S42: Design the anchor-guided loss function. To simultaneously maintain the realism and semantic consistency of the generated results, construct the anchor-guided loss function: in, This refers to the number of vehicles participating in the scene. The generated vehicle trajectory sequence; For the target anchor point sequence; The smoothed L1 loss function measures the spatiotemporal deviation between the generated trajectory and the target anchor point. This loss term is calculated in each inverse denoising process of the diffusion model, guiding the generated trajectory to gradually approach the preset anchor point through gradient guidance, ensuring that the generated result is consistent with the target pattern at both the dynamic and semantic levels.

[0048] S5: Construct a scenario rationality pruning index system. Based on the scenario rationality index, evaluate the generated scenario in multiple dimensions, including kinematic feasibility and scenario consistency. Automatically remove interaction scenarios that are physically or inconsistently unreasonable to ensure the credibility of the generated results.

[0049] Furthermore, step S5 specifically includes the following sub-steps: S51: Social Consistency Detection. Based on the temporal position and bounding box of each subject in the generated scene, this function detects whether any two subjects have spatial overlap at the same time step. If the bounding boxes of any two subjects intersect or overlap, the scene is determined to have social inconsistency, meaning that the interaction behavior does not conform to reasonable social cooperation rules. This detection is used to eliminate irrational interactions caused by generation deviations (such as not avoiding collisions, occupying the same spatial position simultaneously, etc.).

[0050] S52: Behavioral Compliance Detection. Based on high-definition maps (HD Map) and lane geometry information, a compliance analysis is performed on the generated trajectory, including two indicators: Off-road detection: Determines whether the vehicle's trajectory has deviated from the passable area (such as leaving the lane boundary or entering a non-driving area). Wrong-direction detection: Calculate the angle between the vehicle's heading angle and the lane direction. If the deviation exceeds 90° and lasts for more than 1 second, it is considered as wrong-direction behavior.

[0051] If any entity in the scenario exhibits the aforementioned violations, the scenario will be deemed unreasonable at the behavioral level and will be removed.

[0052] S53: Kinematic Feasibility Detection. Calculate the dynamic characteristics (velocity, acceleration, steering curvature, etc.) of the generated trajectory and check whether the vehicle motion conforms to physical constraints. If any vehicle's motion parameters exceed a reasonable threshold range, the scenario is considered dynamically infeasible.

[0053] S54: Comprehensive Judgment and Pruning of Reasonableness. Based on the above three categories of reasonableness indicators (social consistency, behavioral compliance, and kinematic feasibility), if a scenario violates any one of these indicators, it is judged as an unreasonable scenario and removed during the pruning process.

[0054] Ultimately, only the interaction scenarios that simultaneously satisfy all reasonable constraints are retained to form a credible set of physically feasible scenarios, ensuring that the generated results conform to real driving logic in terms of social coordination, rule compliance, and dynamic constraints.

[0055] S6: Integrates semantic generation, spatiotemporal optimization and controllable diffusion modules to complete the entire process of generating physically feasible and semantically complete interactive scenarios from raw data, enabling the large-scale and systematic expansion of autonomous driving scenarios.

[0056] Furthermore, step S6 specifically includes the following sub-steps: S61: Module Integration. The semantic generation model, spatiotemporal network optimization module, and anchor point controllable diffusion model are integrated to form an interactive scene generation framework.

[0057] S62: Scene Generation and Execution. Input natural driving data, and sequentially perform semantic pattern exhaustion, spatiotemporal anchor point optimization, and controllable diffusion sampling to generate an interactive scene that conforms to physical constraints and semantic consistency.

[0058] S63: Result Output and Expansion. After reasonable pruning of the generated results, a set of physically feasible and semantically complete interactive scenarios is output, realizing the large-scale and systematic expansion of autonomous driving simulation scenarios.

[0059] like Figure 1 As shown, the above method is used in the embodiment: First, step S1 is performed to analyze the interaction scenario and extract conflict information. In this embodiment, the high-density driving interaction dataset InterHub is selected, and the interaction event index information of the Waymo dataset that meets the requirements is filtered out from it. Based on this index information, the corresponding interaction fragments are extracted from the Waymo dataset. Potential conflict points are identified, and a structured conflict information set is constructed, including: a conflict point set. Participating vehicle collection Passage priority Travel time .

[0060] Secondly, steps S2 and S3 are executed to construct a complete modal semantic generation module and a spatiotemporal anchor point optimization module. In this embodiment, Figure 2The complete modal semantic generation module shown is used to perform semantic layer parsing of the conflict point relationships in the scene. It adopts a reasoning method based on combinatorial graphs to generate the passage order combination of each vehicle at the local and global levels. Figure 1 (B) The spatiotemporal anchor point optimization module is implemented based on MILP (Mixed Integer Linear Programming) modeling and solved using the Gurobi optimizer. During the solution process, a minimum time deviation constraint is introduced to ensure that the optimization results closely match the natural driving distribution while maintaining consistent traffic flow. After parameter tuning, the average solution time of this optimization module does not exceed 0.5 seconds per scenario.

[0061] Next, perform step S4 to train and deploy the anchor-guided controlled diffusion model, such as... Figure 1 As shown in (C). In this embodiment, the diffusion model adopts the improved Versatile Behavior Diffusion (VBD) framework, and the model converges after being trained on the Waymo dataset for 16 epochs. Its sampling part consists of a temporal conditional diffusion sampling network, which takes as input the anchor guidance signal and the vehicle historical state sequence, and outputs the global scene trajectory distribution.

[0062] Further, step S5 is executed to perform a rationality assessment and automatic pruning of the diffusion generation results. This embodiment adopts a multi-dimensional rationality assessment system, including three parts: social consistency, behavioral compliance, and kinematic constraint detection.

[0063] Finally, execute step S6 to integrate all sub-modules and generate scenes in batches.

[0064] This embodiment comprehensively evaluates driving interaction events in various traffic scenarios. From the InterHub dataset, 188 interaction events were selected as input, ultimately generating over 540 sets of semantically complete interaction scenario samples.

[0065] like Figure 3 As shown in (A), in order to verify the applicability and completeness of the method of the present invention in generating interactive modalities in different scenarios, a systematic statistical analysis was conducted on the generation results from two aspects: the effect of single-vehicle dynamics constraints and the effect of multi-vehicle coordination constraints, based on the semantically complete modal set.

[0066] First, this invention constructs a semantically complete modality set by exhaustively enumerating all possible passage sequences through a semantic modality generation module.

[0067] Subsequently, using the spatiotemporal anchor point optimization module, a complete set of interactions that meet the dynamic feasibility requirements is obtained by considering the dynamic constraints of a single vehicle. Figure 3(B) shows that in about half of the scenarios, all semantic patterns can be retained, while in the remaining scenarios, some patterns are eliminated because they do not meet the acceleration or velocity constraints.

[0068] Based on this, a set of interactive modes that satisfy multi-vehicle coordination constraints is further generated through a controllable diffusion sampling module guided by anchor points. Figure 3 (C) The results show that about 10% of the scenarios can achieve all coordinated feasible modes, while the remaining modes are excluded because conflicting constraints cannot be satisfied simultaneously.

[0069] In summary, this invention can generate comprehensive and well-coordinated interactive scenarios while ensuring physical feasibility, achieving a complete generation process from semantic exhaustion to physical constraints.

[0070] To verify the complete interaction modality generation effect in a dual-vehicle interaction scenario, this embodiment... Figure 4 This demonstrates the interaction generation results when two vehicles merge into the same road segment. In the initial scenario, vehicle 5 is going straight and vehicle 6 is turning right, indicating a potential merging conflict. After generation using the method of this invention, two physically feasible interaction modalities are obtained: In the first mode, vehicle 5 passes the point of conflict first, and vehicle 6 slows down to avoid it; In the second mode, vehicle 6 accelerates to enter first, while vehicle 5 decelerates to follow.

[0071] The results show that the present invention can automatically deduce all dynamically feasible interaction modes of two-vehicle interaction scenarios under the same initial conditions, and truly reflect the "priority-yield" relationship in human driving.

[0072] To verify the complete interaction modality generation effect in multi-vehicle interaction scenarios, this embodiment... Figure 5 This demonstrates a multi-agent interaction between a left-turning vehicle and two oncoming straight-going vehicles at an intersection. Through the reasoning and generation methods of this invention, three dynamically feasible interaction modes were obtained: In the first mode, two vehicles going straight slow down to give way, and vehicles turning left pass first; In the second mode, vehicles going straight ahead accelerate and vehicles behind slow down to give way, while vehicles turning left pass between the two vehicles going straight. In the third mode, two vehicles going straight pass in turn, while vehicles turning left slow down to give way.

[0073] Theoretically, this scenario can generate four modes, but because a non-interactive background car restricts the movement space of the following car, the fourth mode is automatically determined to be unrealizable during the spatiotemporal optimization stage.

[0074] Therefore, this invention can not only exhaustively enumerate all theoretical interaction modes, but also automatically identify and eliminate unreasonable solutions based on physical and environmental constraints, thereby realizing the generation of realistic and complete physically feasible scenarios.

[0075] This embodiment provides a method for generating interactive modal scenarios based on a spatiotemporally optimized controllable diffusion model. This method can be applied to the research and development and verification of autonomous driving technology: generating all possible evolution scenarios corresponding to interactive events for training and stress testing of algorithm models; or being applied to the construction of virtual test fields to efficiently verify the safety of the system in extreme or rare interactions, providing data benchmarks for the formulation of industry testing standards and regulations; or being applied as a simulation testing tool to serve the autonomous driving development and iteration of car manufacturers and technology companies.

[0076] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An interactive multi-modal scene generation method based on a spatio-temporal optimized controllable diffusion model, characterized in that, The method specifically comprises: S1. Obtain natural driving interaction data, identify and extract key conflict points from the natural driving interaction data, and construct a structured conflict information set using the extracted key conflict points; S2. Construct a complete interaction mode semantic generation model, based on the structured conflict information set, use the complete interaction mode semantic generation model to exhaustively combine semantic interaction modes, and generate a scene semantic layer interaction mode set; S3. Construct a space-time network optimization model, optimize the semantic interaction mode set under the constraint of the constraint condition, and convert it into a complete interaction mode anchor point sequence set; S4. Construct an anchor point controllable diffusion sampling model, use the complete interaction mode anchor point sequence set as a condition guide, control the diffusion sampling process, and generate an interaction mode scene vehicle trajectory; S5. Construct a scene rationality pruning index, evaluate the rationality of the generated interaction mode scene vehicle trajectory, eliminate unreasonable interaction mode scene vehicle trajectories, and realize the generation of an interaction mode scene.

2. The interactive multi-modal scene generation method based on the spatio-temporal optimized controllable diffusion model according to claim 1, characterized in that, The S1 specifically comprises: S1.

1. Screen out scene segments that meet the interaction intensity from the obtained natural driving interaction data, and extract interaction vehicle trajectories and corresponding scene map information; S1.

2. Based on the extracted vehicle trajectory, identify key positions that exist potential mutual interference in space-time as key conflict points, and construct a structured conflict information set according to the key conflict points and associated vehicle dynamic characteristics.

3. The interactive multi-modal scene generation method based on the spatio-temporal optimized controllable diffusion model according to claim 1, characterized in that, The scene semantic layer interaction mode set includes a local semantic mode set and a global interaction mode set, and its acquisition process includes the following steps: Based on the structured conflict information set, identify the vehicle set participating in the interaction at the key conflict point, enumerate the vehicle through sequence arrangement combination, and capture the local interaction result at the key conflict point, to generate a local semantic mode set; By globally combining all the local semantic modes, the feasible through sequence of each key conflict point is combined and mapped on the scene level to construct a global interaction mode set.

4. The interactive multi-modal scene generation method based on the spatio-temporal optimized controllable diffusion model according to claim 1, characterized in that, The spatiotemporal network optimization model is constructed by constructing a key conflict point-time joint graph With the semantic interaction mode as input, an anchor point set corresponding to a complete interaction mode satisfying the dynamic constraint is obtained by graph optimization solving, wherein represents a set of spatiotemporal nodes, is a set of directed edges representing node transitions.

5. The interactive multi-modal scene generation method based on the spatio-temporal optimized controllable diffusion model according to claim 1, characterized in that, The optimization objective function of the space-time network optimization model is to minimize the time deviation of each vehicle at the key conflict point, and the formula is: wherein, represents a set of all vehicles participating in the interaction in the scenario, represents a set of critical conflict points, represents a vehicle passes through a critical conflict point after optimization, represents a vehicle actual passing time observed in natural driving data, represents a global objective function value, which measures the consistency of the optimization result with the real data in the time dimension.

6. The interactive multi-modal scene generation method based on the spatio-temporal optimized controllable diffusion model according to claim 1, characterized in that, The constraint conditions include: Space-time consistency constraint: the vehicle maintains path continuity in the space-time network; Passing order constraint: the passing order of the vehicle at the key conflict point meets the expected interaction semantic requirements of the interaction logic; Dynamics constraint: the speed, acceleration and time interval of the vehicle meet the physical feasibility.

7. The interactive multi-modal scene generation method based on the spatio-temporal optimized controllable diffusion model according to claim 1, characterized in that, The anchor point controllable diffusion sampling model is represented as: In the formula, is the diffusion model unguided adjustment at time is the basic prediction mean value of the sample , is the variance matrix, is an anchor-based guiding objective function for constraining the generation result to be close to the spatio-temporal position of the target anchor, is the current diffusion state sample in the generation process, is the gradient calculation of the sample, is a given target interaction modality sequence sample, is the number of participating vehicles in the scene; is the generated vehicle trajectory sequence; is the target anchor sequence; is a smooth L1 loss function for measuring the deviation of the generated trajectory and the target anchor in space and time, is the diffusion model guided adjustment at time is the prediction mean value of the sample .

8. The interactive multi-modal scene generation method based on the spatio-temporal optimized controllable diffusion model according to claim 1, characterized in that, The sampling process of the anchor point controllable diffusion sampling model adopts a denoising guide mechanism based on specific interaction mode anchor point conditions, uses the different interaction mode anchor point set output by the space-time optimization model as the condition input of the diffusion sampling process, dynamically adjusts the sampling direction according to the anchor point constraint in the reverse denoising process, and generates an interaction mode scene vehicle trajectory that meets the target interaction mode requirements on the semantic level.

9. The interactive multi-modal scene generation method based on the spatio-temporal optimized controllable diffusion model according to claim 1, characterized in that, The scene rationality pruning index includes: Social consistency index: detect whether there is envelope overlap at the same time for each subject; Behavior compliance index: detect whether there is a boundary crossing or reverse behavior; Kinematic feasibility index: check whether the vehicle acceleration and steering curvature exceed the set threshold value; When the scene violates any index, it is determined to be unreasonable and is rejected.

10. The interactive multi-modal scene generation method based on the spatio-temporal optimized controllable diffusion model according to claim 1, characterized in that, The method further comprises integrating the complete interaction modal semantic generation model, the spatio-temporal network optimization model and the anchor controllable diffusion sampling model to form an interaction scene generation framework.

Images