End-to-end autonomous driving real adversarial scenario generation and closed-loop verification system and method

Abstract

The present disclosure relates to an end-to-end autonomous driving real confrontation scene generation and closed-loop verification system and method, which is used to solve the problem that the prior art cannot deploy and verify the end-to-end model in the confrontation traffic flow created in the simulation world. The system sequentially executes the following modules for each segment in the natural traffic flow dataset to realize the closed-loop verification of the real confrontation scene of the ego vehicle: an adversarial traffic flow module, which loads a segment of a natural traffic flow dataset, simulates a topological simulation environment, generates multiple candidate round vehicle trajectories based on the topological simulation environment, executes the round vehicle with the highest candidate round vehicle trajectory, and generates an adversarial traffic flow; a driving image generator, which generates a real driving traffic environment image based on a control condition containing traffic information of the simulation environment; and an end-to-end model under test, which generates a trajectory of the ego vehicle executed in the adversarial traffic flow based on the real driving traffic environment image, and the topological simulation environment dynamically changes with the execution of the ego vehicle trajectory.

Description

Technical Field

[0001] This disclosure relates to end-to-end autonomous driving systems, and more particularly to an end-to-end autonomous driving real-world adversarial scenario generation and closed-loop verification system and method. Background Technology

[0002] With the rapid development of intelligent driving technology, end-to-end autonomous driving systems, with their unified perception-decision-control integrated architecture, have gradually become a key focus of research and industry. This system can directly generate driving action commands from image perception input, greatly simplifying the structure of traditional modular systems and improving generalization ability and end-to-end performance.

[0003] However, existing end-to-end autonomous driving systems may still face systemic or sporadic failures in safety-critical scenarios, leading to new safety hazards. Therefore, safety verification of end-to-end autonomous driving is particularly important. Adversarial traffic flow is a crucial method and approach for generating safety-critical scenarios. Current adversarial traffic flow testing primarily relies on virtual simulation environments, such as CARLA and MetaDrive. These simulation environments are mostly based on simple geometric modeling and low-fidelity visual rendering, rather than real-world driving images. For end-to-end models trained on real driving images, there is a significant visual gap between the simulation world and the real world, making it difficult to meet the visual requirements for adversarial loop closure verification in real-world scenarios. Therefore, it is impossible to deploy and verify end-to-end models in adversarial traffic flows created in a simulated world. Summary of the Invention

[0004] The purpose of this disclosure is to propose an end-to-end autonomous driving realistic adversarial scenario generation and closed-loop verification system. This system can controllably generate high-fidelity driving images from traffic information in a simulated environment depicting adversarial traffic flow, and complete closed-loop experimental verification of the end-to-end autonomous driving system under adversarial safety-critical scenarios. The specific technical solution is as follows.

[0005] The first aspect of this disclosure is an end-to-end autonomous driving realistic adversarial scenario generation and closed-loop verification system. The system sequentially executes the following modules for each segment of a natural traffic flow dataset to achieve closed-loop verification of the autonomous vehicle's realistic adversarial scenario, including: an adversarial traffic flow module configured to load a natural traffic flow dataset segment, simulate and generate a topology simulation environment, generate multiple candidate vehicle trajectories based on the generated topology simulation environment, and have an adversarial vehicle execute the candidate vehicle trajectory with the highest adversarial power to generate an adversarial traffic flow; a driving image generator configured to generate a realistic driving traffic environment image based on control conditions containing traffic information from the simulation environment; and an end-to-end test model configured to generate a trajectory executed by the autonomous vehicle in the adversarial traffic flow based on the realistic driving traffic environment image, wherein the topology simulation environment dynamically changes as the autonomous vehicle's trajectory is executed.

[0006] In one embodiment of the above technical solution, the step of generating adversarial traffic flow includes: in the first round, recording the vehicle trajectory in the natural traffic flow; in the second round, based on the recorded vehicle trajectory, performing an adversarial evaluation on each candidate weekly vehicle trajectory, and selecting the candidate weekly vehicle trajectory with the highest adversarial performance as the execution trajectory of the adversarial weekly vehicle.

[0007] In one embodiment of the above technical solution, the adversarial nature of the candidate vehicle trajectory is evaluated using the following scoring function: the higher the score, the stronger the adversarial nature. , In the formula: Representing candidate trajectories The prior probability, This indicates that the adversarial zonal vehicle is in the candidate trajectory. The collision intensity between the vehicle and the recorded trajectory, parameters and These are the preset weight hyperparameters, To be based on the trajectory Calculate the time step of the first collision. It is the attenuation factor; Represents the absolute value of jerk. As a penalty for escalation, .

[0008] In one embodiment of the above technical solution, based on the generated topology simulation environment, the DenseTNT model is used to generate multiple candidate circumferential vehicle trajectories, and the prior probability of each candidate circumferential vehicle trajectory is output.

[0009] In one embodiment of the above technical solution, the step of acquiring traffic information in the simulated environment includes: projecting the 3D bounding box of the surrounding vehicles and the map onto the camera view of the vehicle, acquiring the 3D bounding box of the surrounding vehicles from the camera view and its coordinates, and acquiring the lane lines from the camera view.

[0010] In one embodiment of the above technical solution, the driving image generator consists of a conditional control unit and an image denoising generation backbone network. The conditional control unit embeds the information encoding features obtained based on surrounding vehicle positions, camera parameters, and text prompts into the latent space through a cross-attention mechanism, and injects the structural encoding obtained based on map projection and surrounding vehicle 3D bounding box projection into the upsampling module of the Unet model through ControlNet, thereby ensuring that the traffic semantic information of the generated driving traffic environment image is controllable. The image denoising generation backbone network is based on a flow matching model framework and implemented using the UNet model. In the flow matching model framework, the ordinary differential equation of the flow matching model is used to describe the evolution process of data distribution with a continuous-time dynamic equation, and the optimal alignment of the network at each time step is achieved by minimizing the flow matching loss.

[0011] In one embodiment of the above technical solution, during the training process of the driving image generator, ControlNet is initialized by the pre-trained UNet downsampling module, and the text encoder parameters of the text prompt input are frozen; during the inference phase, Euler's method is used to perform discrete integration on the ordinary differential equation, thereby achieving high-fidelity image generation in fewer time steps.

[0012] A second aspect of this disclosure is a computer-readable storage medium storing a computer program that can be loaded by a processor and execute the system described in any of the preceding claims.

[0013] A third aspect of this disclosure is a method for generating and verifying adversarial real-world driving scenarios for end-to-end autonomous driving. This method achieves real-world adversarial scenario closed-loop verification of the vehicle by repeating the following steps: loading a fragment of a natural traffic flow dataset; simulating and generating a topology simulation environment; generating multiple candidate vehicle trajectories based on the generated topology simulation environment; executing the candidate vehicle trajectory with the highest adversarial level for an adversarial vehicle to generate an adversarial traffic flow; generating a realistic driving traffic environment image based on control conditions containing traffic information from the simulation environment; and generating the trajectory executed by the vehicle in the adversarial traffic flow based on the realistic driving traffic environment image, wherein the topology simulation environment dynamically changes as the vehicle's trajectory is executed.

[0014] The beneficial technical effects of this disclosure are: in a real driving image environment, it efficiently generates adversarial traffic flow, realizing a comprehensive platform for end-to-end model adversarial closed-loop verification, which can be adapted to different test models to verify the performance and safety of end-to-end autonomous driving systems in real safety-critical scenarios. It can be used for stress testing of autonomous vehicles in adversarial traffic processes, improving the safety and stability of autonomous vehicles. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the overall architecture of an end-to-end autonomous driving real-world adversarial scenario generation and closed-loop verification system in one implementation method.

[0017] Figure 2 This is a schematic diagram of an anti-traffic flow generation method in one implementation.

[0018] Figure 3This is a schematic diagram of rapid driving image generation based on a flow matching model in one implementation. Detailed Implementation

[0019] Existing end-to-end autonomous driving systems may face systemic or sporadic failures in safety-critical scenarios, leading to new safety hazards. Therefore, safety verification is crucial. Adversarial traffic flow is an important way to generate safety-critical scenarios; however, existing adversarial traffic flow testing mainly relies on virtual simulation environments such as CARLA and MetaDrive. These simulation environments are based on simple geometric modeling and low-fidelity visual rendering, using non-real-world driving images. This results in a significant visual gap compared to end-to-end models trained on real driving images, making it difficult to meet the visual requirements for adversarial loop closure verification in real-world scenarios. Consequently, it is impossible to deploy and verify end-to-end models within simulated adversarial traffic flows. Therefore, this disclosure proposes an adversarial real-world driving scenario generation and loop closure verification system for end-to-end autonomous driving.

[0020] The following provides a clear and complete description of how the technical solution of this case is implemented. Obviously, the described implementation methods are only a part of the implementation methods of this case, and not all of them. Based on the implementation methods in this case, all other implementation methods obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0021] Figure 1 This is one implementation of the overall architecture of an end-to-end autonomous driving adversarial real-world driving scenario generation and closed-loop verification system. As can be seen from this architecture, the system includes an adversarial traffic flow module, a driving image generator module, and an end-to-end test model module. Through the collaboration of these modules, efficient and realistic adversarial closed-loop verification of the end-to-end autonomous driving system under driving scenarios is achieved.

[0022] (a) Traffic Flow Countermeasure Module

[0023] The adversarial traffic flow module can generate adversarial vehicle trajectories based on natural traffic flow using a multimodal trajectory prediction model, ensuring the rationality and adversarial nature of the entire adversarial traffic flow. The adversarial vehicle trajectories are those that conflict with or collide with the idiosyncratic vehicle trajectories.

[0024] The adversarial traffic flow module can be driven by a traditional topology-based autonomous driving simulator, fully utilizing the physical simulation capabilities of traditional simulators to generate a corresponding topological simulation environment based on natural traffic flow. The adversarial traffic flow module reads traffic information from the topological simulation environment, obtains 3D bounding boxes for surrounding vehicles and a simulation environment map, projects these boxes onto the camera viewpoint adapted to the end-to-end model under test for preprocessing, obtains the camera view's 3D bounding boxes for surrounding vehicles and lane lines, and transmits the preprocessed traffic information to the driving image generator module. The preprocessed traffic information also includes the original corner coordinates of the 3D bounding boxes for surrounding vehicles.

[0025] See Figure 2 The method for generating adversarial traffic flow is illustrated. When simulating the driving environment, simulations are performed based on segments of a natural traffic flow dataset, generating a topological simulation environment within each segment. Adversarial traffic flow is generated using each segment of the natural traffic flow dataset. The generation process consists of two rounds, both loading segments of the same natural traffic flow dataset and reconstructing the same traffic environment in the simulation environment. Because different end-to-end autonomous driving models exhibit varying performance in environmental metrics such as vehicle speed and acceleration, a two-round adversarial approach is employed to ensure applicability across different tested models.

[0026] Specifically, before generating adversarial traffic flows, an adversarial cycle vehicle is first selected. Figure 2 The green car), the vehicle is controlled by the end-to-end test model. In the first round, a segment of natural traffic flow is replayed, allowing the vehicle ( Figure 2 The trajectories of all vehicles except the red car are based on segments of the natural traffic flow dataset. During this process, adversarial vehicles record their own trajectories, including their complete trajectories and / or behaviors within the natural traffic flow. Before the second round, the adversarial vehicles are connected to a multimodal trajectory prediction model, which generates multiple adversarial vehicle trajectories as candidate trajectories. Figure 2 The diagram illustrates four candidate car trajectories. Using the car's own trajectory recorded in the first round, each adversarial car trajectory is evaluated for its adversarial effectiveness. In the second round, when an adversarial car executes the highest adversarial car trajectory, a segment replay is performed on all cars except the car itself and the adversarial car. Figure 2 In the second round, the trajectory of the candidate vehicle with the highest adversarial strength collided with the trajectory of the original vehicle.

[0027] Figure 1The three modules shown form a closed-loop experiment, which runs in both rounds. This allows the adversarial vehicle trajectory to be flexibly adjusted based on the trajectory preferences of different tested models in the first round of natural traffic flow, such as speed and acceleration parameter preferences, thereby improving the adaptability to the adversarial aspects of the tested models. The multimodal trajectory prediction model is a method that combines multiple data sources (such as images, text, sensor signals, etc.) to improve the accuracy of predicting future motion paths, outputting multiple trajectories under natural conditions. In this application, the multimodal trajectory prediction model preferably adopts the DenseTNT model, which generates multiple candidate vehicle trajectories with rationality and adversarial nature based on a real-time generated topology simulation environment, and outputs the prior probability of each candidate vehicle trajectory.

[0028] To determine an adversarial vehicle trajectory that is both adversarial and conforms to the laws of physics, this application employs a multiplicative scoring function for adversarial evaluation. This function comprehensively considers three key factors: (i) the reasonableness of the prediction model, (ii) the probability of collision with the vehicle's trajectory, and (iii) the smoothness of the motion.

[0029] Specifically, for the i-th candidate weekly vehicle trajectory Its scoring function is defined as: , , Representing candidate trajectories The prior probability of the candidate trajectory. A multimodal trajectory prediction model can be used to generate the trajectory. This model is trained using natural traffic flow data; therefore, the prior probability of the trajectory represents the candidate weekly vehicle trajectory selected under natural conditions. The probability is used to ensure that the generated adversarial trajectory still conforms to the real driving distribution, thereby improving the rationality of the trajectory. This indicates that the adversarial zonal vehicle is in the candidate trajectory. The collision intensity between the vehicle and the recorded trajectory is designed so that the earlier the collision occurs, the better. The higher the value, the better. In the definition, candidate trajectory Both the vehicle's recorded trajectory and the opposing vehicle's trajectory have time attributes, allowing for the calculation of the space occupied by the vehicle and the opposing vehicle at each moment. Therefore, collision detection is performed based on the overlap of the occupied spaces of the vehicle and the opposing vehicle at the same moment, indicating whether a collision has occurred. (See attached diagram.) Figure 2 As shown in the middle image, the numbers in the image are trajectory markers. To be based on the trajectory Calculate the time step of the first collision. As a decay factor, this design ensures a higher trajectory score for early collisions, giving it an advantage in the selection process. Represents the absolute value of jerk. The acceleration penalty is used to punish candidate trajectories for excessively abrupt changes in acceleration, ensuring the physical plausibility of adversarial trajectories. .parameter and These are preset weighted hyperparameters used to balance the relative importance of antagonism and motion smoothness. Jerk is the rate of change of acceleration, a physical quantity used to describe how quickly acceleration changes.

[0030] Ultimately, the candidate trajectory with the highest score was selected as the adversarial circumvention trajectory. This design ensures that the selected adversarial circumvention trajectory maintains consistency with the prediction model while also posing a challenge to the vehicle in an adversarial manner, avoiding unrealistic or unstable motion patterns.

[0031] (ii) Driving Image Generator

[0032] The driving image generator module uses a flow matching model and the Euler method to quickly denoise noisy images and generate realistic driving images. During the denoising process, ControlNet and an attention mechanism are used for conditional injection, taking traffic information, surrounding vehicle positions, camera parameters, and text prompts as control conditions to generate high-quality driving images consistent with the control conditions. The traffic information includes the map projection from the camera's perspective and the 3D bounding box projection of surrounding vehicles. Figure 1 In the text, “denoising × N” means denoising is performed in N iterations, where N is a preset value.

[0033] As attached Figure 3 As shown, in one implementation, the flow matching model consists of two parts: a condition control unit and an image denoising generation backbone network.

[0034] The image denoising generation backbone network is used to efficiently iteratively denoise Gaussian noise images under control constraints, outputting latent features of the real image. These latent features are then input into a decoder to obtain a visualized real image. The image denoising generation backbone network is based on a flow matching model framework and constructed using the UNet model. The UNet model consists of a downsampling module and an upsampling module. Traditional diffusion models use stochastic differential equations (SDEs) for denoising, while this invention utilizes the deterministic ordinary differential equation (ODE) form of the flow matching model to describe the evolution of the data distribution using a continuous-time dynamic equation. Its core ODE form is expressed as follows: , This represents the image at time step t after denoising. For continuous time steps, This is the optimal flow vector field predicted by UNet, used to approximate the direction of data flow from noise to the actual data flow. The input downsampling module and the embedded information encoding are concatenated and fused. Guided by the information encoding and structure encoding, the upsampling module outputs the latent features of the real image.

[0035] The condition control unit performs multimodal encoding and fusion of different types of control conditions. The control conditions in this application include two categories: high-dimensional structural conditions and low-dimensional informational conditions. Informational conditions include text prompts, vehicle positions, and camera parameters. These are encoded by text encoders, camera encoders, and position encoders, respectively. After being fused into information codes, these codes are embedded into the latent space (i.e., the upsampling and downsampling modules of the UNet model) using a cross-attention mechanism. Text prompts include, for example, sunny weather and urban road conditions. Structural conditions include map projection and 3D bounding box projections of surrounding vehicles. Structural codes are obtained based on this information. These structural codes are injected into the upsampling module of the UNet model via ControlNet, which is initialized by the downsampling module of a pre-trained UNet model.

[0036] During training, the text encoder parameters for the input text prompts are frozen and not updated. A fast image generation model based on a flow matching model is trained together with a decoder, which generates realistic images based on latent features. Optimal alignment of the network at each time step is achieved by minimizing the following flow matching loss: [loss function]. as follows: ,in and That is and , representing noise-free real samples and pure Gaussian noise, respectively. This refers to the direction of flow from sample data to noise data, which serves as the true value label.

[0037] After training, this invention utilizes the Euler method to perform discrete integration on the aforementioned core ordinary differential equations during the inference phase, enabling high-fidelity image generation in fewer time steps. .

[0038] This solution can complete the image denoising and generation process in less than 10 steps, which significantly improves the generation speed compared with the traditional diffusion model, and effectively ensures the image quality and controllability of traffic elements.

[0039] This solution controls the generation of road layout, surrounding vehicle distribution, ambient lighting, and weather based on traffic information, camera parameters, and text prompts, outputting controllable and visually realistic driving images to provide high-quality image input for subsequent end-to-end autonomous driving modules under test.

[0040] (III) End-to-end test model

[0041] The end-to-end test model is a virtual vehicle with an end-to-end model on it. The end-to-end model loads generated real-world images and obtains corresponding control decisions. These control decisions are trajectories / actions. The actions can be throttle, braking, or steering.

[0042] See Figure 1 The lower end-to-end test model module receives generated real driving images and outputs trajectories / actions. In the topology simulation environment of the adversarial traffic flow module, the vehicle trajectory is executed according to the vehicle's actions, and the adversarial traffic flow driven by the traditional simulator steps and updates the simulation world.

[0043] In summary, the adversarial traffic flow module, driving image generator module, and end-to-end test model module, through their collaboration, can achieve efficient and realistic end-to-end adversarial closed-loop verification of autonomous driving systems in driving scenarios.

[0044] (iv) System operation and deployment

[0045] Regarding the introduction of vehicle-in-the-loop (VIL) technology, one implementation method employs a real-driving scenario verification process that creates virtual adversarial entities for the real vehicle. (See attached...) Figure 1 As shown, a virtual simulation environment identical to the test track where the real vehicle is located is constructed using a digital twin approach. ROS allows for synchronization of the vehicle's position with the real world. This environment can be driven by a traditional autonomous driving simulator with traffic topology relationships. The vehicle's state in the simulation environment is updated in real time based on the real vehicle's sensors to maintain virtual-real synchronization.

[0046] In the closed-loop experiment, this system only needs to generate a virtual adversarial vehicle entity that interacts with the autonomous vehicle in the simulated traffic flow environment, and the driving image generation module generates real driving images of the corresponding adversarial scenario in real time. After receiving the real driving image input, the end-to-end test model outputs a trajectory or control action, and the real vehicle performs the corresponding operation in the real environment; at the same time, the autonomous vehicle in the simulation environment updates its position and attitude synchronously according to the real vehicle's state, thus forming a virtual-real interactive adversarial closed-loop verification platform.

[0047] This process, while maintaining virtual adversarial traffic flow interaction, introduces the dynamic characteristics of real vehicles, further enhancing the realism of adversarial closed-loop verification, while avoiding the dangers of real adversarial physical closed-loop experiments, providing more comprehensive technical support for the closed-loop verification of real adversarial scenarios for end-to-end autonomous driving systems.

[0048] In summary, this application provides a method for generating and verifying adversarial real-world driving scenarios for end-to-end autonomous driving. This method achieves closed-loop verification of the vehicle's real-world adversarial scenarios by repeating the following steps:

[0049] A fragment of a natural traffic flow dataset is loaded, and a topology simulation environment is generated. Based on the generated topology simulation environment, multiple candidate vehicle trajectories are generated. Adversarial vehicles execute the candidate vehicle trajectory with the highest adversariality in the topology simulation environment to generate adversarial traffic flow. A realistic driving traffic environment image is generated based on control conditions, including traffic information from the simulation environment. Based on the realistic driving traffic environment image, the trajectory executed by the vehicle in the adversarial traffic flow is generated. As the vehicle's trajectory is executed, the topology simulation environment dynamically changes. The adversarial vehicle trajectory is a vehicle trajectory that conflicts with or collides with the vehicle's trajectory.

[0050] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0051] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0052] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0053] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0054] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0055] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0056] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0057] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0058] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. An end-to-end autonomous driving real adversarial scene generation and closed-loop verification system, characterized in that, The system sequentially executes the following modules on each segment in the natural traffic flow dataset to realize self-vehicle real closed-loop verification of the confrontation scene, including: The traffic flow module is configured to load a natural traffic flow data set segment, simulate a generated topology simulation environment, generate a plurality of candidate round car trajectories based on the generated topology simulation environment, and execute the most antagonistic candidate round car trajectory for the antagonistic round car to generate an antagonistic traffic flow. The steps include: selecting an antagonistic round car, in the first round, replaying the natural traffic flow segment, making the trajectories of other vehicles except the ego car follow the natural traffic flow data set segment, and recording the ego car trajectory during this process; before the second round, connecting the antagonistic round car to the multi-modal trajectory prediction model, and generating a plurality of antagonistic round car trajectories as candidate round car trajectories using the multi-modal trajectory prediction model; and performing antagonistic evaluation on each antagonistic round car trajectory using the ego car trajectory recorded in the first round: , , represents the prior probability of the candidate trajectory , represents the collision intensity of the antagonistic round car on the candidate trajectory with the ego car recorded trajectory, parameters and are preset weight hyperparameters, is the time step of the first collision calculated according to the trajectory , is a decay factor; represents the absolute value of jerk, is a jerk penalty, , the higher the score, the stronger the antagonism; in the second round, the antagonistic round car executes the highest antagonistic round car trajectory, and all other vehicles except the ego car and the antagonistic round car execute the segment playback; the antagonistic round car trajectory is a round car trajectory that conflicts or collides with the ego car trajectory; A driving image generator configured to generate a real driving traffic environment image based on a control condition containing simulation environment traffic information; An end-to-end tested model configured to generate a trajectory executed by the ego vehicle in the confrontation traffic flow based on the real driving traffic environment image, and the topological simulation environment dynamically changes with the execution of the ego vehicle trajectory.

2. The system of claim 1, wherein, Based on the generated topological simulation environment, a DenseTNT model is used to generate multiple candidate round-trip trajectories, and the prior probability of each candidate round-trip trajectory is output.

3. The system of claim 1, wherein, The simulation environment traffic information acquisition step includes: projecting the round-trip vehicle 3D bounding box and the map into the camera view of the ego vehicle, obtaining the camera view round-trip vehicle 3D bounding box and its coordinates, and obtaining the camera view lane line.

4. The system of claim 1, wherein: The driving image generator is composed of a condition control unit and an image denoising generation backbone network; The condition control unit encodes the information obtained based on the round-trip vehicle position, camera parameters and text prompt words into features and embeds them into the latent space through cross-attention mechanism, and encodes the structure obtained based on map projection and round-trip vehicle 3D bounding box projection into ControlNet and injects it into the up-sampling module of the UNet model, thereby ensuring that the traffic semantic information of the generated driving traffic environment image is controllable; The image denoising generation backbone network is based on a flow matching model framework and uses a UNet model to realize it. In the flow matching model framework, the ordinary differential equation of the flow matching model is used to realize the evolution process of describing the data distribution with continuous time dynamic equation, and the optimal alignment of the network at each time step is realized by minimizing the flow matching loss.

5. The system of claim 4, wherein, During the training process of the driving image generator, the ControlNet is initialized by the pre-trained UNet down-sampling module, and the text encoder parameters input by the text prompt word are frozen; in the inference stage, the ordinary differential equation is discretely integrated by using the Euler method, thereby realizing high-fidelity image generation with fewer time steps.

6. A computer-readable storage medium, characterized in that: A computer program capable of being loaded and executed by a processor to implement the system of any one of claims 1 to 5 is stored.

7. An end-to-end autonomous driving real adversarial scene generation and closed-loop verification method, characterized in that, The real confrontation scene closed-loop verification of the ego vehicle is realized by repeating the following steps, including: Load a segment of natural traffic flow dataset, generate a topological simulation environment, generate a plurality of candidate round car trajectories based on the generated topological simulation environment, and the most resistant round car performs the most resistant candidate round car trajectory, and generates an adversarial traffic flow, the steps comprising: selecting an adversarial round car, in the first round, playing back the natural traffic flow segment, making the trajectories of other vehicles except the ego vehicle follow the segment of the natural traffic flow dataset, and recording the ego vehicle trajectory of the adversarial round car in the process; before the second round, the adversarial round car is connected to the multi-modal trajectory prediction model, and a plurality of adversarial round car trajectories are generated as candidate round car trajectories by using the multi-modal trajectory prediction model; using the ego vehicle trajectory recorded in the first round, each adversarial round car trajectory is evaluated for resistance: , , represents the prior probability of the candidate trajectory , represents the collision intensity of the adversarial round car on the candidate trajectory with the ego vehicle recorded trajectory, parameters and are preset weight hyperparameters, is the time step of the first collision calculated according to the trajectory , is the decay factor; represents the absolute value of jerk, is the jerk penalty, , the higher the score, the stronger the resistance; in the second round, the adversarial round car executes the highest resistance round car trajectory, and all except the ego vehicle and the adversarial round car execute the segment playback, and the adversarial round car trajectory is the round car trajectory that conflicts or collides with the ego vehicle trajectory; Generating a real driving traffic environment image based on a control condition containing simulation environment traffic information; Generating a trajectory executed by the ego vehicle in the confrontation traffic flow based on the real driving traffic environment image, and the topological simulation environment dynamically changes with the execution of the ego vehicle trajectory.

Images