Simulation test method, device and electronic equipment for autonomous driving
Patent Information
- Application Number
- CN202510014912.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-03
- Publication Date
- 2026-07-03
Smart Images
Figure CN122333937A_ABST
Abstract
Description
Technical Field
[0001] This application relates to vehicle technology, and more particularly to a simulation test method, apparatus, electronic device, computer-readable storage medium, and computer program product for autonomous driving. Background Technology
[0002] Autonomous driving technology is a hot topic in the automotive field. It relies on the collaborative efforts of artificial intelligence, computer vision, radar, monitoring devices, and global positioning systems to enable vehicles to drive safely and automatically without any human intervention. Simulation testing is frequently used in the research and development of autonomous driving technology. Simulation testing is a testing method that uses virtual simulation technology to simulate and model road environments, traffic, and perception systems, thereby achieving comprehensive and efficient verification of autonomous driving functions.
[0003] The solutions provided by related technologies typically involve constructing simulated driving scenarios and then conducting simulation tests based on these scenarios. However, the constructed simulated driving scenarios are difficult to directly convert into file formats supported by the simulator, requiring additional steps for format conversion and importing into the simulator, which increases the complexity of the operation and the possibility of errors. Summary of the Invention
[0004] This application provides a simulation testing method, apparatus, electronic device, computer-readable storage medium, and computer program product for autonomous driving, which can automatically convert simulated driving scenarios into scenario description files that conform to the simulator file format for simulation testing, thereby improving the simplicity and success rate of simulation testing.
[0005] The technical solution of this application is implemented as follows:
[0006] This application provides a simulation testing method for autonomous driving, including:
[0007] Static analysis is performed on the simulated driving scene to obtain static information of the scene elements;
[0008] The dynamic information of the scene elements is obtained by performing dynamic analysis on the simulated driving scene.
[0009] Determine a predefined file structure that conforms to the simulator file format; wherein, the predefined file structure defines entity nodes and storyboards;
[0010] The static information of the scene elements is mapped to the node description of the entity nodes in the predefined file structure, and the dynamic information of the scene elements is mapped to the story description of the storyboard in the predefined file structure, so as to obtain a scene description file based on the predefined file structure.
[0011] Import the scenario description file into the simulator for simulation testing.
[0012] This application provides a simulation testing device for autonomous driving, comprising:
[0013] The static parsing module is used to perform static parsing on the simulated driving scene to obtain static information of the scene elements;
[0014] The dynamic parsing module is used to perform dynamic parsing processing on the simulated driving scene to obtain the dynamic information of the scene elements;
[0015] The file structure determination module is used to determine a predefined file structure that conforms to the simulator file format; wherein, the predefined file structure defines entity nodes and storyboards;
[0016] The generation module is used to map the static information of the scene elements to the node description of the entity nodes in the predefined file structure, and to map the dynamic information of the scene elements to the story description of the storyboard in the predefined file structure, so as to obtain a scene description file based on the predefined file structure.
[0017] The import module is used to import the scene description file into the simulator for simulation testing.
[0018] This application provides an electronic device, including:
[0019] Memory, used to store executable instructions;
[0020] The processor is used to implement the autonomous driving simulation test method provided in this application when executing executable instructions stored in the memory.
[0021] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the autonomous driving simulation test method provided in this application.
[0022] This application provides a computer program product including executable instructions for implementing the autonomous driving simulation test method provided in this application when executed by a processor.
[0023] This application has the following beneficial effects:
[0024] This application performs static analysis on a simulated driving scene to obtain static information of scene elements; performs dynamic analysis on the simulated driving scene to obtain dynamic information of scene elements; determines a predefined file structure conforming to the simulator file format; wherein, the predefined file structure defines entity nodes and storyboards; maps the static information of scene elements to node descriptions of entity nodes in the predefined file structure, and maps the dynamic information of scene elements to story descriptions of storyboards in the predefined file structure, obtaining a scene description file based on the predefined file structure; and imports the scene description file into the simulator for simulation testing. On the one hand, the scene description file includes both static and dynamic information of scene elements, which can accurately and comprehensively represent the simulated driving scene, realizing an effective conversion from the simulated driving scene to the scene description file; on the other hand, the scene description file is generated based on the predefined file structure, and the predefined file structure conforms to the simulator file format, which means that the scene description file can be directly imported into the simulator for use without additional format conversion steps, improving the simplicity and success rate of simulation testing. Attached Figure Description
[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1 This is a schematic diagram of the architecture of the autonomous driving simulation test system provided in the embodiments of this application;
[0027] Figure 2 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;
[0028] Figure 3A This is a first flowchart illustrating the simulation testing method for autonomous driving provided in this application embodiment;
[0029] Figure 3B This is a second flowchart illustrating the simulation testing method for autonomous driving provided in this application embodiment;
[0030] Figure 4 This is a flowchart illustrating the scene transition section provided in an embodiment of this application. Detailed Implementation
[0031] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0032] In the following description, references to "some embodiments" describe a subset of all possible embodiments; however, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. In the following description, the term "a plurality of" means at least two.
[0033] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.
[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0035] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for simulating autonomous driving. It can automatically convert simulated driving scenarios into scene description files conforming to the simulator file format for simulation testing, thereby improving the simplicity and success rate of simulation testing. The following describes exemplary applications of the electronic device provided in this application. The electronic device provided in this application can be implemented as various types of terminal devices or as a server.
[0036] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the autonomous driving simulation test system 100 provided in this application embodiment. The terminal device 400 is connected to the server 200 through the network 300, wherein the network 300 can be a wide area network, a local area network, or a combination of both. The autonomous driving simulation test method provided in this application embodiment can be implemented by the terminal device 400, by the server 200, or by the terminal device 400 and the server 200 working together; there is no limitation on this.
[0037] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server. Terminal device 400 may be a computer, vehicle terminal, etc., but is not limited thereto. Terminal device 400 and server 200 may be directly or indirectly connected via wired or wireless communication, and this application embodiment does not impose any restrictions.
[0038] Taking the example of a terminal device provided in this application embodiment, it can be understood that in the case where the electronic device is a server, Figure 2 Some parts of the structure shown (such as the user interface, presentation module, and input processing module) can be omitted. See also Figure 2 , Figure 2 This is a schematic diagram of the structure of the terminal device 400 provided in the embodiments of this application. Figure 2 The terminal device 400 shown includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.
[0039] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.
[0040] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.
[0041] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.
[0042] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.
[0043] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.
[0044] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;
[0045] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.
[0046] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;
[0047] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.
[0048] In some embodiments, the autonomous driving simulation testing device provided in this application can be implemented in software. Figure 2 A simulation test apparatus 455 for autonomous driving, stored in memory 450, is shown. This apparatus can be software in the form of programs and plug-ins, and includes the following software modules: a static parsing module 4551, a dynamic parsing module 4552, a file structure determination module 4553, a generation module 4554, and an import module 4555. These modules are logically connected and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.
[0049] The simulation testing method for autonomous driving provided in this application will be described in conjunction with exemplary applications and implementations of the electronic devices provided in the embodiments of this application.
[0050] See Figure 3A , Figure 3A This is a flowchart illustrating an autonomous driving simulation testing method provided in this application embodiment. The autonomous driving simulation testing method provided in this application embodiment can be implemented by electronic devices, such as terminal devices and / or servers, and will be combined with... Figure 3A The steps shown are explained.
[0051] In step 101, static analysis is performed on the simulated driving scene to obtain static information of the scene elements.
[0052] Here, the constructed simulated driving scenario undergoes static analysis to obtain the static information of the scenario elements. Scenario elements refer to entities involved in traffic, such as vehicles, pedestrians, cones, and traffic lights. The static information of scenario elements refers to their static attributes at a specific point in time, such as type and size. It is worth noting that a simulated driving scenario typically includes multiple scenario elements.
[0053] In some embodiments, the static information of scene elements obtained by statically analyzing the simulated driving scene can be achieved in the following way: for any frame of the simulated driving scene, the following processing is performed: identify scene elements in any frame of the image; extract information from the scene elements in any frame of the image to obtain the static information of the scene elements in any frame of the image.
[0054] Typically, a simulated driving scenario is a dynamic driving process described by multiple frames of images. Here, we take any one frame (named the target image) as an example to illustrate the static analysis process. For instance, first, scene elements in the target image are identified (located), and then information is extracted from these scene elements to obtain their static information. In some embodiments, the above operations can be performed on each frame of the simulated driving scenario to exhaustively identify all scene elements in the simulated driving scenario.
[0055] The above method uses computer vision technology, which can accurately identify scene elements in simulated driving scenarios and extract sufficient information.
[0056] In some embodiments, the above-mentioned identification of scene elements in any frame of an image can be achieved by performing any of the following processes: semantic segmentation of any frame of an image to obtain multiple image regions; wherein each image region corresponds to a scene element; object detection of any frame of an image to obtain scene elements in any frame of an image.
[0057] Here, two methods are provided for identifying scene elements in a target image:
[0058] 1) Semantic segmentation of the target image yields multiple image regions, each corresponding to a scene element. Semantic segmentation aims to classify each pixel in the target image to determine its corresponding scene element. The resulting image regions display the semantics of the corresponding scene element at the pixel level, thereby achieving fine-grained segmentation and in-depth understanding of the target image. In some embodiments, semantic segmentation can be implemented using algorithms such as grayscale segmentation and conditional random fields, or it can be implemented using deep learning algorithms, such as semantic segmentation models like U-Net and DeepLab. Based on the semantic segmentation, information can be extracted from each obtained image region to obtain the static information (such as size) of the scene element corresponding to that region. Through semantic segmentation, different scene elements in the target image can be accurately distinguished, providing rich spatial information, and it also has good adaptability to complex target images.
[0059] 2) Perform object detection on the target image to obtain scene elements within the target image. Object detection aims to detect and locate scene elements in the target image and provide a bounding box for each scene element. In some embodiments, object detection can be implemented using deep learning algorithms, such as object detection models like YOLO, SSD, and Faster R-CNN. Based on the object detection, information can be extracted from the bounding boxes of the scene elements to obtain their static information (e.g., size). Object detection enables fast and accurate identification of scene elements in the target image, while also exhibiting good robustness to interference factors such as occlusion and lighting changes in the target image.
[0060] In step 102, the simulated driving scene is dynamically analyzed to obtain dynamic information of the scene elements.
[0061] Here, the simulated driving scene is dynamically analyzed to obtain the dynamic information of the scene elements. Unlike static information, dynamic information describes the changes of scene elements over time and is used to characterize the dynamic changes of scene elements.
[0062] In some embodiments, the above-mentioned dynamic analysis and processing of the simulated driving scene to obtain the dynamic information of the scene elements can be achieved in the following way: the scene elements in the simulated driving scene are dynamically statistically analyzed according to the time axis to obtain the dynamic information of the scene elements.
[0063] Here, in each frame of the simulated driving scene, information such as the coordinates, speed, and direction of the scene elements can be obtained. Therefore, the scene elements in multiple frames can be dynamically statistically analyzed in order from front to back along the time axis to obtain the dynamic information of the scene elements. This dynamic information of the scene elements can characterize the dynamic changes that occur in the scene elements as the time axis progresses.
[0064] In step 103, a predefined file structure conforming to the simulator file format is determined; wherein, the predefined file structure defines entity nodes and storyboards.
[0065] Here, the simulator file format refers to the file format applicable (supported) by the simulator (or simulation software). This application embodiment does not limit the simulator file format used; for example, it could be the OpenScenario format, which is a format based on extensible markup language (XML). In this application embodiment, a predefined file structure conforming to the simulator file format can be determined. The predefined file structure can be manually defined. The predefined file structure defines entity nodes and a storyboard. Entity nodes refer to entities in the simulated driving scenario, and the storyboard refers to a complete scene description, such as "who does what, when, and where." It is worth noting that in addition to defining entity nodes and a storyboard, the predefined file structure also defines other necessary information, such as a file header and parameter declarations.
[0066] In step 104, the static information of scene elements is mapped to the node description of entity nodes in the predefined file structure, and the dynamic information of scene elements is mapped to the story description of storyboards in the predefined file structure, thus obtaining a scene description file based on the predefined file structure.
[0067] The predefined file structure is only a skeleton and does not contain actual content. In this embodiment, the static information of scene elements is mapped to the node description of entity nodes in the predefined file structure, and the dynamic information of scene elements is mapped to the story description of storyboards in the predefined file structure, so as to obtain a scene description file based on the predefined file structure.
[0068] In some embodiments, the above-mentioned mapping of static information of scene elements to node descriptions of entity nodes in a predefined file structure can be achieved by performing information filling processing on entity nodes in the predefined file structure based on the static information of scene elements to obtain node descriptions of entity nodes in the predefined file structure.
[0069] After obtaining the static information of the scene elements, information is populated into the entity nodes in the predefined file structure based on this static information. This means using the static information of the scene elements as the attribute values of the entity nodes (the attributes of the entity nodes are already defined in the predefined file structure), and the entity node attributes and their corresponding values are collectively referred to as the node description of the entity node. For example, if an entity node has a dimension attribute, the corresponding attribute value would be 4718mm × 1802mm × 1471mm. This information population method accurately maps scene elements to entity nodes, ensuring the accuracy of the entity node descriptions.
[0070] In some embodiments, the above-mentioned mapping of dynamic information of scene elements to story descriptions in a predefined file structure can be achieved in the following way: the dynamic information of scene elements is described and processed according to the story structure of the storyboard in the predefined file structure to obtain the behavior, stage, event and action of the scene elements; the storyboard in the predefined file structure is populated with information according to the behavior, stage, event and action of the scene elements to obtain the story description of the storyboard in the predefined file structure.
[0071] Here, the predefined file structure contains a story structure with a storyboard. Based on this story structure, the dynamic information of scene elements can be described and processed to obtain the behavior, stages, events, and actions of the scene elements. A behavior represents a complete sequence of actions, defining when it starts and ends, and includes multiple stages. A stage consists of one or more events. An event defines the triggering conditions and the actions to be performed. Actions represent specific operations, such as changing speed, changing lanes, or waiting. Then, based on the behavior, stages, events, and actions of the scene elements, the storyboard in the predefined file structure is populated with information to obtain the story description of the storyboard in the predefined file structure. This method ensures that the obtained story description conforms to the story structure and accurately describes the events occurring in the simulated driving scenario.
[0072] In step 105, the scene description file is imported into the simulator for simulation testing.
[0073] Here, the obtained scene description file is imported into the simulator for simulation testing. Since the simulator's file format is the same as the simulator's own, it ensures that the simulator is compatible with the scene description file, meaning the scene description file can run normally in the simulator.
[0074] In some embodiments, before importing the scene description file into the simulator for simulation testing, the autonomous driving simulation testing method further includes: performing format verification on the scene description file according to the simulator file format; the above-mentioned importing of the scene description file into the simulator for simulation testing can be achieved in such a way that when the scene description file format verification passes, the scene description file is imported into the simulator for simulation testing.
[0075] Here, after generating the scene description file, its format can be validated according to the simulator's file format. For example, if the simulator file format is OpenScenario, an XML validation tool or library (such as lxml) can be used to validate the scene description file, essentially verifying whether it conforms to the OpenScenario XML Schema specification. When the scene description file fails the format validation, an alarm can be triggered based on the content of the failed file, prompting relevant personnel to perform manual repairs. When the scene description file passes the format validation, it can be imported into the simulator for simulation testing, thereby improving the usability of the scene description file and the success rate of simulation tests.
[0076] like Figure 3A As shown, this embodiment of the application performs static parsing processing on the simulated driving scene to obtain static information of scene elements; performs dynamic parsing processing on the simulated driving scene to obtain dynamic information of scene elements; determines a predefined file structure that conforms to the simulator file format; wherein, the predefined file structure defines entity nodes and storyboards; maps the static information of scene elements to node descriptions of entity nodes in the predefined file structure, and maps the dynamic information of scene elements to story descriptions of storyboards in the predefined file structure, obtaining a scene description file based on the predefined file structure; and imports the scene description file into the simulator for simulation testing. On the one hand, the scene description file includes both static and dynamic information of scene elements, which can accurately and comprehensively represent the simulated driving scene, realizing an effective conversion from the simulated driving scene to the scene description file; on the other hand, the scene description file is generated based on the predefined file structure, and the predefined file structure conforms to the simulator file format, which means that the scene description file can be directly imported into the simulator for use without additional format conversion steps, thereby improving the simplicity and success rate of simulation testing.
[0077] In some embodiments, see Figure 3B , Figure 3B This is a flowchart illustrating an autonomous driving simulation testing method provided in an embodiment of this application, based on... Figure 3A Before step 101, steps 201 to 202 can also be performed.
[0078] In step 201, the generative adversarial network is trained based on sample data; wherein, the sample data includes sample driving scenarios and semantic data of sample driving scenarios; the generative adversarial network includes a generator and a discriminator.
[0079] Here, sample data can be obtained and used to train Generative Adversarial Networks (GANs), also known as Generative Adversarial Models.
[0080] It's worth noting that the sample data includes sample driving scenarios and their semantic data. Sample driving scenarios can be real driving scenarios or historically generated simulated driving scenarios (the reasonableness of simulated driving scenarios can be judged manually, and only those deemed reasonable are used as sample driving scenarios). The semantic data of the sample driving scenarios represents the semantics within them, enabling the generative adversarial network to fully understand the scenarios. This semantic data can include static and dynamic semantic data. Static semantic data distinguishes different scene elements within the sample driving scenario; for example, labeling the preceding vehicle as number 0, the pedestrian as number 1, and the cone as number 2. Dynamic semantic data represents the actions performed by scene elements, such as the preceding vehicle cutting in, cutting out, or turning. The semantic data of the sample driving scenarios can be manually labeled to ensure its accuracy.
[0081] It's worth noting that Generative Adversarial Networks (GANs) consist of a generator and a discriminator. During model training, the generator aims to generate fake data to deceive the discriminator as much as possible, while the discriminator aims to accurately distinguish between real and fake data. By training the GAN, the generator's data generation capability can be effectively enhanced.
[0082] In step 202, a simulated driving scenario is generated by the generator in the trained generative adversarial network.
[0083] After completing the model training phase, the model inference phase can begin. The model inference phase utilizes only the generator in the trained generative adversarial network (GAN). Specifically, the generator in the trained GAN generates simulated driving scenarios and corresponding semantic data, which are then used in subsequent steps. In some embodiments, the simulated driving scenarios and corresponding semantic data generated by the generator in the trained GAN can be used to construct new sample data.
[0084] In some embodiments, the above-mentioned training of the generative adversarial network based on sample data can be achieved by: generating comparison data through a generator in the generative adversarial network; wherein the comparison data includes a driving scene to be compared and semantic data of the driving scene to be compared; performing discrimination processing on the sample data and the comparison data through a discriminator in the generative adversarial network to obtain a discrimination result; calculating the loss values corresponding to the generator and the discriminator respectively based on the discrimination result, and performing adversarial training on the generator and the discriminator based on the loss values corresponding to the generator and the discriminator respectively.
[0085] Here, we illustrate the model training phase with an example. First, the generator in the generative adversarial network generates comparison data (i.e., the fake data mentioned above). This comparison data includes the driving scenario to be compared and its semantic data. Then, the sample data and the comparison data are input into the discriminator, which performs discrimination processing to obtain a discrimination result. This result includes the probability that the sample data is true and the probability that the comparison data is true. Next, based on the discrimination result, the loss values for the generator and discriminator are calculated, and adversarial training is performed on both the generator and discriminator based on these loss values. Through this adversarial training, the generator's data generation ability becomes increasingly stronger, meaning the generated data becomes closer to the sample data. Simultaneously, the discriminator's data discrimination ability becomes increasingly stronger, meaning it can more accurately distinguish between the sample data and the comparison data. Ultimately, the generator and discriminator reach a steady state, completing the entire model training phase.
[0086] In some embodiments, the generation of comparison data by a generator in a generative adversarial network can be achieved by performing any of the following processes: generating random noise vectors using a generator in a generative adversarial network to obtain comparison data; or generating scene constraint parameters using a generator in a generative adversarial network to obtain comparison data.
[0087] Here, during the model training phase, the generator's input can be a random noise vector or scene constraint parameters. The scene constraint parameters are essentially constraints that ensure the generated driving scenario conforms to them. These constraints can include at least one of the following: lane parameters (e.g., number of vehicles, lane length and width), environmental parameters (e.g., day or night, rain or sunshine), or events (e.g., the time at which the vehicle accelerates). Random noise vectors offer greater randomness and are suitable for scenarios requiring greater diversity in driving scenarios. Scene constraint parameters provide stronger constraints and are suitable for scenarios requiring simulation testing of specific driving scenarios (e.g., simulating a three-lane driving scenario). The scene constraint parameters correspond to sample driving scenarios (i.e., the sample driving scenarios satisfy the constraints represented by the scene constraint parameters). During model training, sample driving scenarios corresponding to various scene constraint parameters can be prepared. This allows the generator to learn how to accurately generate driving scenarios based on the scene constraint parameters, and ultimately supports user-defined scene constraint parameters during the model inference phase.
[0088] It's worth noting that the generator's input during the model inference phase is the same as its input during the model training phase. For example, if the generator's input during model training is a random noise vector, then the generator's input during model inference will also be a random noise vector. Similarly, if the generator's input during model training is scene constraint parameters, then the generator's input during model inference will also be scene constraint parameters. This ensures that the data generation capabilities learned by the generator during model training can be effectively applied during model inference.
[0089] In some embodiments, after generating a simulated driving scene through a generator in a trained generative adversarial network, the simulation testing method for autonomous driving further includes: optimizing the simulated driving scene; wherein the optimization process includes at least one of image enhancement and illumination optimization.
[0090] Here, after generating a simulated driving scene using a generator in a trained generative adversarial network, the simulated driving scene can be optimized to improve its realism. This optimization includes at least one of the following:
[0091] 1) Image enhancement. For example, image enhancement includes at least one of contrast enhancement, sharpening, and color correction. Contrast enhancement is used to adjust the contrast in the simulated driving scene, making the bright and dark areas more distinct; sharpening is used to enhance the edge details in the simulated driving scene, making road signs, vehicle outlines, etc. clearer, thereby improving the simulation accuracy of the simulated driving scene; color correction is used to adjust the color balance of the simulated driving scene, making the colors more realistic and natural, avoiding distortion or excessive vibrancy.
[0092] 2) Lighting optimization. For example, lighting optimization includes at least one of dynamic lighting simulation, shadow processing, reflection processing, and global illumination processing. Dynamic lighting simulation is used to dynamically adjust the intensity and direction of light in the simulated driving scene according to changes in time, weather, and geographical location to simulate a realistic lighting environment. Shadow processing is used to add shadow effects to scene elements in the simulated driving scene to enhance the sense of three-dimensionality and make the simulated driving scene more realistic. Reflection processing is used to simulate the reflection effects of scene elements such as road surfaces and vehicles, especially important on roads with strong sunlight or wet surfaces. Global illumination processing is used to use global illumination technology to ensure that the lighting effects in the simulated driving scene are consistent and natural, avoiding sudden or inconsistent lighting.
[0093] like Figure 3B As shown, this embodiment trains a generative adversarial network (GAN) based on sample data; the sample data includes sample driving scenarios and semantic data of the sample driving scenarios; the GAN includes a generator and a discriminator; and the generator in the trained GAN generates simulated driving scenarios. This embodiment automatically generates simulated driving scenarios using a trained generator, avoiding the human and time costs associated with manually constructing simulated driving scenarios, while also improving the quality of the simulated driving scenarios and thus enhancing the effectiveness of simulation testing.
[0094] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario. The embodiments of this application mainly include two parts: scene generation and scene transformation, which will be described separately below.
[0095] I. Scene Generation Section.
[0096] In the scene generation section, generative adversarial networks are used to intelligently generate realistic simulated driving scenarios, including scene elements such as roads, vehicles, and buildings. The scene generation section can be implemented through the following steps.
[0097] 1) Data preparation: Collect real driving scenarios and corresponding labeled semantic data.
[0098] 2) Model Training: The generator and discriminator are trained through adversarial training. Model training consists of two main parts: generator training and discriminator training, which will be explained separately below.
[0099] a) Generator training refers to the generator generating fake images (i.e., images in the driving scene to be compared) and attempting to deceive the discriminator into believing that the fake images generated by the generator are real. The training objective of the generator is to maximize the probability that the discriminator considers the fake images to be real, and the loss function is as follows:
[0100] LG =-log(D(G(z)))
[0101] Among them, L G denoted by ; z represents the generator loss value; represents the generator input (random noise vector or scene constraint parameters); G(z) represents the fake image generated by the generator; D(G(z)) represents the probability that the discriminator considers the fake image generated by the generator to be real.
[0102] b) Discriminator training refers to the process where the discriminator receives real images (i.e., images from real driving scenarios) and fake images, and learns to distinguish between real and fake images. The discriminator's loss function is as follows:
[0103] L D =-(log(D(x))+log(1-D(G(z))))
[0104] Among them, L D denoted by , where x represents the real image; and D(x) represents the probability that the discriminator considers the real image to be real.
[0105] 3) Model inference: Here, the trained generator generates a simulated driving scenario based on the input random noise vector or scene constraint parameters.
[0106] 4) Optimization processing: Image enhancement and lighting optimization are performed on the simulated driving scene to improve the realism and quality of the simulated driving scene.
[0107] II. Scene Transition Section.
[0108] For ease of understanding, taking the OpenScenario format as an example, the scene conversion section is used to convert the simulated driving scene into an OpenScenario file that conforms to the OpenScenario format. As an example, embodiments of this application provide... Figure 4 The diagram illustrates a process for scene transformation. For the generated simulated driving scene, the static and dynamic information of scene elements are first parsed from the scene. Then, scene description information conforming to the OpenScenario format (including node descriptions of entity nodes and story descriptions of storyboards) is generated and finally exported as an OpenScenario file. The following steps provide more detailed examples.
[0109] 1) Define the OpenScenario structure.
[0110] Here, we create the basic structure of the OpenScenario file (corresponding to the predefined file structure above). OpenScenario files are XML-based, therefore they need to be defined according to the OpenScenario XML Schema. The example implementation steps are as follows:
[0111] ① File header: Contains version information and namespace declarations, such as specifying that the matching OpenScenario version is OpenScenario 2.0. This helps the simulator correctly parse and execute the instructions in the file. Additionally, the OpenScenario specification uses XML Schema (XSD) files to define its data structures, therefore the namespace and XSD location need to be specified.
[0112] ② Parameter Declaration: Defines global parameters in the simulated driving scenario. Global parameters are used to define and manage common variables throughout the simulated driving scenario. Global parameters can be shared among multiple entity nodes or behaviors and are used to control various scene elements in the simulated driving scenario. For example, global parameters can be declared through... <parameterdeclarations>Labels are used for declaration, typically including types such as numeric, string, and boolean. For example, an integer parameter type can be defined in global parameters, so the number of vehicles, lane numbers, and cones in the simulated driving scenario can be described using the integer parameter type. A floating-point parameter type can be defined in global parameters, so the vehicle speed, acceleration, and heading angle in the simulated driving scenario can be described using the floating-point parameter type.
[0113] ③ Entity Node Definition: Define various entity nodes that may appear in the simulated driving scenario (define which entities, how they operate, and what they obtain), such as vehicles, pedestrians, objects, environment entities, and traffic lights. By defining various entity nodes, all traffic participants and their detailed attributes in the OpenScenario file are obtained. Entity nodes can be defined using Python's xml.etree.ElementTree. The definitions of entity nodes are shown in Table 1.
[0114] Table 1 Entity Node Definition Table
[0115]
[0116] ④ Storyboard Definition: Used to define the actions and interactions of various entity nodes in the simulated driving scenario. The storyboard refines the simulated driving scenario layer by layer through Acts, Maneuvers, Events, and Actions. An Act represents a complete sequence of actions, defining when it begins and ends; an Act includes multiple Maneuvers; a Maneuver consists of one or more Events; an Event defines the triggering conditions and the actions to be performed; and an Action represents the specific operation. For example, the Act might be "The vehicle enters the ramp at second 0 and enters the main road at second 2, completing a merging action"; the merging action includes three Maneuvers: the exit ramp Maneuver, the acceleration Maneuver, and the entry into the main road Maneuver; an Event might be an acceleration action performed at second 1 (time-triggered) or at the edge of the exit ramp (position-triggered); Actions might include changing speed, changing lanes, or waiting.
[0117] 2) Analyze the simulated driving scenario.
[0118] On the one hand, computer vision techniques (such as semantic segmentation and / or object detection) are used to analyze images in simulated driving scenarios to identify different scene elements, and then static information of the scene elements, such as type and size, is extracted.
[0119] On the other hand, the scene elements in the simulated driving scenario are dynamically statistically analyzed according to the time axis to obtain the dynamic information of the scene elements. The dynamic information is used to characterize the dynamic changes of the scene elements as the time axis progresses.
[0120] 3) Map scene elements.
[0121] On one hand, based on the relationship between scene elements and entity nodes in the OpenScenario structure, the static information of scene elements is mapped to the node descriptions of entity nodes. Essentially, this process involves populating the attributes of entity nodes. For example, vehicle position, speed, acceleration, and travel path are mapped to the Entities and Maneuver nodes in the OpenScenario structure; road type, length, width, and curvature are mapped to the RoadNetwork node; pedestrian position, speed, and walking path are mapped to the Entities and Pedestrian nodes; and traffic light position and status (red, yellow, green) are mapped to the TrafficSignal node.
[0122] On the other hand, the dynamic information of the scene elements is re-described according to the story structure of the storyboard to obtain the story description of the storyboard.
[0123] Then, the scene description information (including node descriptions of entity nodes and story descriptions of storyboards) is exported as an OpenScenario file conforming to the OpenScenario format.
[0124] 4) Format validation.
[0125] Here, the OpenScenario file is format-validated to ensure that it conforms to the OpenScenario XML Schema specification. This can be done using XML validation tools or libraries (such as lxml).
[0126] The embodiments of this application have at least the following technical effects:
[0127] 1) Generative adversarial networks are used to intelligently generate realistic driving simulation scenarios. Through learning and simulation of a large amount of real driving scenario data, the generated driving simulation scenarios have a high degree of realism and diversity, and can cover a variety of complex driving situations, providing more comprehensive data support for simulation testing.
[0128] 2) Allows users to dynamically set scenario constraint parameters according to their needs, such as road type, number of vehicles, weather conditions, etc., so as to simulate different driving scenarios and meet various simulation test requirements.
[0129] 3) It can automatically convert simulated driving scenarios into OpenScenario files that conform to the OpenScenario format, and then import the OpenScenario files into the simulator for simulation testing. The whole process does not require user operation, which can improve the simplicity and success rate of simulation testing.
[0130] The following continues to describe the exemplary structure of the autonomous driving simulation test device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the autonomous driving simulation test device 455 in the memory 450 may include: a static parsing module 4551, used to perform static parsing processing on the simulated driving scene to obtain static information of scene elements; a dynamic parsing module 4552, used to perform dynamic parsing processing on the simulated driving scene to obtain dynamic information of scene elements; a file structure determination module 4553, used to determine a predefined file structure that conforms to the simulator file format; wherein, the predefined file structure defines entity nodes and storyboards; a generation module 4554, used to map the static information of scene elements to node descriptions of entity nodes in the predefined file structure, and to map the dynamic information of scene elements to story descriptions of storyboards in the predefined file structure, to obtain a scene description file based on the predefined file structure; and an import module 4555, used to import the scene description file into the simulator for simulation testing.
[0131] In some embodiments, the dynamic parsing module 4552 is further configured to: dynamically analyze the scene elements in the simulated driving scenario according to the time axis to obtain the dynamic information of the scene elements.
[0132] In some embodiments, the static parsing module 4551 is further configured to: perform information filling processing on entity nodes in the predefined file structure based on the static information of scene elements to obtain node descriptions of entity nodes in the predefined file structure.
[0133] In some embodiments, the dynamic parsing module 4552 is further configured to: describe the dynamic information of scene elements according to the story structure of the storyboard in the predefined file structure to obtain the behavior, stage, event and action of the scene elements; and fill the storyboard in the predefined file structure with information according to the behavior, stage, event and action of the scene elements to obtain the story description of the storyboard in the predefined file structure.
[0134] In some embodiments, the autonomous driving simulation test device 455 further includes a format verification module, used to: verify the format of the scene description file according to the simulator file format; the import module 4554 is also used to: import the scene description file into the simulator for simulation testing when the scene description file format verification passes.
[0135] In some embodiments, the autonomous driving simulation testing device 455 further includes a training module for: training a generative adversarial network based on sample data; wherein the sample data includes sample driving scenarios and semantic data of the sample driving scenarios; the generative adversarial network includes a generator and a discriminator; the autonomous driving simulation testing device 455 further includes a generation module for: generating simulated driving scenarios through the generator in the trained generative adversarial network.
[0136] In some embodiments, the training module is further configured to: generate comparison data through a generator in a generative adversarial network; wherein the comparison data includes a comparison driving scenario and semantic data of the comparison driving scenario; perform discrimination processing on the sample data and the comparison data through a discriminator in a generative adversarial network to obtain a discrimination result; calculate the loss values corresponding to the generator and the discriminator respectively based on the discrimination result, and perform adversarial training on the generator and the discriminator based on the loss values corresponding to the generator and the discriminator respectively.
[0137] In some embodiments, the training module is further configured to: perform any of the following processes: generate random noise vectors using a generator in a generative adversarial network to obtain comparison data; or generate scene constraint parameters using a generator in a generative adversarial network to obtain comparison data.
[0138] In some embodiments, the autonomous driving simulation test apparatus 455 further includes a generation module for: optimizing the simulated driving scene; wherein the optimization process includes at least one of image enhancement and lighting optimization.
[0139] This application provides a computer program product or computer program, which includes executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, causing the electronic device to implement the autonomous driving simulation testing method described above in this application.
[0140] This application provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, they will cause the processor to implement the autonomous driving simulation test method provided in this application.
[0141] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.
[0142] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
[0143] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).
[0144] As an example, executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.
[0145] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.< / parameterdeclarations>
Claims
1. A simulation testing method for autonomous driving, characterized in that, include: Static analysis is performed on the simulated driving scene to obtain static information of the scene elements; The dynamic information of the scene elements is obtained by performing dynamic analysis on the simulated driving scene. Determine a predefined file structure that conforms to the simulator file format; wherein, the predefined file structure defines entity nodes and storyboards; The static information of the scene elements is mapped to the node description of the entity nodes in the predefined file structure, and the dynamic information of the scene elements is mapped to the story description of the storyboard in the predefined file structure, so as to obtain a scene description file based on the predefined file structure. Import the scenario description file into the simulator for simulation testing.
2. The method according to claim 1, characterized in that, The process of dynamically analyzing the simulated driving scenario to obtain the dynamic information of the scenario elements includes: The scene elements in the simulated driving scenario are dynamically statistically analyzed according to the time axis to obtain the dynamic information of the scene elements.
3. The method according to claim 1, characterized in that, The step of mapping the static information of the scene elements to the node descriptions of entity nodes in the predefined file structure includes: Based on the static information of the scene elements, information filling processing is performed on the entity nodes in the predefined file structure to obtain the node descriptions of the entity nodes in the predefined file structure.
4. The method according to claim 1, characterized in that, The step of mapping the dynamic information of the scene elements to the story description of the storyboard in the predefined file structure includes: The dynamic information of the scene elements is described and processed according to the story structure of the storyboard in the predefined file structure to obtain the behavior, stage, event and action of the scene elements; Based on the behavior, stage, event, and action of the scene elements, information filling processing is performed on the storyboard in the predefined file structure to obtain the story description of the storyboard in the predefined file structure.
5. The method according to claim 1, characterized in that, Before importing the scene description file into the simulator for simulation testing, the method further includes: The scene description file is format-validated according to the simulator file format. The step of importing the scene description file into the simulator for simulation testing includes: When the scene description file format is validated, the scene description file is imported into the simulator for simulation testing.
6. The method according to any one of claims 1 to 5, characterized in that, Before performing static analysis on the simulated driving scene to obtain static information of the scene elements, the method further includes: The generative adversarial network is trained based on sample data; wherein, the sample data includes sample driving scenarios and semantic data of the sample driving scenarios; the generative adversarial network includes a generator and a discriminator; Simulated driving scenarios are generated using a generator in a trained generative adversarial network.
7. The method according to claim 6, characterized in that, The training of the generative adversarial network based on sample data includes: Comparison data is generated using a generator in a generative adversarial network; wherein, the comparison data includes a driving scenario to be compared and semantic data of the driving scenario to be compared; The sample data and the data to be compared are processed by a discriminator in a generative adversarial network to obtain a discrimination result; Calculate the loss values corresponding to the generator and discriminator based on the discrimination results, and perform adversarial training on the generator and discriminator based on the loss values corresponding to the generator and discriminator.
8. The method according to claim 7, characterized in that, The generation of comparison data through a generator in a generative adversarial network includes: Perform any of the following processes: Random noise vectors are generated and processed using a generator in a generative adversarial network to obtain the data to be compared. The scene constraint parameters are generated and processed by the generator in the generative adversarial network to obtain the data to be compared.
9. The method according to claim 6, characterized in that, After generating the simulated driving scenario through the generator in the trained generative adversarial network, the method further includes: The simulated driving scenario is optimized; wherein the optimization includes at least one of image enhancement and lighting optimization.
10. A simulation testing device for autonomous driving, characterized in that, include: The static parsing module is used to perform static parsing on the simulated driving scene to obtain static information of the scene elements; The dynamic parsing module is used to perform dynamic parsing processing on the simulated driving scene to obtain the dynamic information of the scene elements; The file structure determination module is used to determine a predefined file structure that conforms to the simulator file format; wherein, the predefined file structure defines entity nodes and storyboards; The generation module is used to map the static information of the scene elements to the node description of the entity nodes in the predefined file structure, and to map the dynamic information of the scene elements to the story description of the storyboard in the predefined file structure, so as to obtain a scene description file based on the predefined file structure. The import module is used to import the scene description file into the simulator for simulation testing.
11. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method according to any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the method according to any one of claims 1 to 9 when executed by a processor.