Optimization method and device for automatic driving simulation test

Abstract

The technical scheme of the invention discloses an optimization method and device for an automatic driving simulation test, and the method comprises the steps: data collection: installing a data collection device, and collecting video data and LiDAR point cloud data; data preprocessing: performing denoising and filtering processing on LiDAR point cloud data, extracting video frames and labeling dynamic and static elements; and static background and dynamic foreground object separation: separating a dynamic object from the static background by applying a point cloud segmentation technology. And static background model generation: generating a high-fidelity static environment model by using the point cloud data. And generating a dynamic foreground object model and a time sequence track: generating a four-dimensional dynamic model in combination with time information, tracking a dynamic object and adding a timestamp. And plug-in development: developing a plug-in supporting a format, so that the traditional simulation software can import and display the high-fidelity model. And model integration and simulation: importing the high-fidelity model into simulation software, replacing an original environment, and adding dynamic elements through a plug-in interface.

Description

Technical Field

[0001] The technical solution of the present application relates to the technical field of autonomous driving simulation testing, and specifically, to an optimization method and device for autonomous driving simulation testing. Background Art

[0002] With the rapid development of autonomous driving technology, it has become crucial to ensure the safety and reliability of autonomous driving systems. Traditional testing methods, such as road testing, are realistic but costly and difficult to cover all possible driving scenarios. In addition, testing in certain extreme situations (such as extreme weather conditions or sudden accident scenarios) is difficult to conduct safely or effectively in reality. Simulation testing can not only simulate a variety of complex traffic scenarios in a virtual environment, but also evaluate the performance of autonomous driving systems in different environments by adjusting parameters. This method can accelerate the R&D process and reduce development costs, while also discovering and correcting system defects in a more systematic way. However, the effectiveness of simulation testing depends on the authenticity and real-time nature of scene rendering. Summary of the invention

[0003] The technical solution of the present application provides an optimization method and device for autonomous driving simulation testing, so as to overcome at least one technical problem existing in the prior art.

[0004] According to a first aspect of the technical solution embodiment of the present application, a method for optimizing an autonomous driving simulation test is provided, comprising:

[0005] Based on the data acquisition device, the video data and LiDAR point cloud data of the real scene that needs to be reconstructed are synchronously acquired; wherein the data acquisition device includes a vehicle-mounted camera and a laser radar, and the data acquisition device is installed on an experimental vehicle traveling in the real scene, the vehicle-mounted camera is used to acquire the video data, and the laser radar is used to acquire the LiDAR point cloud data;

[0006] Performing a video frame extraction operation on the video data to obtain a number of video frames, and performing data annotation on the number of video frames frame by frame based on motion characteristics of elements included in the video frames to distinguish dynamic foreground objects and static background elements included in each of the number of video frames; then, tracking dynamic objects in the video data, and then annotating the three-dimensional coordinate data and time stamp information of the dynamic foreground objects included in each video frame to obtain a number of annotated video frames;

[0007] Separating the dynamic objects from the static background in the LiDAR point cloud data to distinguish static and dynamic elements, and obtaining LiDAR point cloud data after point cloud segmentation;

[0008] Inputting the LiDAR point cloud data after point cloud segmentation and the several video frames after annotation into the 3DGS model for parallel training to obtain a 3DGS model of a static background and a 3DGS model of a dynamic foreground object;

[0009] The 3DGS model of the static background and the 3DGS model of the dynamic foreground object are integrated into the autonomous driving simulation software for testing the autonomous driving vehicle in the form of a plug-in.

[0010] Preferably, before performing a video frame extraction operation on the video data to obtain a number of video frames, the step includes:

[0011] The collected LiDAR point cloud data is subjected to denoising and filtering, specifically comprising: first, identifying and removing outliers or noise points in the LiDAR point cloud data; then, using statistical filtering or geometric filtering to smooth the LiDAR point cloud data and retain key structural information in the point cloud; wherein statistical filtering is based on the characteristics of points in the neighborhood, the characteristics including the mean value or the median, to remove points that deviate from the normal range; and geometric filtering utilizes the spatial relationship between points to remove points that do not conform to the expected geometric shape.

[0012] Preferably, before performing a video frame extraction operation on the video data to obtain a number of video frames, the method further includes: performing hardware time synchronization on the vehicle-mounted camera and the laser radar, specifically including:

[0013] The vehicle-mounted camera and the lidar are synchronized with a common time source through an external global positioning system timing device or a high-precision clock built into the vehicle. Both the vehicle-mounted camera and the lidar record data based on the common time source, thereby ensuring the consistency of the data collection timestamp.

[0014] Preferably, the tracking of the dynamic object in the video data specifically includes:

[0015] S1, selecting a reference frame and adjacent frames, specifically comprising: first, selecting a frame from the video data as a reference frame, and then selecting one or more subsequent frames as adjacent frames;

[0016] S2, calculating pixel gradient, specifically including: for each pixel point in the reference frame and the adjacent frame, calculating its brightness gradient;

[0017] S3, matching feature points: using pixel gradient information, finding the corresponding relationship between the same feature points in the reference frame and the adjacent frames;

[0018] S4, estimating motion vector: based on the correspondence between feature points, estimate the motion vector of each pixel from the reference frame to the adjacent frame through an iterative optimization algorithm, wherein the vector includes displacement components of the pixel in the horizontal and vertical directions;

[0019] S5. Timestamp allocation: Combining the estimated motion vector and the shooting time interval of the video frame, time information is allocated to each pixel of the dynamic object to form a dynamic model with a time dimension.

[0020] Preferably, the tracking of the dynamic object in the video data specifically includes:

[0021] S1. Feature detection: In each frame of the video data, a feature detection algorithm is applied to extract stable feature points; for the detected feature points, a descriptor is calculated, where the descriptor is used to describe a vector of image content around the feature points;

[0022] S2, feature description: Match feature points by comparing feature descriptors between two or more consecutive frames;

[0023] S3, geometric verification: In order to further eliminate false matches, the random sampling consensus algorithm RANSAC method is used to estimate the transformation matrix based on the matching point pairs, and only the matching point pairs that meet the geometric constraints are retained;

[0024] S4, trajectory tracking: through cross-frame feature matching and geometric verification, the trajectory of feature points over time is established, and a unique identifier is assigned to each continuously tracked feature point sequence to indicate that it belongs to the same dynamic object;

[0025] S5. Timestamp annotation: Whenever a feature point is successfully matched in a frame and assigned to a trajectory, the precise time information of the frame is recorded as a timestamp. In this way, as time goes by, the trajectory of each dynamic object is associated with a series of timestamps, forming a dynamic object motion trajectory with time information. The above process is used to track dynamic objects in the video and add a timestamp to the motion trajectory of each object.

[0026] Preferably, the step of integrating the 3DGS model of the static background and the 3DGS model of the dynamic foreground object into the autonomous driving simulation software for testing the autonomous driving vehicle in the form of a plug-in specifically includes:

[0027] S1. Model import, specifically including: importing the 3DGS model of the static background generated by the 3D Gaussian sputtering technology into the autonomous driving simulation software;

[0028] S2, scene replacement and fine-tuning, specifically including: replacing the basic model in the original simulation environment of the autonomous driving simulation software with the 3DGS model of the static background, and adjusting the 3DGS model of the static background according to specific simulation requirements to ensure that it fits the map layout and proportion of the autonomous driving simulation software;

[0029] S3, dynamic element integration, specifically includes: in the simulation scene that has integrated the 3DGS model of the static background, through the developed plug-in, allowing the user to add and run the dynamic scene elements in the 3DGS model of the dynamic foreground object.

[0030] According to a second aspect of the technical solution embodiment of the present application, there is provided an optimization device for autonomous driving simulation testing, comprising:

[0031] A data acquisition module, used for synchronously acquiring video data and LiDAR point cloud data of a real scene that needs to be reconstructed based on a data acquisition device; wherein the data acquisition device includes a vehicle-mounted camera and a laser radar, and the data acquisition device is installed on an experimental vehicle traveling in the real scene, the vehicle-mounted camera is used to acquire the video data, and the laser radar is used to acquire the LiDAR point cloud data;

[0032] A video data processing module is used to perform a video frame extraction operation on the video data to obtain a number of video frames, and to perform data annotation on the number of video frames frame by frame based on motion characteristics of elements included in the video frames to distinguish dynamic foreground objects and static background elements included in each of the number of video frames; then, track dynamic objects in the video data, and then annotate the three-dimensional coordinate data and time stamp information of the dynamic foreground objects included in each video frame to obtain a number of annotated video frames;

[0033] A point cloud segmentation module is used to separate the dynamic objects from the static background in the LiDAR point cloud data to distinguish static and dynamic elements, and obtain the LiDAR point cloud data after point cloud segmentation;

[0034] A 3DGS training module is used to input the LiDAR point cloud data after point cloud segmentation and the several video frames after annotation into a 3DGS model for parallel training to obtain a 3DGS model of a static background and a 3DGS model of a dynamic foreground object;

[0035] A plug-in integration module is used to integrate the 3DGS model of the static background and the 3DGS model of the dynamic foreground object into the autonomous driving simulation software for testing the autonomous driving vehicle in the form of a plug-in.

[0036] One embodiment of this specification can achieve at least the following beneficial effects:

[0037] 1. High-fidelity scene rendering: By combining 3D Gaussian sputtering 3DGS technology and time dimension information, the technical solution of this application can generate extremely detailed static background and dynamic foreground object models, greatly improving the realism of the simulation scene and making the test environment closer to the complexity of the real world.

[0038] 2. Improve model generation efficiency and automation: By utilizing automated data processing processes, including point cloud segmentation and dynamic object tracking, the technical solution of this application significantly reduces the workload of manual modeling, improves the efficiency of model generation, enables the system to adapt to environmental changes more quickly, and ensures the timeliness of the model.

[0039] 3. Taking into account both real-time and dynamic scene simulation: The optimized rendering method ensures that real-time requirements are met while providing high-precision models. This is crucial for simulating rapidly changing traffic conditions in autonomous driving, such as emergency avoidance and response to unexpected accidents, which enhances the effectiveness of the test.

[0040] 4. Enhanced simulation platform flexibility and scenario customization capabilities: The proposed plug-in solution (applicable to software such as EU5 or Unity) not only facilitates the integration of high-fidelity models into existing simulation platforms, but also provides users with an interactive interface to adjust and customize dynamic scenarios, making it easy to create diverse and targeted test cases based on test requirements.

[0041] To sum up, the technical solution of the technical solution of the present application not only improves the visual realism and dynamic interaction capabilities of autonomous driving simulation tests, but also effectively improves the efficiency and accuracy of simulation tests through automated tools and real-time rendering technology, providing a more powerful and reliable testing environment for the development and verification of autonomous driving systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the technical solution of the present application or the technical solution in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the technical solution of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A flowchart of an optimization method for an autonomous driving simulation test provided in an embodiment of this specification;

[0044] Figure 2 A schematic diagram of the overall process of an optimization method for an autonomous driving simulation test provided in an embodiment of this specification;

[0045] Figure 3 An example 1 for describing a dynamic foreground object and a static background in an optimization method for an autonomous driving simulation test provided in an embodiment of this specification;

[0046] Figure 4 Example 2 for describing dynamic foreground objects and static backgrounds in an optimization method for an autonomous driving simulation test provided in an embodiment of this specification;

[0047] Figure 5 Example 3 for describing dynamic foreground objects and static backgrounds in an optimization method for an autonomous driving simulation test provided in an embodiment of this specification;

[0048] Figure 6 Example 4 for describing dynamic foreground objects and static backgrounds in an optimization method for an autonomous driving simulation test provided in an embodiment of this specification;

[0049] Figure 7 Example 5 for describing dynamic foreground objects and static backgrounds in an optimization method for an autonomous driving simulation test provided in an embodiment of this specification;

[0050] Figure 8 Example 6 for describing dynamic foreground objects and static backgrounds in an optimization method for an autonomous driving simulation test provided in an embodiment of this specification;

[0051] Fig. 9 Example 7 for describing dynamic foreground objects and static backgrounds in an optimization method for an autonomous driving simulation test provided in an embodiment of this specification;

[0052] Fig.10 Example 8 for describing dynamic foreground objects and static backgrounds in an optimization method for an autonomous driving simulation test provided in an embodiment of this specification;

[0053] Fig.11 Example 9 for describing dynamic foreground objects and static backgrounds in an optimization method for an autonomous driving simulation test provided in an embodiment of this specification;

[0054] Fig.12 An example 10 for describing dynamic foreground objects and static backgrounds in an optimization method for an autonomous driving simulation test provided in an embodiment of this specification;

[0055] Fig.13 A schematic diagram of the structure of an optimization device for autonomous driving simulation testing provided in an embodiment of this specification. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of one or more embodiments of this specification clearer, the technical solutions of one or more embodiments of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of one or more embodiments of this specification.

[0057] It should be understood that although the terms first, second, third, etc. may be used in this application document to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other.

[0058] Figure 1 The following is a flow chart of an optimization method for an autonomous driving simulation test applied to a service platform provided in an embodiment of the present specification. From a program perspective, the execution subject of the process can be a program installed on a server of the service platform.

[0059] like Figure 1 As shown, the process may include the following steps.

[0060] Step 102: Synchronously collect video data and LiDAR point cloud data of a real scene that needs to be reconstructed based on a data acquisition device; wherein the data acquisition device includes a vehicle-mounted camera and a laser radar, and the data acquisition device is installed on an experimental vehicle traveling in the real scene, the vehicle-mounted camera is used to collect the video data, and the laser radar is used to collect the LiDAR point cloud data.

[0061] In this step, the on-board cameras are usually installed on the front, back and sides of the experimental vehicle to capture visual information of the surrounding environment from multiple angles. The laser radar (LiDAR) is usually installed on the roof or high position of the front of the vehicle. Such a position helps to obtain a wider horizontal field of view and a longer detection distance, so as to more comprehensively collect the three-dimensional spatial point cloud data around the vehicle, including road conditions, obstacle locations and contour information of the surrounding environment.

[0062] In the technical solution of the present application, LiDAR (Light Detection and Ranging) point cloud data refers to a collection of three-dimensional space points formed by emitting laser pulses through the LiDAR sensor and receiving the reflected signals, and calculating the three-dimensional space coordinates (X, Y, Z) of each point in the surrounding environment based on the round-trip time of light and the angle information of the emitted laser. These point cloud data can be used to construct a three-dimensional model of the environment, which contains precise location information of objects such as terrain, buildings, vehicles, pedestrians, etc., and is crucial for environmental perception, map construction and path planning in autonomous driving technology.

[0063] Video data and LiDAR data are collected simultaneously and used to build high-fidelity autonomous driving simulation test scenes. The correlation between the two data is reflected in the fact that they jointly provide complete information about the scene, that is, video data is captured by the on-board camera and mainly contains visual information of the scene, such as color, texture, and appearance characteristics of traffic participants. LiDAR (laser radar) data provides depth information of the scene, that is, the relative distance and spatial layout between objects, which is crucial for building accurate three-dimensional structures. The combined use of these two types of data, by matching and fusing the rich visual details of the video image with the precise spatial information of LiDAR, can comprehensively and accurately obtain static and dynamic information in the scene. Such data association not only ensures the high authenticity of the reconstructed scene, but also lays the foundation for subsequent steps such as separation of static background and dynamic foreground objects and model generation.

[0064] Step 104: performing a video frame extraction operation on the video data to obtain a number of video frames, and annotating the number of video frames frame by frame based on motion characteristics of elements included in the video frames to distinguish dynamic foreground objects and static background elements included in each of the number of video frames; then, tracking dynamic objects in the video data, and then annotating the three-dimensional coordinate data and timestamp information of the dynamic foreground objects included in each video frame to obtain a number of annotated video frames.

[0065] In this step, the frames extracted from the video data will be annotated to distinguish dynamic foreground objects (such as vehicles and pedestrians) from static backgrounds (such as buildings and roads). At the same time, in order to track dynamic objects in the video, deep learning or computer vision technology can be used. Specific methods include optical flow and feature point matching. The optical flow method can analyze the motion information of pixels between adjacent frames to determine the object's direction and speed of movement. Feature point matching is to find and match points with unique features between different frames to track the position changes of the object. The combination of these two methods can effectively track the motion trajectory of dynamic objects in the video.

[0066] In the technical solution of this application, the feature point matching method can be used to track dynamic objects in the video and mark the timestamp for each object. Specifically, the following steps can be followed: 1. Feature detection: First, in each frame of the video sequence, a feature detection algorithm (such as SIFT, SURF, ORB, etc.) is applied to extract stable feature points. These algorithms can identify features that remain unchanged even when the image scale, rotation or illumination changes. 2. Feature description: For the detected feature points, its descriptor is calculated, which is a vector describing the image content around the feature point. The descriptor should have good distinguishability, that is, similar feature points should have similar descriptors in different images, while different feature points should have obvious differences. 3. Feature matching: Between two or more consecutive frames, feature points are matched by comparing feature descriptors. Commonly used methods include nearest neighbor matching, ratio testing, etc. to ensure that the correct correspondence is found and false matches are filtered out. 4. Geometric verification: In order to further eliminate false matches, methods such as random sampling consensus algorithm RANSAC can be used to estimate the transformation matrix (such as translation, rotation) based on matching point pairs, and only matching point pairs that meet geometric constraints (such as coplanarity) are retained. 5. Trajectory tracking: Through cross-frame feature matching and geometric verification, the trajectory of feature points over time can be established, and a unique identifier is assigned to each continuously tracked sequence of feature points to indicate that it belongs to the same dynamic object. 6. Timestamp annotation: Whenever a feature point is successfully matched in a frame and assigned to a trajectory, the precise time information of the frame is recorded as a timestamp. In this way, over time, the trajectory of each dynamic object is associated with a series of timestamps, forming a dynamic object motion trajectory with time information. Through the above process, it is possible to use the feature point matching method to track dynamic objects in the video and accurately add a timestamp to the motion trajectory of each object, which is crucial for subsequent dynamic model generation and simulation testing.

[0067] At the same time, considering that the optical flow method is a computer vision technology used to analyze the changes in pixel intensity between consecutive frames in a video sequence, and then estimate the movement of objects in the image plane. In the technical solution of this application, the optical flow method can also be used to track dynamic objects in the video and add a timestamp to each object. The specific operation can be as follows:

[0068] 1. Select the reference frame and adjacent frames: First, select a frame from the video sequence as the reference frame, and then select one or more frames thereafter as adjacent frames. 2. Calculate pixel gradients: For each pixel in the reference frame and adjacent frames, calculate its brightness gradient (i.e., the direction and magnitude of brightness change). This step is to determine the trend of pixel movement. 3. Match feature points: Use pixel gradient information to find the correspondence between the reference frame and the same feature points in the adjacent frames. These feature points are usually areas with obvious brightness changes in the image, such as edges or texture changes. 4. Estimate motion vectors: Based on the correspondence between feature points, the motion vector (i.e., optical flow) of each pixel from the reference frame to the adjacent frame is estimated through an iterative optimization algorithm. This vector contains the displacement components of the pixel in the horizontal and vertical directions. 5. Timestamp allocation: Combine the estimated motion vector and the shooting time interval of the video frame to assign time information to each pixel of the dynamic object to form a dynamic model with a time dimension. Through the above steps, the technical solution of this application uses the optical flow method to achieve accurate estimation of the motion trajectory of dynamic foreground objects, which is crucial for generating high-fidelity autonomous driving simulation test scenes. Based on the above analysis, a three-dimensional coordinate sequence that changes over time, i.e., a dynamic trajectory, is constructed for each detected dynamic object. At the same time, a corresponding timestamp is assigned to each key point or frame on the trajectory to record the specific time information of the object appearing in the video. This step ensures that the motion process of the dynamic object can be closely integrated with the time dimension, providing a basis for the subsequent generation of a four-dimensional dynamic model. Through the above process, the optical flow method not only helps track and understand the motion patterns of dynamic objects in the video, but also through the annotation of timestamps, the motion trajectory of each object can be associated with a specific time, providing detailed time dynamic information for generating high-fidelity autonomous driving simulation test scenarios.

[0069] In this step, time information is first introduced to combine the three-dimensional coordinate data of dynamic foreground objects at different time points to create a continuous four-dimensional dynamic model, which not only contains spatial information but also incorporates the time dimension. In order to accurately describe the motion trajectory of dynamic objects, deep learning or computer vision techniques, such as optical flow and feature point matching, are used to track dynamic objects in the video and mark each object with a timestamp. In this way, the position changes of dynamic objects in three-dimensional space are combined with the corresponding time data to form a detailed behavior trajectory. These generated four-dimensional dynamic models can realistically simulate and reproduce complex traffic conditions, involving scenarios such as vehicle driving and pedestrian movement, enhancing the authenticity and effectiveness of autonomous driving simulation tests.

[0070] In video data, there is no uniform standard for the number of frames extracted per second, which usually depends on the specific application requirements and system processing capabilities. For the optimization method of autonomous driving simulation test, the key is to ensure that the number of extracted frames can accurately reflect the changes in dynamic scenes without bringing too much computational burden to subsequent processing. The common practice is to range from 24 to 60 frames per second. The specific value needs to be determined based on the experimental environment, the required picture smoothness, and the efficiency of processing these frame data. In this case, the specific number of extracted frames should be determined based on the above factors before actual operation.

[0071] Step 106: Separate the dynamic objects from the static background in the LiDAR point cloud data to distinguish static and dynamic elements, and obtain LiDAR point cloud data after point cloud segmentation.

[0072] In the technical solution of the present application, the dynamic objects in the LiDAR point cloud data are separated from the static background. This process finely distinguishes which points belong to the static environment (such as buildings, roads) and which belong to potentially moving objects (such as vehicles, pedestrians) based on the objects' motion state, shape, size and other characteristics.

[0073] Step 108: Input the LiDAR point cloud data after point cloud segmentation and the several annotated video frames into the 3DGS model for parallel training to obtain a 3DGS model of a static background and a 3DGS model of a dynamic foreground object.

[0074] In the technical solution of the present application, the input data of the three-dimensional Gaussian sputtering 3DGS technology mainly includes two types: 1. Point cloud data: This is obtained from the data acquisition equipment installed on the experimental vehicle, such as the vehicle-mounted camera and laser radar (LiDAR). LiDAR point cloud data provides depth information of the scene, while the camera captures visual information. These data are used together to construct a static background model. 2. Time series data: Especially for dynamic foreground objects, in addition to the three-dimensional spatial coordinates, time information is also introduced. By tracking dynamic objects in the video and adding a timestamp to each object, a continuous four-dimensional dynamic model is generated, which contains the position information of the dynamic objects changing over time. Therefore, the input data of the 3DGS technology involves both spatial information (point cloud data) and time information (time series data).

[0075] During the 3DGS model training process, the algorithm parameters may need to be carefully adjusted to achieve the best model reconstruction effect. This includes but is not limited to point cloud density, Gaussian function parameter settings, and model smoothness, etc. The purpose is to maximize the realism and accuracy of the model while maintaining real-time performance. After completing the initial training, the generated static background model is verified to check whether there are any missing or inaccurate areas. Depending on the verification results, it may be necessary to return to the previous step to fine-tune the parameters or add additional data input to further improve the quality and completeness of the model. Through the above steps, based on the denoised and segmented point cloud data, the 3DGS technology can generate a highly realistic static background model, providing a more accurate virtual environment foundation for autonomous driving simulation testing.

[0076] In the 3DGS training process, static background and dynamic foreground objects are combined together for training according to their real positions. This means using the point cloud data collected by the lidar and the scene pictures extracted from the video, separating the static background (such as buildings and roads) from the dynamic foreground objects (such as vehicles and pedestrians) through point cloud segmentation technology, and then recombining them according to their actual positional relationship. The purpose of this is to generate a high-fidelity three-dimensional scene model that includes both static environments and dynamic elements, and these elements maintain their relative positions in the real world. This combination helps ensure that the final generated simulation scene can more accurately reflect the actual situation, thereby improving the authenticity and effectiveness of autonomous driving simulation tests.

[0077] Step 110: Integrate the 3DGS model of the static background and the 3DGS model of the dynamic foreground object into the autonomous driving simulation software for testing the autonomous driving vehicle in the form of a plug-in.

[0078] In this step, plug-ins in supported formats are developed for traditional autonomous driving simulation software, such as plug-ins for platforms such as EU5 or Unity. The purpose of these plug-ins is to enable the simulation software to import and display high-fidelity models generated by 3DGS technology. The developed plug-ins include a user-friendly interactive interface that facilitates users to edit and adjust the imported models. The design ensures the smoothness of the model during loading, rendering and editing, and ensures compatibility with existing simulation systems.

[0079] Importing high-fidelity static background models generated by 3D Gaussian sputtering technology into autonomous driving simulation software, such as CARLA, involves converting and optimizing the model format to ensure compatibility with the simulation platform. Scene replacement and fine-tuning: Replacing the original basic model in the simulation environment with a high-fidelity model means upgrading the default scene configuration of the simulation platform with a highly detailed and realistic model. According to specific simulation requirements, these high-fidelity models also need to be fine-tuned to ensure that they accurately fit the map layout and scale of the simulation platform, such as road width, building location, etc. Dynamic element integration: In the simulation scene that has integrated high-fidelity static background, users can add and run dynamic scene elements, such as vehicles and pedestrians, through specially developed plug-ins (such as those based on EU5 or Unity). These dynamic elements can reproduce complex traffic interactions and flows in the simulation environment based on the previously generated time-series trajectory information. This series of operations ensures that the simulation environment is not only visually close to reality, but also simulates complex traffic behaviors in the real world in terms of dynamic interactions, thereby improving the test quality and reliability of the autonomous driving system.

[0080] In this step, plug-in development can mainly include the following: 1. Target determination: Determine the simulation software platform that the plug-in needs to support, such as EU5 or Unity, to ensure that the generated plug-in is compatible with these mainstream simulation software. 2. Functional design: Design the functional architecture of the plug-in to meet the basic needs of importing high-fidelity models generated by 3DGS, editing and adjusting model parameters, and displaying dynamic and static models in the simulation environment. At the same time, consider user-friendliness and provide an intuitive interactive interface. 3. Interface development: Develop an interface for communicating with the simulation software so that the plug-in can read and write necessary data to realize model import, loading, rendering and editing operations. 4. Model adaptation: Ensure that the model format generated by 3DGS can be correctly parsed by the plug-in and displayed correctly in the simulation software. It may be necessary to convert or optimize the model data to meet the specific requirements of the simulation platform. 5. Performance optimization: Optimize the code and rendering process of the plug-in to ensure that when loading and displaying high-fidelity models, it will not significantly affect the operating efficiency of the simulation software and maintain real-time performance. 6. User interface development: Create an easy-to-use graphical interface that allows users to easily browse, select, place and adjust the model's location, direction and other properties, as well as set the timing trajectory of dynamic objects. 7. Testing and debugging: Test the functional integrity, stability and compatibility of the plug-in in different simulation scenarios, debug and fix any problems found, and ensure that the plug-in performs as expected in the target simulation software. Through these steps, the developed plug-in can effectively integrate high-fidelity models into simulation software, improving the scene realism and customization flexibility of autonomous driving simulation testing.

[0081] use Figure 1The method in the invention has the following technical effects: 1. High-fidelity scene rendering: By combining the three-dimensional Gaussian sputtering 3DGS technology and time dimension information, the technical solution of the present application can generate extremely detailed static background and dynamic foreground object models, which greatly improves the realism of the simulation scene and makes the test environment closer to the complexity of the real world. 2. Improve the efficiency and automation of model generation: By utilizing automated data processing processes, including point cloud segmentation and dynamic object tracking, the technical solution of the present application significantly reduces the workload of manual modeling, improves the efficiency of model generation, enables the system to adapt to environmental changes more quickly, and ensures the timeliness of the model. 3. Taking into account both real-time and dynamic scene simulation: The optimized rendering method ensures that while providing a high-precision model, it meets the real-time requirements, which is crucial for simulating rapidly changing traffic conditions in autonomous driving, such as emergency avoidance of vehicles and response to emergencies, and enhances the effectiveness of the test. 4. Enhanced simulation platform flexibility and scene customization capabilities: The proposed plug-in solution (applicable to software such as EU5 or Unity) not only facilitates the integration of high-fidelity models into existing simulation platforms, but also provides users with an interactive interface to adjust and customize dynamic scenes, so that diverse and targeted test cases can be easily created according to test requirements. In summary, the technical solution of the technical solution of this application not only improves the visual realism and dynamic interaction capabilities of autonomous driving simulation tests, but also effectively improves the efficiency and accuracy of simulation tests through automated tools and real-time rendering technology, providing a more powerful and reliable testing environment for the development and verification of autonomous driving systems.

[0082] based on Figure 1 The method, the examples of this specification also provide some specific implementation plans of the method, which are described below.

[0083] In an optional embodiment technical solution, before performing a video frame extraction operation on the video data to obtain a number of video frames, the process may include:

[0084] The collected LiDAR point cloud data is subjected to denoising and filtering, specifically comprising: first, identifying and removing outliers or noise points in the LiDAR point cloud data; then, using statistical filtering or geometric filtering to smooth the LiDAR point cloud data and retain key structural information in the point cloud; wherein statistical filtering is based on the characteristics of points in the neighborhood, the characteristics including the mean value or the median, to remove points that deviate from the normal range; and geometric filtering utilizes the spatial relationship between points to remove points that do not conform to the expected geometric shape.

[0085] In an optional embodiment technical solution, before performing a video frame extraction operation on the video data to obtain a number of video frames, the method further includes: performing hardware time synchronization on the vehicle-mounted camera and the laser radar, which may specifically include:

[0086] The vehicle-mounted camera and the lidar are synchronized with a common time source through an external global positioning system timing device or a high-precision clock built into the vehicle. Both the vehicle-mounted camera and the lidar record data based on the common time source, thereby ensuring the consistency of the data collection timestamp.

[0087] In an optional embodiment technical solution, the 3DGS model of the static background and the 3DGS model of the dynamic foreground object are integrated into the autonomous driving simulation software for testing the autonomous driving vehicle in the form of a plug-in, which may specifically include:

[0088] S1. Model import, specifically including: importing the 3DGS model of the static background generated by the 3D Gaussian sputtering technology into the autonomous driving simulation software;

[0089] S2, scene replacement and fine-tuning, specifically including: replacing the basic model in the original simulation environment of the autonomous driving simulation software with the 3DGS model of the static background, and adjusting the 3DGS model of the static background according to specific simulation requirements to ensure that it fits the map layout and proportion of the autonomous driving simulation software;

[0090] S3, dynamic element integration, specifically includes: in the simulation scene that has integrated the 3DGS model of the static background, through the developed plug-in, allowing the user to add and run the dynamic scene elements in the 3DGS model of the dynamic foreground object.

[0091] At the same time, by modifying the three-dimensional coordinates of the dynamic foreground object, the behavior of the vehicle in the original scene can be changed. Specifically, during the 3DGS training process, the static background and dynamic foreground objects are combined together for training according to their real positions. During the rendering process, by modifying the three-dimensional coordinates of the foreground object (such as a vehicle), different behavior changes of the vehicle can be achieved, such as changing lanes, removing, accelerating or decelerating, or editing into a collision scene, etc., so that videos of different scenes can be rendered to meet the diverse simulation test needs.

[0092] Based on the same idea, the embodiments of this specification also provide a device corresponding to the above method. Fig.13 The embodiments of this specification provide corresponding to Figure 1 A schematic diagram of the structure of an optimization device for autonomous driving simulation testing. Fig.13 As shown, the device may include:

[0093] The data acquisition module 1302 is used to synchronously acquire video data and LiDAR point cloud data of a real scene that needs to be reconstructed based on a data acquisition device; wherein the data acquisition device includes a vehicle-mounted camera and a laser radar, and the data acquisition device is installed on an experimental vehicle traveling in the real scene, the vehicle-mounted camera is used to acquire the video data, and the laser radar is used to acquire the LiDAR point cloud data.

[0094] The video data processing module 1304 is used to perform a video frame extraction operation on the video data to obtain a number of video frames, and to perform data annotation on the number of video frames frame by frame based on the motion characteristics of the elements included in the video frames to distinguish the dynamic foreground objects and static background elements included in each of the number of video frames; then, track the dynamic objects in the video data, and then annotate the three-dimensional coordinate data and timestamp information of the dynamic foreground objects included in each video frame to obtain a number of annotated video frames.

[0095] The point cloud segmentation module 1306 is used to separate the dynamic objects from the static background in the LiDAR point cloud data to distinguish static and dynamic elements, and obtain the LiDAR point cloud data after point cloud segmentation.

[0096] The 3DGS training module 1308 is used to input the LiDAR point cloud data after point cloud segmentation and the several video frames after annotation into the 3DGS model for parallel training to obtain a 3DGS model of a static background and a 3DGS model of a dynamic foreground object.

[0097] The plug-in integration module 1310 is used to integrate the 3DGS model of the static background and the 3DGS model of the dynamic foreground object into the autonomous driving simulation software for testing the autonomous driving vehicle in the form of a plug-in.

[0098] It is understood that the above modules refer to computer programs or program segments for executing one or more specific functions. In addition, the distinction between the above modules does not mean that the actual program codes must also be separated.

[0099] The above describes specific embodiments of the present specification. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0100] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other.

[0101] The devices, equipment and methods provided in the embodiments of this specification correspond to each other, and therefore, the devices and equipment also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding devices and equipment will not be repeated here.

[0102] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0103] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0104] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0105] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0106] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0107] Those skilled in the art will appreciate that the embodiments of the technical solutions of the present application may be provided as methods, systems, or computer program products. Therefore, the technical solutions of the present application may be in the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware. Moreover, the technical solutions of the present application may be in the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0108] The technical solution of the present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the technical solution of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0109] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0111] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0112] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0113] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0114] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0115] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0116] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0117] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An optimization method for autonomous driving simulation testing, characterized in that: include: Based on the data acquisition device, the video data and LiDAR point cloud data of the real scene that needs to be reconstructed are synchronously acquired; wherein the data acquisition device includes a vehicle-mounted camera and a laser radar, and the data acquisition device is installed on an experimental vehicle traveling in the real scene, the vehicle-mounted camera is used to acquire the video data, and the laser radar is used to acquire the LiDAR point cloud data; Performing a video frame extraction operation on the video data to obtain a number of video frames, and performing data annotation on the number of video frames frame by frame based on motion characteristics of elements included in the video frames to distinguish dynamic foreground objects and static background elements included in each of the number of video frames; then, tracking dynamic objects in the video data, and then annotating the three-dimensional coordinate data and time stamp information of the dynamic foreground objects included in each video frame to obtain a number of annotated video frames; Separating the dynamic objects from the static background in the LiDAR point cloud data to distinguish static and dynamic elements, and obtaining LiDAR point cloud data after point cloud segmentation; Inputting the LiDAR point cloud data after point cloud segmentation and the several video frames after annotation into the 3DGS model for parallel training to obtain a 3DGS model of a static background and a 3DGS model of a dynamic foreground object; The 3DGS model of the static background and the 3DGS model of the dynamic foreground object are integrated into the autonomous driving simulation software for testing the autonomous driving vehicle in the form of a plug-in.

2. The optimization method for autonomous driving simulation test according to claim 1, characterized in that: The step of performing a video frame extraction operation on the video data to obtain a number of video frames includes: The collected LiDAR point cloud data is subjected to denoising and filtering, specifically comprising: first, identifying and removing outliers or noise points in the LiDAR point cloud data; then, using statistical filtering or geometric filtering to smooth the LiDAR point cloud data and retain key structural information in the point cloud; wherein statistical filtering is based on the characteristics of points in the neighborhood, the characteristics including the mean value or the median, to remove points that deviate from the normal range; and geometric filtering utilizes the spatial relationship between points to remove points that do not conform to the expected geometric shape.

3. The optimization method for autonomous driving simulation test according to claim 1, characterized in that: Before performing a video frame extraction operation on the video data to obtain a number of video frames, the method further includes: performing hardware time synchronization on the vehicle-mounted camera and the laser radar, specifically including: The vehicle-mounted camera and the lidar are synchronized with a common time source through an external global positioning system timing device or a high-precision clock built into the vehicle. Both the vehicle-mounted camera and the lidar record data based on the common time source, thereby ensuring the consistency of the data collection timestamp.

4. The optimization method for autonomous driving simulation test according to claim 1, characterized in that: The tracking of the dynamic object in the video data specifically includes: S1, selecting a reference frame and adjacent frames, specifically comprising: first, selecting a frame from the video data as a reference frame, and then selecting one or more subsequent frames as adjacent frames; S2, calculating pixel gradient, specifically including: for each pixel point in the reference frame and the adjacent frame, calculating its brightness gradient; S3, matching feature points: using pixel gradient information, finding the corresponding relationship between the same feature points in the reference frame and the adjacent frames; S4, estimating motion vector: based on the correspondence between feature points, estimate the motion vector of each pixel from the reference frame to the adjacent frame through an iterative optimization algorithm, wherein the vector includes displacement components of the pixel in the horizontal and vertical directions; S5. Timestamp allocation: Combining the estimated motion vector and the shooting time interval of the video frame, time information is allocated to each pixel of the dynamic object to form a dynamic model with a time dimension.

5. The optimization method for autonomous driving simulation test according to claim 1, characterized in that: The tracking of the dynamic object in the video data specifically includes: S1. Feature detection: In each frame of the video data, a feature detection algorithm is applied to extract stable feature points; for the detected feature points, a descriptor is calculated, where the descriptor is used to describe a vector of image content around the feature points; S2, feature description: Match feature points by comparing feature descriptors between two or more consecutive frames; S3, geometric verification: In order to further eliminate false matches, the random sampling consensus algorithm RANSAC method is used to estimate the transformation matrix based on the matching point pairs, and only the matching point pairs that meet the geometric constraints are retained; S4, trajectory tracking: through cross-frame feature matching and geometric verification, the trajectory of feature points over time is established, and a unique identifier is assigned to each continuously tracked feature point sequence to indicate that it belongs to the same dynamic object; S5. Timestamp annotation: Whenever a feature point is successfully matched in a frame and assigned to a trajectory, the precise time information of the frame is recorded as a timestamp. In this way, as time goes by, the trajectory of each dynamic object is associated with a series of timestamps, forming a dynamic object motion trajectory with time information. The above process is used to track dynamic objects in the video and add a timestamp to the motion trajectory of each object.

6. The optimization method for autonomous driving simulation test according to claim 1, characterized in that: The step of integrating the 3DGS model of the static background and the 3DGS model of the dynamic foreground object into the autonomous driving simulation software for testing the autonomous driving vehicle in the form of a plug-in specifically includes: S1. Model import, specifically including: importing the 3DGS model of the static background generated by the 3D Gaussian sputtering technology into the autonomous driving simulation software; S2, scene replacement and fine-tuning, specifically including: replacing the basic model in the original simulation environment of the autonomous driving simulation software with the 3DGS model of the static background, and adjusting the 3DGS model of the static background according to specific simulation requirements to ensure that it fits the map layout and proportion of the autonomous driving simulation software; S3, dynamic element integration, specifically includes: in the simulation scene that has integrated the 3DGS model of the static background, through the developed plug-in, allowing the user to add and run the dynamic scene elements in the 3DGS model of the dynamic foreground object.

7. An optimization device for autonomous driving simulation testing, characterized in that: include: A data acquisition module, used for synchronously acquiring video data and LiDAR point cloud data of a real scene that needs to be reconstructed based on a data acquisition device; wherein the data acquisition device includes a vehicle-mounted camera and a laser radar, and the data acquisition device is installed on an experimental vehicle traveling in the real scene, the vehicle-mounted camera is used to acquire the video data, and the laser radar is used to acquire the LiDAR point cloud data; A video data processing module is used to perform a video frame extraction operation on the video data to obtain a number of video frames, and to perform data annotation on the number of video frames frame by frame based on motion characteristics of elements included in the video frames to distinguish dynamic foreground objects and static background elements included in each of the number of video frames; then, track dynamic objects in the video data, and then annotate the three-dimensional coordinate data and time stamp information of the dynamic foreground objects included in each video frame to obtain a number of annotated video frames; A point cloud segmentation module is used to separate the dynamic objects from the static background in the LiDAR point cloud data to distinguish static and dynamic elements, and obtain the LiDAR point cloud data after point cloud segmentation; A 3DGS training module is used to input the LiDAR point cloud data after point cloud segmentation and the several video frames after annotation into a 3DGS model for parallel training to obtain a 3DGS model of a static background and a 3DGS model of a dynamic foreground object; A plug-in integration module is used to integrate the 3DGS model of the static background and the 3DGS model of the dynamic foreground object into the autonomous driving simulation software for testing the autonomous driving vehicle in the form of a plug-in.

Images