An optimization method and device for autonomous driving simulation test

Abstract

The technical scheme of the application discloses an optimization method and device for automatic driving simulation test, comprising: data acquisition: install data acquisition equipment, collect video data and LiDAR point cloud data. Data preprocessing: denoising and filtering LiDAR point cloud data, extracting video frames and labeling dynamic and static elements. Static background and dynamic foreground object separation: applying point cloud segmentation technology to separate dynamic objects and static background. Static background model generation: generating a high-fidelity static environment model using point cloud data. Dynamic foreground object model and time sequence trajectory generation: generating a four-dimensional dynamic model combined with time information, tracking dynamic objects and adding timestamps. Plug-in development: developing a plug-in supporting format to enable traditional simulation software to import and display high-fidelity models. Model integration and simulation: importing high-fidelity models into simulation software, replacing the original environment, and adding dynamic elements through the plug-in interface.

Description

TECHNICAL FIELD

[0001] The technical solution of the present application relates to the field of automatic driving simulation testing, in particular, to an automatic driving simulation testing optimization method and device. BACKGROUND

[0002] With the rapid development of automatic driving technology, it is crucial to ensure the safety and reliability of automatic driving systems. Traditional testing methods, such as road testing, are real but costly and difficult to cover all possible driving scenarios. In addition, testing in certain extreme situations (such as extreme weather conditions or sudden accident scenarios) is difficult to safely or effectively conduct in reality. Simulation testing not only simulates a variety of complex traffic scenarios in a virtual environment, but also evaluates the performance of the automatic driving system in different environments by adjusting parameters. This method can accelerate the development process, reduce development costs, and also systematically find and correct system defects. However, the effectiveness of simulation testing depends on the realism and real-time performance of scene rendering. SUMMARY

[0003] The technical solution of the present application provides an automatic driving simulation testing optimization method and device to overcome at least one technical problem in the prior art.

[0004] According to a first aspect of an embodiment of the technical solution of the present application, an automatic driving simulation testing optimization method is provided, comprising:

[0005] synchronously collecting video data and LiDAR point cloud data of a real scene that needs to be reconstructed based on a data collection device; wherein the data collection device includes a vehicle-mounted camera and a laser radar, the data collection device is installed on an experimental vehicle driving in the real scene, the vehicle-mounted camera is used to collect the video data, and the laser radar is used to collect the LiDAR point cloud data;

[0006] performing a video frame extraction operation on the video data to obtain a plurality of video frames, performing data labeling on the plurality of video frames frame by frame based on the motion characteristics of the elements included in the video frames to distinguish between dynamic foreground objects and static background elements included in each video frame in the plurality of video frames; then, tracking dynamic objects in the video data, and further labeling three-dimensional coordinate data and timestamp information of the dynamic foreground objects included in each video frame to obtain a plurality of labeled video frames;

[0007] separating dynamic objects from static backgrounds in the LiDAR point cloud data to distinguish between static and dynamic elements, and obtaining LiDAR point cloud data after point cloud segmentation;

[0008] inputting the LiDAR point cloud data after the point cloud segmentation and the video frames after the labeling into a 3DGS model for parallel training to obtain a 3DGS model of static background and a 3DGS model of dynamic foreground objects;

[0009] integrating the 3DGS model of static background and the 3DGS model of dynamic foreground objects into an automatic driving simulation software for testing an automatic driving vehicle in the form of plug-ins.

[0010] Preferably, before the video frame extraction operation on the video data to obtain a plurality of video frames, the method further comprises:

[0011] performing denoising and filtering processing on the collected LiDAR point cloud data, specifically including: firstly, identifying and removing outliers or noise points in the LiDAR point cloud data; then, using statistical filtering or geometric filtering to smooth the LiDAR point cloud data and retain key structural information in the point cloud; wherein the statistical filtering removes points deviating from the normal range based on the characteristics of the points in the neighborhood, including the mean or median; the geometric filtering removes points that do not conform to the expected geometric shape by using the spatial relationship between the points.

[0012] Preferably, before the video frame extraction operation on the video data to obtain a plurality of video frames, the method further comprises: synchronizing the vehicle-mounted camera and the laser radar in hardware time, specifically including:

[0013] synchronizing the vehicle-mounted camera and the laser radar with a common time source through an external global positioning system time device or a high-precision clock built in the vehicle, and both the vehicle-mounted camera and the laser radar record data according to the common time source, thereby ensuring the consistency of the data acquisition time stamp.

[0014] Preferably, the tracking of the dynamic objects in the video data specifically includes:

[0015] S1, selecting a reference frame and adjacent frames, specifically including: firstly, selecting a frame from the video data as a reference frame, and then selecting one or more frames after the reference frame as adjacent frames;

[0016] S2, calculating pixel gradient, specifically including: calculating the brightness gradient of each pixel point in the reference frame and the adjacent frames;

[0017] S3, matching feature points: using the pixel gradient information to find the corresponding relationship of the same feature points in the reference frame and the adjacent frames;

[0018] S4, estimating motion vector: based on the correspondence of feature points, the motion vector of each pixel from the reference frame to the adjacent frame is estimated by an iterative optimization algorithm, and the vector contains the displacement components of the pixel in the horizontal and vertical directions;

[0019] S5, timestamp assignment: combining the estimated motion vector and the shooting time interval of the video frame, the time information of each dynamic object pixel is assigned to form a dynamic model with time dimension.

[0020] Preferably, the tracking of the dynamic object in the video data specifically includes:

[0021] S1, feature detection: in each frame of the video data, a feature detection algorithm is applied to extract stable feature points; for the detected feature points, the descriptor is calculated, which is used to describe the vector of the image content around the feature point;

[0022] S2, feature description: between two or more consecutive frames, the feature points are matched by comparing the feature descriptors;

[0023] S3, geometric verification: in order to further eliminate false matches, the random sample consensus algorithm (RANSAC) is used to estimate the transformation matrix based on the matching point pairs, and only the matching point pairs that meet the geometric constraints are retained;

[0024] S4, trajectory tracking: through cross-frame feature matching and geometric verification, the trajectory of the feature points over time is established, and a unique identifier is assigned to each feature point sequence that is continuously tracked, indicating that it belongs to the same dynamic object;

[0025] S5, timestamp labeling: whenever a feature point is successfully matched in a frame and is assigned to a certain trajectory, the exact time information of the frame is recorded as a timestamp, so that over time, the trajectory of each dynamic object is associated with a series of timestamps, forming a dynamic object motion trajectory with time information. Through the above process, the dynamic object in the video is tracked, and a timestamp is added to the motion trajectory of each object.

[0026] Preferably, the 3DGS model of the static background and the 3DGS model of the dynamic foreground object are integrated into the automatic driving simulation software for testing the automatic driving vehicle in the form of plug-ins, specifically including:

[0027] S1, model import, specifically including: importing the 3DGS model of the static background generated by the 3D Gaussian sputtering technology into the automatic driving simulation software;

[0028] S2, scene replacement and fine-tuning, specifically including: replacing the 3DGS model of the static background with the basic model in the original simulation environment of the automatic driving simulation software, and adjusting the 3DGS model of the static background according to specific simulation requirements to ensure that it fits the map layout and scale of the automatic driving simulation software;

[0029] S3, dynamic element integration, specifically including: in the simulation scene in which the 3DGS model of the static background has been integrated, through the developed plug-in, the user adds and runs the dynamic scene elements in the 3DGS model of the dynamic foreground object.

[0030] According to the second aspect of the technical scheme embodiment of the present application, an automatic driving simulation test optimization device is provided, comprising:

[0031] A data acquisition module is configured to synchronously acquire video data and LiDAR point cloud data of a real scene requiring scene reconstruction based on a data acquisition device; wherein the data acquisition device comprises a vehicle-mounted camera and a laser radar, the data acquisition device is installed on an experimental vehicle running in the real scene, the vehicle-mounted camera is configured to acquire the video data, and the laser radar is configured to acquire the LiDAR point cloud data;

[0032] A video data processing module is configured to perform a video frame extraction operation on the video data to obtain a plurality of video frames, perform data labeling on the plurality of video frames frame by frame based on the motion characteristics of elements included in the video frames to distinguish dynamic foreground objects and static background elements included in each video frame in the plurality of video frames, track the dynamic objects in the video data, and then label the three-dimensional coordinate data and timestamp information of the dynamic foreground objects included in each video frame, to obtain a plurality of labeled video frames;

[0033] A point cloud segmentation module is configured to separate dynamic objects from static backgrounds in the LiDAR point cloud data to distinguish static and dynamic elements, and obtain LiDAR point cloud data after point cloud segmentation;

[0034] A 3DGS training module is configured to input the LiDAR point cloud data after point cloud segmentation and the plurality of labeled video frames into a 3DGS model for parallel training to obtain a 3DGS model of a static background and a 3DGS model of a dynamic foreground object;

[0035] A plug-in integration module is configured to integrate the 3DGS model of the static background and the 3DGS model of the dynamic foreground object into an automatic driving simulation software for testing an automatic driving vehicle in the form of a plug-in.

[0036] The present specification one embodiment can achieve at least the following beneficial effects:

[0037] 1. High-fidelity scene rendering: By combining 3D Gaussian sputtering (3DGS) technology and time dimension information, the present application can generate extremely detailed static background and dynamic foreground object models, greatly improving the realism of the simulation scene, making the test environment more close to the complexity of the real world.

[0038] 2. Improve model generation efficiency and automation: Using automated data processing procedures, including point cloud segmentation and dynamic object tracking, the present application significantly reduces the workload of manual modeling, improves the efficiency of model generation, and enables the system to adapt to environmental changes more quickly, ensuring the timeliness of the model.

[0039] 3. Real-time and dynamic scene simulation: Optimized rendering methods ensure high-precision models while meeting real-time requirements, which is crucial for simulating rapidly changing traffic conditions in autonomous driving, such as emergency avoidance and handling of unexpected accidents, enhancing the effectiveness of testing.

[0040] 4. Enhanced simulation platform flexibility and scene customization: The proposed plug-in solution (applicable to EU5 or Unity software) not only facilitates the integration of high-fidelity models into existing simulation platforms, but also provides users with an interactive interface to adjust and customize dynamic scenes, allowing easy creation of diverse and targeted test cases according to testing needs.

[0041] In summary, the present application not only improves the visual realism and dynamic interaction capabilities of autonomous driving simulation testing, but also effectively improves the efficiency and accuracy of simulation testing through automated tools and real-time rendering technology, providing a more powerful and reliable test environment for the development and validation of autonomous driving systems. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the technical solutions in the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0043] Figure 1 A flowchart of an optimization method for autonomous driving simulation testing provided by an embodiment of the present specification;

[0044] Figure 2 A flowchart of an optimization method for autonomous driving simulation testing provided by an embodiment of the present specification;

[0045] Figure 3 An example 1 for describing dynamic foreground objects and static background in an optimization method of an autonomous driving simulation test provided by an embodiment of the present specification;

[0046] Figure 4 An example 2 for describing dynamic foreground objects and static background in an optimization method of an autonomous driving simulation test provided by an embodiment of the present specification;

[0047] Figure 5 An example 3 for describing dynamic foreground objects and static background in an optimization method of an autonomous driving simulation test provided by an embodiment of the present specification;

[0048] Figure 6 An example 4 for describing dynamic foreground objects and static background in an optimization method of an autonomous driving simulation test provided by an embodiment of the present specification;

[0049] Figure 7 An example 5 for describing dynamic foreground objects and static background in an optimization method of an autonomous driving simulation test provided by an embodiment of the present specification;

[0050] Figure 8 An example 6 for describing dynamic foreground objects and static background in an optimization method of an autonomous driving simulation test provided by an embodiment of the present specification;

[0051] Figure 9 An example 7 for describing dynamic foreground objects and static background in an optimization method of an autonomous driving simulation test provided by an embodiment of the present specification;

[0052] Figure 10 An example 8 for describing dynamic foreground objects and static background in an optimization method of an autonomous driving simulation test provided by an embodiment of the present specification;

[0053] Figure 11 An example 9 for describing dynamic foreground objects and static background in an optimization method of an autonomous driving simulation test provided by an embodiment of the present specification;

[0054] Figure 12 An example 10 for describing dynamic foreground objects and static background in an optimization method of an autonomous driving simulation test provided by an embodiment of the present specification;

[0055] Figure 13 A structural schematic diagram of an optimization device of an autonomous driving simulation test provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0056] In order to make the purposes, technical solutions and advantages of one or more embodiments of the present specification clearer, the technical solutions of one or more embodiments of the present specification will be described below in conjunction with specific embodiments of the present specification and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present specification, rather than all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of one or more embodiments of the present specification.

[0057] It should be understood that although the terms first, second, third, etc. can be used in this application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other.

[0058] Figure 1 The flowchart of the application of the optimization method for autonomous driving simulation test of the service platform provided by the embodiments of the present specification. From the perspective of program, the execution subject of the flowchart can be the program of the server carried on the service platform.

[0059] As shown in Figure 1 The flowchart can include the following steps.

[0060] Step 102: synchronously collecting video data and LiDAR point cloud data of a real scene which needs to be reconstructed based on a data collection device; wherein the data collection device includes a vehicle-mounted camera and a laser radar, the data collection device is installed on an experimental vehicle running in the real scene, the vehicle-mounted camera is used to collect the video data, and the laser radar is used to collect the LiDAR point cloud data.

[0061] In this step, the vehicle-mounted camera is usually installed on the front, back and side of the experimental vehicle in order to capture visual information of the surrounding environment from multiple angles. The laser radar (LiDAR) is usually installed on the roof or the front end of the vehicle at a high position. Such position is helpful to obtain a wider horizontal field of view and a longer detection distance, so as to more comprehensively collect three-dimensional space point cloud data of the surrounding environment of the vehicle, including road surface conditions, obstacle positions and contour information of the surrounding environment.

[0062] In the technical solution of the present application, LiDAR (Light Detection and Ranging) point cloud data refers to a series of three-dimensional space point sets formed by calculating the three-dimensional space coordinates (X, Y, Z) of each point in the surrounding environment according to the round-trip time of light and the angle information of the emitted laser, through the emission of laser pulses by a LiDAR sensor and the reception of the reflected signals. These point cloud data can be used to construct a three-dimensional model of the environment, including the precise position information of objects such as terrain, buildings, vehicles, and pedestrians, and are crucial for environment perception, map construction, and path planning in autonomous driving technology.

[0063] Video data and LiDAR data are simultaneously collected and used to construct a high-fidelity autonomous driving simulation test scene. The correlation between the two data is reflected in the fact that both provide complete information about the scene. Video data is captured by vehicle-mounted cameras and mainly contains visual information about the scene, such as color, texture, and the appearance characteristics of traffic participants. LiDAR (Laser Radar) data provides depth information about the scene, i.e., the relative distances and spatial layouts of various objects, which is crucial for constructing accurate three-dimensional structures. The combination of these two types of data enables comprehensive and accurate acquisition of static and dynamic information in the scene by matching and fusing the rich visual details of video images with the precise spatial information of LiDAR. Such data correlation not only ensures the high realism of the reconstructed scene, but also lays the foundation for subsequent steps such as separation of static backgrounds and dynamic foreground objects and model generation.

[0064] Step 104: Perform video frame extraction on the video data to obtain a number of video frames, and perform data labeling on each of the number of video frames based on the motion characteristics of the elements included in the video frames to distinguish between dynamic foreground objects and static background elements included in each of the number of video frames. Then, track the dynamic objects in the video data, and further label the three-dimensional coordinate data and timestamp information of the dynamic foreground objects included in each video frame to obtain a number of labeled video frames.

[0065] In this step, the frames extracted from the video data are labeled to distinguish between dynamic foreground objects (such as vehicles and pedestrians) and static backgrounds (such as buildings and roads). At the same time, in order to track the dynamic objects in the video, deep learning or computer vision techniques can be used, including optical flow and feature point matching. Optical flow can analyze the motion information of pixels between adjacent frames to determine the direction and speed of object motion, while feature point matching finds and matches points with unique features between different frames to track the position changes of objects. The combination of these two methods can effectively track the motion trajectories of dynamic objects in the video.

[0066] In the technical solution of the present application, the feature point matching method can be used to track dynamic objects in the video and label timestamps for each object. Specifically, the following steps can be followed: 1. Feature detection: First, in each frame of the video sequence, a feature detection algorithm (such as SIFT, SURF, ORB, etc.) is applied to extract stable feature points. These algorithms can identify features that remain unchanged even under changes in image scale, rotation, or illumination. 2. Feature description: For the detected feature points, calculate their descriptors, which are vectors that describe the image content around the feature points. Descriptors should have good discriminability, i.e., similar feature points in different images should have similar descriptors, while different feature points should have significant differences. 3. Feature matching: Between consecutive two or more frames, match feature points by comparing feature descriptors. Common methods include nearest neighbor matching, ratio testing, etc., to ensure correct correspondence and filter out false matches. 4. Geometric verification: To further eliminate false matches, methods such as RANSAC can be used to estimate transformation matrices (such as translation, rotation) based on matching point pairs, and only keep matching point pairs that meet geometric constraints (such as coplanarity). 5. Trajectory tracking: Through cross-frame feature matching and geometric verification, the trajectory of a feature point over time can be established, and a unique identifier is assigned to each continuously tracked feature point sequence, indicating that it belongs to the same dynamic object. 6. Timestamp labeling: Whenever a feature point is successfully matched in a frame and assigned to a certain trajectory, record the exact time information of that frame as a timestamp. In this way, over time, the trajectory of each dynamic object is associated with a series of timestamps, forming a dynamic object motion trajectory with time information. Through the above process, the feature point matching method can be used to track dynamic objects in the video and accurately label timestamps for each object's motion trajectory, which is crucial for subsequent dynamic model generation and simulation testing.

[0067] At the same time, considering that the optical flow method is a computer vision technology used to analyze the changes in pixel intensity between consecutive frames in a video sequence, and then estimate the motion of objects in the image plane. In the technical solution of the present application, the optical flow method can also be used to track dynamic objects in the video and add timestamps for each object. Specifically, the following operations can be performed:

[0068] 1. Select reference frame and adjacent frames: First, select one frame from the video sequence as the reference frame, then select one or more frames after it as the adjacent frames. 2. Calculate pixel gradients: For each pixel point in the reference frame and adjacent frames, calculate its brightness gradient (i.e. the direction and size of brightness change), this step is to determine the trend of pixel movement. 3. Match feature points: Use pixel gradient information to find the corresponding relationship of the same feature points in the reference frame and adjacent frames, these feature points are usually areas with obvious brightness changes in the image, such as edges or texture changes. 4. Estimate motion vectors: Based on the correspondence of feature points, estimate the motion vector (i.e. optical flow) of each pixel from the reference frame to the adjacent frame through iterative optimization algorithm. This vector contains the displacement components of the pixel in the horizontal and vertical directions. 5. Time stamp assignment: Combine the estimated motion vector and the shooting time interval of the video frame to assign time information to each pixel point of the dynamic object, forming a dynamic model with time dimension. Through the above steps, the technical scheme of the present application uses the optical flow method to realize the accurate estimation of the motion trajectory of the dynamic foreground object, which is crucial for generating high-fidelity automatic driving simulation test scenes. Based on the above analysis, a three-dimensional coordinate sequence that changes over time, i.e. a dynamic trajectory, is constructed for each dynamic object detected. At the same time, a corresponding time stamp is assigned to each key point or frame on the trajectory, recording the specific time information of the object appearing in the video. This step ensures that the motion process of the dynamic object can be closely combined with the time dimension, providing a basis for subsequent four-dimensional dynamic model generation. Through the above process, the optical flow method not only helps to track and understand the motion patterns of dynamic objects in the video, but also makes each object's motion trajectory associated with specific time through the labeling of time stamps, providing detailed temporal dynamic information for generating high-fidelity automatic driving simulation test scenes.

[0069] In this step, first introduce time information, combine the three-dimensional coordinate data of dynamic foreground objects at different time points to create a continuous four-dimensional dynamic model, this model not only contains spatial information, but also integrates time dimension. In order to accurately describe the motion trajectory of dynamic objects, deep learning or computer vision technology such as optical flow method and feature point matching is used to track dynamic objects in the video and label time stamps for each object. In this way, the position change of dynamic objects in three-dimensional space is combined with the corresponding time data to form a detailed behavior trajectory. These generated four-dimensional dynamic models can realistically simulate and reproduce complex traffic conditions involving vehicle driving, pedestrian movement and other scenarios, enhancing the authenticity and effectiveness of automatic driving simulation testing.

[0070] In video data, there is no uniform standard for the number of frames extracted per second, which usually depends on the specific application requirements and system processing capacity. For the automatic driving simulation test optimization method, the key is to ensure that the extracted frame rate can accurately reflect the changes of dynamic scenes and will not bring too much computational burden to the subsequent processing. The common practice is from 24 frames per second to 60 frames per second, and the specific value needs to be determined according to the experimental environment, the required smoothness of the picture and the efficiency of processing these frame data. In this case, the specific extraction frame rate should be determined according to the above factors before actual operation.

[0071] Step 106: separating the dynamic objects in the LiDAR point cloud data from the static background to distinguish static and dynamic elements, and obtaining the LiDAR point cloud data after point cloud segmentation.

[0072] In the technical solution of the present application, the dynamic objects in the LiDAR point cloud data are separated from the static background, and this process distinguishes which points belong to the static environment (such as buildings, roads) and which points belong to the objects that may move (such as vehicles, pedestrians) according to the motion state, shape, size and other characteristics of the objects.

[0073] Step 108: inputting the LiDAR point cloud data after point cloud segmentation and the video frames after labeling into the 3DGS model for parallel training to obtain the 3DGS model of the static background and the 3DGS model of the dynamic foreground objects.

[0074] In the technical solution of the present application, the input data of the three-dimensional Gaussian sputtering 3DGS technology mainly includes two kinds: 1. Point cloud data: This is obtained from data acquisition devices installed on experimental vehicles, such as vehicle-mounted cameras and laser radars (LiDAR). LiDAR point cloud data provides depth information of the scene, while cameras capture visual information. These data are used together to construct a static background model. 2. Time series data: Especially for dynamic foreground objects, in addition to three-dimensional spatial coordinates, time information is also introduced. By tracking dynamic objects in the video and adding timestamps to each object, a continuous four-dimensional dynamic model is generated, which contains the position information of dynamic objects changing over time. Therefore, the input data of the 3DGS technology involves both spatial information (point cloud data) and time information (time series data).

[0075] During the training process of the 3DGS model, detailed adjustments to the algorithm's parameters may be necessary to achieve the best model reconstruction results. This includes but is not limited to point cloud density, Gaussian function parameter settings, and model smoothness, with the goal of maximizing the realism and accuracy of the model while maintaining real-time performance. After completing the initial training, the generated static background model is verified to check for any missing or inaccurate areas. Based on the verification results, it may be necessary to return to the previous step for parameter fine-tuning or add additional data inputs to further improve the quality and completeness of the model. Through the above steps, based on the denoised and segmented point cloud data, the 3DGS technology can generate a highly realistic static background model, providing a more accurate virtual environment foundation for autonomous driving simulation testing.

[0076] In the 3DGS training process, combining static backgrounds and dynamic foreground objects according to their real positions means using the point cloud data collected by the laser radar and the scene pictures extracted from the video, separating the static background (such as buildings, roads) from the dynamic foreground objects (such as vehicles, pedestrians) through point cloud segmentation technology, and then recombining them according to their actual position relationship. The purpose of this is to generate a high-fidelity three-dimensional scene model that includes both static environments and dynamic elements, and these elements maintain their relative positions in the real world. This combination helps ensure that the final simulation scene can more accurately reflect the actual situation, thereby improving the authenticity and effectiveness of autonomous driving simulation testing.

[0077] Step 110: Integrate the 3DGS model of the static background and the 3DGS model of the dynamic foreground objects into the autonomous driving simulation software for testing autonomous driving vehicles in the form of plug-ins.

[0078] In this step, plug-ins supporting the format of traditional autonomous driving simulation software are developed, such as plug-ins for EU5 or Unity platforms. The purpose of these plug-ins is to enable the simulation software to import and display high-fidelity models generated by 3DGS technology. The developed plug-ins include user-friendly interfaces that facilitate users to perform editing and adjustment operations on imported models, ensuring smoothness in the loading, rendering, and editing processes of the models, and ensuring compatibility with existing simulation systems.

[0079] The high-fidelity static background model generated through 3D Gaussian sputtering technology is imported into an autonomous driving simulation software, such as CARLA. This process involves model format conversion and optimization to ensure compatibility with the simulation platform. Scene replacement and fine-tuning: Replace the original simulation environment's base model with the high-fidelity model, which means upgrading the simulation platform's default scene configuration with highly detailed and realistic models. According to specific simulation needs, these high-fidelity models need to be carefully adjusted to ensure they accurately match the simulation platform's map layout and scale, such as road width and building location. Dynamic element integration: In the simulation scene that has integrated high-fidelity static backgrounds, users can add and run dynamic scene elements, such as vehicles and pedestrians, through specially developed plugins (such as EU5 or Unity). These dynamic elements, based on previously generated time sequence trajectory information, can reproduce complex traffic interactions and flow in the simulation environment. This series of operations ensures that the simulation environment not only visually resembles reality but also simulates complex traffic behavior in dynamic interactions, thereby improving the testing quality and reliability of autonomous driving systems.

[0080] In this step, plugin development can mainly include the following: 1. Target determination: Determine the simulation software platform that the plugin needs to support, such as EU5 or Unity, to ensure that the generated plugin is compatible with these mainstream simulation software. 2. Function design: Design the functional architecture of the plugin, which needs to meet the basic requirements of importing high-fidelity models generated by 3DGS, editing and adjusting model parameters, and displaying dynamic and static models in the simulation environment. At the same time, consider user friendliness and provide an intuitive interface. 3. Interface development: Develop interfaces for communication with simulation software, so that the plugin can read and write necessary data to achieve model import, loading, rendering, and editing operations. 4. Model adaptation: Ensure that the model format generated by 3DGS can be correctly parsed by the plugin and correctly displayed in the simulation software. It may be necessary to convert or optimize model data to meet the specific requirements of the simulation platform. 5. Performance optimization: Optimize the code and rendering process of the plugin to ensure that loading and displaying high-fidelity models do not significantly affect the running efficiency of the simulation software, maintaining real-time performance. 6. User interface development: Create an easy-to-use graphical interface that allows users to easily browse, select, place, and adjust model properties such as position and direction, as well as set up time sequence trajectories for dynamic objects. 7. Testing and debugging: Test the plugin's functional integrity, stability, and compatibility in different simulation scenarios, debug and fix any problems found, and ensure that the plugin performs as expected in the target simulation software. Through these steps, the developed plugin can effectively integrate high-fidelity models into simulation software, improving the scene realism and customization flexibility of autonomous driving simulation testing.

[0081] Adopting Figure 1The method described herein has the following technical effects: 1. High-fidelity scene rendering: By combining 3D Gaussian sputtering 3DGS technology and temporal dimension information, the technical solution of this application can generate extremely detailed static background and dynamic foreground object models, greatly improving the realism of the simulation scene and making the test environment closer to the complexity of the real world. 2. Improved model generation efficiency and automation: Utilizing automated data processing workflows, including point cloud segmentation and dynamic object tracking, the technical solution of this application significantly reduces the workload of manual modeling, improves the efficiency of model generation, enables the system to adapt to environmental changes more quickly, and ensures the timeliness of the model. 3. Balancing real-time performance and dynamic scene simulation: The optimized rendering method ensures that while providing high-precision models, it also meets real-time requirements. This is crucial for simulating rapidly changing traffic conditions in autonomous driving, such as emergency avoidance and handling of sudden accidents, enhancing the effectiveness of the test. 4. Enhanced Simulation Platform Flexibility and Scene Customization Capabilities: The proposed plug-in solution (applicable to software such as EU5 or Unity) not only facilitates the integration of high-fidelity models into existing simulation platforms but also provides users with an interactive interface to adjust and customize dynamic scenes, enabling the easy creation of diverse and targeted test cases based on testing needs. In summary, the technical solution presented in this application not only improves the visual realism and dynamic interactivity of autonomous driving simulation testing but also effectively enhances the efficiency and accuracy of simulation testing through automated tools and real-time rendering technology, providing a more powerful and reliable testing environment for the development and verification of autonomous driving systems.

[0082] based on Figure 1 In addition to the method described herein, this specification also provides some specific implementation schemes of the method, which will be described below.

[0083] In an optional embodiment, the step of performing video frame extraction on the video data to obtain a number of video frames may include:

[0084] The acquired LiDAR point cloud data is subjected to denoising and filtering processes, specifically including: first, identifying and removing outliers or noise points in the LiDAR point cloud data; then, using statistical filtering or geometric filtering to smooth the LiDAR point cloud data while retaining key structural information in the point cloud; wherein, statistical filtering is based on the characteristics of points in the neighborhood, including the average or median, to remove points that deviate from the normal range; and geometric filtering utilizes the spatial relationships between points to eliminate points that do not conform to the expected geometric shape.

[0085] In an optional embodiment, before performing video frame extraction on the video data to obtain a number of video frames, the method further includes: hardware time synchronization of the vehicle-mounted camera and the LiDAR, which may specifically include:

[0086] The vehicle-mounted camera and the laser radar are synchronized with a common time source by an external global positioning system time device or a high-precision clock built in the vehicle, and the vehicle-mounted camera and the laser radar record data according to the common time source, so as to ensure consistency of data acquisition timestamps.

[0087] In an optional embodiment, the 3DGS model of the static background and the 3DGS model of the dynamic foreground object are integrated into the automatic driving simulation software for testing the automatic driving vehicle in the form of a plug-in, which can specifically include:

[0088] S1, model import, specifically including: importing the 3DGS model of the static background generated by the 3D Gaussian sputtering technology into the automatic driving simulation software;

[0089] S2, scene replacement and fine-tuning, specifically including: replacing the 3DGS model of the static background with the basic model in the original simulation environment of the automatic driving simulation software, and adjusting the 3DGS model of the static background according to specific simulation requirements to ensure that it fits the map layout and scale of the automatic driving simulation software;

[0090] S3, dynamic element integration, specifically including: in the simulation scene in which the 3DGS model of the static background has been integrated, through the developed plug-in, the user adds and runs the dynamic scene elements in the 3DGS model of the dynamic foreground object.

[0091] Meanwhile, by modifying the three-dimensional coordinates of the dynamic foreground object, the behavior of the vehicle in the original scene can be changed. Specifically, in the 3DGS training process, the static background and the dynamic foreground object are combined according to the real position for training. In the rendering process, by modifying the three-dimensional coordinates of the foreground (such as the vehicle), different behaviors of the vehicle can be realized, such as lane changing, removing, accelerating or decelerating, or editing into a collision scene, etc., so that videos of different scenes can be rendered to meet the diversified simulation test requirements.

[0092] Based on the same idea, the present specification embodiment also provides a device corresponding to the above method. Figure 13 The structure of an automatic driving simulation test optimization device corresponding to Figure 1 provided by the present specification embodiment is shown in the structure diagram. As Figure 13 shown, the device can include:

[0093] The data acquisition module 1302 is configured to synchronously acquire video data and LiDAR point cloud data of a real scene requiring scene reconstruction based on a data acquisition device; wherein the data acquisition device comprises a vehicle-mounted camera and a laser radar, the data acquisition device is installed on an experimental vehicle running in the real scene, the vehicle-mounted camera is configured to acquire the video data, and the laser radar is configured to acquire the LiDAR point cloud data.

[0094] The video data processing module 1304 is configured to perform a video frame extraction operation on the video data to obtain a plurality of video frames, perform data labeling on the plurality of video frames frame by frame based on motion features of elements included in the video frames, so as to distinguish dynamic foreground objects and static background elements included in each video frame in the plurality of video frames, then track dynamic objects in the video data, and further label three-dimensional coordinate data and timestamp information of the dynamic foreground objects included in each video frame, so as to obtain a plurality of labeled video frames.

[0095] The point cloud segmentation module 1306 is configured to separate dynamic objects from static backgrounds in the LiDAR point cloud data, so as to distinguish static elements and dynamic elements, and obtain LiDAR point cloud data after point cloud segmentation.

[0096] The 3DGS training module 1308 is configured to input the LiDAR point cloud data after point cloud segmentation and the plurality of labeled video frames into a 3DGS model for parallel training, so as to obtain a 3DGS model of a static background and a 3DGS model of a dynamic foreground object.

[0097] The plug-in integration module 1310 is configured to integrate the 3DGS model of the static background and the 3DGS model of the dynamic foreground object into an automatic driving simulation software for testing an automatic driving vehicle in the form of plug-ins.

[0098] It can be understood that each of the above modules refers to a computer program or program segment, and is configured to perform one or more specific functions. In addition, the division of each of the above modules does not mean that the actual program code must also be separated.

[0099] The above describes specific embodiments of the present specification, and in some cases, the actions or steps recorded in the claims can be executed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

[0100] The various embodiments in the specification are described in progressive manner, and the same or similar parts between the various embodiments can be mutually referred to.

[0101] The device, equipment and method provided by the embodiments of the present specification are corresponding, and therefore the device and equipment also have similar beneficial technical effects to the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding device and equipment will not be described here again.

[0102] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than the order in which the acts or steps are recited in the embodiments and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or necessary.

[0103] In the 1990s, it was relatively easy to distinguish whether an improvement in a technology was a hardware improvement (e.g., an improvement in the circuit structure of a diode, transistor, switch, etc.) or a software improvement (an improvement in a method flow). However, as technology has evolved, many improvements in method flows today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flows into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using hardware entity modules. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A designer programs a digital system "integrated" on a PLD by himself, without having to ask a chip manufacturer to design and manufacture a special integrated circuit chip. Moreover, instead of manually manufacturing integrated circuit chips, this programming is now mostly implemented using "logic compiler" software, which is similar to the software compiler used when developing programs, and the original code before compilation must also be written in a specific programming language, which is called a hardware description language (HDL), and there are many types of HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that it is easy to obtain a hardware circuit that implements a logical method flow by simply logically programming the method flow in the above-mentioned hardware description languages and programming it into an integrated circuit.

[0104] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can also be implemented to perform the same functions in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can even be considered as both a software module implementing a method and a structure within a hardware component.

[0105] The systems, apparatuses, modules or units illustrated by the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0106] For the sake of description, the above apparatuses are described in various units by functions respectively. Of course, the functions of each unit can be implemented in the same or multiple software and / or hardware in the implementation of the present application.

[0107] Those skilled in the art will understand that the embodiments of the technical solutions of the present application can be provided as a method, a system or a computer program product. Therefore, the technical solutions of the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the technical solutions of the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0108] The technical solutions of the present application are described with reference to flowcharts and / or block diagrams of the embodiments of the technical solutions of the present application, methods, devices (systems), and computer program products. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks. Figure 1 one or more flows and / or blocks.

[0109] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks. Figure 1 one or more flows and / or blocks.

[0110] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks. Figure 1 one or more flows and / or blocks.

[0111] In a typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memories.

[0112] The memory can include non-persistent memory in the computer readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer readable medium.

[0113] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0114] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but also other elements not explicitly listed or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0115] Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0116] The present application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0117] The above merely provides an example of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the scope of claims of the present application.

Claims

1. An optimization method for autonomous driving simulation testing, characterized in that, include: The system synchronously collects video data and LiDAR point cloud data of the real scene that needs to be reconstructed using data acquisition equipment; wherein, the data acquisition equipment includes a vehicle-mounted camera and a LiDAR, the data acquisition equipment is installed on an experimental vehicle traveling in the real scene, the vehicle-mounted camera is used to collect the video data, and the LiDAR is used to collect the LiDAR point cloud data; The video data is subjected to video frame extraction to obtain a number of video frames. Based on the motion characteristics of the elements included in the video frames, the number of video frames are labeled frame by frame to distinguish the dynamic foreground objects and static background elements included in each video frame. Then, the dynamic objects in the video data are tracked, and the three-dimensional coordinate data and timestamp information of the dynamic foreground objects included in each video frame are labeled to obtain a number of labeled video frames. The dynamic objects in the LiDAR point cloud data are separated from the static background to distinguish between static and dynamic elements, resulting in LiDAR point cloud data after point cloud segmentation. The LiDAR point cloud data after point cloud segmentation and several labeled video frames are input into the 3DGS model for parallel training to obtain a 3DGS model of static background and a 3DGS model of dynamic foreground object. The 3DGS models of the static background and the dynamic foreground objects are integrated into autonomous driving simulation software for testing autonomous vehicles in the form of plugins.

2. The optimization method for autonomous driving simulation testing according to claim 1, characterized in that, Before performing video frame extraction on the video data to obtain a number of video frames, the following steps are included: The acquired LiDAR point cloud data is subjected to denoising and filtering processes, specifically including: first, identifying and removing outliers or noise points in the LiDAR point cloud data; then, using statistical filtering or geometric filtering to smooth the LiDAR point cloud data while retaining key structural information in the point cloud; wherein, statistical filtering is based on the characteristics of points in the neighborhood, including the average or median, to remove points that deviate from the normal range; and geometric filtering utilizes the spatial relationships between points to eliminate points that do not conform to the expected geometric shape.

3. The optimization method for autonomous driving simulation testing according to claim 1, characterized in that, Before performing video frame extraction on the video data to obtain a number of video frames, the method further includes: hardware time synchronization of the vehicle-mounted camera and the lidar, specifically including: The vehicle-mounted camera and the lidar are synchronized with a common time source by an external GPS timing device or a high-precision clock built into the vehicle. The vehicle-mounted camera and the lidar record data according to this common time source, thereby ensuring the consistency of data acquisition timestamps.

4. The optimization method for autonomous driving simulation testing according to claim 1, characterized in that, The tracking of dynamic objects in the video data specifically includes: S1. Selecting a reference frame and adjacent frames, specifically including: First, selecting a frame from the video data as a reference frame, and then selecting one or more subsequent frames as adjacent frames. S2. Calculate the pixel gradient, specifically including: for each pixel in the reference frame and adjacent frames, calculate its brightness gradient; S3. Matching feature points: Using pixel gradient information, find the correspondence between the same feature points in the reference frame and adjacent frames; S4. Estimating motion vectors: Based on the correspondence of feature points, the motion vector of each pixel from the reference frame to the adjacent frame is estimated through an iterative optimization algorithm. The vector includes the displacement components of the pixel in the horizontal and vertical directions. S5. Timestamp Allocation: Combining the estimated motion vectors and the shooting time interval of video frames, time information is assigned to each pixel of the dynamic object to form a dynamic model with a time dimension.

5. The optimization method for autonomous driving simulation testing according to claim 1, characterized in that, The tracking of dynamic objects in the video data specifically includes: S1. Feature Detection: In each frame of the video data, a feature detection algorithm is applied to extract stable feature points; for each detected feature point, its descriptor is calculated, which is used to describe the vector of image content around the feature point. S2. Feature Description: Matching feature points by comparing feature descriptors between two or more consecutive frames; S3. Geometric Verification: To further eliminate incorrect matches, the Random Sampling Consensus (RANSAC) method is used to estimate the transformation matrix based on the matching point pairs, and only matching point pairs that meet the geometric constraints are retained. S4. Trajectory Tracking: Through cross-frame feature matching and geometric verification, the trajectory of feature points over time is established, and a unique identifier is assigned to each continuously tracked feature point sequence to indicate that it belongs to the same dynamic object. S5. Timestamp annotation: Whenever a feature point is successfully matched in a frame and assigned to a trajectory, the precise time information of that frame is recorded as a timestamp. In this way, as time goes by, the trajectory of each dynamic object is associated with a series of timestamps, forming a dynamic object motion trajectory with time information. The above process is used to track dynamic objects in the video and add a timestamp to the motion trajectory of each object.

6. The optimization method for autonomous driving simulation testing according to claim 1, characterized in that, The integration of the 3DGS model of the static background and the 3DGS model of the dynamic foreground object into the autonomous driving simulation software for testing autonomous vehicles, in the form of a plugin, specifically includes: S1. Model import, specifically including: importing the 3DGS model of the static background generated by 3D Gaussian sputtering technology into the autonomous driving simulation software; S2. Scene replacement and fine-tuning, specifically including: replacing the basic model in the original simulation environment of the autonomous driving simulation software with the 3DGS model of the static background, and adjusting the 3DGS model of the static background according to specific simulation requirements to ensure that it fits the map layout and scale of the autonomous driving simulation software. S3. Dynamic element integration, specifically including: in a simulation scene that has already integrated the static background 3DGS model, through a developed plugin, users can add and run dynamic scene elements in the 3DGS model of the dynamic foreground object.

7. An optimization device for autonomous driving simulation testing, characterized in that, include: A data acquisition module is used to simultaneously acquire video data and LiDAR point cloud data of the real scene that needs to be reconstructed based on a data acquisition device; wherein, the data acquisition device includes a vehicle-mounted camera and a LiDAR, the data acquisition device is installed on an experimental vehicle traveling in the real scene, the vehicle-mounted camera is used to acquire the video data, and the LiDAR is used to acquire the LiDAR point cloud data; The video data processing module is used to perform video frame extraction on the video data to obtain a number of video frames. Based on the motion characteristics of the elements included in the video frames, the module performs data annotation on each of the video frames to distinguish between dynamic foreground objects and static background elements in each video frame. Then, the module tracks the dynamic objects in the video data and annotates the three-dimensional coordinate data and timestamp information of the dynamic foreground objects included in each video frame to obtain the annotated video frames. The point cloud segmentation module is used to separate dynamic objects from static backgrounds in the LiDAR point cloud data to distinguish between static and dynamic elements, thereby obtaining LiDAR point cloud data after point cloud segmentation. The 3DGS training module is used to input the LiDAR point cloud data after point cloud segmentation and several labeled video frames into the 3DGS model for parallel training, so as to obtain the 3DGS model of static background and the 3DGS model of dynamic foreground object. The plug-in integration module is used to integrate the 3DGS model of the static background and the 3DGS model of the dynamic foreground object into the autonomous driving simulation software for testing autonomous vehicles in the form of plug-ins.

Images