Automatic driving test data enhancement method

Abstract

The invention discloses an automatic driving test data enhancement method, and the main design concept of the method lies in that a large amount of generalization approaching real rendering is realized based on original collection data, specifically, test data not only comprise original data, but also comprises data which is subjected to 3D reconstruction and rationality judgment and processing, and the test data is more accurate. Therefore, generalization of the scene is realized and a more reasonable rendering scene graph is obtained. Then, through an attack resisting mechanism of the scene picture, a large number of corner scenes are further added, and the test data to be used are strengthened and generalized again; the problems of insufficient automatic driving simulation test data, lack of corner scenes and the like are solved, the scene rendering effect is effectively improved, and the test database is enriched and strengthened, so that the training test requirement of an automatic driving large model is met, and the robustness of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving test technology, and in particular to an autonomous driving test data enhancement method. Background Art

[0002] With the rapid development of autonomous driving technology, autonomous driving models and algorithms are gradually transitioning from prior-based perception-decision-planning and control methods to large models. The maturity and reliability of large models depend on data training and verification covering various scenarios. Therefore, a database with large data volume and wide coverage is essential for the mature application of large models, especially the mining and reconstruction of corner scenes, which is crucial to the development of autonomous driving.

[0003] The test data for autonomous driving comes from data collection scenarios, shadow mode fast upload scenarios, simulation and construction of virtual scenarios, etc. For regulatory control, a scenario library that directly transmits perception results can be used for testing and verification. For perception models or large models, they rely more on real data and close-to-real scene conversion data, mainly pictures of the camera footage and related parameter data. The scene data needs to cover a wide range, especially rare corner scenes, which are particularly important for model training and verification. The strengthening of autonomous driving test data means more aggressive processing of the relevant scenes / data collected or established by simulation to obtain scenes that are more in line with dangerous scenarios or rare corner scenes for autonomous driving.

[0004] One of the current methods for enhancing autonomous driving test data is the Open series of standard structured data based on data acquisition scenarios, including Opendrive (static scenes), OpenScenario (dynamic scenes), OpenCRG (road details), etc. In the process of generating simulation scenarios, some of the data are generalized to cover aggressive driving. For example, the target vehicle's speed, lane change time, and driving habits. At the same time, weather and other conditions can be added through simulation software. When generating simulation scenes, the data is converted into a virtual scene through rendering by the simulation engine. This technology is adapted to regulatory control algorithms, but because the rendering of the simulation software will cause distortion, and the rendering itself cannot achieve 100% restoration, the credibility of the input of the end-to-end large model is controversial, and the effect is not ideal. Summary of the invention

[0005] In view of the above, the present invention aims to provide an autonomous driving test data enhancement method to solve the above-mentioned technical problems.

[0006] The technical solution adopted by the present invention is as follows:

[0007] The present invention provides an autonomous driving test data enhancement method, which includes:

[0008] Collect test scene data to obtain original data;

[0009] Label the original data and reconstruct the 3D scene;

[0010] The initial reconstruction scene data is increased, decreased and labeled based on rationality;

[0011] The reconstructed image data after rational processing is merged with the original image data to generate adversarial images and annotate them;

[0012] All annotated data are stored for testing purposes.

[0013] In at least one possible implementation, the rationality processing includes:

[0014] Make a reasonable judgment on the reconstructed image, and add or delete frames based on the judgment result;

[0015] The reconstructed point cloud data is matched with corresponding frames using the image data after rational processing, and invalid point cloud data is deleted.

[0016] In at least one possible implementation, the rationality judgment includes detecting whether the reconstructed image data has at least one of the following anomalies: image display ghosting, objects crossing in the image, and target vehicles or people floating in the image.

[0017] In at least one possible implementation manner, the interpolating or deleting the picture according to the judgment result includes:

[0018] After removing abnormal reconstructed images, detect the number of missing frames;

[0019] When the number of missing frames is less than or equal to a preset first frame number threshold, directly copying a corresponding number of previous image frames as supplements for the missing frames;

[0020] When the number of missing frames exceeds the first frame number threshold and is less than or equal to the second frame number threshold, frame supplementation is performed using motion interpolation;

[0021] When the number of missing frames exceeds the second frame number threshold, the corresponding reconstructed scene data is deleted.

[0022] In at least one possible implementation, generating an adversarial image includes performing local and global perturbations on the image, wherein the global perturbation includes at least one of the following: noise attack, weather condition attack, and image blur attack; and the local perturbation includes at least one of the following: light and shadow attack, foreign object attack, rain, mud and soil splashing attack, and road damage attack.

[0023] In at least one possible implementation, the 3D scene reconstruction includes: generating a generalized scene group using a 3D reconstruction model for image data and point cloud data obtained by de-framing the collected video.

[0024] Compared with the prior art, the main design concept of the present invention is to achieve a large number of generalizations of near-real renderings based on the original collected data. Specifically, the test data includes not only the original data, but also the data reconstructed by 3D and processed for rationality judgment, thereby achieving the generalization of the scene and obtaining a more reasonable rendering scene graph; then, through the adversarial attack mechanism of the scene image, a large number of corner scenes are further added, and the test data to be used is strengthened and generalized again. The present invention solves the problems of insufficient test data and lack of corner scenes in autonomous driving simulation, effectively improves the scene rendering effect, and enriches and strengthens the test database, thereby meeting the training and testing requirements of large autonomous driving models and improving the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described below with reference to the accompanying drawings, in which:

[0026] Figure 1 A schematic diagram of a flow chart of an autonomous driving test data enhancement method provided by an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of a method for judging the rationality of a reconstructed scene graph provided by an embodiment of the present invention;

[0028] Figure 3 A schematic diagram of a motion interpolation frame supplementation method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0030] The present invention proposes an embodiment of an autonomous driving test data enhancement method. Specifically, Figure 1 shown, including:

[0031] Step S1, collecting test scene data to obtain original data;

[0032] In actual operation, the collected raw data can be cleaned and sliced ​​to obtain the original data of the scene fragment.

[0033] Step S2: annotate the original data and simultaneously reconstruct the scene using the 3D reconstruction model;

[0034] Specifically, the collected video is deframed and converted to obtain frame-by-frame image data and point cloud data. In addition to being directly sent to the annotation module, it can also be input into, for example, but not limited to, a 3D Gaussian sputtering module (3D Gaussian sputtering: a rasterization technology described by 3D Gaussian distribution for real-time radiation field rendering, with high quality and real-time rendering capabilities; the basic idea of ​​this technology is to estimate an explicit sparse point cloud from a set of images, and for each point in the point cloud, a Gaussian ellipsoid probability prediction model is constructed. Through neural network learning, a discrete representation similar to voxels is obtained, which supports multi-angle rendering and rasterization), and the scene is reconstructed using a 3D reconstruction model to generate a generalized scene group.

[0035] In this example, 3D Gaussian sputtering reconstruction reconstructs the original data to generate a 3D perspective scene. Through perspective transformation and target object transformation, richer transformation scene images can also be generated, thereby obtaining a large number of reconstructed images.

[0036] Step S3, adding or subtracting the initial reconstructed scene data based on rationality and marking it;

[0037] Next, the rationality of the reconstructed image is judged, and the image is deleted or frames are added accordingly; the reconstructed point cloud data is matched with corresponding frames using the rationally processed image data, and invalid point cloud data is deleted, thereby obtaining the final reconstructed scene data containing the image and point cloud.

[0038] Regarding the rationality judgment, it is explained here that the image data of the reconstructed scene is judged whether there is serious ghosting, whether objects are crossed, and whether the target vehicle / pedestrian is suspended. After screening, the reconstructed scene image is supplemented with frames; and when the consecutive missing frames exceed 3 frames (can be changed to other frame numbers according to the specific frame rate), the reconstructed scene can be deleted without subsequent annotation and storage processing; the point cloud data of the reconstructed scene can be obtained through the simulation software radar sensor, and when it is determined to delete a scene after the rationality judgment of the reconstructed scene image, the corresponding point cloud data must also be deleted. On the contrary, when the data is judged to be correct or becomes valid through frame supplementation, the point cloud data can enter the subsequent annotation and storage processing. Through the above process, the present invention avoids the limitations of 3D scene reconstruction and the roughness problems such as ghosting of rendered data when the perspective changes.

[0039] For the specific implementation logic of rationality judgment of reconstructed image data, please refer to Figure 2 As shown in the figure: when one frame is missing, the previous frame is directly copied as the missing frame; when two or three frames are missing, motion interpolation is used to fill in the missing frames. For further details, the specific implementation process of motion interpolation can be referred to Figure 3 Illustration: Motion curve fitting is performed on the first two frames of the missing frame, that is, the first frame after the missing frame, a total of 3 frames. The color blocks of the previous frame F1 are used as queries one by one, and the nine-square grid (when the edge color blocks are not enough for the nine-square grid, only the existing color blocks are queried) with the current color block as the center of the F2 frame to perform color block matching to determine the best matching position; similarly, the corresponding matching position with F0 is queried backward, and the corresponding matching positions of the three frames are fitted to interpolate the image of the middle missing frame. It can be understood by those skilled in the art that in this example, the motion interpolation frame filling present invention uses a total of 3 frames for fitting. In actual operation, several frames can be added forward and backward to further refine the fitting.

[0040] Step S4, fusing the reconstructed image data processed by rationality with the original image data to generate an adversarial image and annotate it;

[0041] As mentioned above, on the one hand, the reconstructed scene data is sent to the labeling module as generalized test data; on the other hand, the image data of the reconstructed scene and the collected original frame-by-frame images can be sent to the adversarial attack module (adversarial attack: a specific attack method against machine learning models, whose purpose is to deceive the model and cause model output errors by making small and carefully designed changes to the input data) to generate adversarial images and send them to the labeling module.

[0042] To elaborate, the adversarial attack data includes two parts: adversarial attack processing of original data and adversarial attack processing of 3D reconstruction data. The adversarial attack module generates an adversarial image by performing adversarial disturbances on the above two parts of image data. The adversarial disturbances include local disturbances and global disturbances. Global disturbances specifically include noise attacks, weather condition attacks, and image blur attacks. Among them, noise also specifically includes Gaussian noise, uniform noise, impulse noise, etc., weather conditions also specifically include rain, snow, fog, frost, splashing, etc., and image blur also specifically includes object motion blur, camera shooting blur, etc.; local disturbances specifically include light and shadow attacks, foreign objects, rain, mud and soil splashing, road damage, etc. Through the above process, the present invention overcomes the defect that scene reconstruction can only control the motion of objects based on elements in the original data, and can also effectively enhance the data for aspects involving extreme weather and data interference.

[0043] Step S5: store all the marked data for testing.

[0044] Finally, the initially collected data is stored in the database after the above processing. It can be understood that the labeled data sent to the database contains at least three parts of data: the collected original data, 3D reconstruction data, and generated adversarial image data, which greatly enhances the scene data required for autonomous driving testing.

[0045] In summary, the main design concept of the present invention is to achieve a large number of generalizations of near-real renderings based on the original collected data. Specifically, the test data includes not only the original data, but also the data that is reconstructed using 3D and has been judged and processed for rationality, thereby achieving the generalization of the scene and obtaining a more reasonable rendering scene graph; then, through the adversarial attack mechanism of the scene image, a large number of corner scenes are further added, and the test data to be used is strengthened and generalized again. The present invention solves the problems of insufficient test data and lack of corner scenes in autonomous driving simulation, effectively improves the scene rendering effect, and enriches and strengthens the test database, thereby meeting the training and testing requirements of large autonomous driving models and improving the robustness of the model.

[0046] If the expressions expressing orientation are mentioned in the embodiments of the present invention, they are relative concepts based on the embodiments. In addition, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, wherein a, b, c can be single or multiple.

[0047] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings, but the above is only a preferred embodiment of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred methods can be reasonably combined and matched into a variety of equivalent schemes by those skilled in the art without departing from or changing the design ideas and technical effects of the present invention; therefore, the present invention is not limited to the scope of implementation shown in the drawings, and all changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which still do not exceed the spirit covered by the specification and drawings, should be within the protection scope of the present invention.

Claims

1. A method for enhancing autonomous driving test data, characterized in that: include: Collect test scene data to obtain original data; Label the original data and reconstruct the 3D scene; The initial reconstruction scene data is increased, decreased and labeled based on rationality; The reconstructed image data after rational processing is merged with the original image data to generate adversarial images and annotate them; All annotated data are stored for testing purposes.

2. The method for enhancing autonomous driving test data according to claim 1, characterized in that: The reasonableness processing includes: Make a reasonable judgment on the reconstructed image, and add or delete frames based on the judgment result; The reconstructed point cloud data is matched with the corresponding frames using the image data after rational processing, and the invalid point cloud data is deleted.

3. The method for enhancing autonomous driving test data according to claim 2, characterized in that: The rationality judgment includes detecting whether the reconstructed image data has at least one of the following anomalies: image display ghosting, objects crossing in the image, and target vehicles or people floating in the image.

4. The method for enhancing autonomous driving test data according to claim 2, characterized in that: The picture interpolation or deletion according to the judgment result includes: After removing abnormal reconstructed images, detect the number of missing frames; When the number of missing frames is less than or equal to a preset first frame number threshold, directly copying a corresponding number of previous image frames as supplements for the missing frames; When the number of missing frames exceeds the first frame number threshold and is less than or equal to the second frame number threshold, frame supplementation is performed using motion interpolation; When the number of missing frames exceeds the second frame number threshold, the corresponding reconstructed scene data is deleted.

5. The method for enhancing autonomous driving test data according to claim 2, characterized in that: The generating of adversarial images includes local perturbations and global perturbations on the images, wherein the global perturbations include at least one of the following: noise attacks, weather condition attacks, and image blur attacks; the local perturbations include at least one of the following: light and shadow attacks, foreign body attacks, rain, mud and soil splashing attacks, and road damage attacks.

6. The method for enhancing autonomous driving test data according to any one of claims 1 to 5, characterized in that: The 3D scene reconstruction includes: generating a generalized scene group using a 3D reconstruction model by transforming the acquired video into image data and point cloud data by frame decomposition.

Images