Data set construction method and system for automatic driving and computer device

Abstract

The invention relates to the field of data set construction, and discloses a data set construction method and system for automatic driving and a computer device, and the method comprises the steps: collecting all-round video data on a driving path through a plurality of cameras, and collecting point cloud data on the driving path through a laser radar; performing internal reference calibration among the plurality of cameras to obtain internal reference of the cameras; performing external parameter calibration between the plurality of cameras and the laser radar to obtain external parameters from the plurality of cameras to the laser radar; labeling instances in the point cloud data by using a 3D labeling tool, obtaining instance labeling information and generating a target frame; and based on the acquired video, the point cloud data, the internal and external parameters, the instance labeling information and the target frame, integrating a data set used for automatic driving algorithm training and verification. According to the method and the device, the data set acquired by the user can be converted into data in a similar Nuscenes format, so that the self-acquired data set is used for training and verifying a plurality of existing driving perception algorithms.

Description

Technical Field

[0001] The present application relates to the field of data set construction, and more specifically, to a data set construction method, system and computer device for autonomous driving. Background Art

[0002] With the rapid development of artificial intelligence technology, intelligent driving algorithms and autonomous driving systems are also constantly improving. For example, autonomous driving perception algorithms such as BEVFormer, BEVFusion, BEVDet, and FlashOcc are constantly refreshing the accuracy rankings and improving the robustness of the results. However, these algorithms are trained and verified on large public data sets such as Nuscenes. At the same time, the data structure of Nuscenes is very complex, and there is no method to generate open source data structures. Therefore, how to construct our self-collected data into a data format that can be directly called by the perception algorithm for training and verification is an urgent problem to be solved. Summary of the invention

[0003] In order to solve the above technical problems, the present application provides a data set construction method, system and computer device for autonomous driving. Through the steps of the specific data set construction method, the calibrated sensor internal parameters, external parameters and instance information around the autonomous driving path are organized and generated into a data set that can be directly called by the autonomous driving algorithm for training and verification.

[0004] Specifically, the technical solution of this application is as follows:

[0005] In a first aspect, the present application discloses a method for constructing a data set for autonomous driving, comprising:

[0006] Collect surround video data on the driving path by using multiple cameras installed on the data collection vehicle, and collect point cloud data on the driving path by using a laser radar installed on the data collection vehicle;

[0007] Performing internal parameter calibration between the plurality of cameras to obtain camera internal parameters; performing external parameter calibration between the plurality of cameras and the laser radar to obtain external parameters from the plurality of cameras to the laser radar;

[0008] Annotate the instances in the point cloud data using a 3D annotation tool, obtain instance annotation information and generate a target frame;

[0009] Based on the surround video data, the point cloud data, the camera intrinsic parameters, the extrinsic parameters from the camera to the lidar, the instance annotation information and the target box, a data set for autonomous driving algorithm training and verification is integrated and generated.

[0010] In some embodiments, the step of integrating and generating a data set for autonomous driving algorithm training and verification based on the surround view video data, the point cloud data, the camera intrinsic parameters, the external parameters from the camera to the lidar, the instance annotation information, and the target box comprises the following steps:

[0011] The dataset used for autonomous driving algorithm training and verification is the Nuscenes dataset;

[0012] Obtain the instance annotation information corresponding to each frame of the point cloud data, integrate all the instance annotation information, and construct an instance information file;

[0013] Based on the instance information file and the instance annotation information, integrating all the target frame information to generate a target frame information file;

[0014] Based on the target frame information file, integrating the frame number information of the surround view video data and the point cloud data to generate a frame number information file;

[0015] Integrate the internal and external parameters of the camera and the laser radar to construct an internal and external parameter information file;

[0016] Based on the target frame information file, integrating the detail information of the surround view video data and the point cloud data to construct a sampling data file;

[0017] Integrate the frame number information file to construct a scene information file;

[0018] Based on the internal and external reference information files, integrating the sensor information of the camera and the laser radar to construct a sensor information file;

[0019] The Nuscenes data set is constructed based on the instance information file, the target frame information file, the frame number information file, the internal and external parameter information file, the sampling data file, the scene information file, and the sensor information file.

[0020] In some implementations, the step of using a 3D annotation tool to annotate instances in the point cloud data, obtaining instance annotation information, and generating a target frame comprises the following steps:

[0021] Annotate the instances in the point cloud data using the Sustechpoints 3D annotation tool to obtain the instance annotation information, wherein the instance annotation information includes the name of the instance, the number of occurrences of the instance, the location of the instance, the direction of the instance, and the size of the instance in each frame of the point cloud data;

[0022] The target frame is generated one by one for each instance of the anchor frame in the point cloud data of each frame.

[0023] In some implementations, the instance information file includes an index of the instance, a category of the instance, a name of the instance, a number of occurrences of the instance in previous and next frames, and a number of the instance;

[0024] The target frame information file includes the index of the target frame, the timestamp of the point cloud data corresponding to the target frame, the file name of the corresponding point cloud data, the index of the corresponding point cloud data frame, the index of the instance corresponding to the target frame, the position of the instance, the size of the instance, the quaternion obtained by converting the direction of the instance, the degree of occlusion of the instance, and the indexes of the target frames before and after;

[0025] The frame number information file includes: the index of the data frame of the point cloud data, the timestamp of the point cloud data, the index of the corresponding scene, and the index of the previous and next timestamps;

[0026] The internal and external parameter information file includes: the index of the internal and external parameter information, the index of the sensor information, the internal parameter matrix, the external parameter translation matrix, and the external parameter rotation matrix;

[0027] The sampling data file includes: the index of the surround view video data or the point cloud data, the timestamp of the point cloud data, the index of the data frame of the surround view video data or the point cloud data, whether the data frame is a key frame, the suffix of the data format, the height of the surround view video data, the width of the surround view video data, the file path of the surround view video data or the point cloud data, and the position and posture of the data collection vehicle;

[0028] The scene information file includes: an index of the scene, an index of the data frame where the scene appears for the first time, and an index of the data frame where the scene appears for the last time;

[0029] The sensor information file includes: the index of the internal and external parameter information, the mode of the sensor, and the installation location of the sensor.

[0030] In some implementations, the step of performing internal calibration between the plurality of cameras comprises the following steps:

[0031] The internal reference calibration between the multiple cameras is performed using Zhang Zhengyou calibration method.

[0032] In some implementations, the extrinsic calibration between the plurality of cameras and the laser radar comprises the following steps:

[0033] The lidar2camera tool is used to perform external parameter calibration between the multiple cameras and the laser radar.

[0034] In a second aspect, the present application provides a data set construction system for autonomous driving, comprising:

[0035] Multiple cameras are installed on the data collection vehicle to collect surround video data on the driving path;

[0036] A laser radar, installed on the data collection vehicle, for collecting point cloud data on the driving path;

[0037] An internal parameter calibration module, used to perform internal parameter calibration between the plurality of cameras to obtain camera internal parameters;

[0038] An external parameter calibration module, used to perform external parameter calibration between the plurality of cameras and the laser radar, and obtain external parameters from the plurality of cameras to the laser radar;

[0039] A point cloud annotation module, used to annotate instances in the point cloud data using a 3D annotation tool, obtain instance annotation information and generate a target frame;

[0040] A data set integration module is used to integrate and generate a data set for autonomous driving algorithm training and verification based on the surround view video data, the point cloud data, the camera intrinsic parameters, the external parameters from the camera to the lidar, the instance annotation information and the target box.

[0041] In some implementations, the data set integration module is specifically used to: obtain the instance annotation information corresponding to each frame of the point cloud data, integrate all the instance annotation information, and construct an instance information file;

[0042] Based on the instance information file and the instance annotation information, integrating all the target frame information to generate a target frame information file;

[0043] Based on the target frame information file, integrating the frame number information of the surround view video data and the point cloud data to generate a frame number information file;

[0044] Integrate the internal and external parameters of the camera and the laser radar to construct an internal and external parameter information file;

[0045] Based on the target frame information file, integrating the detail information of the surround view video data and the point cloud data to construct a sampling data file;

[0046] Integrate the frame number information file to construct a scene information file;

[0047] Based on the internal and external reference information files, integrating the sensor information of the camera and the laser radar to construct a sensor information file;

[0048] A data set for autonomous driving algorithm training and verification is constructed based on the instance information file, the target frame information file, the frame number information file, the internal and external parameter information file, the sampling data file, the scene information file, and the sensor information file.

[0049] In some embodiments, the instance annotation information includes the name of the instance, the number of occurrences of the instance, the location of the instance, the direction of the instance, and the size of the instance in each frame of the point cloud data;

[0050] The instance information file includes the index of the instance, the category of the instance, the name of the instance, the number of times the instance appears in the previous and next frames, and the number of the instance;

[0051] The target frame information file includes the index of the target frame, the timestamp of the point cloud data corresponding to the target frame, the file name of the corresponding point cloud data, the index of the corresponding point cloud data frame, the index of the instance corresponding to the target frame, the position of the instance, the size of the instance, the quaternion obtained by converting the direction of the instance, the degree of occlusion of the instance, and the indexes of the target frames before and after;

[0052] The frame number information file includes: the index of the data frame of the point cloud data, the timestamp of the point cloud data, the index of the corresponding scene, and the index of the previous and next timestamps;

[0053] The internal and external parameter information file includes: the index of the internal and external parameter information, the index of the sensor information, the internal parameter matrix, the external parameter translation matrix, and the external parameter rotation matrix;

[0054] The sampling data file includes: the index of the surround view video data or the point cloud data, the timestamp of the point cloud data, the index of the data frame of the surround view video data or the point cloud data, whether the data frame is a key frame, the suffix of the data format, the height of the surround view video data, the width of the surround view video data, the file path of the surround view video data or the point cloud data, and the position and posture of the data collection vehicle;

[0055] The scene information file includes: an index of the scene, an index of the data frame where the scene appears for the first time, and an index of the data frame where the scene appears for the last time;

[0056] The sensor information file includes: the index of the internal and external parameter information, the mode of the sensor, and the installation location of the sensor.

[0057] In a third aspect, the present application also discloses a computer device, which at least includes a data set construction system for autonomous driving as described in any one of the above embodiments.

[0058] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0059] Through the method and system for constructing a dataset for autonomous driving described in this application, the dataset collected by oneself can be converted into data in a format similar to Nuscenes, so that researchers and developers can directly use these data to train and verify various autonomous driving perception algorithms. Not only can the collected data be fully utilized to improve the availability and value of the data, but also the research and development of autonomous driving technology can be promoted. In addition, this conversion method also helps promote the standardization of datasets and facilitate data sharing and collaboration in the field of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The preferred implementation scheme will be described below in a clear and understandable manner with reference to the accompanying drawings to further illustrate the above-mentioned characteristics, technical features, advantages and implementation methods of the present application.

[0061] Figure 1 A flowchart of the steps of an embodiment of a method for constructing a data set for autonomous driving according to the present application;

[0062] Figure 2 This is a structural block diagram of an embodiment of a data set construction system for autonomous driving in the present application. DETAILED DESCRIPTION

[0063] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0064] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections.

[0065] In order to simplify the drawings, only the parts related to the invention are schematically shown in each figure, and they do not represent the actual structure of the product. In addition, in order to simplify the drawings and facilitate understanding, in some figures, only one of the parts with the same structure or function is schematically drawn or marked. In this article, "one" not only means "only one", but also means "more than one".

[0066] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0067] In this document, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0068] In a specific implementation, the terminal device described in the embodiments of the present application includes, but is not limited to, other portable devices such as mobile phones, laptop computers, tutoring machines, or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads). It should also be understood that in some embodiments, the terminal device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0069] In addition, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0070] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the specific implementation methods of the present application will be described below with reference to the accompanying drawings. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0071] With the rapid development of artificial intelligence technology, intelligent driving algorithms and autonomous driving systems are constantly being innovated. Autonomous driving perception algorithms such as BEVFormer, BEVFusion, BEVDet, and FlashOcc are constantly refreshing the accuracy rankings and improving the robustness of the results. However, these algorithms are trained and verified on large public data sets such as Nuscenes. At the same time, the data structure of Nuscenes is very complex, and there is no method to generate open source data structures. Therefore, how to construct our self-collected data into a data format that can be trained and verified by autonomous driving perception algorithms is an urgent problem to be solved.

[0072] Among the existing solutions, most of them convert the labeled data into KITTI data format and then modify the network parameter configuration. This method requires a thorough understanding of each algorithm and is inefficient. Therefore, if you convert your collected data set into data in a format similar to Nuscenes, you can directly use it to quickly train and verify multiple autonomous driving perception algorithms.

[0073] In order to achieve the above technical effects, the present application provides an embodiment of a method for constructing a data set for autonomous driving, refer to the attached specification Figure 1 , including the following steps:

[0074] S100, collecting surround video data on a driving path through a plurality of cameras installed on a data collection vehicle, and collecting point cloud data on the driving path through a laser radar installed on the data collection vehicle.

[0075] S200, performing internal parameter calibration between the plurality of cameras to obtain camera internal parameters.

[0076] S300, calibrating the external parameters between the multiple cameras and the laser radar, and obtaining the external parameters from the multiple cameras to the laser radar.

[0077] S400: Use a 3D annotation tool to annotate instances in the point cloud data, obtain instance annotation information and generate a target frame.

[0078] S500, based on the surround view video data, the point cloud data, the camera intrinsic parameters, the external parameters from the camera to the lidar, the instance annotation information and the target box, integrate and generate a data set for autonomous driving algorithm training and verification.

[0079] Specifically, multiple cameras are dispersedly installed on the side walls of the data collection vehicle to provide a panoramic view of the vehicle, and then the data collected by multiple cameras are spliced ​​into a seamless panoramic image to obtain surround video data; the laser radar is installed on the roof of the data collection vehicle. Of course, the number of cameras used and the installation positions of the cameras and radars are determined according to actual conditions.

[0080] Implementation method 1 of this embodiment: Based on the above embodiment, step S200 performs internal parameter calibration between the multiple cameras to obtain the camera internal parameters, specifically including using Zhang Zhengyou calibration method to perform internal parameter calibration on the camera. Zhang Zhengyou calibration method is also known as the checkerboard calibration method, which is a widely used camera calibration technology. This method mainly estimates the intrinsic parameters of the camera by photographing a calibration plate with a known geometric pattern (usually a checkerboard or a dot grid). The following are the specific steps for performing internal parameter calibration on multiple cameras using Zhang Zhengyou calibration method:

[0081] S210: Use each camera to photograph a preset calibration plate.

[0082] S220, obtaining a calibration image taken by each of the cameras, and finding the position of the target corner point on the calibration image using a corner point detection algorithm.

[0083] S230, estimating the internal parameters of each camera through a calibration algorithm. The internal parameters include the focal length, principal point coordinates, pixel size, etc. of the camera.

[0084] In other implementations of this embodiment, other existing camera calibration methods may also be used to achieve the technical effect of this embodiment.

[0085] Implementation method 2 of this embodiment: On the basis of the above embodiment, step S300, performs external parameter calibration between multiple cameras and the laser radar, and obtains external parameters from multiple cameras to the laser radar, specifically including using SenseTime's open source external parameter calibration method lidar2camera to perform external parameter calibration from multiple cameras to radars. SenseTime's open source lidar2camera tool is a software for external parameter calibration between laser radar (LiDAR) and cameras, which can help users determine the relative position and posture of multiple cameras relative to a laser radar. The following are the general steps for using lidar2camera to perform external parameter calibration from multiple cameras to radars:

[0086] S310: Use the laser radar and the camera to photograph a preset calibration plate in the same coordinate system.

[0087] S320, using lidar2camera tool to detect feature points on the calibration plate.

[0088] S330, performing feature point matching between the point cloud data generated by the laser radar and the surround view video data generated by the camera.

[0089] S340, based on feature point matching, estimating the external parameters of each camera relative to the laser radar, the external parameters including a translation matrix and a rotation matrix from the camera coordinate system to the radar coordinate system.

[0090] In other implementations of this embodiment, other existing calibration methods between cameras and radars may also be used to achieve the technical effects of this embodiment.

[0091] Implementation method 3 of this embodiment: Based on the above embodiment, step S400 uses a 3D annotation tool to annotate instances in the point cloud data, obtain instance annotation information and generate a target frame, including the following steps:

[0092] S410, using Sustechpoints 3D annotation tool to annotate instances in the point cloud data to obtain instance annotation information, wherein the instance annotation information includes the name of the instance appearing in each frame of the point cloud data, the number of occurrences of the instance, the location of the instance, the direction of the instance, and the size of the instance.

[0093] S420, generating the target frame one by one for the instances of the anchor frame in each frame of the point cloud data.

[0094] Specifically, Sustechpoints 3D annotation tool is a common tool for point cloud data annotation. It is often used in fields such as autonomous driving, robot navigation, and 3D scene reconstruction to help researchers and developers annotate different instances in point cloud data.

[0095] An instance usually refers to an object that can be individually identified and distinguished in three-dimensional space. In the Sustechpoints 3D annotation tool, instance annotation of point cloud data means assigning a label or identifier to each independent object in the point cloud to distinguish different objects.

[0096] The general steps for using Sustechpoints 3D annotation tools to annotate point cloud data include:

[0097] Import the point cloud data into the Sustechpoints 3D annotation tool. Adjust the display parameters of the point cloud as needed, determine the instance categories that need to be annotated, such as people, trees, photovoltaic modules, buildings, etc., and select the corresponding categories in the tool for automatic annotation.

[0098] In other implementations of this embodiment, other existing point cloud annotation methods may also be used to achieve the technical effect of this embodiment.

[0099] Another embodiment of a method for constructing a data set for autonomous driving is disclosed in the present application. Based on an embodiment of the above method, step S500, integrating and generating a data set for autonomous driving algorithm training and verification based on the surround view video data, the point cloud data, the camera intrinsic parameters, the external parameters from the camera to the lidar, the instance annotation information, and the target box, comprises the following steps:

[0100] Based on the collected surround video data, point cloud data, camera intrinsic parameters, camera-to-radar extrinsic parameters, instance annotation information, and target frame, the following seven json format files are generated, including: instance information file, target frame information file, frame number information file, internal and external parameter information file, sampling data file, scene information file, and sensor information file.

[0101] The Nuscenes dataset is constructed based on the above 7 json target format files.

[0102] In some implementations of this embodiment, the step S500 specifically includes:

[0103] S510, obtaining the instance annotation information corresponding to each frame of the point cloud data, integrating all the instance annotation information, and constructing an instance information file; the instance information file includes the index of the instance, the category of the instance, the name of the instance, the number of times the instance appears in the previous and next frames, and the number of the instance.

[0104] S520, based on the instance information file and the instance annotation information, integrating all the target frame information to generate a target frame information file; the target frame information file includes the index of the target frame, the timestamp of the point cloud data corresponding to the target frame, the corresponding file name of the point cloud data, the corresponding index of the point cloud data frame, the index of the instance corresponding to the target frame, the position of the instance, the size of the instance, the quaternion obtained by converting the direction of the instance, the degree of occlusion of the instance, and the indexes of the target frames before and after.

[0105] S530, based on the target frame information file, integrate the frame number information of the surround video data and the point cloud data to generate a frame number information file; the frame number information file includes: the index of the data frame of the point cloud data, the timestamp of the point cloud data, the index of the corresponding scene, and the index of the previous and next timestamps.

[0106] S540, integrating the internal and external parameters of the camera and the laser radar to construct an internal and external parameter information file; the internal and external parameter information file includes: an index of internal and external parameter information, an index of the sensor information, an internal parameter matrix, an external parameter translation matrix, and an external parameter rotation matrix.

[0107] S550, based on the target frame information file, integrate the detailed information of the surround view video data and the point cloud data to construct a sampling data file; the sampling data file includes: the index of the surround view video data or the point cloud data, the timestamp of the point cloud data, the index of the data frame of the surround view video data or the point cloud data, whether the data frame is a key frame, the suffix of the data format, the height of the surround view video data, the width of the surround view video data, the file path of the surround view video data or the point cloud data, and the position and posture of the data acquisition vehicle.

[0108] S560, integrating the frame number information file to construct a scene information file; the scene information file includes: an index of the scene, an index of the data frame where the scene appears for the first time, and an index of the data frame where the scene appears for the last time.

[0109] S570, based on the internal and external parameter information file, integrating the sensor information of the camera and the laser radar, and constructing a sensor information file; the sensor information file includes: an index of the internal and external parameter information, a modality of the sensor, and an installation location of the sensor.

[0110] S580, constructing a data set for autonomous driving algorithm training and verification based on the instance information file, the target box information file, the frame number information file, the internal and external parameter information file, the sampling data file, the scene information file, and the sensor information file.

[0111] Another embodiment of a method for constructing a data set for autonomous driving in this application provides a detailed explanation of the construction process and method of seven json files in the Nuscenes data format. The specific steps are as follows:

[0112] S1. Use a data collection vehicle to collect 6-way surround video data and 1-way laser radar data. In this embodiment, take 6 cameras as an example, which are installed in the front, rear, left front, left rear, right front, and right rear of the data collection vehicle. The 6-way surround video data collected by the 6 cameras are placed in the form of pictures in the six folders CAM_BACK, CAM_BACK_LEFT, CAM_BACK_RIGHT, CAM_FRONT, CAM_FRONT_LEFT, and CAM_FRONT_RIGHT. The six folder names correspond to the distribution positions of the 6 cameras in the collection vehicle. The laser radar collects point cloud data stored in bin point cloud format and placed in the LIDAR_TOP folder.

[0113] S2. Use the camera intrinsic parameter calibration method to calibrate the internal parameters of the six cameras respectively, and the obtained internal parameters are a 4X4 internal parameter matrix; use the external parameter calibration method to calibrate the external parameters of the six cameras from camera to lidar, and the obtained external parameters are a 3x3 external parameter matrix.

[0114] S3. Use the internal parameters of the front camera CAM_FRONT and the external parameters from the front camera to the lidar to construct a front.json (front-end information) file. front.json contains two keys, extrinsic and intrinsic. Among them, extrinsic stores the external parameter matrix of the front camera, and intrinsic stores the internal parameter matrix from the camera to the lidar. front.json is a necessary internal and external parameter file for Sustechpoints annotation tool to annotate 3D point cloud data. Use Sustechpoints 3D annotation tool to annotate point cloud data. After annotation, each point cloud data will generate a corresponding label.json (instance annotation information) file. Each label in the label.json file contains five keys: obj_id, obj_type, position, rotation, and scale. obj_type is the name of the target, such as people, cars, components, etc. obj_id is the number of times the target appears in the previous and next frame images, position is the position of the target relative to the vehicle coordinate system, rotation is the orientation of the target relative to the vehicle coordinate system, and scale is the size of the target relative to the vehicle coordinate system.

[0115] S4. Write the label.json file generated in step 3 into the instance.json file (instance information file). instance.json is a collection containing all instance information, and is a file containing all instance information. In all point cloud data, there are a total of multiple different instances. Specifically, different people, different cars, and different components are different instances. Each instance contains five keys: token, category_token, nbr_annotations, category, and id. Token is a randomly generated 32-bit string, which is the index of the instance information. Category_token is a 32-bit string corresponding to the category (car, person, panel) to which this instance belongs. Category is the category name corresponding to this instance (car, person, panel). nbr_annoataions is the number of times this instance appears in the previous and next frames. ID refers specifically to an instance number. Specifically: Taking this batch of data as an example, there are a total of 334 point cloud data. After annotation, a total of 334 label.json files are generated. There are a total of 35 instances in this batch of data. Each instance appears multiple times in the previous and next frames. The sum of nbr_annotations of each instance is the total number of targets, 1303, that is, a total of 1303 3D target boxes are annotated.

[0116] S5. Write the information of the instance.json file and label.json generated in step 4 into the sample_annotation.json file (target box information file). sample_annotation.json is a collection of all target box information and is a file of all target box information. Each target box information contains 11 keys: token, timestamp, sample_token, instance_token, translation, size, rotation, visibility_token, attribute_token, prev and next. Among them, token is a randomly generated 32-bit string, which is the index of all information of this target box; timestamp is the timestamp (time point) of collecting the current point cloud information, and the timestamp value is set to the file name of the collected point cloud data, and the data type is integer; sample_token is a randomly generated 32-bit string, which is the string value of the token of the sample.json file in step 6. One pcd point cloud data corresponds to one sample_token; instance_token is the instance token to which the target box belongs, and its value is equal to the token value in the instance.json file; translation number The value is equal to the position value in the annotated label.json file; the size value is equal to the scale value in the annotated label.json file; the rotation value is equal to the rotation value in the label.json file after quaternion conversion; visibility_token indicates the degree of occlusion of the point cloud target, which is set to 4 (unobstructed) here; attribute_token indicates the attribute of the target box, indicating whether it is moving or still. Our cars are set to moving, and a 32-bit string is randomly generated here and set to a fixed value; prev and next are the string values ​​of the front and back tokens of the front and back target boxes.

[0117] S6. Write the information of the sample_annotation.json file generated in step 5 into the sample.json file (frame number information file). sample.json is the frame number information of the collected data file. The sample.json file contains five keys: token, timestamp, scene_toke, prev, and next. The token value is the value of sample_token in the sample_annotation.json file; timestamp reads the timestamp value from the sample_annotation.json file and writes it after deduplication; scene_token is the scene token, and the value is obtained from the token in the scene.json file; prev and next are the token values ​​corresponding to the previous and next timestamps.

[0118] S7, calibrated_sensor.json (internal and external parameter information file) is a file containing the internal and external parameter information of a laser radar and 6 surround view cameras. Each internal and external parameter information contains five keys: token, sensor_token, translation, rotation, and camera_intrinsic. Token is a randomly generated 32-bit string. The value of sensor_token is equal to the token value of the sensor.json file, which is the token corresponding to the 7 sensors. Camera_intrinsic is directly written by reading the internal parameter file of the second step calibration. The external parameters translation and rotation are also converted by reading the external parameter file of the second step calibration. The laser radar internal parameter is written as a null value.

[0119] S8. Write the information of the sample_annotation.json file generated in step 5 into the sample_data.json file, which is a file that stores more detailed information about each image and point cloud data. The sample_data.json file contains 12 keys: token, timestamp, sample_token, is_key_frame, calibrated_sensor_token, fileformat, height, width, filename, ego_pose_token, prev, and next. token is a randomly generated 32-bit string, which is the index of each image point cloud data information; timestamp and sample_token read the timestamp value from the sample_annotation.json file and write it directly, all the values ​​of is_key_frame are written as true, indicating that this image is a key frame, and all our annotated data are key frames; the value of calibrated_sensor_token is the token value from calibrated_sensor_token.json and written; fileformat is the suffix of the file of this data, point cloud data is written as bin, image data is written as jpg, height and width are the height and width of the image file, the two keys of point cloud data are written as empty values, filename is the file path information of the image and point cloud files, ego_pose_token is the current position of the vehicle, and the value is the value of the current token.

[0120] S9, scene.json (scene information file) is the scene information of the collected data, including token, log_token, nbr_samples, first_sample_token and last_sample_token. Token is a randomly generated 32-bit string, log_token is also a randomly generated 32-bit string, first_sample_token and last_sample_token are the first and last token values ​​of sample.json.

[0121] S10, sensor.json (sensor information file) is a file that saves information of 7 sensors. Each sensor contains three keys: token, channel and modality. The token value is read and written from sensor_token in calibrated_sensor.json. Modality is the modal information of the sensor, which is divided into camera modality and lidar point cloud modality. Channel is the sensor installation position. The camera is divided into CAM_BACK, CAM_BACK_LEFT, CAM_BACK_RIGHT, CAM_FRONT, CAM_FRONT_LEFT and CAM_FRONT_RIGHT, and the radar is LIDAR_TOP.

[0122] S11. Through the above process and program, the annotation information and sensor information can be written into seven json files: instance.json (instance information file), sample_annotation.json (target box information file), sample.json (frame number information file), calibrated_sensor.json (internal and external reference information file), sample_data.json (sampling data file), scene.json (scene information file) and sensor.json (sensor information file). These files constitute the nuscenes data format, which can be adapted to a variety of autonomous driving algorithms.

[0123] In this embodiment, a token represents a collection of information, and the stored information can be indexed through this token. In a JSON file, data is written in the form of a key / value pair (name / value), and a "key" refers to the first part of a key-value pair, which is used to identify the name or attribute of the second part "value".

[0124] Through the construction process and method of the dataset in the Nuscenes data format described in this application, the dataset collected by oneself can be converted into data in a similar format to Nuscenes, so that researchers and developers can easily use these data to train and verify various driving perception algorithms. Not only can the collected data be fully utilized to improve the availability and value of the data, but also the research and development of autonomous driving technology can be promoted. In addition, this conversion method also helps to promote the standardization of datasets and facilitate data sharing and collaboration in the field of autonomous driving.

[0125] Based on the same technical concept, the present application also discloses a data set construction system for autonomous driving, which can be used to implement any of the above-mentioned data set construction methods for autonomous driving. Specifically, an embodiment of a data set construction system for autonomous driving of the present application is shown in the attached specification. Figure 2 As shown, including:

[0126] Multiple cameras are installed in a distributed manner on the data collection vehicle to collect surround video data on the driving path.

[0127] A laser radar is installed on the data collection vehicle and is used to collect point cloud data on the driving path.

[0128] The internal parameter calibration module is used to perform internal parameter calibration between the multiple cameras to obtain camera internal parameters.

[0129] The external parameter calibration module is used to perform external parameter calibration between the multiple cameras and the laser radar, and obtain the external parameters from the multiple cameras to the laser radar.

[0130] The point cloud annotation module is used to use a 3D annotation tool to annotate instances in the point cloud data, obtain instance annotation information and generate a target frame.

[0131] A data set integration module is used to integrate and generate a data set for autonomous driving algorithm training and verification based on the surround view video data, the point cloud data, the camera intrinsic parameters, the external parameters from the camera to the lidar, the instance annotation information and the target box.

[0132] The present application provides another embodiment of a data set construction system for autonomous driving. Based on the above system embodiment, the intrinsic parameter calibration module is also used to estimate the intrinsic parameters of each camera through the Zhang Zhengyou calibration algorithm.

[0133] The external parameter calibration module is also used to estimate the external parameters of each camera relative to the laser radar using the lidar2camera tool. The external parameters include the translation matrix and the rotation matrix from the camera coordinate system to the radar coordinate system.

[0134] The point cloud annotation module is also used to annotate the point cloud data using the Sustechpoints 3D annotation tool to generate instance information on the driving path, wherein the instance information includes the name, number, location information, size information and angle information of the identified and annotated instances.

[0135] In one implementation of this embodiment, the dataset construction system for autonomous driving is a robot operating system, and the system is configured with an environment that can be used for dataset construction, specifically including lidar2camera tools, Sustechpoints 3D annotation tools, etc.

[0136] The present application provides another embodiment of a data set construction system for autonomous driving. Based on the above system embodiment, the instance annotation information includes the name of the instance appearing in each frame of the point cloud data, the number of times the instance appears, the location of the instance, the direction of the instance, and the size of the instance.

[0137] The data set integration module is specifically used to: obtain the instance annotation information corresponding to each frame of the point cloud data, integrate all the instance annotation information, and construct an instance information file; the instance information file includes the index of the instance, the category of the instance, the name of the instance, the number of times the instance appears in the previous and next frames, and the number of the instance.

[0138] Based on the instance information file and the instance annotation information, the information of all the target boxes is integrated to generate a target box information file; the target box information file includes the index of the target box, the timestamp of the point cloud data corresponding to the target box, the corresponding file name of the point cloud data, the corresponding index of the point cloud data frame, the index of the instance corresponding to the target box, the position of the instance, the size of the instance, the quaternion obtained by converting the direction of the instance, the degree of occlusion of the instance, and the indexes of the target boxes before and after.

[0139] Based on the target frame information file, the frame number information of the surround video data and the point cloud data is integrated to generate a frame number information file; the frame number information file includes: the index of the data frame of the point cloud data, the timestamp of the point cloud data, the index of the corresponding scene, and the index of the previous and next timestamps.

[0140] Integrate the internal and external parameters of the camera and the laser radar to construct an internal and external parameter information file; the internal and external parameter information file includes: an index of internal and external parameter information, an index of the sensor information, an internal parameter matrix, an external parameter translation matrix, and an external parameter rotation matrix.

[0141] Based on the target frame information file, the detailed information of the surround view video data and the point cloud data is integrated to construct a sampling data file; the sampling data file includes: the index of the surround view video data or the point cloud data, the timestamp of the point cloud data, the index of the data frame of the surround view video data or the point cloud data, whether the data frame is a key frame, the suffix of the data format, the height of the surround view video data, the width of the surround view video data, the file path of the surround view video data or the point cloud data, and the position and posture of the data acquisition vehicle.

[0142] The frame number information file is integrated to construct a scene information file; the scene information file includes: an index of the scene, an index of the data frame where the scene appears for the first time, and an index of the data frame where the scene appears for the last time.

[0143] Based on the internal and external parameter information file, the sensor information of the camera and the lidar is integrated to construct a sensor information file; the sensor information file includes: the index of the internal and external parameter information, the mode of the sensor, and the installation position of the sensor.

[0144] A data set for autonomous driving algorithm training and verification is constructed based on the instance information file, the target frame information file, the frame number information file, the internal and external parameter information file, the sampling data file, the scene information file, and the sensor information file.

[0145] Based on the same concept, the present application also discloses a computer device, which at least includes a data set construction system for autonomous driving as described in any one of the above embodiments.

[0146] The present application discloses a method, system and computer device for constructing a data set for autonomous driving with the same technical concept. The technical details of the embodiments of the two are applicable to each other. To reduce repetition, they will not be described in detail here.

[0147] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned program modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program units or modules to complete all or part of the functions described above. The program modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a processing unit, and the above-mentioned integrated unit can be implemented in the form of hardware or in the form of software program units. In addition, the specific names of the program modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0148] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0149] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0150] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0151] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for constructing a data set for autonomous driving, characterized in that: The steps include: Collect surround video data on the driving path by using multiple cameras installed on the data collection vehicle, and collect point cloud data on the driving path by using a laser radar installed on the data collection vehicle; Performing internal parameter calibration between the plurality of cameras to obtain camera internal parameters; performing external parameter calibration between the plurality of cameras and the laser radar to obtain external parameters from the plurality of cameras to the laser radar; Annotate the instances in the point cloud data using a 3D annotation tool, obtain instance annotation information and generate a target frame; Based on the surround video data, the point cloud data, the camera intrinsic parameters, the extrinsic parameters from the camera to the lidar, the instance annotation information and the target box, a data set for autonomous driving algorithm training and verification is integrated and generated.

2. A method for constructing a data set for autonomous driving according to claim 1, characterized in that: The step of integrating and generating a data set for autonomous driving algorithm training and verification based on the surround view video data, the point cloud data, the camera intrinsic parameters, the external parameters from the camera to the lidar, the instance annotation information, and the target frame comprises the following steps: The dataset used for autonomous driving algorithm training and verification is the Nuscenes dataset; Obtain the instance annotation information corresponding to each frame of the point cloud data, integrate all the instance annotation information, and construct an instance information file; Based on the instance information file and the instance annotation information, integrating all the target frame information to generate a target frame information file; Based on the target frame information file, integrating the frame number information of the surround view video data and the point cloud data to generate a frame number information file; Integrate the internal and external parameters of the camera and the laser radar to construct an internal and external parameter information file; Based on the target frame information file, integrating the detail information of the surround view video data and the point cloud data to construct a sampling data file; Integrate the frame number information file to construct a scene information file; Based on the internal and external reference information files, integrating the sensor information of the camera and the laser radar to construct a sensor information file; The Nuscenes data set is constructed based on the instance information file, the target frame information file, the frame number information file, the internal and external parameter information file, the sampling data file, the scene information file, and the sensor information file.

3. The method for constructing a data set for autonomous driving according to claim 2, characterized in that: The method of using a 3D annotation tool to annotate instances in the point cloud data, obtaining instance annotation information and generating a target frame comprises the following steps: Annotate the instances in the point cloud data using the Sustechpoints 3D annotation tool to obtain the instance annotation information, wherein the instance annotation information includes the name of the instance, the number of occurrences of the instance, the location of the instance, the direction of the instance, and the size of the instance in each frame of the point cloud data; The target frame is generated one by one for each instance of the anchor frame in the point cloud data of each frame.

4. A method for constructing a data set for autonomous driving according to claim 3, characterized in that: The instance information file includes the index of the instance, the category of the instance, the name of the instance, the number of times the instance appears in the previous and next frames, and the number of the instance; The target frame information file includes the index of the target frame, the timestamp of the point cloud data corresponding to the target frame, the file name of the corresponding point cloud data, the index of the corresponding point cloud data frame, the index of the instance corresponding to the target frame, the position of the instance, the size of the instance, the quaternion obtained by converting the direction of the instance, the degree of occlusion of the instance, and the indexes of the target frames before and after; The frame number information file includes: the index of the data frame of the point cloud data, the timestamp of the point cloud data, the index of the corresponding scene, and the index of the previous and next timestamps; The internal and external parameter information file includes: the index of the internal and external parameter information, the index of the sensor information, the internal parameter matrix, the external parameter translation matrix, and the external parameter rotation matrix; The sampling data file includes: the index of the surround view video data or the point cloud data, the timestamp of the point cloud data, the index of the data frame of the surround view video data or the point cloud data, whether the data frame is a key frame, the suffix of the data format, the height of the surround view video data, the width of the surround view video data, the file path of the surround view video data or the point cloud data, and the position and posture of the data collection vehicle; The scene information file includes: an index of the scene, an index of the data frame where the scene appears for the first time, and an index of the data frame where the scene appears for the last time; The sensor information file includes: the index of the internal and external parameter information, the mode of the sensor, and the installation location of the sensor.

5. A method for constructing a data set for autonomous driving according to any one of claims 1 to 4, characterized in that: The internal parameter calibration between the plurality of cameras comprises the following steps: The internal reference calibration between the multiple cameras is performed using Zhang Zhengyou calibration method.

6. A method for constructing a data set for autonomous driving according to any one of claims 1 to 4, characterized in that: The extrinsic calibration between the plurality of cameras and the laser radar comprises the following steps: The lidar2camera tool is used to perform external parameter calibration between the multiple cameras and the laser radar.

7. A data set construction system for autonomous driving, characterized in that: include: Multiple cameras are installed on the data collection vehicle to collect surround video data on the driving path; A laser radar, installed on the data collection vehicle, for collecting point cloud data on the driving path; An internal parameter calibration module, used to perform internal parameter calibration between the plurality of cameras to obtain camera internal parameters; An external parameter calibration module, used to perform external parameter calibration between the plurality of cameras and the laser radar, and obtain external parameters from the camera to the laser radar; A point cloud annotation module, used to annotate instances in the point cloud data using a 3D annotation tool, obtain instance annotation information and generate a target frame; A data set integration module is used to integrate and generate a data set for autonomous driving algorithm training and verification based on the surround view video data, the point cloud data, the camera intrinsic parameters, the external parameters from the camera to the lidar, the instance annotation information and the target box.

8. A data set construction system for autonomous driving according to claim 7, characterized in that: The data set integration module is specifically used to: obtain the instance annotation information corresponding to each frame of the point cloud data, integrate all the instance annotation information, and construct an instance information file; Based on the instance information file and the instance annotation information, integrating all the target frame information to generate a target frame information file; Based on the target frame information file, integrating the frame number information of the surround view video data and the point cloud data to generate a frame number information file; Integrate the internal and external parameters of the camera and the laser radar to construct an internal and external parameter information file; Based on the target frame information file, integrating the detail information of the surround view video data and the point cloud data to construct a sampling data file; Integrate the frame number information file to construct a scene information file; Based on the internal and external reference information files, integrating the sensor information of the camera and the laser radar to construct a sensor information file; A data set for autonomous driving algorithm training and verification is constructed based on the instance information file, the target frame information file, the frame number information file, the internal and external parameter information file, the sampling data file, the scene information file, and the sensor information file.

9. A data set construction system for autonomous driving according to claim 8, characterized in that: The instance annotation information includes the name of the instance, the number of occurrences of the instance, the location of the instance, the direction of the instance, and the size of the instance in each frame of the point cloud data; The instance information file includes the index of the instance, the category of the instance, the name of the instance, the number of times the instance appears in the previous and next frames, and the number of the instance; The target frame information file includes the index of the target frame, the timestamp of the point cloud data corresponding to the target frame, the file name of the corresponding point cloud data, the index of the corresponding point cloud data frame, the index of the instance corresponding to the target frame, the position of the instance, the size of the instance, the quaternion obtained by converting the direction of the instance, the degree of occlusion of the instance, and the indexes of the target frames before and after; The frame number information file includes: the index of the data frame of the point cloud data, the timestamp of the point cloud data, the index of the corresponding scene, and the index of the previous and next timestamps; The internal and external parameter information file includes: the index of the internal and external parameter information, the index of the sensor information, the internal parameter matrix, the external parameter translation matrix, and the external parameter rotation matrix; The sampling data file includes: the index of the surround view video data or the point cloud data, the timestamp of the point cloud data, the index of the data frame of the surround view video data or the point cloud data, whether the data frame is a key frame, the suffix of the data format, the height of the surround view video data, the width of the surround view video data, the file path of the surround view video data or the point cloud data, and the position and posture of the data collection vehicle; The scene information file includes: an index of the scene, an index of the data frame where the scene appears for the first time, and an index of the data frame where the scene appears for the last time; The sensor information file includes: the index of the internal and external parameter information, the mode of the sensor, and the installation location of the sensor.

10. A computer device, characterized in that: The computer device includes at least a data set construction system for autonomous driving as described in any one of claims 7-9.

Images