An automatic driving public data set intelligent retrieval system based on multi-task

Abstract

The application relates to a multi-task-based intelligent query system for an automatic driving public data set, which comprises a data analysis layer configured to analyze a plurality of original data sets, extract data, and store the extracted data in different databases to obtain a plurality of sub-databases; a data conversion layer configured to convert the data analyzed by the data analysis layer into a fused data structure, wherein each entity data is subjected to module abstraction, and each automatic driving parameter is subjected to form conversion and unification; a data warehouse layer storing the data fused from different data sets; a retrieval module configured to search and call data from the data warehouse; and a visualization module configured to display the retrieval result of the retrieval module.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data retrieval, in particular to an intelligent retrieval system for an automatic driving public data set based on multi-task. BACKGROUND

[0002] The tasks under the direction of automatic driving mainly include perception (camera, laser radar, multi-sensor hybrid), lane line detection, route planning, etc. Among them, the camera and laser radar in the perception direction are one of the most widely used directions, which can provide important guidance information for planning control. The existing automatic driving data set has great unevenness in the distribution of scenes, weather and markers, which leads to poor effect of the perception algorithm model in many small sample size long-tail scenes, and large perception error in actual conditions.

[0003] Many methods have been used in the academic field in recent years to make special breakthroughs for long-tail distribution samples, but due to the relatively small sample size of long-tail distribution scenes, some extreme cases often cannot be well identified and classified. Model training and optimization under a single data set are prone to underfitting in specific scenarios due to the limited amount of data in such special scenarios, thereby affecting the overall performance, especially when the efficiency is low when quickly retrieving specific weather and specific object types.

[0004] How to enable the perception model to train across multiple data sets and train or model optimize for relatively poor accuracy categories is an important problem to improve the generality of the model. SUMMARY

[0005] The task of the present application is to provide an intelligent retrieval system for an automatic driving public data set based on multi-task, which fuses different data sets into a total database and develops retrieval and visualization functions. The system can realize multi-label data retrieval across data sets, and provides interface type and front-end visualization type data display methods. The entire retrieval system can be applied to the training and optimization of automatic driving algorithm models.

[0006] To solve the problems in the prior art, the present application provides an intelligent retrieval system for an automatic driving public data set based on multi-task, which comprises:

[0007] A data analysis layer configured to analyze a plurality of original data sets, extract data, and store the extracted data in different databases to obtain a plurality of sub-databases;

[0008] A data conversion layer configured to convert the data analyzed by the data analysis layer into a fused data structure, wherein each entity data is abstracted by a module, and each automatic driving parameter is uniformly converted in form;

[0009] a data warehouse layer, which stores data fused from different data sets;

[0010] a retrieval module configured to search and call data from the data warehouse; and

[0011] a visualization module configured to display the retrieval results of the retrieval module.

[0012] In an embodiment of the present application, a plurality of sub-databases after data structure conversion by the data conversion layer are fused to obtain a total database, wherein the total database is divided into a relational database and an ElasticSearch non-relational database with the same data, and the data warehouse contains the two total databases.

[0013] In an embodiment of the present application, the fusion of the plurality of sub-databases after data structure conversion by the data conversion layer comprises:

[0014] unifying the coordinate systems in the two sub-databases;

[0015] verifying whether the coordinate conversion is correct through annotation projection verification; and

[0016] unifying the label systems in the two sub-databases.

[0017] In an embodiment of the present application, the annotations of the objects in the sub-data sets are unified under the vehicle coordinate system, then converted to the world coordinate system through the pose matrix, and finally converted to the sensor coordinate system through the sensor external matrix.

[0018] In an embodiment of the present application, the objects under the converted coordinate system are projected to the initial coordinate system, and whether the coordinate conversion is correct is verified according to whether the projected coordinates are the same as the coordinates under the initial coordinate system, wherein the object annotation coordinates after the coordinate system conversion are projected to the image captured by the vehicle-mounted camera, 2D and 3D annotation box diagrams are drawn, and whether the matrix conversion and the coordinate conversion are correct is determined according to whether the 2D and 3D annotation box diagrams enclose the objects on the image.

[0019] In an embodiment of the present application, a unified label system is established, the objects under the labels in the plurality of sub-databases are re-assigned labels, and the re-assigned labels are the same as the meanings expressed by the original labels.

[0020] In an embodiment of the present application, the plurality of original data sets are autonomous driving data sets Waymo and nuScenes.

[0021] The data analysis layer parses and extracts data from multiple original data sets, and stores the data in a relational database and a non-relational database according to the characteristics of the data, to obtain a sub-database Waymo and a sub-database nuScenes, wherein the point cloud and the picture in the original data set are stored in the non-relational database, and the data other than the point cloud and the picture are stored in the relational database, and the relational database and the non-relational database are connected through a key.

[0022] In an embodiment of the present application, the data conversion layer unifies the data types of the corresponding data in the two sub-databases and the data units.

[0023] In an embodiment of the present application, the retrieval module communicates with the data warehouse layer and the visualization module through a Restful style backend data service interface.

[0024] In an embodiment of the present application, the visualization module selects multiple retrieval conditions, the retrieval module receives the retrieval conditions to retrieve the total database, returns all data in the total database meeting the conditions through the Restful style backend data service interface, and then displays the retrieval results containing pictures in the visualization module.

[0025] The present application has at least the following advantages: the disclosed automatic driving public data set intelligent retrieval system based on multi-task can fuse different data sets into a total database, develop retrieval and visualization functions based on the total database, output a data interface capable of providing stable access retrieval and a front-end visualization service module capable of freely combining indexes and displaying retrieval results, realize multi-label data retrieval function across data sets, provide interface type and front-end visualization type data display modes, and apply the entire retrieval system to training and optimization of an automatic driving algorithm model to avoid model underfitting caused by small amount of special scene data; the system can greatly improve the retrieval speed of developers for different weather, different scenes and different scene objects of automatic driving data. BRIEF DESCRIPTION OF DRAWINGS

[0026] To further clarify the above and other advantages and features of the embodiments of the present application, more particular description of the embodiments of the present application will be rendered by reference to the appended drawings. It is appreciated that these drawings depict only typical embodiments of the application and are therefore not to be considered limiting of its scope.

[0027] Figure 1 Fig. 1 shows a schematic diagram of an automatic driving public data set intelligent query system based on multi-task according to an embodiment of the present application;

[0028] Figure 2A schematic diagram of the parsed Waymo database according to an embodiment of the present application is shown;

[0029] Figure 3 A schematic diagram of the parsed NuScenes database according to an embodiment of the present application is shown; and

[0030] Figure 4 A schematic diagram of the total database nuWay according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] It should be noted that the components in the various figures can be shown exaggerated in number and / or size for the sake of illustration, and are not necessarily drawn to scale.

[0032] In the present application, each embodiment is merely intended to illustrate the scheme of the present application, and should not be understood as limiting.

[0033] In the present application, unless specifically indicated, the quantifier “one”, “a” does not exclude the scenario of multiple elements.

[0034] It should also be noted here that, in the embodiments of the present application, only a part of the components or assemblies can be shown for the sake of clarity and simplicity, but those skilled in the art can understand that, under the teaching of the present application, the required components or assemblies can be added according to the specific scene needs.

[0035] It should also be noted here that, within the scope of the present application, the phrases “same”, “equal”, “equal to” do not mean that the two values are absolutely equal, but allow a certain reasonable error, that is, the phrases also cover “substantially same”, “substantially equal”, “substantially equal to”.

[0036] It should also be noted here that, in the description of the present application, the orientations or positional relationships indicated by the terms “center”, “longitudinal”, “transverse”, “upper”, “lower”, “front”, “rear”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inner”, “outer” are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not mean that the devices or elements indicated thereby must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms “first”, “second” are only for the purpose of description, and cannot be understood as explicitly or implicitly indicating relative importance.

[0037] In addition, the numbering of the steps of each method of the present application does not limit the execution order of the method steps. Unless specifically indicated, each method step can be executed in a different order.

[0038] Figure 1A schematic diagram of a multi-task based automatic driving public dataset intelligent query system is shown.

[0039] As shown in the figure, the multi-task based automatic driving public dataset intelligent query system includes a data module, a retrieval module and a visualization module. The data module, the retrieval module and the visualization module are connected through an interface, the retrieval module can search and call data from the data module, and then display in the visualization module. The retrieval module can perform multi-label simultaneous search, which can greatly improve the retrieval speed of developers on different weather, different scenes and different scene objects in the automatic driving data. Figure 1

[0040] The data module includes a data parsing layer, a data conversion layer and a data warehouse layer.

[0041] The data parsing layer parses different datasets, extracts data, and stores the extracted data in different databases. The open source automatic driving datasets Waymo and nuScenes are parsed to extract data, and the extracted data is stored in different relational databases and non-relational databases respectively. The data structure relationship of the data stored in the relational database and the non-relational database is consistent with the data structure relationship in the original dataset Waymo and nuScenes. The data parsing layer only converts the dataset to the database, and the display form of the automatic driving parameters in the data remains unchanged. The data of the original dataset Waymo and nuScenes is stored in the database to obtain two sub-databases Waymo and nuScenes. The data structure in the two databases, i.e. the table structure, is the same.

[0042] The data conversion layer converts the parsed data obtained by the data parsing layer to the fused data structure, abstracts each entity data, and uniformly converts each automatic driving parameter. Abstracting each entity data is to classify the data of the same system into the same module, for example, classifying the automatic driving parameters related to the sensor into the sensor module. The data converted by the data conversion layer is consistent with the final fused data structure. The purpose of setting the data conversion layer is to consider that the current data is stored in the relational database and the non-relational database, and it is extremely troublesome to convert a large amount of data and delete errors. Therefore, the data conversion layer (processing layer) is set to isolate the final data warehouse layer.

[0043] In the data conversion layer, the types of data in the Waymo database and the nuScenes database are unified. The data types of the corresponding data in the two databases are unified, and the data units are unified, for example, the data of the vehicle speed field in the two databases are both floating point type, and the unit is kilometer / hour.

[0044] ​The data conversion layer to the data warehouse layer undergoes a data fusion process. The data fusion process includes coordinate conversion, label projection verification, label system merging, and the like.

[0045] The coordinate systems of the original data sets Waymo and nuScenes are different, and need to be converted into a unified coordinate system. The coordinate conversion process needs to go through the conversion of the vehicle coordinate system, the world coordinate system, and the sensor coordinate system. First, the labels of the objects are unified in the vehicle coordinate system, and then converted to the world coordinate system through the pose matrix, and finally converted to the sensor coordinate system through the sensor extrinsic matrix. Here, the sensors include cameras, radars, and lidars, all of which are on the vehicle. Both Waymo and nuScenes need to go through the above coordinate conversion.

[0046] The correctness of the coordinate conversion is verified by label projection verification. The objects in the converted coordinate system are projected onto the initial coordinate system (vehicle coordinate system), and the correctness is verified according to whether the projected coordinates are the same as the coordinates in the initial coordinate system. Specifically, the object label coordinates after coordinate system conversion are projected onto the image taken by the vehicle-mounted camera, and 2D and 3D label box diagrams are drawn. According to whether the 2D and 3D label box diagrams enclose the objects on the image, it is determined whether the matrix conversion and coordinate conversion are correct.

[0047] The objects in the original data sets Waymo and nuScenes are divided into various categories, such as people, animals, vehicles, etc., and vehicles can be further divided into bicycles, trucks, cars, etc. The classification of objects in the original data sets Waymo and nuScenes is not completely the same, for example, compared with the nuScenes data set, the Waymo data set has a more detailed division of vehicle categories. Here, the object labels of the two data sets need to be unified. A unified label system is established to reassign the labels of the objects in the original data sets Waymo and nuScenes. A new semantic is constructed that can describe both original label systems. After reassigning the labels, the same meaning as the original label is expressed. The unified label system includes labels such as people, vehicles (cars, trucks), etc.

[0048] After the fusion of the Waymo database and the nuScenes database, the fused data is stored in a relational database and a non-relational database to obtain a total database nuWay. The total database nuWay is in the data warehouse layer, which provides query data. The bottom table for query can be designed according to actual application. The total database has two forms, a relational database and a non-relational database (ElasticSearch database). Storing data in the ElasticSearch database is beneficial for fast retrieval.

[0049] The retrieval module and the data warehouse layer communicate through a Restful style backend data service interface. The retrieval module and the visualization module communicate through a Restful style backend data service interface. When the visualization module selects multiple retrieval conditions (tags or tasks), the retrieval module receives the retrieval conditions, retrieves the total database, returns all data in the total database that meets the conditions through the interface, and then displays the search results containing pictures in the visualization module. Depending on the actual application, the relational database or ElasticSearch database can be selected for retrieval. All retrieval conditions include dataset, category, point cloud density (point cloud reflectivity), weather, camera, 2D annotation box, 3D annotation box, segmentation number, frame number, and part or all of the retrieval conditions can be selected during retrieval.

[0050] Figure 2 A schematic diagram of the parsed Waymo database according to an embodiment of the application is shown; Figure 3 A schematic diagram of the parsed NuScenes database according to an embodiment of the application is shown.

[0051] The data of the original dataset Waymo is organized in basic units (files with the extension.tfrecord) of a certain period of autonomous driving scene, which contains various sensor parameters, vehicle information, annotation information, etc., but the association between these basic units is weak and difficult to retrieve efficiently.

[0052] The data in the original dataset Waymo and NuScenes exists in their own specific data format, which is difficult to read, and different formats of data are difficult to fuse, so the data in the original dataset Waymo and NuScenes needs to be extracted and stored in a unified format to prepare for subsequent data fusion.

[0053] The original dataset Waymo and NuScenes are parsed, and the pictures and laser radar point clouds are stored in a non-relational database, such as object storage (OSS), and the rest of the data is stored in a relational database, such as MySQL, and the relational database and the non-relational database are connected. The original dataset Waymo and NuScenes contain pictures and laser radar point clouds, which are not suitable for storage in a relational database, so the pictures and laser radar point clouds are stored in the OSS, and the key (here is the time) and the autonomous driving data in the relational database are associated, such as the current pose of the car, the speed, etc. The key (Key) value is stored in the table of the OSS, and the parsed data is uploaded to the OSS while inserting the metadata, maintaining the data correspondence in the original dataset.

[0054] AsFigure 2 and Figure 3 As shown in FIG. 1, the original datasets Waymo and NuScenes are parsed and stored in the database to obtain the Waymo database and the NuScenes database. The boxes in the figure represent tables, and the arrows represent the key between the two.

[0055] The Waymo database includes a scene module, a sample module, a sensor module, an annotation and object module, a sample data module, and an OSS module.

[0056] For performance, use, and resource considerations, a relational database is not suitable for storing files such as pictures and point clouds. It is appropriate to expand and store such data on the object storage (Object Storage Service, OSS) for easy downloading and overall data statistics. However, it is necessary to ensure that the key (Key) value stored in the table of the relational database corresponds correctly. The key value is stored in the table, and the parsed and inserted metadata is uploaded at the same time to ensure consistency of the correspondence. The pictures taken by the on-board camera and the laser radar point cloud are stored in the OSS, and are associated with other data in the relational database through the key (time), such as the current pose, speed, and the like of the vehicle in the relational database.

[0057] The scene module includes tables named scene_context and scene_meta. The scene_context table includes three fields: meta, calibrations_camera, and calibrations_lidar. The scene_meta includes three fields: time_of_day, location, and weather.

[0058] The sample module includes a table named frame. The frame table includes four fields: scene_id, frame_id, timestamp, and pose.

[0059] The sensor module includes tables named radar (sensor_calibrations_lidar) and camera (sensor_calibrations_camera). The sensor calibration radar table contains five fields: channel, modality, lidar mid-beam inclination (beam_inclinations), lidar mid-beam inclination range (beam_inclinations_range), and extrinsic. The sensor calibration camera table contains six fields: channel, modality, intrinsic, extrinsic, width and height (width_height), and rolling shutter.

[0060] The annotation and object module includes a frame-projected-lidar-label table named frame- object, a frame-camera-label table, a frame-laser-label table, an instance-camera-label table, an instance-laser-label table, a type table, and a difficulty table. The frame-object-lidar-label table includes twelve fields: label id, timestamp, channel, center x, center y, wide, length, speed x, speed y, accel x, accel y, and type. The frame-camera-label table includes seven fields: label id, camera channel, center x, center y, wide, length, and type. The frame-laser-label table includes thirteen fields: label id, center x, center y, center z, wide, length, height, heading, speed x, speed y, accel z, accel x, and type. The instance-camera-label table and the instance-laser-label table each include two fields: label id and type. The type table includes two fields: type id and type name. The difficulty table includes two fields: difficulty level and description.

[0061] The sample data module includes tables named frame-camera image, frame-pcd, and frame-lidar seg. The frame-camera image table includes fifteen fields: scene id, frame id, camera channel, storage path, pose, pose timestamp, shutter, camera trigger time, camera readout done time, velocity v x, velocity v y, velocity v z, velocity w x, velocity w y, and velocity w z. The frame-pcd table includes five fields: scene id, frame id, segment frame id, timestamp, and pcd file path. The frame-lidar seg table includes five fields: scene id, frame id, segment frame id, timestamp, and seg pcd file path.

[0062] The basic position information can be represented by three parts: translation (point position), size (length, width, height / length, width), and rotation (rotation). The rotation representation method is different, and it is recommended to integrate the final representation using quaternions. The conversion can be performed using the scipy package.

[0063] Waymo objects are trackable, using numbers.

[0064] The NuScenes sub-database includes a scene module, an annotation module, and a sample data module.

[0065] The scene module contains tables named scene, map, log, and weather. The scene table contains seven fields: token, name, description, log_token, log_samples, nbr_samples, first_samples_token, and last_samples_token. The map table contains four fields: token, log_token, category, and filename. The log table contains five fields: token, logfile, vehicle, data_captured, and location. The weather table contains two fields: weather and log_token.

[0066] The annotation module contains tables named visibility, sample_annotation, attribute, category, and instance. The visibility table contains three fields: token, level, and description. The sample_annotation table contains fourteen fields: token, sample_token, instance_token, attribute_token, visibility_token, size, rotation, num_lidar_pts, num_radar_pts, next, prev, 3D_object, and 2D_object. The attribute table contains three fields: token, name, and description. The category table contains four fields: token, name, description, and index. The instance table contains five fields: token, category_token, nbr_annotations, first_annotation_token, and last_annotation_token.

[0067] The sample data module contains tables named sensor, calibrated_sensor, ego_pose, lidarseg, and sample_data.

[0068] The sensor table contains three fields: token, level, and modality. The calibrated_sensor table contains five fields: token, sensor_token, translation, rotation, and camera_intrinsic. The ego_pose table contains four fields: token, translation, rotation, and timestamp. The lidarseg table contains three fields: token, filename, and sample_data_token. The sample_data table contains twelve fields: token, sample_token, ego_pose, calibrated_sensor_token, filename, fileformat, wide, height, timestamp, is_key_frame, next, and prev.

[0069] All global primary keys in the NuScenes sub-database are string-type tokens.

[0070] Attributes are changeable properties under the same category, such as the state of a car (stopped, moving).

[0071] calibrated_sensor is the parameter of a certain sensor on a certain car, where the external parameter is relative to a certain ego vehicle body frame.

[0072] Categories contain large classes of entities, such as people and cars, and subcategories under the large class can have multiple layers.

[0073] ego_pose: pose of ego vehicle at a certain time point in the global coordinate system of the map of the log, output of the lidar map-based localization algorithm.

[0074] instance: refers to an entity, such as a specific vehicle. The instance table enumerates all observed objects, but the entity tracking does not cross scenes.

[0075] Lidarseg: based on the radar point cloud of a keyframe, map sample_data and nuScenes laser radar annotations, i.e. annotation file and sample data associated with lidarseg.

[0076] log: log is the unit of information capture, and the log table records some meta-information of information capture, such as the name of the vehicle collecting data, the date and place of collection, etc.

[0077] map: some map information of the log, the category refers to some surface and sidewalk information.

[0078] scene: 20s long continuous frame, extracted from the log, and multiple scenes can be derived from a log. In addition, the identification of instances is not cross-scene.

[0079] dataset: collected sensor data. When is_key_frame=true, the time point of collecting this batch of data will be close.

[0080] sample_annotations: annotation of the entity. The position information of the annotation is in the global coordinate system

[0081] visibility: visibility of instance in all six images. Four levels: 0-40%, 40%-60%, 60%-80%, 80-100%.

[0082] description in the scene is a series of labels separated by commas, generally recorded at 2Hz for 20s, so generally there are about 40-40 sample quantities.

[0083] Figure 4 A schematic diagram of the total database nuWay according to an embodiment of the application is shown.

[0084] As Figure 4The same and similar modules in Waymo sub-database and Nusence sub-database are fused, and the total database after fusion contains scene and sample modules, object modules, annotation modules, sensor data modules and OSS modules. The solid line box in the figure represents a table, and the arrow represents a key between the two.

[0085] The scene and sample module contains tables named dateset, segment and sample. The dateset table contains two fields: id and metadata. The segment table contains five fields: id, tags, location, time and dataset_id. The sample table contains three fields: id, scene and sequence.

[0086] The object module contains tables named instance and category. The instance table contains three fields: id, scene and category_id. The category table contains three fields: id, original category and universal category.

[0087] The sensor data module contains tables named sample_data and calibrated_sensor. The sample_data table contains six fields: id, file_type, sensor_id, pose, sample_id and file_path. The calibrated_sensor table contains five fields: id, channel, modality, instrinsic parameters and extrinsic parameters.

[0088] The annotation module includes tables named 2D annotation (annotation_2D), 3D annotation (annotation_3D), and 3D annotation object wide (annotation_3D_project_wide). The 2D annotation table includes five fields: id, object id, data id, 2D location, and size. The 3D annotation table includes five fields: id, object id, 3D location, rotation, and size. The 3D annotation object wide table includes five fields: 3D annotation id, object id, data id, data id, and corners.

[0089] Embodiments can be provided as a computer program product, which can include one or more machine-readable media having stored thereon machine-executable instructions that, when executed by one or more machines, such as a computer, networked computers, or other electronic devices, can cause the one or more machines to perform the operations described herein. The machine-readable media can include, but is not limited to, floppy diskettes, optical disks, CD-ROMs (compact disc read-only memories), and magneto-optical disks, ROMs (read-only memories), RAMs (random access memories), EPROMs (erasable programmable read-only memories), EEPROMs (electrically erasable programmable read-only memories), magnetic or optical cards, flash memory, or other type of media / machine-readable media suitable for storing electronic instructions.

[0090] Furthermore, embodiments can be downloaded as a computer program product, wherein the program can be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by way of one or more data signals embodied in and / or modulated by a carrier wave or other propagation medium via a communication link (e.g., a modem and / or a network connection). Accordingly, the machine-readable medium can include a carrier wave that includes or embodies the program.

[0091] The application has at least the following beneficial effects: the automatic driving public data set intelligent retrieval system based on multi-task disclosed in the application fuses different data sets into a total database, develops retrieval and visualization functions on the basis of the total database, outputs a set of data interfaces capable of providing stable access retrieval and a front-end visualization service module capable of freely combining indexes and displaying retrieval results, the system can realize multi-label data retrieval function across data sets, simultaneously provides data display modes of interface type and front-end visualization type, the whole retrieval system can be applied to training and optimization of automatic driving algorithm models, and model underfitting caused by small amount of special scene data can be avoided; the system can greatly improve the retrieval speed of developers for different weather, different scenes and different scene objects of automatic driving data.

[0092] While some embodiments of the application have been described in the present application, it is understood that those skilled in the art can make various modifications, substitutions and improvements without departing from the scope of the application. The appended claims are intended to define the scope of the application and thereby encompass methods and structures within the scope of the claims themselves and their equivalent transformations.

Claims

1. A multi-task based intelligent query system for autonomous driving public data sets, comprising: a data parsing layer configured to parse a plurality of original data sets, extract data, and store the extracted data in different databases to obtain a plurality of sub-databases; a data conversion layer configured to convert the data parsed by the data parsing layer into a fused data structure, wherein each entity data is subjected to module abstraction, and each autonomous driving parameter is subjected to formal conversion and unification; a data warehouse layer storing data fused from different data sets; a retrieval module configured to search and call data from the data warehouse; and a visualization module configured to display the retrieval results of the retrieval module, wherein the retrieval results containing pictures are displayed on the visualization module. The plurality of original data sets are autonomous driving data sets Waymo and nuScenes. The data parsing layer parses and extracts data from the plurality of original data sets, and stores the data in a relational database and a non-relational database according to the characteristics of the data to obtain a sub-database Waymo and a sub-database nuScenes, wherein the point cloud and the picture in the original data set are stored in the non-relational database, and the data other than the point cloud and the picture are stored in the relational database, and the relational database and the non-relational database are connected through a key. The plurality of sub-databases subjected to data structure conversion by the data conversion layer are fused to obtain a total database, wherein the total database includes a relational database and a non-relational database ElasticSearch, and the data warehouse includes the two total databases. The fusion of the plurality of sub-databases subjected to data structure conversion by the data conversion layer includes: unifying the coordinate systems in the two sub-databases; verifying whether the coordinate conversion is correct through annotation projection verification; and unifying the label systems in the two sub-databases. The annotations of the objects in the sub-databases are unified in the vehicle coordinate system, and then converted to the world coordinate system through a pose matrix, and finally converted to the sensor coordinate system through a sensor external matrix. 2.The multi-task based autonomous driving open dataset intelligent query system of claim 1, wherein, The objects in the converted coordinate system are projected to the initial coordinate system, and whether the coordinate conversion is correct is verified according to whether the projected coordinates are the same as the coordinates in the initial coordinate system, wherein the object annotation coordinates after the coordinate system conversion are projected to the image taken by the vehicle-mounted camera, and 2D and 3D annotation box diagrams are drawn, and whether the matrix conversion and the coordinate conversion are correct is determined according to whether the 2D and 3D annotation box diagrams enclose the objects on the image. 3.The multi-task based autonomous driving open dataset intelligent query system of claim 1, wherein, A unified label system is established, the labels of the objects in the plurality of sub-databases are re-assigned, and the labels are the same as the original labels in meaning.

4. The multi-task based automated driving open dataset intelligent query system of claim 1, wherein, The data conversion layer unifies the data types and the data units of the corresponding data in the two sub-databases. 5.The multi-task based autonomous driving open dataset intelligent query system of claim 1, wherein, The retrieval module, the data warehouse layer, and the visualization module communicate through a Restful style backend data service interface. 6.The multi-task based autonomous driving open dataset intelligent query system of claim 1, wherein, ​ 7. The multi-task based automated driving open dataset intelligent query system according to claim 6, characterized in that, The visualization module selects multiple search conditions, the search module receives the search conditions to search the total database, returns all data meeting the conditions in the total database through a Restful style backend data service interface, and then displays the search results containing pictures in the visualization module.

Images