Method and device for converting data in rosbag format into nuScenes format

Abstract

The invention discloses a method and device for converting data in a rosbag format into a nuScenes format, and relates to the field of artificial intelligence. A plurality of rosbag data frames belonging to the rosbag format are obtained; acquiring a corresponding relationship between the acquisition timestamps of the key frames in the plurality of rosbag data frames and the speedometer information corresponding to the key frames; the plurality of rosbag data frames are converted into a plurality of nuScenes data frames in a nuScenes format, and the plurality of nuScenes data frames in the nuScenes format are obtained; based on the acquisition timestamps of the plurality of nuScenes data frames and the acquisition timestamps of the plurality of pieces of speedometer information, constructing a corresponding relationship between the nuScenes data frames and the speedometer information; and storing the plurality of nuScenes data frames and the corresponding relationship between the nuScenes data frames and the odometer information into a pre-constructed nuScenes database, so that the purpose of converting the data in the rosbag format into the data in the nuScenes format is achieved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for converting data in a rosbag format into a nuScenes format. Background Art

[0002] As an autonomous driving dataset, the nuScenes dataset has the advantages of large data volume, unified format, and complete data structure. Therefore, the nuScenes dataset is often used as data for model training.

[0003] The data collected by the ROS (Robot Operating System) hardware platform is in rosbag format. Since the data in rosbag format and the data in nuScenes format are not consistent, it is impossible to train the model with the data in rosbag format and nuScenes format.

[0004] Based on this, there is an urgent need for a method to convert data in rosbag format into data in nuScenes format. Summary of the invention

[0005] In view of the above problems, this application provides a method and device for converting data in rosbag format to nuScenes format, so as to achieve the purpose of converting data in rosbag format to nuScenes format. The specific scheme is as follows:

[0006] In a first aspect, the present application provides a method for converting data in rosbag format into nuScenes format, comprising:

[0007] Obtain multiple rosbag data frames in rosbag format, wherein the rosbag data frames include data, sensor type for collecting the data, sensor name for collecting the data, collection timestamp for collecting the data, calibration data of the sensor for collecting the data, ID of the sensor for collecting the data, and message topic corresponding to the data; the sensor types include camera, inertial measurement unit, lidar, and millimeter wave radar;

[0008] Obtain the corresponding relationship between the acquisition timestamp of the key frames in multiple rosbag data frames and the odometer information corresponding to the key frames;

[0009] Convert the multiple rosbag data frames into multiple nuScenes data frames in nuScenes format;

[0010] Based on the acquisition timestamps of the plurality of nuScenes data frames and the acquisition timestamps of the plurality of odometer information, constructing a correspondence between the nuScenes data frames and the odometer information;

[0011] The plurality of nuScenes data frames and the correspondence between the nuScenes data frames and the odometer information are stored in a pre-constructed nuScenes database.

[0012] In a possible implementation, the step of constructing a correspondence between the nuScenes data frame and the odometer information based on the acquisition timestamps of the plurality of nuScenes data frames and the acquisition timestamps of the plurality of odometer information includes:

[0013] Creating a first token for each of the plurality of odometer information;

[0014] Store the odometer information, the first token of the odometer information, and the acquisition timestamp of the odometer information into a nuScenes odometer queue in a nuScenes format;

[0015] For a nuScenes data frame containing point cloud data, compress multiple nuScenes data frames collected by the same radar whose maximum difference of acquisition timestamps is less than or equal to the target difference into the same point cloud set to obtain multiple point cloud sets, wherein the radar is a millimeter wave radar or a laser radar;

[0016] For each of the point cloud sets, determining the smallest acquisition timestamp among the multiple nuScenes data frames contained in the point cloud set as the acquisition timestamp of the point cloud set;

[0017] For each of the point cloud sets, determining the odometer information whose acquisition timestamp is closest to the acquisition timestamp of the point cloud set as the odometer information of the point cloud set;

[0018] Creating a second token for each of the plurality of point cloud sets;

[0019] Store the point cloud set, the second token of the point cloud set, and the acquisition timestamp of the point cloud set in a first nuScenes message queue in nuScenes format;

[0020] For each of the nuScenes data frames captured by the camera, determine the odometer information whose acquisition timestamp is closest to the acquisition timestamp of the nuScenes data frame as the odometer information of the nuScenes data frame;

[0021] Allocate a third token to each of the multiple nuScenes data frames captured by the camera;

[0022] The nuScenes data frame collected by the camera, the third token corresponding to the nuScenes data frame, the odometer information corresponding to the nuScenes data frame, and the setting calibration information of the camera are stored in a second nuScenes message queue.

[0023] In a possible implementation, the step of storing the plurality of nuScenes data frames and the correspondence between the nuScenes data frames and the odometer information in a pre-built nuScenes database includes:

[0024] Create a target folder, a samples folder, a sweeps folder, and a calibrate_sensor.json folder in the nuScenes database;

[0025] Create ego_pose.json file, sample.json file, sample_data.json file and scene.json file in the target folder;

[0026] Create a file with a suffix of .pcd, a file with a suffix of .pcd.bin, and a file with a suffix of .jpg in the nuScenes database;

[0027] Convert the odometer information corresponding to the plurality of point cloud sets and the odometer information corresponding to the nuScenes data frame collected by the camera into pose information relative to a preset body coordinate system, and write them into the calibrate_sensor.json folder;

[0028] If the radar corresponding to the first nuScenes message queue is a millimeter wave radar, write the information stored in the first nuScenes message queue into a file with a suffix of .pcd;

[0029] If the radar corresponding to the first nuScenes message queue is a laser radar, the information stored in the first nuScenes message queue is written into a file with a suffix of .pcd.bin;

[0030] Write the information stored in the second nuScenes message queue into a file with a suffix of .jpg;

[0031] The nuScenes data frames respectively corresponding to the multiple key frames are stored in the scene.json file;

[0032] Store the information stored in the nuScenes odometry queue into the ego_pose.json file;

[0033] The preset scene information of the point cloud data and the preset scene information of the nuScenes data frame collected by the camera are stored in the scene.json file;

[0034] Store the preceding token and the succeeding token of the first token of each odometer information, the preceding token and the succeeding token of the second token of each point cloud set, the preceding token and the succeeding token of the third token of each nuScenes data frame acquired by the camera, the acquisition timestamp of each first token, the acquisition timestamp of each second token, and the acquisition timestamp of each third token into the samples folder;

[0035] Store the preceding token and the succeeding token of the first token of each of the odometer information, the preceding token and the succeeding token of the second token of each of the point cloud sets, the preceding token and the succeeding token of the third token of each of the nuScenes data frames collected by the camera, the acquisition timestamp of each of the first tokens, the acquisition timestamp of each of the second tokens, the acquisition timestamp of each of the third tokens, whether each of the point cloud sets includes a key frame, whether each of the nuScenes data frames collected by the camera is a key frame, the storage path of each of the point cloud sets, the storage path of each of the nuScenes data frames collected by the camera, the first token of the odometer information of each of the point cloud sets, and the first token of the odometer information of a nuScenes data frame collected by the camera into the sample_data.json file;

[0036] Store the nuScenes data frames corresponding to the multiple key frames respectively into the samples folder;

[0037] The nuScenes data frames other than the key frame among the multiple nuScenes data frames are stored in the sweeps folder.

[0038] In a possible implementation, the step of obtaining the correspondence between the acquisition timestamps of the key frames in the multiple rosbag data frames and the odometer information corresponding to the key frames includes:

[0039] Acquire multiple key frames from multiple rosbag data frames whose sensor type is laser radar;

[0040] For each of the key frames, based on the first rosbag data frame, the second rosbag data frame and the key frame, determine the odometer information of the key frame to obtain the odometer information corresponding to the multiple key frames respectively; the first rosbag data frame is acquired by a camera, the data in the first rosbag data frame is an image, the absolute value of the difference between the acquisition timestamp of the first rosbag data frame and the acquisition timestamp of the key frame is less than or equal to a preset duration, the second rosbag data frame is acquired by an inertial measurement unit, and the data in the second rosbag data frame is IMU posture data; the absolute value of the difference between the acquisition timestamp of the second rosbag data frame and the acquisition timestamp of the key frame is less than or equal to the preset duration;

[0041] A correspondence between the acquisition timestamp of the key frame and the odometer information corresponding to the key frame is constructed.

[0042] In a possible implementation, the step of acquiring multiple key frames from multiple rosbag data frames whose sensor type is a laser radar includes:

[0043] Acquire multiple candidate key frames from multiple rosbag data frames whose sensor type is laser radar;

[0044] Constructing a KD tree based on the plurality of candidate key frames;

[0045] A plurality of key frames are obtained from the KD tree based on a preset recursive function.

[0046] In a possible implementation, for each of the key frames, based on the first rosbag data frame, the second rosbag data frame and the key frame, determining the odometer information of the key frame to obtain the odometer information corresponding to the multiple key frames, respectively, includes:

[0047] For each of the key frames, obtain the first rosbag data frame and the second rosbag data frame whose absolute value of the difference between the acquisition timestamp and the acquisition timestamp of the key frame is less than or equal to the preset duration;

[0048] For each of the key frames, obtaining the zero bias of the IMU pose data based on the first rosbag data frame, the second rosbag data frame and the key frame through a zero bias estimation integrator;

[0049] For each of the key frames, obtaining candidate odometer information of the key frame based on the zero bias, the first rosbag data frame, the second rosbag data frame and the key frame through a mileage calculation integrator;

[0050] Determine the initial pose information;

[0051] Convert the candidate odometer information corresponding to each of the plurality of key frames into relative pose information based on the initial pose information;

[0052] Constructing a preliminary local map based on the relative posture information respectively corresponding to the plurality of key frames;

[0053] Based on the preliminary local map, obtain optimized relative pose information corresponding to a plurality of the key frames respectively;

[0054] Update the preliminary local map based on the optimized relative pose information respectively corresponding to the plurality of key frames to obtain a candidate local map;

[0055] Updating the candidate local map by closed-loop detection technology to obtain a target local map;

[0056] The odometer information respectively corresponding to the multiple key frames is obtained from the target local map.

[0057] In a possible implementation, before the step of converting the plurality of rosbag data frames into a plurality of nuScenes data frames in nuScenes format, the method further includes:

[0058] Determine the target message topic of the rosbag data frame of the sensor ID to be completed;

[0059] Obtaining a rosbag data frame having the target message topic from the plurality of rosbag data frames;

[0060] Adjust the data frame ID of the rosbag data frame with the target message topic to the set ID.

[0061] In a possible implementation, before the step of converting the plurality of rosbag data frames into a plurality of nuScenes data frames in nuScenes format, the method further includes:

[0062] Determine a target rosbag data frame of the calibration data of the sensor to be completed from the multiple rosbag data frames;

[0063] Fill in the target rosbag data frame with set calibration data.

[0064] In a possible implementation, the step of converting the plurality of rosbag data frames into a plurality of nuScenes data frames in nuScenes format includes:

[0065] Acquire multiple target rosbag data frames whose sensor type is the target sensor type from the multiple rosbag data frames;

[0066] Based on the nuScenes configuration file, the rosbag fields in the multiple target rosbag data frames are converted into nuScenes fields and the field value types of the rosbag fields in the multiple target rosbag data frames are converted into the field value types of the nuScenes fields, so as to obtain the multiple nuScenes data frames; the nuScenes configuration file includes the correspondence between the preset nuScenes fields and the rosbag fields and the correspondence between the preset field value types of the nuScenes fields and the field value types of the rosbag fields;

[0067] Store the multiple nuScenes data frames into the nuScenes structure.

[0068] A second aspect of the present application provides a device for converting data in rosbag format into nuScenes format, including:

[0069] The first acquisition module is used to acquire multiple rosbag data frames in the rosbag format, wherein the rosbag data frames include data, the type of sensor that collects the data, the name of the sensor that collects the data, the acquisition timestamp of the data, the calibration data of the sensor that collects the data, the ID of the sensor that collects the data, and the message topic corresponding to the data; the sensor types include camera, inertial measurement unit, lidar and millimeter wave radar;

[0070] The second acquisition module is used to obtain the corresponding relationship between the acquisition timestamp of the key frame in multiple rosbag data frames and the odometer information corresponding to the key frame;

[0071] A conversion module, used for converting the plurality of rosbag data frames into a plurality of nuScenes data frames in a nuScenes format;

[0072] A construction module, configured to construct a correspondence between the nuScenes data frame and the odometer information based on a collection timestamp of a plurality of the nuScenes data frames and a collection timestamp of a plurality of the odometer information;

[0073] A storage module is used to store the plurality of nuScenes data frames and the correspondence between the nuScenes data frames and the odometer information in a pre-built nuScenes database.

[0074] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the method for converting data in rosbag format into nuScenes format according to the first aspect or any implementation method of the first aspect.

[0075] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0076] The memory is used to store computer programs;

[0077] The processor is used to execute the computer program so that the electronic device can implement the method of converting data in rosbag format into nuScenes format according to the first aspect or any implementation method of the first aspect.

[0078] In a fifth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can convert data in the rosbag format of the above-mentioned first aspect or any implementation method of the first aspect into a method of nuScenes format.

[0079] By means of the above technical scheme, the present application provides a method for converting data in rosbag format into nuScenes format, obtaining multiple rosbag data frames in rosbag format; obtaining the correspondence between the acquisition timestamps of key frames in multiple rosbag data frames and the odometer information corresponding to the key frames; converting multiple rosbag data frames into multiple nuScenes data frames in nuScenes format; constructing the correspondence between nuScenes data frames and odometer information based on the acquisition timestamps of multiple nuScenes data frames and the acquisition timestamps of multiple odometer information; storing multiple nuScenes data frames and the correspondence between nuScenes data frames and odometer information in a pre-built nuScenes database, thereby achieving the purpose of converting data in rosbag format into data in nuScenes format. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0081] Figure 1 A schematic diagram of a system architecture provided for this application;

[0082] Figure 2 A schematic diagram of an optional hardware structure of an electronic device 200 provided in this application;

[0083] Figure 3 A flowchart of a method for converting data in rosbag format to nuScenes format provided in an embodiment of the present application;

[0084] Figure 4 A schematic diagram of format conversion provided in an embodiment of the present application;

[0085] Figure 5 A schematic diagram of the structure of a device for converting data in rosbag format into nuScenes format provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0087] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0088] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0089] See also Figure 1 , Figure 1A schematic diagram of a system architecture is shown. The system may include a ROS (Robot Operating System) hardware platform 100 , a rosbag database 400 , an electronic device 200 , and a nuScenes database 300 .

[0090] Among them, an application or a web page can be installed on the electronic device 200, and the above application and web page can provide an interface. The electronic device 200 can receive relevant parameters entered by the user on the interface, for example, instructions for obtaining multiple rosbag data frames from the rosbag database 400, and sending the above parameters to the rosbag database 400. The rosbag database 400 can obtain processing results such as multiple rosbag data frames based on the received parameters, and return the processing results to the electronic device 200.

[0091] Next describe Figure 1 The structure of the ROS hardware platform 100 in FIG. 100 includes, but is not limited to, one or more millimeter wave radars, one or more laser radars, one or more cameras, and an inertial measurement unit (IMU). The ROS hardware platform 100 can be deployed on a train, a high-speed train, or an autonomous driving vehicle.

[0092] Exemplarily, the ROS hardware platform 100 stores the collected multiple rosbag data frames in a rosbag database.

[0093] Figure 2 An optional hardware structure diagram of the electronic device 200 is shown.

[0094] Exemplarily, the electronic device 200 may be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a server or a server cluster, etc., and the embodiments of the present application do not impose any limitations on this.

[0095] refer to Figure 2As shown, the electronic device 200 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 The electronic device is merely an example and does not limit the electronic device, and may include more or less components than those shown in the figure, or may combine certain components, or may include different components.

[0096] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes contact point coordinate information. The touch screen 131 can provide an input interface and an output interface between the electronic device 200 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0097] Among them, the input device 132 can receive input data and the like.

[0098] The display unit 140 can be used to display information input by the user or information provided to the user, various menus of the electronic device 200, interactive interfaces, file display and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 can be used to display interfaces, processing results, etc.

[0099] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0100] The processor 170 is the control center of the electronic device 200. It uses various interfaces and lines to connect various parts of the entire electronic device 200. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the electronic device 200 and processes data, thereby controlling the electronic device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor, memory, can be implemented on a single chip, and in some embodiments, they can also be implemented separately on independent chips. The processor 170 can also be used to generate corresponding operation control signals, send them to the corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0101] Among them, the memory 120 can be used to store software codes related to the method of converting data in rosbag format into nuScenes format, the processor 170 can execute the steps of the method of converting data in rosbag format into nuScenes format, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to implement corresponding functions.

[0102] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0103] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0104] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0105] The electronic device 200 also includes a power source 190 (such as a battery) for supplying power to various components. Preferably, the power source can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0106] The electronic device 200 also includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the electronic device 200 to communicate with other devices, or may be used to connect a charger to charge the electronic device 200 .

[0107] Although not shown, the electronic device 200 may also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied to Figure 2 In the electronic device 200 shown.

[0108] It should be understood that the above-mentioned electronic device 200 can be a centralized or distributed device, and the processor of the above-mentioned electronic device 200 (such as the processor 170) can be a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, a DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, an FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0109] Reference Figure 3 , Figure 3 A flowchart of a method for converting data in rosbag format to nuScenes format provided in an embodiment of the present application is shown as follows: Figure 3 As shown, a method for converting data in rosbag format to nuScenes format provided in an embodiment of the present application may include steps S301 to S305, and these steps are described in detail below.

[0110] Step S301: Obtain multiple rosbag data frames in rosbag format.

[0111] The rosbag data frame includes data, the type of sensor that collects the data, the name of the sensor that collects the data, the collection timestamp of the data, the calibration data of the sensor that collects the data, the ID of the sensor that collects the data, and the message topic corresponding to the data; the sensor types include cameras, inertial measurement units, lidars, and millimeter-wave radars.

[0112] It is understood that one or more rosbag data frames can be stored in the same data packet. A data packet is a file that the ROS hardware platform uses to store and replay rosbag data frames.

[0113] The relevant nouns are explained below.

[0114] The collection timestamp of the collected data refers to the collection time of the collected data.

[0115] The calibration data of the sensor that collects data is the parameters set in the sensor by the manufacturer of the sensor. If the manufacturer does not set the calibration data of the sensor, the calibration data of the sensor can be determined manually based on the data collected by the sensor.

[0116] Exemplarily, the calibration data of the sensor includes, but is not limited to: instantaneous field of view (IFOV), relative sensitivity (RSR), and noise equivalent radiation (NER). The calibration data of the sensor is crucial to ensure the accuracy and reliability of the sensor. For example, through the calibration data of the sensor, the system errors of the sensor, such as zero drift and insufficient sensitivity, can be eliminated to ensure the consistency and accuracy between the data output by the sensor and the actual measurement value.

[0117] The communication mechanism of data in Rosbag format is the message sending and receiving mechanism, which is divided into three types: topic, service and action. The message topic mentioned in this application is the specific content of the "topic" of the message sending and receiving mechanism.

[0118] It can be understood that if the sensor type is a camera, the data in the rosbag data frame is an image; if the sensor type is a lidar or a millimeter-wave radar, the data in the rosbag data frame is point cloud data; if the sensor type is an inertial measurement unit, the data in the rosbag data frame is IMU pose data.

[0119] It can be understood that the IMU posture data measured by the inertial measurement unit includes three-axis angular velocity and three-axis acceleration information.

[0120] Step S302: Obtain the correspondence between the acquisition timestamps of key frames in multiple rosbag data frames and the odometer information corresponding to the key frames.

[0121] It is understandable that there are multiple ways to implement step S302. The embodiment of the present application provides but is not limited to the following method, which includes the following steps A1 to A3 during implementation.

[0122] Step A1: Acquire multiple key frames from multiple rosbag data frames whose sensor type is lidar.

[0123] Exemplarily, the implementation method of step A1 includes the following steps A11 to A13.

[0124] Step A11: Obtain multiple candidate key frames from the multiple rosbag data frames whose sensor type is lidar.

[0125] It can be understood that the data contained in the rosbag data frame whose sensor type is lidar is point cloud data.

[0126] Exemplarily, point cloud data can be subjected to point cloud distortion processing. Point cloud distortion processing is an important step in point cloud data processing, especially when using LiDAR for environmental scanning. Distortion is usually caused by the movement of LiDAR, resulting in a mismatch between the collected point cloud data and the position of the actual environment. To solve this problem, it is necessary to combine the point cloud data with the IMU pose data and eliminate this distortion through motion compensation.

[0127] It is understandable that point cloud data includes: three-dimensional coordinates (XYZ) and laser reflection intensity (Intensity), where the laser reflection intensity is related to the surface material and roughness of the target, the laser incident angle, the laser wavelength, and the energy density of the lidar. The IMU contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometer detects the acceleration signal of the target in the carrier coordinate system in three independent axes, while the gyroscope detects the angular velocity signal of the carrier relative to the navigation coordinate system, measures the angular velocity and acceleration of the target in three-dimensional space, and uses this to calculate the target's posture.

[0128] Exemplarily, multiple candidate key frames can be obtained from multiple rosbag data frames at preset intervals.

[0129] Step A12: construct a KD tree based on the multiple candidate key frames.

[0130] Step A13: Acquire a plurality of the key frames from the KD tree based on a preset recursive function.

[0131] It is understandable that in the process of building a local map based on SLAM technology, if all rosbag data frames are used to build a local map, complex calculations such as feature extraction, matching, and pose estimation are required for all point cloud data, which greatly increases the amount of calculation and leads to a decrease in real-time performance. Therefore, by selecting representative key frames from multiple candidate key frames, we avoid processing a large number of redundant candidate key frames and reduce invalid calculations.

[0132] Based on all candidate keyframes and the odometer information corresponding to all candidate keyframes, a KD (k-dimensional tree) tree is constructed. Exemplarily, a nearest neighbor search can be performed based on a given candidate keyframe as an index, that is, one or more candidate keyframes closest to a given candidate keyframe (query point) are found and used as keyframes; Exemplarily, a radius search can be performed based on a given candidate keyframe as an index, that is, all candidate keyframes within a specified radius threshold with respect to a given candidate keyframe (query point) are found and used as keyframes. Exemplarily, to avoid keyframes being too close, downsampling is used to limit the number of keyframes.

[0133] Exemplarily, the preset recursive function may traverse each node of the KD tree and decide whether to determine the candidate key frame as a key frame according to a given condition (eg, the distance from the query point).

[0134] Step A2: For each of the key frames, based on the first rosbag data frame, the second rosbag data frame and the key frame, determine the odometer information of the key frame to obtain the odometer information corresponding to the multiple key frames respectively.

[0135] The first rosbag data frame is collected by a camera, and the data in the first rosbag data frame is an image. The absolute value of the difference between the collection timestamp of the first rosbag data frame and the collection timestamp of the key frame is less than or equal to the preset duration. The second rosbag data frame is collected by an inertial measurement unit, and the data in the second rosbag data frame is IMU posture data. The absolute value of the difference between the collection timestamp of the second rosbag data frame and the collection timestamp of the key frame is less than or equal to the preset duration.

[0136] Exemplarily, the preset duration may be determined based on actual conditions, and is not limited in the embodiments of the present application.

[0137] There are many ways to implement step A2. The embodiment of the present application provides but is not limited to the following method, which includes steps A21 to A30.

[0138] Step A21: For each of the key frames, obtain the first rosbag data frame and the second rosbag data frame whose absolute value of the difference between the acquisition timestamp and the acquisition timestamp of the key frame is less than or equal to the preset duration.

[0139] It is understandable that the acquisition timestamp of the point cloud data collected by the lidar may be inconsistent with the acquisition timestamp of the IMU pose data collected by the IMU. Assuming that the acquisition timestamp of the key frame is 15:05:10:29 ms on November 20, 2024, and assuming that the preset duration is 2 milliseconds, the acquisition timestamp of the first rosbag data frame belongs to [15:05:10:27 ms on November 20, 2024, 15:05:10:31 ms on November 20, 2024], and the acquisition timestamp of the second rosbag data frame belongs to [15:05:10:27 ms on November 20, 2024, 15:05:10:31 ms on November 20, 2024].

[0140] Step A22: For each of the key frames, the zero bias of the IMU pose data is obtained based on the first rosbag data frame, the second rosbag data frame and the key frame through a zero bias estimation integrator.

[0141] Step A23: For each of the key frames, the candidate odometer information of the key frame is obtained through the odometer calculation integrator based on the zero bias, the first rosbag data frame, the second rosbag data frame and the key frame.

[0142] For example, the IMU pre-integration thread is a key part in the SLAM (Simultaneous Localization and Mapping) system. The IMU pre-integration thread usually includes two main integrators: the Bias Estimation Integrator and the Odometry Estimation Integrator.

[0143] In the context of sensors, an integrator may be used to process sensor readings to estimate the sensor's bias. Bias is the value of a sensor's output when there is no measured input, which is usually due to sensor imperfections or environmental factors. With an integrator, the output signal of the sensor can be accumulated over time to estimate the bias. This approach is particularly useful for sensors that require stable output over a long period of time, such as gyroscopes and accelerometers. In an IMU, gyroscopes are used to measure angular velocity, while accelerometers are used to measure linear acceleration. The output signals of these sensors need to be integrated to estimate the pose and position of the target. However, due to the imperfections of the sensors themselves, such as bias and random errors, errors will accumulate during the integration process. Therefore, these errors need to be estimated and compensated to improve the accuracy of pose and position estimates.

[0144] The main task of the bias estimation integrator is to estimate the bias of the IMU, which is the readings of the accelerometer and gyroscope when there is no motion. These readings are usually biased due to the imperfections of the sensors. The odometer calculation integrator is used to estimate the changes in the IMU's posture during motion, including position, velocity, and attitude.

[0145] IMU posture data includes: IMU three-axis angular velocity and three-axis acceleration.

[0146] Step A24: Determine initial pose information.

[0147] Pose initialization is a very important step in SLAM because it provides a reference point for the system, giving a clear starting point for subsequent pose estimation and local map construction.

[0148] Exemplarily, when an object (such as a train or a high-speed train set) deployed with a ROS hardware platform is in a stationary state, posture initialization is performed to obtain initial posture information.

[0149] It is understandable that in dynamic environments or complex scenarios such as mines, the initial pose information of the sensor will affect the stability of the entire system. Accurate pose initialization can reduce jitter or instability in the initial stage of the system, ensuring that it can still maintain reliable pose estimation in complex environments.

[0150] For example, when the object deployed with the ROS hardware platform is in a stationary state, the IMU pose data collected by the IMU can be the initial pose information. This is because in a stationary state, the IMU can provide a relatively accurate pose estimation.

[0151] For example, after obtaining the initial pose information, the pose estimation can be further optimized by an optimization algorithm (such as a nonlinear least squares method) to reduce the accumulated error.

[0152] Step A25: converting the candidate odometer information corresponding to the plurality of key frames into relative pose information based on the initial pose information.

[0153] Step A26: constructing a preliminary local map based on the relative posture information corresponding to the plurality of key frames.

[0154] For example, a preliminary local map can be obtained through a mapping module in SLAM technology.

[0155] It is understandable that SLAM technology is generally divided into five modules: sensor data module, visual odometer module, backend module, mapping module and loop detection module. The sensor data module is used to collect various types of raw data in the actual environment; the visual odometer module is used to estimate the relative position of the moving target at different times; the backend module is used to optimize the cumulative error caused by the visual odometer; the mapping module builds a map based on the motion trajectory obtained by the visual odometer module; the loop detection module eliminates the cumulative error by considering images of the same scene at different times.

[0156] For example, the present application can be applied to a mine transport environment, where there are many dynamic obstacles, such as mining equipment, loading trucks, and personnel. SLAM uses sensor data (such as lidar, cameras, and ultrasonic sensors) to perceive the surrounding environment in real time and generate a local map around the current train. During the journey, the train can update the map in real time and detect surrounding obstacles, dynamically adjust the route, and avoid collisions.

[0157] Step A27: Based on the preliminary local map, the optimized relative posture information corresponding to each of the key frames is obtained.

[0158] Exemplarily, the optimized relative pose information can be obtained by the Gauss-Newton method.

[0159] Exemplarily, the method of obtaining the optimized relative pose information obtained by the Gauss-Newton method includes but is not limited to the following methods: obtaining the curvature of the point in the point cloud data; determining the point whose curvature is greater than or equal to the set threshold 1 as the corner point, and determining the point whose curvature is less than the set threshold 1 as the plane point; for the plane point, determining the line closest to the plane point in the preliminary local map, and calculating the vertical distance between the plane point and the line; for the corner point, determining the point closest to the angle in the preliminary local map, and determining the target distance between the corner point and the point closest to the angle. Determine that the plane point whose vertical distance is less than or equal to the set threshold 2 needs to be optimized; determine that the corner point whose target distance is less than or equal to the set threshold 3 needs to be optimized. Optimize the points that need to be optimized based on the preset error function.

[0160] Since the preset error function quantifies the difference between the corner points and plane points that need to be optimized and the corner points and plane points estimated and calculated based on the current relative pose information, the optimized relative pose information is more accurate than the relative position information before optimization.

[0161] Exemplarily, the method for calculating the curvature of each point A includes: taking 5 points on the left and right of the current point, and calculating according to the following formula:

[0162]

[0163] Among them, c represents the curvature, S represents the point set containing 5 points, ri represents the coordinates of point A, i is greater than or equal to 1 and less than or equal to 5. rj represents the coordinates of the jth point.

[0164] Exemplarily, corner points and plane points may be published to ROS topics respectively.

[0165] It is understandable that because the laser radar scanning method is line scanning, therefore, on the same line, only 5 points are taken on the left and right, so the corner points extracted are line corner points.

[0166] Step A28: updating the preliminary local map based on the optimized relative pose information respectively corresponding to the plurality of key frames to obtain a candidate local map.

[0167] Step A29: Update the candidate local map through closed-loop detection technology to obtain a target local map.

[0168] Step A30: Obtain the odometer information corresponding to the multiple key frames respectively from the target local map.

[0169] Exemplarily, the present application is applied to a mine application scenario, and the object deployed with the ROS hardware platform is a train. Because the transportation routes in the mine are often repetitive, trains frequently run on the same track. The candidate local map combined with the closed-loop detection technology can identify the area where the train returns to where it has traveled, correct the trajectory, reduce the cumulative error caused by long-term driving, and ensure accurate navigation under long-term and high-load conditions. When the train completes an ore transport and returns to the mine, the candidate local map will detect the same part of the track. The system can correct the posture error through the loop detection mechanism to ensure that the train maintains a high navigation accuracy in subsequent transportation to obtain the target local map.

[0170] For example, the problem of adjusting relative pose information and building a map can be converted into a graph optimization problem, i.e., the process of constructing a factor graph. In graph optimization, nodes represent odometer information. The edges between two nodes represent the constraint relationship between two nodes, i.e., factors. Common factors include: Laser Odometry Factors, IMU Factors, and Loop Closure Factors. Combining the above nodes and edges can construct a factor graph.

[0171] In an embodiment of the present application, the nodes in the factor graph include historical nodes and current nodes. The historical node refers to the relative pose information collected historically, and the current node refers to the relative pose information collected currently. The laser odometer factor is used to provide motion constraints between adjacent frames, the IMU factor provides additional constraints through high-frequency motion estimation, and the loopback factor corrects the global drift error by detecting repeatedly visited scenes. Exemplarily, the converged odometer information can be published to a ROS topic.

[0172] Step A3: constructing a correspondence between the acquisition timestamp of the key frame and the odometer information corresponding to the key frame.

[0173] It can be understood that the acquisition timestamp of the key frame is the acquisition timestamp of the odometer information corresponding to the key frame.

[0174] Step S303: Convert the multiple rosbag data frames into multiple nuScenes data frames in nuScenes format.

[0175] Exemplarily, a data packet (rosbag) includes one or more rosbag data frames. The data packet is stored in a rosbag database in a serialized form in a computer.

[0176] For example, there are two main formats of data packets (rosbag), namely: SQLite3 and rosbagv2.

[0177] If the ROS hardware platform is the ROS2 platform, the format of the data packet collected by the ROS2 platform is the Fast-CDR format. The format of the rosbag database storage is the SQLite3 format. The rosbag database has the advantages of small source code size (250KB), fast operation, large single file storage capacity (>1TB), and strong compatibility (can be shared on machines with different byte orders). The SQLite3 access interface is provided in the ROS2 platform. In the process of reading the data packet (rosbag) collected by the ROS2 platform into the rosbag database, the serialization format of the data packet (rosbag) needs to be set to cdr. Call the rosbag2_storage plug-in, instantiate the sequence reader, and read the description file and deserialized data of the data packet (rosbag) into the structure. The description file contains: file storage format, serialization format, data packet time length, ROS topic name (i.e. message topic), message type, and number of messages. The structure contains data information corresponding to the description file.

[0178] As an autonomous driving dataset, the nuScenes dataset has the advantages of large data volume, unified format, and complete data structure. Therefore, the nuScenes dataset is often used as training data for models. The nuScenes dataset is a large-scale autonomous driving dataset that provides rich resources for training models.

[0179] The applicant collects multiple rosbag data frames based on the ROS hardware platform, and the rosbag data frames are in the rosbag format. The ROS hardware platform includes sensors such as millimeter-wave radar, lidar, camera, and IMU. There are the following problems in collecting multiple rosbag data frames through the ROS hardware platform: the rosbag data frame includes data, the type of sensor that collects the data, the name of the sensor that collects the data, the acquisition timestamp of the collected data, the calibration data of the sensor that collects the data, the ID of the sensor that collects the data, and the message topic corresponding to the data. Due to the different sensor drivers provided by different manufacturers, there are cases where the sensor ID is missing in some rosbag data frames.

[0180] It is understandable that the ROS hardware platform usually does not consider the adaptability to the model that needs to be trained later when collecting data packets. Therefore, the sensor ID in the data frame is different from the sensor ID required by the model to be trained. Therefore, it is necessary to fill in the sensor ID to meet the requirements. The specific method may include the following steps B1 to B3.

[0181] Step B1: Determine the target message topic of the rosbag data frame of the sensor ID to be completed.

[0182] You can manually set the sensor IDs of the rosbag data frames with the target message topics that need to be completed. After determination, you can traverse all rosbag data frames to obtain the rosbag data frames with the target message topics.

[0183] Step B2: Obtain a rosbag data frame having the target message topic from the multiple rosbag data frames.

[0184] Step B3: Adjust the ID of the data frame of the rosbag data frame having the target message topic to the set ID.

[0185] Exemplarily, an instantiated deserialization reader can be used to read a rosbag data frame (located in a data packet) with a target message topic into a corresponding structure according to its type; after replacing the ID of the original data frame in the structure with the set ID, the data in the structure is reserialized and converted into a data packet through an instantiated serialization writer.

[0186] A deserializer is a process or tool that converts data back to its original structure in programming.

[0187] It is understandable that some sensors are non-standard sensors, and non-standard sensors do not have calibration data, so the calibration data of the sensors needs to be supplemented. The specific method includes the following steps C1 to C2.

[0188] Step C1: Determine a target rosbag data frame for completing the calibration data of the sensor from the plurality of rosbag data frames.

[0189] Step C2: Fill in the target rosbag data frame with calibration data.

[0190] Exemplarily, the calibration message content and calibration message type of the missing sensor calibration data may be preset.

[0191] Exemplarily, for a camera, the calibration message type is CameraInfo, and the calibration message content includes camera intrinsic parameters, camera distortion parameters, image width and height. The process of obtaining the calibration message content includes: creating a new message whose acquisition timestamp is synchronized with the camera's acquisition timestamp. Writing the camera intrinsic parameters and distortion parameters obtained by Zhang Zhengyou's calibration method into the calibration message content. Finally, the calibration message content is written into the target rosbag data frame through the instantiated serialized writer.

[0192] Step S304: constructing a correspondence between the nuScenes data frames and the odometer information based on the acquisition timestamps of the plurality of nuScenes data frames and the acquisition timestamps of the plurality of odometer information.

[0193] Step S305: storing the plurality of nuScenes data frames and the correspondence between the nuScenes data frames and the odometer information in a pre-constructed nuScenes database.

[0194] An embodiment of the present application provides a method for converting data in rosbag format into nuScenes format, obtaining multiple rosbag data frames in rosbag format; obtaining the correspondence between the acquisition timestamp of key frames in multiple rosbag data frames and the odometer information corresponding to the key frames; converting the multiple rosbag data frames into multiple nuScenes data frames in nuScenes format; constructing the correspondence between the nuScenes data frames and the odometer information based on the acquisition timestamps of the multiple nuScenes data frames and the acquisition timestamps of the multiple odometer information; storing the multiple nuScenes data frames and the correspondence between the nuScenes data frames and the odometer information in a pre-constructed nuScenes database, thereby achieving the purpose of converting data in rosbag format into data in nuScenes format.

[0195] Exemplarily, there are multiple ways to implement step S303. The embodiment of the present application provides but is not limited to the following method, which includes the following steps D1 to D3 during implementation.

[0196] Step D1: Acquire multiple target rosbag data frames whose sensor type is the target sensor type from the multiple rosbag data frames.

[0197] It is understandable that the data frame used to train the model can be collected by a lidar, a millimeter-wave radar, or a camera. Based on this, the target sensor type corresponding to the data frame of the training model can be pre-set, so that the target rosbag data frame corresponding to the target sensor type can be converted into a nuScenes data frame.

[0198] Step D2: Based on the nuScenes configuration file, convert the rosbag fields in the multiple target rosbag data frames into nuScenes fields and convert the field value types of the rosbag fields in the multiple target rosbag data frames into the field value types of the nuScenes fields to obtain multiple nuScenes data frames.

[0199] The nuScenes configuration file includes the correspondence between the preset nuScenes fields and the rosbag fields, and the correspondence between the field value types of the preset nuScenes fields and the field value types of the rosbag fields.

[0200] The following example illustrates this.

[0201] Assume that the rosbag field of the sensor type is "CGQLX" and the nuScenes field of the sensor type is "sensor", the nuScenes configuration file includes the correspondence between "CGQLX" and "sensor".

[0202] Assuming that the field value type of the rosbag field of the sensor type is "CGQLX" and the field value type is character type, and the field value type of the nuScenes field of the sensor type is "sensor" and the field value type is integer type, then the nuScenes configuration file includes: the correspondence between "CGQLX" corresponding to the character type and "sensor" corresponding to the integer type.

[0203] For example, the field value of the sensor type rosbag field "CGQLX" is LiDAR or camera or MMWRadar or IMU, and the field value of the sensor type nuScenes field "sensor" is 1 or 2 or 3 or 4, where 1 represents lidar, 2 represents camera, 3 represents millimeter wave radar, and 4 represents IMU.

[0204] It is understandable that if the field values ​​of the rosbag field and the nuScenes field representing the same meaning are different, the corresponding relationship needs to be set. For example, it is necessary to set the corresponding relationship between 1 and LiDAR, 2 and camera, 3 and MMWRadar, and 4 and IMU. This corresponding relationship also needs to be stored in the nuScenes configuration file.

[0205] Step D3: Store the plurality of nuScenes data frames into a nuScenes structure.

[0206] Exemplarily, the nuScenes configuration file may be a URDF configuration file.

[0207] Exemplarily, the nuScenes structure is used as an intermediate, which facilitates storing multiple nuScenes data frames into the nuScenes database.

[0208] As an autonomous driving dataset, the nuScenes dataset has the advantages of large data volume, unified format, and complete data structure. Therefore, the nuScenes dataset is often used as training data for models. The nuScenes dataset is a large-scale autonomous driving dataset that provides rich resources for training models.

[0209] The applicant collects multiple rosbag data frames based on the ROS hardware platform, and the rosbag data frames are in the rosbag format. The ROS hardware platform includes sensors such as millimeter-wave radar, lidar, camera, and IMU. There are the following problems in collecting multiple rosbag data frames through the ROS hardware platform: Since the ROS hardware platform does not include GPS (Global Positioning System) and encoders, the collection timestamp of the IMU's IMU pose data may be different from the collection timestamp of the lidar, camera, and millimeter-wave radar data, making it impossible to obtain the IMU pose data when the lidar, camera, and millimeter-wave radar collect data, that is, the odometer information when the lidar, camera, and millimeter-wave radar collect data is missing.

[0210] Due to the above problems, the rosbag data frame cannot be used for model training. Based on this, the implementation method of step S304 includes the following steps E01 to E10.

[0211] Exemplarily, after obtaining multiple rosbag data frames from the rosbag database, the rosbag data frames can be deserialized so that the information in the rosbag data frames can be read.

[0212] Exemplarily, the correspondence between the odometer information and the collection timestamp obtained from the rosbag database is deserialized so that the correspondence between the odometer information and the collection timestamp can be read.

[0213] Step E01: Create a first token for each of the plurality of odometer information.

[0214] For example, the number of odometer information may be counted. Different odometer information has different first tokens.

[0215] Exemplarily, different first tokens are randomly assigned to different odometer information.

[0216] Step E02: Store the odometer information, the first token of the odometer information and the collection timestamp of the odometer information into a nuScenes odometer queue in a nuScenes format.

[0217] Exemplarily, step E01 and step E02 may be executed by the created odometer conversion thread.

[0218] Exemplarily, writing all odometer information into the nuScenes odometer queue may wake up the sensor sample conversion thread.

[0219] Exemplarily, the sensor sample conversion thread may execute steps E03 to E07.

[0220] Step E03: For the nuScenes data frame containing point cloud data, compress multiple nuScenes data frames collected by the same radar whose maximum difference in acquisition timestamps is less than or equal to the target difference into the same point cloud set to obtain multiple point cloud sets, where the radar is a millimeter wave radar or a lidar.

[0221] Exemplarily, the target difference may be determined based on actual conditions, which is not limited here.

[0222] Exemplarily, a specific implementation method of compressing multiple nuScenes data frames collected by the same radar whose maximum difference of collection timestamps is less than or equal to the target difference into the same point cloud set includes the following steps E31 to E33. Assume that the radar is a millimeter wave radar.

[0223] Step E31: Sort the acquisition timestamps of multiple nuScenes data frames corresponding to the millimeter-wave radar from early to late to obtain a sorting result.

[0224] Step E32: Create a new point cloud collection.

[0225] Step E33: Obtain the pending point cloud data that is adjacent to the point cloud data that has been divided into the point cloud set and has not been divided into the point cloud set from the sorting result; if the absolute value of the difference between the acquisition timestamp of the pending point cloud data and the minimum acquisition timestamp in the new point cloud set in step E32 is less than or equal to the target difference, divide the pending point cloud data into the new point cloud set in step E32; return to step E33; if the absolute value of the difference between the acquisition timestamp of the pending point cloud data and the minimum acquisition timestamp in the new point cloud set in step E32 is greater than the target difference, return to step E32, and divide the pending point cloud data into the new point cloud set created in step E32.

[0226] Step E04: for each of the point cloud sets, determining the smallest acquisition timestamp among the multiple nuScenes data frames included in the point cloud set as the acquisition timestamp of the point cloud set.

[0227] Step E05: for each of the point cloud sets, determine the odometer information whose acquisition timestamp is closest to the acquisition timestamp of the point cloud set as the odometer information of the point cloud set.

[0228] Step E06: Create a second token for each of the plurality of point cloud sets.

[0229] Exemplarily, the total number of point cloud sets is obtained. It is determined whether the point cloud set is the last frame according to the message counter, and if it is the last frame, the second Token of the next frame is set to empty, otherwise, it is set to the random second Token.

[0230] Step E07: Store the point cloud set, the second token of the point cloud set, and the acquisition timestamp of the point cloud set into the first nuScenes message queue in the nuScenes format.

[0231] Exemplarily, after executing step E07, the camera sample conversion thread may be awakened. Exemplarily, the camera sample conversion thread may execute steps E08 to E10.

[0232] Step E08: for each of the nuScenes data frames captured by the camera, determine the odometer information whose acquisition timestamp is closest to the acquisition timestamp of the nuScenes data frame as the odometer information of the nuScenes data frame.

[0233] The data in the nuScenes data frame collected by the camera is an image. For example, the OpenCV API (Application Programming Interface) may be called to convert the image into a bgr8 format.

[0234] Step E09: assigning a third token to each of the plurality of nuScenes data frames captured by the camera.

[0235] Step E10: Store the nuScenes data frame collected by the camera, the third token corresponding to the nuScenes data frame, the odometer information corresponding to the nuScenes data frame, and the camera calibration information in a second nuScenes message queue.

[0236] Exemplarily, the total number of nuScenes data frames captured by the camera can be obtained. According to the message counter, it is determined whether the nuScenes data frame to be assigned the third token is the last frame. If it is the last frame, the next frame Token is set to empty, otherwise, a random third Token is set.

[0237] Exemplarily, the calibration data of the camera includes a camera intrinsic parameter matrix, and the dimension of the camera intrinsic parameter matrix is ​​3x3. The calibration data of the camera may be stored in the second nuScenes message queue.

[0238] Exemplarily, the first nuScenes message queue and the second nuScenes message queue may be the same nuScenes message queue or different nuScenes message queues.

[0239] Exemplarily, there are multiple ways to implement step S305. The embodiment of the present application provides but is not limited to the following method, which includes the following steps F01 to F14 during implementation.

[0240] Step F01: Create a target folder, a samples folder, a sweeps folder, and a calibrate_sensor.json folder in the nuScenes database.

[0241] Exemplarily, the name of the target folder can be set arbitrarily. Exemplarily, the name of the target folder can be v1.0-mini.

[0242] Step F02: Create an ego_pose.json file, a sample.json file, a sample_data.json file, and a scene.json file in the target folder.

[0243] Step F03: Create a file with a suffix of .pcd, a file with a suffix of .pcd.bin, and a file with a suffix of .jpg in the nuScenes database.

[0244] Step F04: converting the odometer information corresponding to the plurality of point cloud sets and the odometer information corresponding to the nuScenes data frame collected by the camera into pose information relative to a preset body coordinate system, and writing the pose information into the calibrate_sensor.json folder.

[0245] Exemplarily, the body coordinate system may be set manually.

[0246] Exemplarily, the odometer information is converted into position and posture information relative to a preset body coordinate system, that is, the coordinates of the sensor are transformed. Exemplarily, the converted position and posture information of the sensor can be saved in the form of quaternion.

[0247] Step F05: If the radar corresponding to the first nuScenes message queue is a millimeter wave radar, the information stored in the first nuScenes message queue is written into a file with a suffix of .pcd.

[0248] Step F06: If the radar corresponding to the first nuScenes message queue is a laser radar, the information stored in the first nuScenes message queue is written into a file with a suffix of .pcd.bin.

[0249] Step F07: Write the information stored in the second nuScenes message queue into a file with a suffix of .jpg.

[0250] Step F08: storing the nuScenes data frames corresponding to the plurality of key frames respectively into the scene.json file.

[0251] Step F09: storing the information stored in the nuScenes odometry queue into the ego_pose.json file.

[0252] Step F10: storing the preset scene information of the point cloud data and the preset scene information of the nuScenes data frame collected by the camera into the scene.json file.

[0253] Step F11: Store the predecessor token and successor token of the first token of each odometry information, the predecessor token and successor token of the second token of each point cloud set, the predecessor token and successor token of the third token of each nuScenes data frame captured by the camera, the acquisition timestamp of each first token, the acquisition timestamp of each second token, and the acquisition timestamp of each third token in the samples folder.

[0254] Step F12: Store the predecessor token and successor token of the first token of each of the odometer information, the predecessor token and successor token of the second token of each of the point cloud sets, the predecessor token and successor token of the third token of each of the nuScenes data frames captured by the camera, the acquisition timestamp of each of the first tokens, the acquisition timestamp of each of the second tokens, the acquisition timestamp of each of the third tokens, whether each of the point cloud sets includes key frames, whether each of the nuScenes data frames captured by the camera is a key frame, the storage path of each of the point cloud sets, the storage path of each of the nuScenes data frames captured by the camera, the first token of the odometer information of each of the point cloud sets, and the first token of the odometer information of a nuScenes data frame captured by the camera into the sample_data.json file.

[0255] Step F13: storing the nuScenes data frames corresponding to the plurality of key frames respectively into the samples folder.

[0256] Step F14: storing the nuScenes data frames other than the key frame in the plurality of nuScenes data frames into the sweeps folder.

[0257] After the development of the method for converting data in rosbag format to nuScenes format provided in the embodiment of the present application is completed, it can be used as a general format conversion tool for converting data in rosbag format to data in nuScenes format. Figure 4 As shown, it is a schematic diagram of format conversion provided in an embodiment of the present application.

[0258] Figure 4 The rosbag data frame obtained from the rosbag database 40 and the nuScenes configuration information, namely the URDF configuration information, are input into the universal format conversion tool 41, and the universal format conversion tool 41 can store the corresponding information into the nuScenes database 42.

[0259] Corresponding nuScenes data may be retrieved or looked up from the nuScenes database 42 for use in training one or more models.

[0260] In the related art, a data conversion script is written for each model in each field, and the rosbag data frame is converted into the data format corresponding to the field, which brings huge manpower consumption. This application generates odometer information through sensor data such as lidar, camera, IMU, etc., which solves the problem of missing odometer data; the sensor ID and sensor calibration data are supplemented by re-recording the rosbag data frame, which solves the problem of missing data composition; the rosbag data frame is parsed by deserialization, and serialized and saved to the nuScenes database, which realizes the purpose of converting the format of the rosbag data frame to the nuScenes database. There is no need to manually write data conversion scripts for the training data of the model in each field, which greatly reduces manpower consumption.

[0261] The above introduces a method for converting data in rosbag format to nuScenes format provided by an embodiment of the present application. The following will introduce a device for executing the method for converting data in rosbag format to nuScenes format.

[0262] See also Figure 5 , Figure 5 A schematic diagram of the structure of a device for converting data in rosbag format into nuScenes format provided in an embodiment of the present application. Figure 5 As shown, the device that converts the data in rosbag format into nuScenes format includes:

[0263] The first acquisition module 501 is used to acquire multiple rosbag data frames in the rosbag format, wherein the rosbag data frames include data, the type of sensor that collects the data, the name of the sensor that collects the data, the acquisition timestamp of the data, the calibration data of the sensor that collects the data, the ID of the sensor that collects the data, and the message topic corresponding to the data; the sensor types include camera, inertial measurement unit, laser radar and millimeter wave radar;

[0264] The second acquisition module 502 is used to obtain the corresponding relationship between the acquisition timestamp of the key frame in multiple rosbag data frames and the odometer information corresponding to the key frame;

[0265] A conversion module 503, used for converting the plurality of rosbag data frames into a plurality of nuScenes data frames in a nuScenes format;

[0266] A construction module 504, configured to construct a correspondence between the nuScenes data frame and the odometer information based on the acquisition timestamps of the plurality of nuScenes data frames and the acquisition timestamps of the plurality of odometer information;

[0267] The storage module 505 is used to store the plurality of nuScenes data frames and the correspondence between the nuScenes data frames and the odometer information in a pre-built nuScenes database.

[0268] In an optional implementation, the building blocks include:

[0269] A first allocation unit, configured to create first tokens for the plurality of odometer information respectively;

[0270] A first storage unit, configured to store the odometer information, the first token of the odometer information, and a collection timestamp of the odometer information into a nuScenes odometer queue in a nuScenes format;

[0271] A partitioning unit is used for compressing multiple nuScenes data frames collected by the same radar whose maximum difference of acquisition timestamps is less than or equal to the target difference, for a nuScenes data frame containing point cloud data, into the same point cloud set to obtain multiple point cloud sets, wherein the radar is a millimeter wave radar or a laser radar;

[0272] A first determining unit is used to determine, for each of the point cloud sets, a minimum acquisition timestamp among a plurality of the nuScenes data frames included in the point cloud set as the acquisition timestamp of the point cloud set;

[0273] A second determining unit is used to determine, for each of the point cloud sets, the odometer information whose acquisition timestamp is closest to the acquisition timestamp of the point cloud set as the odometer information of the point cloud set;

[0274] A second allocation unit, configured to respectively create a second token for each of the plurality of point cloud sets;

[0275] A second storage unit, configured to store the point cloud set, the second token of the point cloud set, and the acquisition timestamp of the point cloud set in a first nuScenes message queue in a nuScenes format;

[0276] A third determining unit is used to determine, for each of the nuScenes data frames captured by the camera, the odometer information whose acquisition timestamp is closest to the acquisition timestamp of the nuScenes data frame as the odometer information of the nuScenes data frame;

[0277] A third allocation unit, configured to allocate third tokens to the multiple nuScenes data frames captured by the camera respectively;

[0278] The third storage unit is used to store the nuScenes data frame collected by the camera, the third token corresponding to the nuScenes data frame, the odometer information corresponding to the nuScenes data frame, and the setting calibration information of the camera in a second nuScenes message queue.

[0279] In an optional implementation, the storage module includes:

[0280] A first creation unit is used to create a target folder, a samples folder, a sweeps folder and a calibrate_sensor.json folder in the nuScenes database;

[0281] A second creation unit is used to create an ego_pose.json file, a sample.json file, a sample_data.json file and a scene.json file in the target folder;

[0282] A third creation unit is used to create a file with a suffix of .pcd, a file with a suffix of .pcd.bin, and a file with a suffix of .jpg in the nuScenes database;

[0283] A conversion unit, used to convert the odometer information corresponding to the plurality of point cloud sets and the odometer information corresponding to the nuScenes data frame collected by the camera into pose information relative to a preset body coordinate system, and write the pose information into the calibrate_sensor.json folder;

[0284] A fourth storage unit, configured to write the information stored in the first nuScenes message queue into a file with a suffix of .pcd if the radar corresponding to the first nuScenes message queue is a millimeter wave radar;

[0285] A fifth storage unit, configured to write the information stored in the first nuScenes message queue into a file with a suffix of .pcd.bin if the radar corresponding to the first nuScenes message queue is a laser radar;

[0286] A sixth storage unit, used for writing the information stored in the second nuScenes message queue into a file with a suffix of .jpg;

[0287] A seventh storage unit, used for storing the nuScenes data frames respectively corresponding to the plurality of key frames into the scene.json file;

[0288] An eighth storage unit, used to store the information stored in the nuScenes odometer queue into the ego_pose.json file;

[0289] A ninth storage unit, used to store the preset scene information of the point cloud data and the preset scene information of the nuScenes data frame collected by the camera into the scene.json file;

[0290] a tenth storage unit, configured to store the preceding token and the succeeding token of the first token of each odometer information, the preceding token and the succeeding token of the second token of each point cloud set, the preceding token and the succeeding token of the third token of each nuScenes data frame acquired by the camera, the acquisition timestamp of each first token, the acquisition timestamp of each second token, and the acquisition timestamp of each third token into the samples folder;

[0291] An eleventh storage unit is used to store the preceding token and the succeeding token of the first token of each of the odometer information, the preceding token and the succeeding token of the second token of each of the point cloud sets, the preceding token and the succeeding token of the third token of each of the nuScenes data frames collected by the camera, the acquisition timestamp of each of the first tokens, the acquisition timestamp of each of the second tokens, the acquisition timestamp of each of the third tokens, whether each of the point cloud sets includes a key frame, whether each of the nuScenes data frames collected by the camera is a key frame, the storage path of each of the point cloud sets, the storage path of each of the nuScenes data frames collected by the camera, the first token of the odometer information of each of the point cloud sets, and the first token of the odometer information of a nuScenes data frame collected by the camera into the sample_data.json file;

[0292] A twelfth storage unit, used for storing the nuScenes data frames respectively corresponding to the multiple key frames into the samples folder;

[0293] A thirteenth storage unit is used to store the nuScenes data frames other than the key frame in the plurality of nuScenes data frames into the sweeps folder.

[0294] In an optional implementation, the second acquisition module includes:

[0295] A first acquisition unit, used for acquiring a plurality of key frames from a plurality of rosbag data frames whose sensor type is a laser radar;

[0296] A fourth determination unit is used to determine the odometer information of each key frame based on the first rosbag data frame, the second rosbag data frame and the key frame to obtain the odometer information corresponding to the multiple key frames respectively; the first rosbag data frame is acquired by a camera, the data in the first rosbag data frame is an image, the absolute value of the difference between the acquisition timestamp of the first rosbag data frame and the acquisition timestamp of the key frame is less than or equal to a preset duration, the second rosbag data frame is acquired by an inertial measurement unit, and the data in the second rosbag data frame is IMU posture data; the absolute value of the difference between the acquisition timestamp of the second rosbag data frame and the acquisition timestamp of the key frame is less than or equal to the preset duration;

[0297] The corresponding relationship building unit is used to build a corresponding relationship between the acquisition timestamp of the key frame and the odometer information corresponding to the key frame.

[0298] In an optional implementation, the first acquiring unit includes:

[0299] A first acquisition subunit is used to acquire multiple candidate key frames from multiple rosbag data frames whose sensor type is a laser radar;

[0300] A first construction subunit, configured to construct a KD tree based on a plurality of the candidate key frames;

[0301] The second acquisition subunit is used to acquire a plurality of the key frames from the KD tree based on a preset recursive function.

[0302] In an optional implementation, the fourth determining unit includes:

[0303] A third acquisition subunit is used to acquire, for each of the key frames, the first rosbag data frame and the second rosbag data frame whose absolute value of the difference between the acquisition timestamp and the acquisition timestamp of the key frame is less than or equal to the preset duration;

[0304] A fourth acquisition subunit is used to obtain, for each of the key frames, a zero bias of the IMU pose data based on the first rosbag data frame, the second rosbag data frame and the key frame through a zero bias estimation integrator;

[0305] A fifth acquisition subunit is used to obtain, for each of the key frames, candidate odometer information of the key frame based on the zero bias, the first rosbag data frame, the second rosbag data frame and the key frame through a mileage calculation integrator;

[0306] A first determination subunit, used to determine initial posture information;

[0307] A conversion subunit, used for converting the candidate odometer information corresponding to the plurality of key frames respectively into relative pose information based on the initial pose information;

[0308] A second construction subunit is used to construct a preliminary local map based on the relative posture information respectively corresponding to the plurality of key frames;

[0309] A sixth acquisition subunit, configured to obtain optimized relative pose information respectively corresponding to a plurality of the key frames based on the preliminary local map;

[0310] a seventh acquisition subunit, configured to update the preliminary local map based on the optimized relative pose information respectively corresponding to the plurality of key frames, so as to obtain a candidate local map;

[0311] an eighth acquisition subunit, configured to update the candidate local map by a closed-loop detection technology to obtain a target local map;

[0312] The ninth acquisition subunit is used to acquire the odometer information corresponding to the multiple key frames respectively from the target local map.

[0313] In an optional implementation, the method further includes:

[0314] The first determination module is used to determine the target message topic of the rosbag data frame of the sensor ID to be completed;

[0315] A third acquisition module is used to acquire a rosbag data frame having the target message topic from the plurality of rosbag data frames;

[0316] An adjustment module is used to adjust the ID of the data frame of the rosbag data frame having the target message topic to a set ID.

[0317] In an optional implementation, the method further includes:

[0318] A second determination module is used to determine a target rosbag data frame of the calibration data of the sensor to be completed from the multiple rosbag data frames;

[0319] A filling module is used to fill in the target rosbag data frame with set calibration data.

[0320] In an optional implementation, the conversion module includes:

[0321] A second acquisition unit is used to acquire a plurality of target rosbag data frames whose sensor type is the target sensor type from the plurality of rosbag data frames;

[0322] A conversion format unit, for converting the rosbag fields in the plurality of target rosbag data frames into nuScenes fields and converting the field value types of the rosbag fields in the plurality of target rosbag data frames into the field value types of the nuScenes fields based on a nuScenes configuration file, so as to obtain a plurality of the nuScenes data frames; the nuScenes configuration file includes a correspondence between preset nuScenes fields and rosbag fields and a correspondence between preset field value types of the nuScenes fields and field value types of the rosbag fields;

[0323] A fourteenth storage unit is used to store the plurality of nuScenes data frames into a nuScenes structure.

[0324] Also provided in an embodiment of the present application is a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements a method for converting data in any rosbag format provided in an embodiment of the present application into data in nuScenes format.

[0325] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device can implement a method for converting data in any rosbag format provided in an embodiment of the present application into a nuScenes format.

[0326] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0327] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0328] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0329] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for converting data in rosbag format to nuScenes format, characterized in that: include: Obtain multiple rosbag data frames in rosbag format, wherein the rosbag data frames include data, sensor type for collecting the data, sensor name for collecting the data, collection timestamp for collecting the data, calibration data of the sensor for collecting the data, ID of the sensor for collecting the data, and message topic corresponding to the data; the sensor types include camera, inertial measurement unit, lidar, and millimeter wave radar; Obtain the corresponding relationship between the acquisition timestamp of the key frames in multiple rosbag data frames and the odometer information corresponding to the key frames; Convert the multiple rosbag data frames into multiple nuScenes data frames in nuScenes format; Based on the acquisition timestamps of the plurality of nuScenes data frames and the acquisition timestamps of the plurality of odometer information, constructing a correspondence between the nuScenes data frames and the odometer information; The plurality of nuScenes data frames and the correspondence between the nuScenes data frames and the odometer information are stored in a pre-constructed nuScenes database.

2. The method for converting data in rosbag format into nuScenes format according to claim 1, characterized in that: The step of constructing a correspondence between the nuScenes data frame and the odometer information based on the acquisition timestamps of the plurality of nuScenes data frames and the acquisition timestamps of the plurality of odometer information comprises: Creating a first token for each of the plurality of odometer information; Store the odometer information, the first token of the odometer information, and the acquisition timestamp of the odometer information into a nuScenes odometer queue in a nuScenes format; For a nuScenes data frame containing point cloud data, compress multiple nuScenes data frames collected by the same radar whose maximum difference of acquisition timestamps is less than or equal to the target difference into the same point cloud set to obtain multiple point cloud sets, wherein the radar is a millimeter wave radar or a laser radar; For each of the point cloud sets, determining the smallest acquisition timestamp among the multiple nuScenes data frames contained in the point cloud set as the acquisition timestamp of the point cloud set; For each of the point cloud sets, determining the odometer information whose acquisition timestamp is closest to the acquisition timestamp of the point cloud set as the odometer information of the point cloud set; Creating a second token for each of the plurality of point cloud sets; Store the point cloud set, the second token of the point cloud set, and the acquisition timestamp of the point cloud set in a first nuScenes message queue in nuScenes format; For each of the nuScenes data frames captured by the camera, determine the odometer information whose acquisition timestamp is closest to the acquisition timestamp of the nuScenes data frame as the odometer information of the nuScenes data frame; Allocate a third token to each of the multiple nuScenes data frames captured by the camera; The nuScenes data frame collected by the camera, the third token corresponding to the nuScenes data frame, the odometer information corresponding to the nuScenes data frame, and the setting calibration information of the camera are stored in a second nuScenes message queue.

3. The method for converting data in rosbag format into nuScenes format according to claim 1, characterized in that: The step of storing the plurality of nuScenes data frames and the correspondence between the nuScenes data frames and the odometer information in a pre-built nuScenes database comprises: Create a target folder, a samples folder, a sweeps folder, and a calibrate_sensor.json folder in the nuScenes database; Create ego_pose.json file, sample.json file, sample_data.json file and scene.json file in the target folder; Create a file with a suffix of .pcd, a file with a suffix of .pcd.bin, and a file with a suffix of .jpg in the nuScenes database; Convert the odometer information corresponding to the plurality of point cloud sets and the odometer information corresponding to the nuScenes data frame collected by the camera into pose information relative to a preset body coordinate system, and write them into the calibrate_sensor.json folder; If the radar corresponding to the first nuScenes message queue is a millimeter wave radar, write the information stored in the first nuScenes message queue into a file with a suffix of .pcd; If the radar corresponding to the first nuScenes message queue is a laser radar, the information stored in the first nuScenes message queue is written into a file with a suffix of .pcd.bin; Write the information stored in the second nuScenes message queue into a file with a suffix of .jpg; The nuScenes data frames respectively corresponding to the multiple key frames are stored in the scene.json file; Store the information stored in the nuScenes odometry queue into the ego_pose.json file; The preset scene information of the point cloud data and the preset scene information of the nuScenes data frame collected by the camera are stored in the scene.json file; Store the preceding token and the succeeding token of the first token of each odometer information, the preceding token and the succeeding token of the second token of each point cloud set, the preceding token and the succeeding token of the third token of each nuScenes data frame acquired by the camera, the acquisition timestamp of each first token, the acquisition timestamp of each second token, and the acquisition timestamp of each third token into the samples folder; Store the preceding token and the succeeding token of the first token of each of the odometer information, the preceding token and the succeeding token of the second token of each of the point cloud sets, the preceding token and the succeeding token of the third token of each of the nuScenes data frames collected by the camera, the acquisition timestamp of each of the first tokens, the acquisition timestamp of each of the second tokens, the acquisition timestamp of each of the third tokens, whether each of the point cloud sets includes a key frame, whether each of the nuScenes data frames collected by the camera is a key frame, the storage path of each of the point cloud sets, the storage path of each of the nuScenes data frames collected by the camera, the first token of the odometer information of each of the point cloud sets, and the first token of the odometer information of a nuScenes data frame collected by the camera into the sample_data.json file; Store the nuScenes data frames corresponding to the multiple key frames respectively into the samples folder; The nuScenes data frames other than the key frame among the multiple nuScenes data frames are stored in the sweeps folder.

4. The method for converting data in rosbag format into nuScenes format according to any one of claims 1 to 3, characterized in that: The step of obtaining the correspondence between the acquisition timestamps of key frames in multiple rosbag data frames and the odometer information corresponding to the key frames includes: Acquire multiple key frames from multiple rosbag data frames whose sensor type is laser radar; For each of the key frames, based on the first rosbag data frame, the second rosbag data frame and the key frame, determine the odometer information of the key frame to obtain the odometer information corresponding to the multiple key frames respectively; the first rosbag data frame is acquired by a camera, the data in the first rosbag data frame is an image, the absolute value of the difference between the acquisition timestamp of the first rosbag data frame and the acquisition timestamp of the key frame is less than or equal to a preset duration, the second rosbag data frame is acquired by an inertial measurement unit, and the data in the second rosbag data frame is IMU posture data; the absolute value of the difference between the acquisition timestamp of the second rosbag data frame and the acquisition timestamp of the key frame is less than or equal to the preset duration; A correspondence between the acquisition timestamp of the key frame and the odometer information corresponding to the key frame is constructed.

5. The method for converting data in rosbag format into nuScenes format according to claim 4, characterized in that: The step of obtaining multiple key frames from multiple rosbag data frames whose sensor type is a laser radar comprises: Acquire multiple candidate key frames from multiple rosbag data frames whose sensor type is laser radar; Constructing a KD tree based on the plurality of candidate key frames; A plurality of key frames are obtained from the KD tree based on a preset recursive function.

6. The method for converting data in rosbag format into nuScenes format according to claim 4, characterized in that: The step of determining the odometer information of each key frame based on the first rosbag data frame, the second rosbag data frame and the key frame to obtain the odometer information corresponding to the plurality of key frames includes: For each of the key frames, obtain the first rosbag data frame and the second rosbag data frame whose absolute value of the difference between the acquisition timestamp and the acquisition timestamp of the key frame is less than or equal to the preset duration; For each of the key frames, obtaining the zero bias of the IMU pose data based on the first rosbag data frame, the second rosbag data frame and the key frame through a zero bias estimation integrator; For each of the key frames, obtaining candidate odometer information of the key frame based on the zero bias, the first rosbag data frame, the second rosbag data frame and the key frame through a mileage calculation integrator; Determine the initial pose information; Convert the candidate odometer information corresponding to each of the plurality of key frames into relative pose information based on the initial pose information; Constructing a preliminary local map based on the relative posture information respectively corresponding to the plurality of key frames; Based on the preliminary local map, obtain optimized relative pose information corresponding to a plurality of the key frames respectively; Update the preliminary local map based on the optimized relative pose information respectively corresponding to the plurality of key frames to obtain a candidate local map; Updating the candidate local map by closed-loop detection technology to obtain a target local map; The odometer information respectively corresponding to the multiple key frames is obtained from the target local map.

7. The method for converting data in rosbag format into nuScenes format according to claim 1, characterized in that: Before the step of converting the plurality of rosbag data frames into a plurality of nuScenes data frames in nuScenes format, the method further includes: Determine the target message topic of the rosbag data frame of the sensor ID to be completed; Obtaining a rosbag data frame having the target message topic from the plurality of rosbag data frames; Adjust the data frame ID of the rosbag data frame with the target message topic to the set ID.

8. The method for converting data in rosbag format into nuScenes format according to claim 1, characterized in that: Before the step of converting the plurality of rosbag data frames into a plurality of nuScenes data frames in nuScenes format, the method further includes: Determine a target rosbag data frame of the calibration data of the sensor to be completed from the multiple rosbag data frames; Fill in the target rosbag data frame with set calibration data.

9. The method for converting data in rosbag format into nuScenes format according to claim 1, characterized in that: The step of converting the plurality of rosbag data frames into a plurality of nuScenes data frames in nuScenes format comprises: Acquire multiple target rosbag data frames whose sensor type is the target sensor type from the multiple rosbag data frames; Based on the nuScenes configuration file, the rosbag fields in the multiple target rosbag data frames are converted into nuScenes fields and the field value types of the rosbag fields in the multiple target rosbag data frames are converted into the field value types of the nuScenes fields, so as to obtain the multiple nuScenes data frames; the nuScenes configuration file includes the correspondence between the preset nuScenes fields and the rosbag fields and the correspondence between the preset field value types of the nuScenes fields and the field value types of the rosbag fields; Store the multiple nuScenes data frames into the nuScenes structure.

10. A device for converting data in rosbag format into nuScenes format, characterized in that: include: The first acquisition module is used to acquire multiple rosbag data frames in the rosbag format, wherein the rosbag data frames include data, the type of sensor that collects the data, the name of the sensor that collects the data, the acquisition timestamp of the data, the calibration data of the sensor that collects the data, the ID of the sensor that collects the data, and the message topic corresponding to the data; the sensor types include camera, inertial measurement unit, lidar and millimeter wave radar; The second acquisition module is used to obtain the corresponding relationship between the acquisition timestamp of the key frame in multiple rosbag data frames and the odometer information corresponding to the key frame; A conversion module, used for converting the plurality of rosbag data frames into a plurality of nuScenes data frames in a nuScenes format; A construction module, configured to construct a correspondence between the nuScenes data frame and the odometer information based on the acquisition timestamps of the plurality of nuScenes data frames and the acquisition timestamps of the plurality of odometer information; A storage module is used to store the plurality of nuScenes data frames and the correspondence between the nuScenes data frames and the odometer information in a pre-built nuScenes database.

Images