Data analysis system, method and product based on multiple sensors

Abstract

The embodiment of the invention provides a data analysis system, method and product based on multiple sensors, and the system comprises a preprocessing module which is used for checking original data; the data loading module is used for performing topic extraction and corresponding data entry extraction on the original data passing the inspection; the analysis management module is used for initializing various analyzers according to parameters in the configuration file; the parser module is used for analyzing the corresponding data items through various parsers according to the parameters in the calibration file and the extracted data items; and the post-processing module is used for carrying out time synchronization and data correction on the analyzed and processed data, and obtaining and storing target data subjected to time synchronization and data correction. The objective of the invention is to improve the analysis efficiency, consistency and expandability of multi-sensor data.

Description

Technical Field

[0001] The present application relates to the field of data analysis technology, and in particular to a multi-sensor based data analysis system, method and product. Background Art

[0002] The development of autonomous driving technology relies on rich and accurate environmental perception capabilities. The limitations of a single sensor make multi-sensor fusion the key to achieving high-precision environmental perception. Multi-sensor systems integrate multiple types of sensors (such as cameras, LiDAR, Radar, IMU, INS, GNSS, etc.) to obtain comprehensive and redundant environmental information to make up for the shortcomings of a single sensor. However, there are many challenges in the real-time and consistency of data analysis for multi-sensor data. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a multi-sensor based data analysis system, method and product, aiming to improve the efficiency, consistency and scalability of multi-sensor data analysis.

[0004] In a first aspect, an embodiment of the present application provides a multi-sensor based data analysis system, the system comprising: a pre-processing module, a data loading module, a analysis management module, a parser module and a post-processing module;

[0005] The preprocessing module is used to check the original data;

[0006] The data loading module is used to extract topics and corresponding data items from the original data checked by the preprocessing module;

[0007] The parsing management module is used to initialize the parsers corresponding to various sensors in the parser module according to the parameters in the configuration file;

[0008] The parser module is used to parse and process the corresponding data entries through the initialized parsers corresponding to various sensors according to the parameters in the calibration file, the extracted topics and the corresponding data entries;

[0009] The post-processing module is used to perform time synchronization and data correction on the data parsed and processed by the parser module, and obtain and store the target data after time synchronization and data correction.

[0010] Optionally, the data loading module includes: a file reading module, a data parsing module, a Topic classification module and a data caching module;

[0011] The file reading module is used to open and read the original data checked by the preprocessing module;

[0012] A data parsing module, used to extract topics and corresponding data items from the original data read by the file reading module;

[0013] Topic classification module, used to classify the extracted topics and corresponding data items;

[0014] The data cache module is used to cache classified data entries.

[0015] Optionally, the system further comprises a configuration manager module; the configuration manager module comprises: a configuration file reader, a calibration file reader and a configuration checker;

[0016] The configuration file reader is used to read the system configuration file and the sensor configuration file and extract key information therein;

[0017] The calibration file reader is used to read the calibration files of various sensors;

[0018] A configuration verifier is used to perform validity verification on the key information extracted by the configuration file reader.

[0019] Optionally, the parser management module includes: a parser initialization module, a parser management module and a parser scheduling module;

[0020] The parser initialization module is used to initialize the parsers corresponding to various sensors in the parser module according to the parameters in the configuration file that has been verified for validity;

[0021] The parser management module is used to perform life cycle management, status monitoring and resource management on all initialized parsers;

[0022] The parser scheduling module is used to schedule the initialized parsers corresponding to the corresponding various sensors to perform parsing work according to data parsing requirements.

[0023] Optionally, the parser module includes: an image parser, a Lidar parser, a Radar parser, a Pose parser, an INS parser and a CAN parser;

[0024] The image analyzer is used to analyze and process the collected image data to obtain frame image data;

[0025] The Lidar parser is used to parse and process the collected point cloud data to obtain target point cloud data of coordinate system one;

[0026] The Radar parser is used to parse and process the collected echo data to extract target level information and original point cloud data;

[0027] The Pose parser is used to parse and process the extracted IMU data and GNSS data to determine the position information and posture information of the vehicle;

[0028] The INS parser is used to parse the extracted INS data to determine the vehicle's position and posture information;

[0029] The CAN analyzer is used to analyze the extracted CAN signal to determine the status information of the vehicle.

[0030] Optionally, the post-processing module includes: a time synchronization module, an image dedistortion module, a data correction module and a result output module;

[0031] The time synchronization module is used to perform time alignment on the parsed data from different sensors;

[0032] The image dedistortion module is used to perform dedistortion processing on the frame image data that has been analyzed;

[0033] The data correction module is used to perform spatial alignment processing and / or data filtering processing on the time-aligned data;

[0034] The result output module is used to format the post-processed data and send the formatted data to the corresponding back-end application module and / or store it in a database.

[0035] Optionally, the Lidar parser includes: a first interface module, a Lidar data preprocessing module and a first result output module;

[0036] The first interface module is used to call the corresponding LiDAR SDK according to the parameters in the configuration file that has been verified for validity, and read the corresponding data entries from the data loading module;

[0037] The Lidar data preprocessing module is used to preprocess the data entries read through the first interface module;

[0038] The first result output module is used to format the data items preprocessed by the Lidar data preprocessing module and output them to a subsequent processing module and / or a database.

[0039] Optionally, the Radar parser includes: a second interface module, a Radar data preprocessing module, a target level information extraction module, an original point extraction module and a second result output module;

[0040] The second interface module is used to call the corresponding Radar SDK according to the parameters in the configuration file that has been verified for validity, and read the corresponding data entries from the data loading module;

[0041] The Radar data preprocessing module is used to preprocess the data entries read through the second interface module;

[0042] The target level information extraction module is used to extract target level information from the data entries preprocessed by the Lidar data preprocessing module;

[0043] The original point extraction module is used to extract original point cloud data from the data entries preprocessed by the Lidar data preprocessing module;

[0044] The second result output module is used to format the target level information and the original point cloud data, and output them to a subsequent processing module and / or a database.

[0045] Optionally, the Pose parser includes: an IMU data parsing module, a GNSS data parsing module and a third result output module;

[0046] The IMU data analysis module is used to analyze and process the extracted IMU data to determine the posture information of the vehicle;

[0047] The GNSS data analysis module is used to analyze and process the extracted GNSS data to determine the location information of the vehicle;

[0048] The third result output module is used to format the fused posture information and position information, and output them to a subsequent processing module and / or a database.

[0049] Optionally, the INS parser comprises: an INS data parsing module, a data correction filtering module, a posture and position calculation module and a fourth result output module;

[0050] The INS data parsing module is used to parse the extracted INS data to determine first information related to the vehicle's position and posture information;

[0051] The data correction and filtering module is used to correct and filter the first information to obtain second information;

[0052] The posture and position calculation module is used to determine the posture information of the vehicle by calculating the second information;

[0053] The fourth result output module is used to format the posture information and output it to a subsequent processing module and / or a database.

[0054] Optionally, the CAN parser comprises: a CAN signal parsing module, a signal processing module and a fifth result output module;

[0055] The CAN signal analysis module is used to analyze and process the extracted CAN signals to determine the values ​​of each signal;

[0056] The signal processing module is used to determine the state information of the vehicle corresponding to each signal according to the determined value of each signal;

[0057] The fifth result output module is used to format the status information and output it to a subsequent processing module and / or a database.

[0058] A second aspect of an embodiment of the present application provides a data analysis method based on multiple sensors, the method comprising:

[0059] Perform pre-processing checks on raw data;

[0060] Extract topics and corresponding data items from raw data that has passed the preprocessing check;

[0061] Initialize the parsers corresponding to various sensors based on the parameters in the read configuration file;

[0062] Based on the parameters in the read calibration file, the extracted topics and the corresponding data items, the initialized parsers corresponding to the various sensors are used to parse and process the corresponding data items;

[0063] Time synchronization and data correction are performed on the analyzed data to obtain and store the target data after time synchronization and data correction.

[0064] A third aspect of an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and running on the processor, wherein when the computer program is executed by the processor, the steps in a multi-sensor based data analysis method as described in the second aspect of the present application are implemented.

[0065] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps in a multi-sensor based data analysis method as described in the second aspect of the present application are implemented.

[0066] The multi-sensor data analysis method provided in this application has the following advantages:

[0067] The embodiment of the present application provides a data parsing system based on multiple sensors, the system includes: a preprocessing module, a data loading module, a parsing management module, a parser module and a post-processing module; the preprocessing module is used to check the raw data; the data loading module is used to extract the topic and the corresponding data entry of the raw data checked by the preprocessing module; the parsing management module is used to initialize the parsers corresponding to various sensors in the parser module according to the parameters in the configuration file; the parser module is used to parse and process the corresponding data entries through the initialized parsers corresponding to various sensors according to the parameters in the calibration file, the extracted topic and the corresponding data entry; the post-processing module is used to perform time synchronization and data correction on the data parsed by the parser module, and obtain and store the target data after time synchronization and data correction. Thus, the present application constructs a system that can be used to parse data collected by multiple sensors at the same time, and divides the system into multiple modules, which are used to parse the raw data from different sensors and convert them into a unified data format for subsequent data fusion and environmental perception. At the same time, the system adopts a modular design to ensure the efficiency and scalability of the parsing process. With this modular concept, when some core contents need to be adjusted, it is only necessary to expand and adjust the corresponding modules. For example, when the configuration information of the system and parser needs to be adjusted, it is only necessary to expand and adjust the corresponding modules of the management configuration files and calibration files. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0069] Figure 1 A schematic diagram of a multi-sensor based data analysis system according to an embodiment of the present application;

[0070] Figure 2 Another schematic diagram of a multi-sensor based data analysis system according to an embodiment of the present application;

[0071] Figure 3 The present invention is a flowchart of a multi-sensor based data analysis method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0073] refer to Figure 1 , Figure 1 This is a schematic diagram of a multi-sensor based data analysis system according to an embodiment of the present application. Figure 1 As shown, the system 100 includes: a preprocessing module 101, a data loading module 102, a parsing management module 103, a parser module 104 and a post-processing module 105; the preprocessing module 101 is used to check the original data; the data loading module 102 is used to extract the topic and the corresponding data items of the original data checked by the preprocessing module; the parsing management module 103 is used to initialize the parsers corresponding to various sensors in the parser module according to the parameters in the configuration file; the parser module 104 is used to parse and process the corresponding data items of each sensor through the initialized parsers corresponding to various sensors according to the parameters in the calibration file, the extracted topic and the corresponding data items; the post-processing module 105 is used to perform time synchronization and data correction on the data parsed by the parser module, and obtain and store the target data after time synchronization and data correction.

[0074] In this embodiment, since various types of sensors are only responsible for collecting data, the present application sets a preprocessing module in the system, which is used to check the raw data collected by the sensor to determine whether the raw data collected by the sensor meets the requirements for subsequent data analysis. The specific inspection content at least includes whether the name of the raw data collected by various sensors is normal; whether the raw data is valid and can be read normally; whether the recording time of the raw data is normal. Only when the raw data passes all the inspection items of the preprocessing module, it is determined that the raw data inspection has passed and can be used for subsequent analysis and processing.

[0075] In this embodiment, in order to ensure that the system can more efficiently parse the raw data collected by multiple sensors, the present application sets up a loading module in the system to read the raw data collected by various sensors that have passed the inspection of the preprocessing module (the raw data is preferably stored as a dat binary file), and parse and extract the data entries corresponding to each different topic one by one according to different topics for subsequent analysis and processing.

[0076] In this embodiment, in order to improve the expansibility of the system, various data components for configuring the system form a configuration file of the system, various data components for configuring the parser for parsing the data collected by various sensors in the system form a configuration file of the parser, and calibration parameters for the parser involved in the parsing process of the parser form a calibration file of the parser. The configuration file of the system, the configuration file of the parser and the calibration file of the parser are read and managed by the corresponding modules. In this way, when it is necessary to adjust the configuration information and calibration information, it is only necessary to expand and adjust the corresponding module and the relevant management files in the corresponding module, thereby ensuring the configurability, flexibility and ease of use of the system, so that users can easily adjust parameters and configure the system.

[0077] In this embodiment, the parsing management module in the system is used to initialize the parsers corresponding to various types of sensors in the parser module based on the parameters in the configuration file, so that each parser can be used to parse the data.

[0078] In this embodiment, the parser module in the system includes a parser corresponding to each type of sensor, and each parser is used to parse and process the data collected by the corresponding sensor. Specifically, based on the parameters in the calibration file, and the topic and corresponding data entry extracted by the data loading module, the parser corresponding to the sensor type is first determined according to the sensor type corresponding to the data entry, and then the parser parses and processes the data entry based on the parameters in the calibration file to obtain the corresponding parsed data.

[0079] In this embodiment, in order to ensure the accuracy and consistency of the data after analysis by various sensors, a post-processing module set by the system performs time synchronization and data correction on the data of various sensors after analysis and processing by the parser module, thereby obtaining corresponding multi-sensor data after time synchronization and data correction, and then stores these data in a local file or a corresponding database for subsequent reasoning.

[0080] The embodiment of the present application provides a data parsing system based on multiple sensors, the system includes: a preprocessing module, a data loading module, a parsing management module, a parser module and a post-processing module; the preprocessing module is used to check the raw data; the data loading module is used to extract the topic and the corresponding data entry of the raw data checked by the preprocessing module; the parsing management module is used to initialize the parsers corresponding to various sensors in the parser module according to the parameters in the configuration file; the parser module is used to parse and process the corresponding data entries through the initialized parsers corresponding to various sensors according to the parameters in the calibration file, the extracted topic and the corresponding data entry; the post-processing module is used to perform time synchronization and data correction on the data parsed by the parser module, and obtain and store the target data after time synchronization and data correction. Thus, the present application constructs a system that can be used to parse data collected by multiple sensors at the same time, and divides the system into multiple modules, which are used to parse the raw data from different sensors and convert them into a unified data format for subsequent data fusion and environmental perception. At the same time, the system adopts a modular design to ensure the efficiency and scalability of the parsing process. With this modular concept, when some core contents need to be adjusted, it is only necessary to expand and adjust the corresponding modules. For example, when the configuration information of the system and parser needs to be adjusted, it is only necessary to expand and adjust the corresponding modules of the management configuration files and calibration files.

[0081] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a data analysis system based on multiple sensors. In the data analysis system based on multiple sensors, the data loading module includes: a file reading module, a data analysis module, a Topic classification module and a data cache module; the file reading module is used to open and read the original data checked by the preprocessing module; the data analysis module is used to extract topics and corresponding data entries from the original data read by the file reading module; the Topic classification module is used to classify the extracted topics and corresponding data entries; the data cache module is used to cache the classified data entries.

[0082] In this embodiment, if Figure 2As shown, the data loading module includes a file reading module, a data parsing module, a Topic classification module and a data caching module. The file reading module is used to open and read the original data that has passed the inspection processed by the preprocessing module. Specifically, the dat file containing the original data that has passed the inspection is opened according to the file path, and then the original data in the dat file is read in the form of data blocks, which can ensure the reading efficiency of large files and handle abnormal situations in the file reading process. For example, when the file does not exist or the reading error occurs, the data reading is terminated and the abnormal data is not used for subsequent parsing processing.

[0083] In this embodiment, for the original data in the dat file read by the file reading module, the data parsing module parses the read binary data to extract the topic and the data entry corresponding to the topic. The binary data is parsed according to the file format, the data entry in the binary data is extracted, and the topic information is extracted from the data entry, so as to extract each topic and the data entry corresponding to each topic. Based on the topic information, the sensor type to which the data entry belongs can be known, so that each data entry can be sent to the corresponding parser for parsing processing based on the topic information.

[0084] In this embodiment, for the topics and data entries corresponding to the topics extracted by the data parsing module, the Topic classification module classifies the parsed data entries by topic, that is, according to the sensor type corresponding to each topic, the topics belonging to the same sensor type and the data entries corresponding to the topic are divided into a category. When these data entries need to be parsed subsequently, the topics belonging to the same category and the data entries corresponding to the topics are sent to the parser corresponding to the category for parsing (for example, each topic in category a and the data entries corresponding to the topics are data collected by the image sensor, so each topic in category a and the data entries corresponding to the topics are distributed to the parser corresponding to the image sensor for parsing).

[0085] In this embodiment, for the data entries of each category obtained by the Topic classification module, the data cache module caches the classified data entries into the memory, and the data cache module manages the size and life cycle of the cache to prevent memory overflow. When parsing is required later, the Topic classification module obtains the data entries by category and distributes them to the corresponding parser for parsing.

[0086] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a data analysis system based on multiple sensors. In the data analysis system based on multiple sensors, the system further includes a configuration manager module; the configuration manager module includes: a configuration file reader, a calibration file reader, and a configuration verifier; the configuration file reader is used to read the system configuration file and the sensor configuration file, and extract the key information therein; the calibration file reader is used to read the calibration files of various sensors; the configuration verifier is used to verify the validity of the key information extracted by the configuration file reader.

[0087] In this embodiment, the purpose of setting up a configuration manager module in the system is to ensure the configurability, flexibility and ease of use of the system so that users can easily adjust parameters and configure the system. The configuration manager module is mainly used to read and manage configuration files and calibration files. With respect to the configuration file reader in the configuration manager module, the configuration file reader is used to read the system configuration file and the sensor configuration file, and extract key information therein. Specifically: the file format of the read configuration file is YAML, JSON format, etc. It supports reading the two configuration files from the local file system or remote server, supports reading of multiple file formats, parses the two configuration files read, and extracts key information therein, such as sensor parameters, system settings, algorithm parameters, etc. At the same time, the configuration file reader also handles abnormal situations in the process of file reading and file parsing, such as file non-existence, format error, etc., to ensure stable operation of the system. If an abnormality occurs when reading the configuration file, an immediate warning is issued.

[0088] In this embodiment, the calibration file reader in the configuration manager module is used to read the calibration files of various sensors. The calibration files record the calibration parameters of the corresponding sensors, such as camera internal parameters, external parameters, image distortion parameters, lidar internal parameters, etc. Specifically: the calibration file reader reads the calibration files of different sensors. The formats of the calibration files include YAML, JSON, etc. For the read calibration files, the calibration parameters of the sensors in the calibration files are parsed and converted into a data structure recognizable by the system.

[0089] In this embodiment, for the key information extracted by the configuration file reader, the configuration checker in the configuration manager module performs validity check on the key information extracted by the configuration file reader to determine the rationality and validity of the key information, and stops the subsequent data parsing action and issues an early warning when an abnormality exists. Specifically: the configuration checker checks whether the extracted key information is within a reasonable range, such as sampling rate, data threshold, filtering parameters, etc.; detects whether there are conflicting or repeated parameters in the configuration file to avoid problems during system operation.

[0090] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a data analysis system based on multiple sensors. In the data analysis system based on multiple sensors, the analysis management module includes: a analyzer initialization module, a analyzer management module and a analyzer scheduling module; the analyzer initialization module is used to initialize the analyzers corresponding to various sensors in the analyzer module according to the parameters in the configuration file that has been verified for validity; the analyzer management module is used to perform life cycle management, status monitoring and resource management on all initialized analyzers; the analyzer scheduling module is used to schedule the initialized analyzers corresponding to various sensors to perform analysis work according to the data analysis requirements.

[0091] In this embodiment, the purpose of setting up a parsing management module in the system is to ensure the flexibility and scalability of the system so that it can adapt to a variety of sensor types and configuration requirements. The parsing management module includes a parser initialization module, a parser management module, and a parser scheduling module. For the parser initialization module, it is used to initialize the parsers corresponding to various sensors in the parser module based on the parameters in the configuration file that have passed the validity check. Specifically: the parser initialization module selects the parser corresponding to the sensor type according to the sensor type and parameters in the configuration file, and then passes the parameters in the configuration file to the parser for initialization settings, and finally dynamically creates and initializes the parser instance to ensure that the parser corresponding to each sensor is initialized and accurate for subsequent data parsing processing.

[0092] In this embodiment, the parser management module in the parsing management module is used to perform life cycle management, status monitoring and resource management for all initialized parsers. Specifically: the initialized parser instance is registered in the parser management module to facilitate subsequent calls and scheduling; the parser management module monitors the running status of the parser in real time, detects whether the parser is working properly, and manages the system resources used by the parser to ensure reasonable resource allocation and avoid resource conflicts and waste.

[0093] In this embodiment, the parser scheduling module in the parsing management module is used to schedule the initialized parsers corresponding to the corresponding various sensors to perform parsing work according to the data parsing requirements, so as to ensure the real-time and coordination of multi-sensor data parsing. Specifically: the parser scheduling module distributes parsing tasks to the corresponding parsers according to the arrival order and timestamp of the extracted data entries, and at the same time, sets priorities for the data entries corresponding to different topics, and reasonably schedules the parsing tasks according to the priority and real-time requirements of the data entry parsing, so as to ensure the priority parsing of key data.

[0094] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a data analysis system based on multiple sensors. In the data analysis system based on multiple sensors, the analyzer module includes: an image analyzer, a Lidar analyzer, a Radar analyzer, a Pose analyzer, an INS analyzer and a CAN analyzer; the image analyzer is used to analyze and process the collected image data to obtain frame image data; the Lidar analyzer is used to analyze and process the collected point cloud data to obtain target point cloud data of coordinate system 1; the Radar analyzer is used to analyze and process the collected echo data to extract target-level information and original point cloud data; the Pose analyzer is used to analyze and process the extracted IMU data and GNSS data to determine the position information and attitude information of the vehicle; the INS analyzer is used to analyze and process the extracted INS data to determine the posture information of the vehicle; the CAN analyzer is used to analyze and process the extracted CAN signal to determine the status information of the vehicle.

[0095] In this embodiment, the parser module of the system includes at least an image parser, a Lidar parser, a Radar parser, a Pose parser, an INS parser and a CAN parser, each of which is used to parse and process the data collected by the corresponding sensor. Specifically: the image parser in the parser module of the system is used to parse and process the collected image data to obtain frame image data. In order to process the video stream, the parser uses FFmpeg to parse the video stream and extract the frame image. The Lidar parser in the parser module of the system is used to parse and process the point cloud data collected by Lidar to obtain the target point cloud data of coordinate system 1. Since different models of Lidar sensors use different SDKs for data reading and parsing, the Lidar parser is used to call the corresponding SDK according to the parameters in the configuration file, and perform preprocessing and feature extraction of the point cloud data. The Radar parser in the parser module of the system is used to parse and process the echo data collected by Radar, extract target-level information (such as the distance, speed, direction, etc. of the target) and original point cloud data, and provide environmental perception information for the autonomous driving system.

[0096] The Pose parser in the parser module of the system is used to parse and process the IMU data and GNSS data extracted by IMU (inertial measurement unit) and GNSS (global navigation satellite system) to determine the vehicle's position information and attitude information, such as calculating the vehicle's displacement, velocity, acceleration and other information. The INS parser in the parser module of the system is used to parse and process the INS data extracted by the inertial navigation system to determine the vehicle's posture information to provide high-precision navigation data for the autonomous driving system. The INS data includes acceleration, angular velocity, azimuth, position and other information. The CAN parser in the parser module of the system is used to parse and process the extracted CAN signals to determine the vehicle's status information. The CAN signal contains the status information of each part of the vehicle, such as speed, steering angle, throttle status, etc., which is crucial to the decision-making and control of the autonomous driving system. This module needs to efficiently parse binary data, extract key CAN signals and process them to provide real-time vehicle status information for the autonomous driving system.

[0097] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a data analysis system based on multiple sensors. In the data analysis system based on multiple sensors, the post-processing module includes: a time synchronization module, an image dedistortion module, a data correction module and a result output module; the time synchronization module is used to perform time alignment on the analyzed data from different sensors; the image dedistortion module is used to perform dedistortion on the analyzed frame image data; the data correction module is used to perform spatial alignment and / or data filtering on the time-aligned data; the result output module is used to format the post-processed data, and send the formatted data to the corresponding back-end application module and / or store it in a database.

[0098] In this embodiment, the post-processing module of the system is used to perform time synchronization and data correction on multi-sensor data, including time synchronization of LiDAR and camera data, image dedistortion and other operations. This module is designed to ensure the consistency of data from different sensors in time and space, and provide high-quality input for subsequent data fusion and analysis. The post-processing module includes a time synchronization module, an image dedistortion module, a data correction module and a result output module. The time synchronization module in the post-processing module is used to time align the parsed data from different sensors to ensure that they are processed on the same time reference. Specifically: the time synchronization module extracts timestamps from the parsed data corresponding to the LiDAR and the camera, and then uses interpolation, extrapolation and other methods to time align the various parsed data. After the time alignment, the time synchronization error between different sensors is corrected.

[0099] In this embodiment, the image dedistortion module in the post-processing module is used to perform dedistortion processing on the frame image data that has been parsed and processed, and correct the image distortion caused by lens distortion. Specifically: the image dedistortion module reads the distortion parameters of the camera from the configuration file or calibration file read by the configuration manager module, and uses the image processing algorithm based on the distortion parameters of the image area to perform dedistortion correction on the frame image data that has been parsed and processed by the parser.

[0100] In this embodiment, the data correction module in the post-processing module is used to perform spatial alignment processing and / or data filtering processing on the time-aligned data. Specifically: the data correction module reads the calibration parameters in the calibration file from the configuration manager module to perform spatial alignment on the Lidar point cloud data and the camera image data after the parser parsing process, and the data correction module filters the Lidar point cloud data after the parsing process by the parser to remove noise points and improve data quality.

[0101] In this embodiment, the result output module in the post-processing module is used to format the post-processed data, and send the formatted data to the corresponding back-end application module and / or store it in the database, where the standard formats include but are not limited to JSON and CSV formats.

[0102] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a data analysis system based on multiple sensors. In the data analysis system based on multiple sensors, the Lidar analyzer includes: a first interface module, a Lidar data preprocessing module and a first result output module; the first interface module is used to call the corresponding LiDAR SDK according to the parameters in the configuration file that has been verified for validity, and read the corresponding data entries from the data loading module; the Lidar data preprocessing module is used to preprocess the data entries read through the first interface module; the first result output module is used to format the data entries preprocessed by the Lidar data preprocessing module, and output them to the subsequent processing module and / or database.

[0103] In this embodiment, the Lidar parser includes a first interface module, a Lidar data preprocessing module and a first result output module. The first interface module is used to call the corresponding LiDAR SDK according to the parameters in the configuration file that has been verified for validity, and read the corresponding data entry from the data loading module. Specifically: the first interface module selects the corresponding SDK according to the LiDAR model in the configuration file, initializes the SDK using the parameters in the configuration file, obtains the data entry corresponding to the LiDAR model from the data loading module, and reads the corresponding data entry from the data loading module and passes it to the Lidar data preprocessing module for processing.

[0104] In this embodiment, the Lidar data preprocessing module is used to preprocess the data entries read through the first interface module. Specifically: the data entries read through the first interface module are filtered out invalid points, such as filtering out nan or (0, 0, 0) invalid points, to improve data quality, reduce point cloud storage space, and convert the data entries (that is, point cloud data) into a unified coordinate system for downstream use. At the same time, according to algorithm requirements, the data entries are converted into the required data format.

[0105] In this embodiment, the first result output module is used to format the data entries preprocessed by the Lidar data preprocessing module and output them to the subsequent processing module and / or database for downstream use.

[0106] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a data analysis system based on multiple sensors. In the data analysis system based on multiple sensors, the Radar analyzer includes: a second interface module, a Radar data preprocessing module, a target level information extraction module, an original point extraction module and a second result output module; the second interface module is used to call the corresponding RadarSDK according to the parameters in the configuration file that has been verified for validity, and read the corresponding data entry from the data loading module; the Radar data preprocessing module is used to preprocess the data entry read through the second interface module; the target level information extraction module is used to extract the target level information from the data entry preprocessed by the Lidar data preprocessing module; the original point extraction module is used to extract the original point cloud data from the data entry preprocessed by the Lidar data preprocessing module; the second result output module is used to format the target level information and the original point cloud data, and output them to the subsequent processing module and / or database.

[0107] In this embodiment, the Radar parser includes a second interface module, a Radar data preprocessing module, a target level information extraction module, an original point extraction module, and a second result output module. The second interface module is used to call the corresponding Radar SDK according to the parameters in the configuration file that has been verified for validity, and read the corresponding data entry from the data loading module. Specifically: the second interface module selects the corresponding SDK according to the radar model in the configuration file, initializes the SDK using the parameters in the configuration file, obtains the data entry corresponding to the radar model from the data loading module, and reads the corresponding data entry from the data loading module and passes it to the Radar data preprocessing module for processing.

[0108] In this embodiment, the Radar data preprocessing module is used to preprocess the data entries read through the second interface module. Specifically: invalid points are filtered out from the data entries read through the second interface module to improve data quality, reduce point cloud storage space, and convert the data entries into the required data format according to algorithm requirements.

[0109] In this embodiment, the target level information extraction module is used to extract target level information from the data entry preprocessed by the Lidar data preprocessing module. Specifically: an algorithm (such as CFAR, DBSCAN) is used to detect the target in the data entry corresponding to the radar echo, and then estimate the target's distance, speed, direction and other parameters.

[0110] In this embodiment, the original point extraction module is used to extract the original point cloud data from the data entry preprocessed by the Lidar data preprocessing module to provide more fine-grained environmental information. Specifically: the data entry corresponding to the radar echo is converted into three-dimensional point cloud data, and then the converted three-dimensional point cloud data is converted into a unified coordinate system to obtain the original point cloud data for downstream use.

[0111] In this embodiment, the second result output module is used to format the target level information and the original point cloud data, and output them to the subsequent processing module and / or database for downstream use.

[0112] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a data analysis system based on multiple sensors. In the data analysis system based on multiple sensors, the Pose analyzer includes: an IMU data analysis module, a GNSS data analysis module and a third result output module; the IMU data analysis module is used to analyze and process the extracted IMU data to determine the posture information of the vehicle; the GNSS data analysis module is used to analyze and process the extracted GNSS data to determine the location information of the vehicle; the third result output module is used to format the fused posture information and location information, and output them to the subsequent processing module and / or database.

[0113] In this embodiment, the Pose parser includes an IMU data parsing module, a GNSS data parsing module and a third result output module. The IMU data parsing module is used to parse and process the extracted IMU data to determine the posture information of the vehicle. Specifically: parse the IMU data to obtain the acceleration information therein, and parse the IMU data to obtain the angular velocity information therein, and calculate the posture information of the vehicle based on the determined acceleration and angular velocity data.

[0114] In this embodiment, the GNSS data parsing module is used to parse the extracted GNSS data to determine the location information of the vehicle. Specifically, the GNSS data is parsed to obtain the longitude and latitude information therein, and the GNSS data is parsed to obtain the altitude information therein, and the location information of the vehicle is calculated based on the determined longitude and latitude and altitude data.

[0115] In this embodiment, the third result output module is used to format the fused posture information and position information and output them to the subsequent processing module and / or database. Specifically: the obtained position information and posture information are fused and then formatted, and then the formatted data is output to the subsequent processing module and / or database for downstream use.

[0116] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a data analysis system based on multiple sensors. In the data analysis system based on multiple sensors, the INS analyzer includes: an INS data analysis module, a data correction and filtering module, a posture and position calculation module, and a fourth result output module; the INS data analysis module is used to analyze and process the extracted INS data to determine the first information related to the vehicle's posture information; the data correction and filtering module is used to correct and filter the first information to obtain the second information; the posture and position calculation module is used to determine the vehicle's posture information by calculating the second information; the fourth result output module is used to format the posture information and output it to a subsequent processing module and / or a database.

[0117] In this embodiment, the INS parser includes an INS data parsing module, a data correction filtering module, a posture and position calculation module and a fourth result output module. The INS data parsing module is used to parse and process the extracted INS data to determine the first information related to the vehicle's posture information. Specifically: parse the INS data to obtain the acceleration information therein, parse the INS data to obtain the angular velocity information therein, parse the INS data to obtain the azimuth information therein, and parse the INS data to obtain the position information therein, all of which are the first information related to the vehicle's posture information.

[0118] In this embodiment, the data correction and filtering module is used to correct and filter the obtained first information to obtain the second information. Specifically: use a filtering algorithm (such as Kalman filtering, mean filtering) to perform noise correction on the INS data, correct sensor errors such as zero bias error, scale factor error, etc., and smooth the data to reduce data jitter. After the first information is corrected and filtered, the corresponding second information is obtained.

[0119] In this embodiment, the attitude and position calculation module is used to determine the vehicle's posture information by calculating the obtained second information. Specifically: the vehicle's attitude information (such as roll angle, pitch angle, yaw angle) is calculated based on the acceleration and angular velocity data after correction and filtering, and the vehicle's position information (such as latitude and longitude, altitude) is calculated based on the second information after correction and filtering.

[0120] In this embodiment, the fourth result output module is used to format the posture information and output it to the subsequent processing module and / or database. Specifically: the obtained posture information is formatted, and then the formatted data is output to the subsequent processing module and / or database for downstream use.

[0121] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a data analysis system based on multiple sensors. In the data analysis system based on multiple sensors, the CAN analyzer includes: a CAN signal analysis module, a signal processing module and a fifth result output module; the CAN signal analysis module is used to analyze and process the extracted CAN signals to determine the values ​​of each signal; the signal processing module is used to determine the status information of the vehicle corresponding to each signal according to the determined values ​​of each signal; the fifth result output module is used to format the status information and output it to the subsequent processing module and / or database.

[0122] In this embodiment, the CAN parser includes a CAN signal parsing module, a signal processing module and a fifth result output module. The CAN signal parsing module is used to parse and process the extracted CAN signals to determine the values ​​of each signal. Specifically: parse the CAN ID, data length, data field and other information in the CAN data frame, and then extract the specific signal value according to the CAN ID and data length.

[0123] In this embodiment, the signal processing module is used to determine the vehicle status information corresponding to each signal according to the determined value of each signal. Specifically: convert the original data field into a specific signal value, such as speed, steering angle, etc., and convert the signal value into an actual unit, such as converting from the original value to km / h or degree.

[0124] In this embodiment, the fifth result output module is used to format the state information obtained by the signal processing module and output it to the subsequent processing module and / or database. Specifically: the state information obtained by the signal processing module is formatted, and then the formatted data is output to the subsequent processing module and / or database for downstream use.

[0125] Based on the same inventive concept, an embodiment of the present application provides a data analysis method based on multiple sensors, such as Figure 3 As shown, the method includes:

[0126] Perform pre-processing checks on raw data;

[0127] Extract topics and corresponding data items from raw data that has passed the preprocessing check;

[0128] Initialize the parsers corresponding to various sensors based on the parameters in the read configuration file;

[0129] Based on the parameters in the read calibration file, the extracted topics and the corresponding data items, the initialized parsers corresponding to the various sensors are used to parse and process the corresponding data items;

[0130] Time synchronization and data correction are performed on the analyzed data to obtain and store the target data after time synchronization and data correction.

[0131] Based on the same inventive concept, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and running on the processor. When the computer program is executed by the processor, it implements the steps in a multi-sensor based data analysis method as described in the first aspect of the present application.

[0132] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in a multi-sensor based data analysis method as described in the first aspect of the present application are implemented.

[0133] As for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0134] It should be noted that, for the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0135] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0136] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0137] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0140] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0141] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0142] The above is a detailed introduction to a multi-sensor based data analysis system, method and product provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multi-sensor based data analysis system, characterized in that: The system includes: a pre-processing module, a data loading module, a parsing management module, a parser module and a post-processing module; The preprocessing module is used to check the original data; The data loading module is used to extract topics and corresponding data items from the original data checked by the preprocessing module; The parsing management module is used to initialize the parsers corresponding to various sensors in the parser module according to the parameters in the configuration file; The parser module is used to parse and process the corresponding data entries through the initialized parsers corresponding to various sensors according to the parameters in the calibration file, the extracted topics and the corresponding data entries; The post-processing module is used to perform time synchronization and data correction on the data parsed and processed by the parser module, and obtain and store the target data after time synchronization and data correction.

2. A multi-sensor based data analysis system according to claim 1, characterized in that: The data loading module includes: a file reading module, a data parsing module, a Topic classification module and a data caching module; The file reading module is used to open and read the original data checked by the preprocessing module; A data parsing module, used to extract topics and corresponding data items from the original data read by the file reading module; Topic classification module, used to classify the extracted topics and corresponding data items; The data cache module is used to cache classified data entries.

3. The multi-sensor based data analysis system according to claim 1, characterized in that: The system further comprises a configuration manager module; the configuration manager module comprises: a configuration file reader, a calibration file reader and a configuration verifier; The configuration file reader is used to read the system configuration file and the sensor configuration file and extract key information therein; The calibration file reader is used to read the calibration files of various sensors; A configuration verifier is used to perform validity verification on the key information extracted by the configuration file reader.

4. The multi-sensor based data analysis system according to claim 1, characterized in that: The parser management module includes: a parser initialization module, a parser management module and a parser scheduling module; The parser initialization module is used to initialize the parsers corresponding to various sensors in the parser module according to the parameters in the configuration file that has been verified for validity; The parser management module is used to perform life cycle management, status monitoring and resource management on all initialized parsers; The parser scheduling module is used to schedule the initialized parsers corresponding to the corresponding various sensors to perform parsing work according to data parsing requirements.

5. The multi-sensor based data analysis system according to claim 1, characterized in that: The parser module includes: an image parser, a Lidar parser, a Radar parser, a Pose parser, an INS parser and a CAN parser; The image analyzer is used to analyze and process the collected image data to obtain frame image data; The Lidar parser is used to parse and process the collected point cloud data to obtain target point cloud data of coordinate system one; The Radar parser is used to parse and process the collected echo data to extract target level information and original point cloud data; The Pose parser is used to parse and process the extracted IMU data and GNSS data to determine the position information and posture information of the vehicle; The INS parser is used to parse the extracted INS data to determine the vehicle's position and posture information; The CAN analyzer is used to analyze the extracted CAN signal to determine the status information of the vehicle.

6. The multi-sensor based data analysis system according to claim 1, characterized in that: The post-processing module includes: a time synchronization module, an image dedistortion module, a data correction module and a result output module; The time synchronization module is used to perform time alignment on the parsed data from different sensors; The image dedistortion module is used to perform dedistortion processing on the frame image data that has been analyzed; The data correction module is used to perform spatial alignment processing and / or data filtering processing on the time-aligned data; The result output module is used to format the post-processed data and send the formatted data to the corresponding back-end application module and / or store it in a database.

7. The multi-sensor based data analysis system according to claim 5, characterized in that: The Lidar parser includes: a first interface module, a Lidar data preprocessing module and a first result output module; The first interface module is used to call the corresponding LiDARSDK according to the parameters in the configuration file that has been verified for validity, and read the corresponding data entries from the data loading module; The Lidar data preprocessing module is used to preprocess the data entries read through the first interface module; The first result output module is used to format the data items preprocessed by the Lidar data preprocessing module and output them to a subsequent processing module and / or a database.

8. The multi-sensor based data analysis system according to claim 5, characterized in that: The Radar parser includes: a second interface module, a Radar data preprocessing module, a target level information extraction module, an original point extraction module, and a second result output module; The second interface module is used to call the corresponding RadarSDK according to the parameters in the configuration file that has been verified for validity, and read the corresponding data entries from the data loading module; The Radar data preprocessing module is used to preprocess the data entries read through the second interface module; The target level information extraction module is used to extract target level information from the data entries preprocessed by the Lidar data preprocessing module; The original point extraction module is used to extract original point cloud data from the data entries preprocessed by the Lidar data preprocessing module; The second result output module is used to format the target level information and the original point cloud data, and output them to a subsequent processing module and / or a database.

9. The multi-sensor based data analysis system according to claim 5, characterized in that: The Pose parser includes: an IMU data parsing module, a GNSS data parsing module and a third result output module; The IMU data analysis module is used to analyze and process the extracted IMU data to determine the posture information of the vehicle; The GNSS data analysis module is used to analyze and process the extracted GNSS data to determine the location information of the vehicle; The third result output module is used to format the fused posture information and position information, and output them to a subsequent processing module and / or a database.

10. The multi-sensor based data analysis system according to claim 5, characterized in that: The INS parser comprises: an INS data parsing module, a data correction filtering module, a posture and position calculation module and a fourth result output module; The INS data parsing module is used to parse and process the extracted INS data to determine first information related to the position and posture information of the vehicle; The data correction and filtering module is used to correct and filter the first information to obtain second information; The posture and position calculation module is used to determine the posture information of the vehicle by calculating the second information; The fourth result output module is used to format the posture information and output it to a subsequent processing module and / or a database.

11. The multi-sensor based data analysis system according to claim 5, characterized in that: The CAN parser comprises: a CAN signal parsing module, a signal processing module and a fifth result output module; The CAN signal analysis module is used to analyze and process the extracted CAN signals to determine the values ​​of each signal; The signal processing module is used to determine the state information of the vehicle corresponding to each signal according to the determined value of each signal; The fifth result output module is used to format the status information and output it to a subsequent processing module and / or a database.

12. A data analysis method based on multiple sensors, characterized in that: The method comprises: Perform pre-processing checks on raw data; Extract topics and corresponding data items from raw data that has passed the preprocessing check; Initialize the parsers corresponding to various sensors based on the parameters in the read configuration file; Based on the parameters in the read calibration file, the extracted topic and the corresponding data items, the initialized parsers corresponding to each type of sensor are used to parse and process the corresponding data items; Time synchronization and data correction are performed on the analyzed data to obtain and store the target data after time synchronization and data correction.

13. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and running on the processor, wherein when the computer program is executed by the processor, the steps in the multi-sensor based data analysis method as claimed in claim 12 are implemented.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the multi-sensor based data analysis method according to claim 12 are implemented.

Images