Offline vehicle data preprocessing system and method

Abstract

The invention discloses an offline vehicle data preprocessing system and method, and belongs to the technical field of vehicle data processing. The system comprises a data analysis module and a data preprocessing module. The data analysis module can identify and load a protocol database file, analyzes an original CAN data file frame by frame based on the file, obtains structured data, and standardizes a data format. And the data preprocessing module preprocesses the analyzed data based on the configuration file. The method comprises the steps of obtaining an original data file and a protocol database file; analyzing the original data to a signal layer based on the protocol database file; executing a preprocessing operation based on the configuration file; and outputting the preprocessed data file. Through combination of protocol analysis and configurable preprocessing, the problems that a traditional method is low in processing efficiency, poor in data quality, asynchronous in multi-source signals and lack of special tools are solved, the data size is remarkably reduced, the data quality is improved, and the method can flexibly adapt to different vehicle types and data analysis requirements.

Description

Technical Field

[0001] This invention belongs to the field of vehicle data processing technology, specifically relating to an offline vehicle data preprocessing system and method. Background Technology

[0002] With the development of intelligent commercial vehicles, the amount of data generated by vehicles every day has increased dramatically, reaching tens of gigabytes (e.g., in ASCII format). This data contains a wealth of vehicle status information, which is crucial for analyzing key performance indicators such as energy efficiency, power performance, and vehicle health.

[0003] However, traditional data processing methods, such as those described in Chinese Patent CN119938763A (Vehicle Measurement Data Conversion and Processing Method, System, Equipment, and Medium), using Matlab models, exhibit poor performance and high PC memory consumption when processing such large-scale data, leading to excessively long processing times and even software crashes, severely impacting data analysis efficiency. Furthermore, the data often contains significant noise, outliers, and heterogeneity from multiple sources, further increasing the difficulty of data analysis.

[0004] Therefore, based on the above problems, there is a need to develop an efficient data preprocessing, cleaning, and synchronization system. Summary of the Invention

[0005] The purpose of this invention is to propose an offline vehicle data preprocessing system and method to solve the problems in the prior art.

[0006] Therefore, the present invention provides an offline vehicle data preprocessing system, comprising: A data parsing module is configured to recognize the loaded protocol database file. The data parsing module parses the original data file frame by frame based on the protocol database file to obtain multiple first data, and standardizes the format of the parsed first data. The data preprocessing module preprocesses the parsed first data based on the configuration file, wherein the preprocessing includes at least message filtering, signal filtering, time filtering and scene segmentation.

[0007] In some embodiments, an interactive interface module is also included, which provides a graphical interface, receives input and configuration files, and displays processing progress and results.

[0008] In some embodiments, the data parsing module is based on functions from the Python function library.

[0009] In some embodiments, the first data includes at least a message ID, a timestamp, a length, and a data value.

[0010] In some embodiments, the data preprocessing module includes a message filtering module, which is used to filter the first data based on a message ID list in a configuration file, so as to retain specified message ID data and write the message ID data into a filtered data file.

[0011] In some embodiments, the data preprocessing module includes a signal filtering module, which parses the first data to the signal layer based on the protocol database file, filters signal data that meets predetermined conditions, cleans invalid values, and writes the filtered signal data into a text file or a table file.

[0012] In some embodiments, the invalid values ​​include at least values ​​that are outside a predetermined range and null values.

[0013] In some embodiments, the data preprocessing module includes a time filtering module, which filters the first data based on preset time conditions to retain data that meets the preset time conditions.

[0014] In some embodiments, the data preprocessing module includes a scene segmentation module. The scene segmentation module parses the first data to the signal layer based on a preset signal, obtains the data time range of multiple scenes, and segments the original file configuration file into data corresponding to multiple scenes and saves them separately.

[0015] On the other hand, an offline vehicle data preprocessing method is also provided, including: Obtain the vehicle's raw data file and the corresponding protocol database file; Based on the protocol database file, the original data file is parsed frame by frame to the signal layer to obtain multiple first data, and the format of the parsed first data is standardized. Based on the configuration file, at least one preprocessing operation is performed on the parsed first data. The preprocessing operation includes at least message filtering, signal filtering, time filtering, scene segmentation, invalid value cleaning, and data synchronization. Output the preprocessed data file.

[0016] Beneficial effects: 1. This invention uses a configurable signal and message filtering function to quickly remove redundant and non-target messages from the original data, solving the problems of excessively long analysis time, high memory consumption, and even software crashes caused by the large amount of data in traditional methods. It achieves the effect of reducing the amount of data by up to 92% and significantly improving the efficiency of subsequent data analysis.

[0017] 2. This invention actively identifies and cleans abnormal out-of-limit values, invalid values, and consecutive identical values ​​in signals through a protocol database file, solving the problem of data quality defects caused by sensor failures, bus interference, and other factors, which in turn cause deviations in analysis results, thereby improving the accuracy and reliability of data.

[0018] 3. This invention uses a multi-strategy time alignment algorithm to synchronize signals with inconsistent periods, solving the problem of signal time dimension misalignment and difficulty in synchronization analysis caused by inconsistent sampling periods of different controllers, thus facilitating subsequent deep calculations such as time integration.

[0019] 4. This invention solves the problem of difficulty in extracting data and the need for manual intervention in specific scenarios, such as driving gear and charging status, by analyzing specific signals such as gear position and charging status. It achieves the effect of automatically splitting the original data into multiple specific scenario data as input for subsequent accurate analysis.

[0020] 5. This invention solves the problem of strong coupling between existing data processing tools and vehicle models, and the need to reconstruct parsing logic when adding new vehicle models or changing signals, by decoupling the whitelist rules of the message / signal configuration file from the DBC database file. This achieves the effect of flexibly matching different platform vehicle models and data analysis needs, and reducing the cost of repeated development.

[0021] 6. This invention can effectively solve the problems of low processing efficiency, uncontrollable data quality, difficulty in scene extraction, and poor tool scalability of traditional methods, significantly improve the efficiency and accuracy of subsequent analysis, and can flexibly adapt to different vehicle models and data analysis needs. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is an architecture diagram of the preprocessing system provided by the present invention.

[0024] Figure 2 This is an architecture diagram of the preprocessing module provided by the present invention.

[0025] Figure 3 This is a schematic diagram of the preprocessing method provided by the present invention.

[0026] Figure 4 The code architecture diagram of the preprocessing system provided by this invention.

[0027] Figure 5 This is a schematic diagram of the interactive interface module provided by the present invention.

[0028] Figure 6 The flowchart for the preprocessing provided by this invention.

[0029] In the diagram: 100, Data parsing module; 200, Data preprocessing module; 210, Message filtering module; 220, Signal filtering module; 230, Time filtering module; 240, Scene segmentation module; 250, Invalid value cleaning module; 260, Data synchronization module; 300, Interactive interface module. Detailed Implementation

[0030] The invention will be more readily understood by referring to the following detailed description of preferred embodiments and included examples. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. In case of conflict, the definitions in this specification shall prevail.

[0031] This invention provides an offline vehicle data preprocessing system and method, solving the problem that existing technologies, due to the large number of non-target message signals generated by commercial vehicles, cannot quickly remove redundant data using traditional manual screening or fixed-rule filtering, resulting in long analysis times, high memory consumption, and even software crashes, failing to meet the needs of real-time or near-real-time data analysis. It also addresses the issue that raw data contains a large amount of noise, outliers, and invalid data due to scenarios such as sensor failures, initial power-on / off system self-tests, and bus interference. If this data is used directly for analysis without cleaning, it will seriously affect the accuracy and reliability of the analysis results. Furthermore, it solves the problem that inconsistent sampling periods between different system controllers, as well as inconsistent sampling information message periods within the same system, can lead to signal misalignment in the time dimension, increasing the complexity and difficulty of data analysis requiring time integration. In addition, it addresses the issue of data analysis involving specific vehicle operation scenarios, such as vehicle in driving and non-driving gears, and vehicle charging and non-charging states. Finally, it solves the problem of the lack of efficient preprocessing and cleaning tools specifically for offline operating data of commercial vehicles, which prevents automated and batch processing of data, increasing manual intervention and costs. At the same time, some current data processing tools are strongly coupled with vehicle models, requiring reconstruction of the parsing logic when adding new vehicle models or changing signals.

[0032] The technical structure of this invention lies in parsing raw CAN data through a protocol database file and introducing configurable preprocessing rules to perform message filtering, signal filtering, invalid value cleaning, time synchronization, and scene segmentation on massive amounts of vehicle data, thereby improving data analysis efficiency and accuracy, supporting platform expansion, and reducing development costs.

[0033] like Figure 1-2 As shown, an offline vehicle data preprocessing system and method includes: The data parsing module 100 is configured to recognize the loaded protocol database file, i.e., the DBC file, a database file used to define the message and signal formats in the CAN network. This file defines the message, signal, and other information of the vehicle's internal network communication. The data parsing module 100 parses the original data file, i.e., the original file recording CAN bus communication, frame by frame based on the protocol database file, such as an ASCII or LOG format file. Based on the definitions in the DBC file, the data parsing module 100 can accurately cut off the loaded original CAN data file to obtain multiple first data, i.e., structured data containing timestamps, message IDs, signal physical values, etc. after parsing, and standardize the format of the parsed first data. Since the first data is in a standardized data format, it is easier for subsequent data modules to call it.

[0034] The data preprocessing module 200 is a key component of the entire system architecture. Based on a configuration file, i.e. a user-defined file containing preprocessing rules such as a message ID whitelist, a signal name list, and a time interval, the data preprocessing module 200 preprocesses the parsed first data. The preprocessing includes at least message filtering, signal filtering, time filtering, and scene segmentation. In some embodiments, it also includes invalid value identification and cleaning, as well as data synchronization.

[0035] The above technical solution enables automated and platform-based preprocessing of massive amounts of raw vehicle data. The data parsing module 100 uses DBC files to accurately translate the raw, unstructured hexadecimal data stream into meaningful signal values, solving the data heterogeneity problem and providing a standardized data foundation for the subsequent data preprocessing module 200. The data preprocessing module 200 then performs precise processing on the standardized data according to the configuration file. Specifically, message filtering directly reduces the data size by retaining key message IDs. Signal filtering further filters target signals and cleans up invalid values ​​at the physical level, improving data quality. Time filtering focuses on specific analysis periods; scene segmentation intelligently divides the data stream into different scene segments based on operating condition signals such as gear position.

[0036] Therefore, it can effectively solve the problems of low processing efficiency, uncontrollable data quality, difficulty in scene extraction, and poor tool scalability of traditional methods, achieving a reduction of more than 90% in data volume, significantly improving the efficiency and accuracy of subsequent analysis, and flexibly adapting to different vehicle models and data analysis needs.

[0037] In some embodiments, such as Figure 5As shown, it also includes an interactive interface module 300, which provides a graphical interface, receives input and configuration files, and displays processing progress and results. Progress can be dynamically displayed via a progress bar. Results can be recorded in text form in the execution log window, including operation records, running status, and final completion information. Specifically, the user interface provides an intuitive operation and display interface, such as providing a file / path input button, allowing users to select the output location of the original .asc file, DBC file, and processed file. It also provides data preprocessing function buttons and format / time selection boxes, facilitating user selection of specific processing functions and parameters. An execution progress bar is displayed as a reference for the processing status. An execution log window is also displayed to record operation records and running logs, facilitating problem tracking and debugging.

[0038] Specifically, the interactive interface module 300 includes a main interface, which primarily comprises a data preprocessing interface and a reserved data analysis tab interface. The data preprocessing interface mainly consists of three parts: file input controls, function execution and configuration selection controls, and execution progress, status, and result display. The file input control allows users to select input ASCII data, a DBC database, a filter configuration file, and the preprocessed output data path. The function execution controls include signal and message filtering, time filtering, and scene segmentation, etc., executing corresponding data cleaning functions. The configuration selection control allows users to define the format of the filtered data output. The execution progress is used to display the data cleaning progress, and the execution log is used to display file selection operation records, key information records during the execution process, and execution completion result information records.

[0039] The above technical solution encapsulates the complex underlying data processing flow into intuitive visual operations through graphical controls. It receives various resources specified by the user through file / path input buttons, and converts the user's preprocessing intentions into system-recognizable instructions through function execution buttons and selection boxes, which are then passed to the data parsing module 100 and the preprocessing module.

[0040] In some embodiments, the data parsing module 100 is based on functions from a Python function library, specifically the `cantools.database` function. Using the `load_file()` method provided by the `cantools.database` library, it accurately reads and parses the DBC file describing the vehicle communication protocol, establishing an internal mapping relationship between message and signal definitions. Subsequently, using methods such as `decode_message()` from this library, it parses the original ASCII data file frame by frame, converting the hexadecimal data of each frame into a dictionary or data frame containing specific signal physical values ​​according to the DBC definition.

[0041] By calling highly specialized existing library functions, the need to develop parsing algorithms from scratch was avoided, significantly reducing code development effort, improving module reliability and parsing accuracy, and solving the problem of errors that easily occur with manual parsing. At the same time, the library functions can automatically handle various complex signal definitions, ensuring the standardization and uniformity of the conversion process from raw data to engineering physical values, providing a standard, high-quality data input foundation for the subsequent data preprocessing module 200.

[0042] In some embodiments, the data preprocessing module 200 includes a message filtering module 210. The message filtering module 210 is used to filter the first data, i.e., the standardized structured data obtained after processing by the data parsing module 100, based on the message ID list in the configuration file, i.e., the whitelist of specific message identifiers that need to be retained as predefined by the user. The message filtering module 210 retains the specified message ID data and writes the message ID data into the filtered data file.

[0043] Specifically, the system reads the user-preset list of message IDs in the configuration file and uses it as the basis for filtering. Then, the module scans the parsed standardized data line by line, compares the message IDs in each line of data with the whitelist, retains only the message data frames whose IDs exist in the whitelist, and reassembles them in the original format (such as ASC format) and writes them into a new data file.

[0044] By employing a whitelist-based filtering mechanism, redundant packets irrelevant to the analysis target (such as entertainment system and diagnostic frames) can be quickly and accurately removed at the initial stage of data processing, significantly reducing the volume of data to be processed at the source. This directly solves the problems of excessive memory consumption, slow processing speed, and even software crashes caused by massive amounts of full-packet data, laying an efficient data foundation for subsequent fine-grained signal layer processing and analysis tasks, and significantly improving the efficiency and stability of the overall data analysis process.

[0045] In some embodiments, the data preprocessing module 200 includes a signal filtering module 220. The signal filtering module 220 parses the first data, i.e., the standardized structured data obtained after processing by the data parsing module 100, to the signal layer, i.e., signal values ​​with actual physical meaning, such as vehicle speed and engine speed, based on the protocol database file, i.e., the DBC file. The filtering meets the predetermined conditions, which usually refers to signal data located in the signal whitelist defined in the configuration file, and cleans invalid values, specifically including values ​​that exceed the predetermined range, null values, and default values ​​of the system's initial self-test, etc., and writes the filtered signal data into a text file or table file, such as CSV format.

[0046] Based on the initial parsing completed by the data parsing module 100, this module accurately filters out the target signal from all parsed signals according to the predefined signal name whitelist in the configuration file. At the same time, combined with the signal validity range defined in the protocol database or the built-in invalid value judgment logic (such as identifying consecutive identical values), the filtered signal data is cleaned to remove abnormal and invalid data points; finally, the cleaned high-quality signal data is output in a text or table format that is easier for statistical analysis.

[0047] This technical solution effectively solves the data quality defects caused by noise, interference, and system default values ​​in the original data. This not only greatly improves the accuracy and reliability of subsequent data analysis results, but also, by outputting structured tabular files (such as CSV), allows the processed data to be directly read by various data analysis tools such as MATLAB and Excel, greatly enhancing the data's versatility and the convenience of subsequent analysis, and realizing the direct conversion from raw data to high-quality analytical data.

[0048] In some embodiments, the data preprocessing module 200 includes a time filtering module 230, which filters the first data based on preset time conditions to retain data that meets the preset time conditions.

[0049] Specifically, the system reads the user's preset time interval conditions and uses them as the time window for data filtering. Then, the module scans the parsed data line by line, compares the timestamp in each line of data with the preset time window, and retains only the data lines whose timestamps fall within the specified time interval based on the conditions, thereby achieving accurate data extraction based on the time dimension.

[0050] In short, by precisely positioning the time window, effective data for a specific time period can be quickly extracted from massive amounts of 24 / 7 vehicle operation data, solving the problems of unclear analysis objectives and low analysis efficiency caused by interference from data throughout the day. This allows users to focus on data analysis during specific operating conditions or event periods, greatly improving the targeting and efficiency of subsequent data analysis, and providing strong data support for application scenarios such as vehicle operation status statistics and specific event analysis.

[0051] In some embodiments, the data preprocessing module 200 includes a scene segmentation module 240. The scene segmentation module 240 parses the first data to the signal layer based on a preset signal, which may be a vehicle gear signal or a charging status signal, etc., to obtain the data time range of multiple scenarios, and segments the original file configuration file into data corresponding to multiple scenarios and saves them separately.

[0052] The system first scans for changes in key signals in the data parsed down to the signal layer based on preset scene segmentation rules (such as identifying driving gears D / R / M and non-driving gears P / N through gear signal values). When a signal value reaches a preset scene switching threshold, the corresponding time point is recorded as the scene boundary. Finally, based on the identified multiple scene time ranges, the original continuous data stream is divided into multiple independent data files, each corresponding to a specific operating scene. By automatically identifying and segmenting different vehicle operating scenes through signal features, the system solves the problems of inefficiency and error-proneness associated with traditional methods that require manual identification and segmentation of scene data.

[0053] In some embodiments, the system further includes an invalid value identification and cleaning module and a data synchronization module 260. The invalid value identification and cleaning module establishes an invalid value criterion library by loading the valid signal value range defined in the protocol database or user-defined threshold rules. It then uses a rule engine to perform logical judgments on a signal-by-signal and data-by-data-point basis, marking and removing data points that meet the invalid value characteristics. Specifically, this includes: comparing signal values ​​with reasonable physical ranges, detecting consecutive identical values ​​exceeding a set number of times, and identifying specific default value patterns during the system initialization phase.

[0054] The data synchronization module 260 establishes a unified time axis and then, for each non-target sampling moment, calculates the estimated signal value at that moment using an interpolation algorithm based on the known signal values ​​of surrounding moments. Linear interpolation performs line fitting between adjacent data points, spline interpolation performs curve fitting, and median filtering uses a sliding window to take the median value to achieve signal smoothing and alignment.

[0055] In some embodiments, such as Figure 4As shown, this demonstrates the code framework of the data preprocessing system. The bottom layer is the data parsing module 100, which primarily calls functions from the Python library such as `cantools`, `datetime`, and `pandas` to read ASCII data files and parse them into a dictionary format containing timestamps, message IDs, and data content. It also reads DBC files, parses the timestamps and physical values ​​of specific defined signals, and outputs the data in dictionary format. The middle layer consists of specific functional modules, including the data preprocessing module 200 and a reserved data analysis module. The data preprocessing module 200 encapsulates data cleaning function interfaces such as signal filtering and message filtering, and is the core module of the data preprocessing system. The data cleaning function interface primarily takes the parsed data dictionary as input. Based on the cleaning configuration conditions for messages / signals / time ranges / signal values ​​in specific scenarios, it cleans the dictionary data, filters abnormal values, and synchronizes time. The output is preprocessed data, generated in standard ASCII file format or a text format suitable for information analysis, which is then used as input parameters by the reserved data analysis module. The top layer is the GUI user interface module 300, which mainly calls functions such as tkinter and matplotlib in the Python library to generate a simple and intuitive operation and display interface for easy user use.

[0056] In some embodiments, data preprocessing is performed for energy consumption analysis.

[0057] Specifically, in order to analyze the energy consumption information of a vehicle in a certain driving scenario, the system obtains the full data of the vehicle in that scenario through near-end collection or remote access, and updates the signal collection whitelist. This data mainly includes battery output voltage and output current information, engine input voltage and output current information, and input voltage and output current signals of high-voltage accessories such as oil pumps and air conditioning compressors, which are required for energy consumption analysis.

[0058] In the user interface of the data preprocessing system, select the input full data, signal acquisition whitelist, vehicle model DBC file, and output path for the preprocessed data file. Configure the output file format to ASCII and execute signal filtering. Wait for the process to complete, and retrieve the preprocessed data from the output path for further energy consumption calculation and analysis.

[0059] In some embodiments, data preprocessing is performed for multiple time periods.

[0060] Specifically, in order to collect the vehicle's operating status information for multiple specific time periods, the system obtains full data covering the time period through near-end collection or remote access, and updates the message whitelist that requires the collection of vehicle operating status information, which may mainly include vehicle ignition status, brake and throttle status, vehicle speed, and vehicle fault information.

[0061] In the data preprocessing system interface, select the input full data and the output path for the preprocessed file, configure multiple time periods for update requirements, and perform time-based filtering. After execution is complete, select the time-filtered data file again for input, add it to the message whitelist, and perform message filtering. After execution is complete, output the data for further operational status statistics.

[0062] In some embodiments, data preprocessing is performed for driving scenarios.

[0063] Specifically, having acquired full daily vehicle data, to analyze energy consumption information in driving scenarios, the full data needs to be segmented into driving data and non-driving data to avoid data from non-driving scenarios affecting the overall energy consumption analysis. Using the data preprocessing interface, select the input full data, the vehicle model DBC file, and the output path for the preprocessed data file, then execute gear segmentation. After execution, full data for both driving and non-driving scenarios can be obtained for further driving scenario energy consumption analysis.

[0064] like Figure 3 and Figure 6 As shown, an offline vehicle data preprocessing method is also provided, including: Obtain the vehicle's raw data file and the corresponding protocol database file; Based on the protocol database file, the original data file is parsed frame by frame to the signal layer to obtain multiple first data, and the format of the parsed first data is standardized. Based on the configuration file, at least one preprocessing operation is performed on the parsed first data. The preprocessing operation includes at least message filtering, signal filtering, time filtering, scene segmentation, invalid value cleaning, and data synchronization. Output the preprocessed data file.

[0065] The above method utilizes DBC files to convert raw data into standardized signal data, resolving the data heterogeneity issue. Then, based on personalized rules in the configuration file, a combination of various preprocessing algorithms is applied to achieve precise data filtering, quality improvement, and format standardization. Finally, a high-quality dataset meeting specific analytical requirements is output. This achieves a one-stop conversion from raw data to analysis-ready data, improving the processing efficiency of massive vehicle data by over 90%, while ensuring the accuracy and reliability of subsequent data analysis, providing a solid data foundation for vehicle performance analysis and optimization.

[0066] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An offline vehicle data preprocessing system, characterized in that, include: A data parsing module is configured to recognize the loaded protocol database file. The data parsing module parses the original data file frame by frame based on the protocol database file to obtain multiple first data, and standardizes the format of the parsed first data. The data preprocessing module preprocesses the parsed first data based on the configuration file, wherein the preprocessing includes at least message filtering, signal filtering, time filtering and scene segmentation.

2. The offline vehicle data preprocessing system according to claim 1, characterized in that, It also includes an interactive interface module, which provides a graphical interface, receives input and configuration files, and displays processing progress and results.

3. The offline vehicle data preprocessing system according to claim 1, characterized in that, The data parsing module is based on functions from the Python function library.

4. The offline vehicle data preprocessing system according to claim 1, characterized in that, The first data includes at least the message ID, timestamp, length, and data value.

5. The offline vehicle data preprocessing system according to claim 1, characterized in that, The data preprocessing module includes a message filtering module, which filters the first data based on the message ID list in the configuration file to retain specified message ID data and writes the message ID data into the filtered data file.

6. The offline vehicle data preprocessing system according to claim 1, characterized in that, The data preprocessing module includes a signal filtering module. The signal filtering module parses the first data to the signal layer based on the protocol database file, filters signal data that meets predetermined conditions, cleans invalid values, and writes the filtered signal data into a text file or a table file.

7. The offline vehicle data preprocessing system according to claim 6, characterized in that, The invalid values ​​include at least values ​​that are outside the predetermined range and null values.

8. The offline vehicle data preprocessing system according to claim 1, characterized in that, The data preprocessing module includes a time filtering module, which filters the first data based on preset time conditions to retain data that meets the preset time conditions.

9. The offline vehicle data preprocessing system according to claim 1, characterized in that, The data preprocessing module includes a scene segmentation module. Based on a preset signal, the scene segmentation module parses the first data to the signal layer, obtains the data time range of multiple scenes, and segments the original file configuration file into data corresponding to multiple scenes and saves them separately.

10. An offline vehicle data preprocessing method, characterized in that, include: Obtain the vehicle's raw data file and the corresponding protocol database file; Based on the protocol database file, the original data file is parsed frame by frame to the signal layer to obtain multiple first data, and the format of the parsed first data is standardized. Based on the configuration file, at least one preprocessing operation is performed on the parsed first data. The preprocessing operation includes at least message filtering, signal filtering, time filtering, scene segmentation, invalid value cleaning, and data synchronization. Output the preprocessed data file.

Images