Rapid preprocessing method suitable for OBD remote emission monitoring data
By converting remote OBD emission monitoring data into columnar storage format and processing using adaptive bucket sorting algorithm, the problem of inefficient data processing is solved, rapid preprocessing is achieved, data quality and processing efficiency are improved, and specific research needs are supported. analysis.
Patent Information
- Application Number
- CN202411176284.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-26
AI Technical Summary
Existing remote OBD emission monitoring data processing has problems of low speed and inefficiency, especially due to the problems of sensor stopping working at low temperatures, data out of order, loss and retransmission, which leads to poor data quality, affecting the efficiency of emission results analysis. .
Use the conversion of original row-type storage data into column-type storage format, and the timeline is sorted through the adaptive bucket sorting algorithm, retransmitted frames and stroke segmentation, clean abnormal data, evaluate data representativeness, combine Apache Parquet format and multiple Thread processing, optimize data reading and processing speed.
It significantly improves the data reading speed and processing efficiency, shortens the sorting time through the adaptive bucket sorting algorithm, and the three-step cleaning strategy effectively clears abnormal data, improves data quality and representativeness, and supports special processing for specific research needs.
Smart Images

Figure CN119066057A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of emission monitoring, and in particular to a fast preprocessing method suitable for OBD remote emission monitoring data. Background Art
[0002] As an important part of road mobile sources, heavy-duty diesel vehicles are the main source of emissions of pollutants such as nitrogen oxides (NOx). Although new energy vehicles have made significant progress in the field of light vehicles, the progress of new energy vehicles in the field of heavy vehicles is relatively slow, and heavy-duty diesel vehicles will still maintain a huge number of vehicles and stable annual production and sales. In order to control its emissions, countries continue to tighten emission standards, and exhaust after-treatment technology is constantly upgraded. However, engines that meet the standards in bench tests often exceed the emission standards in actual driving, so actual road emission monitoring of the whole vehicle has become a necessary means.
[0003] At present, the actual road emission monitoring methods mainly include PEMS test, remote sensing test, vehicle following test, tunnel test and remote OBD. PEMS test is costly, time-consuming and has a small amount of data; while non-contact methods such as remote sensing test, vehicle following test and tunnel test cannot obtain engine data. Remote OBD relies on the original OBD system of the vehicle. It only needs to install an additional communication terminal, with low modification cost, large amount of data that can be obtained, synchronized engine data and long-term tracking and analysis of a single vehicle. It is a very practical emission monitoring method. With the widespread application of remote OBD system, the low speed and low efficiency of massive data processing have become a research bottleneck. OBD remote emission monitoring data not only contains synchronized engine data, but also involves a comprehensive record of vehicle operation status. There are data quality issues that cannot be ignored (such as sensors stop working at low exhaust temperature, insufficient accuracy, data disorder, loss, retransmission during transmission, etc.), which makes data preprocessing a necessary task. Therefore, optimizing the preprocessing process and improving the processing speed are crucial to improving the overall research efficiency. Summary of the invention
[0004] The purpose of the present invention is to provide a fast preprocessing method suitable for OBD remote emission monitoring data, which is aimed at remote large-volume OBD data, and can efficiently identify and process quality problems in the data (such as the sensor stops working at low exhaust temperature, insufficient accuracy, data disorder, loss, retransmission, etc. during transmission), so as to speed up and improve the efficiency of the leading work necessary for emission result analysis.
[0005] In order to achieve the above object, the present invention adopts the following technical solutions:
[0006] A fast preprocessing method for OBD remote emission monitoring data includes the following steps:
[0007] Convert the original row-based data files into column-based storage formats;
[0008] Adaptive bucket sorting algorithm is used to reshape the time sequence of the converted column storage data, remove retransmitted frames, and perform trip segmentation after the time axis is sorted;
[0009] Perform abnormal data cleaning on the sorted data, including identifying and removing invalid values, over-limit values, and unreasonable continuous repeated values;
[0010] Evaluate the mileage coverage and effective data ratio of the cleaned data, and comprehensively evaluate the data representativeness.
[0011] Preferably, the column storage format adopts Apache Parquet format to improve data reading speed and compression efficiency.
[0012] Preferably, the adaptive bucket sorting algorithm includes:
[0013] Traverse the data to determine the timestamp range and divide the time axis within the coverage range into multiple buckets;
[0014] Put the data frame into the corresponding bucket according to the timestamp;
[0015] Quickly sort the data frame in each bucket;
[0016] Merge the data frames in the buckets sequentially to get a data table in time order.
[0017] Preferably, the abnormal data cleaning step includes:
[0018] Based on the standard invalid return value table, identify and clear standard invalid values;
[0019] Based on the effective upper and lower limit tables, identify and eliminate unreasonable over-limit values;
[0020] Identify and remove unreasonable continuous repeated values through a continuous repeated detection algorithm based on differential binarization;
[0021] Exclude data in the engine non-start state before cleaning.
[0022] Preferably, the data representativeness evaluation step comprises:
[0023] Calculate the mileage coverage of the cleaned data;
[0024] The data representativeness is calculated by multiplying the weighted mileage coverage by the effective data ratio.
[0025] Preferably, it also includes special data processing steps for specific research needs, including:
[0026] Reshape the non-negativity of NOx concentration downstream of SCR;
[0027] The jump and drift problems of GPS positioning are corrected by using OBD vehicle speed comparison based on wheel rotation calculation.
[0028] Preferably, it includes: a storage format conversion module, used to convert the original row-type storage data file into a column-type storage format;
[0029] The timeline sorting module uses an adaptive bucket sorting algorithm to sort the converted data on the timeline, removes retransmitted frames based on whether the timestamps of adjacent frames are identical, and performs trip segmentation based on large timestamp jumps and low SCR outlet temperatures;
[0030] Abnormal data cleaning module, used to clean abnormal values in the sorted data;
[0031] Data evaluation module, used to evaluate the mileage coverage and effective data ratio of cleaned data, and comprehensively evaluate the data representativeness;
[0032] Special processing module, used to perform special processing on data for specific research needs.
[0033] The present invention has at least the following beneficial effects:
[0034] Data file preparation: Convert the original row storage file into a column storage format (such as Apache Parquet) to improve data reading speed, reduce IO operations, support selective reading of column features, and optimize storage and compression efficiency.
[0035] Timeline Arrangement: Adopting the adaptive bucket sorting algorithm, the timeline in the range is divided into multiple buckets by traversing the data to determine the timestamp range, and then merged after sorting them separately, effectively shortening the sorting time, and supporting multi-threaded processing to further improve the speed. Retransmitted frames are cleared based on whether the timestamps of adjacent frames are the same, and the trip is segmented based on large timestamp jumps and low SCR outlet temperatures.
[0036] Abnormal data cleaning: Cleaning strategies are formulated for standard invalid values, over-limit values, and unreasonable continuous repeated values. In particular, data in the engine non-start state is excluded before cleaning to reduce the subsequent cleaning workload. After cleaning, the mileage coverage and effective data ratio of the data are evaluated, and the data representativeness is comprehensively evaluated.
[0037] Special processing: Special processing is performed for specific research needs, such as NOx emission analysis and GPS positioning correction. NOx concentration data below 0 is reshaped to non-negativity by taking it as 0, and GPS positioning jump and drift problems are corrected by using OBD vehicle speed comparison based on wheel rotation calculation. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.
[0039] Figure 1 It is a schematic diagram of the adaptive bucket sorting flow chart of the present invention;
[0040] Figure 2 It is a schematic diagram of the process flow of itinerary segmentation of the present invention;
[0041] Figure 3 It is a schematic diagram of the three-step cleaning process of standard invalid values, over-limit values, and unreasonable continuous repeated values of the present invention. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0043] Embodiment 1, refer to Figure 1-3 , a fast preprocessing method suitable for OBD remote emission monitoring data, comprising the following steps:
[0044] Convert the original row-based data files into column-based storage formats;
[0045] Adopt adaptive bucket sorting algorithm to sort the converted data on time axis, remove retransmitted frames according to whether the timestamps of adjacent frames are the same, and perform trip segmentation based on large timestamp jumps and low SCR outlet temperature;
[0046] Perform abnormal data cleaning on the sorted data, including identifying and removing invalid values, over-limit values, and unreasonable continuous repeated values;
[0047] Evaluate the mileage coverage and effective data ratio of the cleaned data, and comprehensively evaluate the data representativeness.
[0048] Data file preparation:
[0049] In the data file preparation part, the original row storage file (usually a CSV file) exported from the database and loaded with remote OBD data is converted into a column storage format. Remote OBD data is transmitted to the supervisory end in real time in frames. The supervisory end usually uses a row-based relational database to adapt to the characteristics of vertical data accumulation. However, row storage has no advantage in the preprocessing and analysis process after exporting data. Its comparison with column storage is shown in Table 1. There are only 20 column features of remote OBD data, which is usually 2 to 5 orders of magnitude lower than the number of frames contained in a single file. The use of column storage can reduce IO operations; most analysis scenarios do not require all remote OBD features, and column storage can also improve the reading speed in the selection of some features; the data formats of column feature items are different, and column storage provides a basis for targeted compression, which not only reduces the file size, but also indirectly improves the reading efficiency because the number of bytes required to be read is reduced.
[0050] Table 1 Comparison of row storage and column storage in various aspects
[0051]
[0052] Timeline arrangement:
[0053] Errors in remote OBD data can be divided into two categories: timeline errors and abnormal data. Timeline errors are caused by abnormalities in the communication process. Data packet loss, improper retransmission, and supplementary transmission are the main causes of missing data frames, multiple data at the same timestamp, and data disorder. This section first solves the disorder problem by sorting, then cleans up redundant frames according to the timestamps of adjacent frames to solve the retransmission problem, then divides the trip according to the long-term vacant context, and finally calculates the mileage coverage of the evaluation data.
[0054] This section uses an adaptive bucket sort to organize the timeline. First, traverse the data to obtain the upper and lower bounds of the timestamp, define each hour in between as a bucket, put each data frame into the corresponding bucket according to the timestamp, and then use any quick sort algorithm to sort the data frames in each bucket. Finally, traverse each bucket and take out the data frames in order to connect them, that is, to obtain a data table in time order. The algorithm is as follows: Figure 1 shown.
[0055] According to the above embodiments, it can be seen that:
[0056] Data file preparation: Convert the original row storage file (usually a CSV file) into a column storage format (such as Apache Parquet) to increase data reading speed, reduce IO operations, support selective reading, and optimize storage and compression efficiency.
[0057] Timeline Arrangement: Adopting adaptive bucket sorting algorithm, the time is divided into multiple buckets by traversing the data to determine the timestamp range, sorting them separately and then merging them, effectively shortening the sorting time, supporting multi-threaded processing to further improve the speed. Retransmitted frames are cleared according to whether the timestamps of adjacent frames are the same, and the trip is segmented based on large timestamp jumps and low SCR outlet temperature.
[0058] Abnormal data cleaning: Cleaning strategies are formulated for standard invalid values, over-limit values, and unreasonable continuous repeated values. In particular, data in the engine non-start state is excluded before cleaning to reduce the subsequent cleaning workload. After cleaning, the mileage coverage and effective data ratio of the data are evaluated, and the data representativeness is comprehensively evaluated.
[0059] Special processing: Special processing is performed for specific research needs, such as NOx emission analysis and GPS positioning correction. The NOx concentration data below 0 is set to 0 to reshape the non-negativity, and the GPS positioning jump and drift problems are corrected by vehicle speed comparison.
[0060] Example 2, refer to Figure 1-3 The column storage format adopts Apache Parquet format to improve data reading speed and compression efficiency.
[0061] The adaptive bucket sorting algorithm includes:
[0062] Traverse the data to determine the timestamp range and divide the time axis within the coverage range into multiple buckets;
[0063] Put the data frame into the corresponding bucket according to the timestamp;
[0064] Quickly sort the data frame in each bucket;
[0065] Merge the data frames in the buckets sequentially to get a data table in time order.
[0066] The abnormal data cleaning step includes:
[0067] Based on the standard invalid return value table, identify and clear standard invalid values;
[0068] Based on the effective upper and lower limit tables, identify and eliminate unreasonable over-limit values;
[0069] Identify and remove unreasonable continuous repeated values through a continuous repeated detection algorithm based on differential binarization;
[0070] Exclude data in the engine non-start state before cleaning.
[0071] The data representativeness evaluation step comprises:
[0072] Calculate the mileage coverage of the cleaned data;
[0073] The data representativeness is calculated by multiplying the weighted mileage coverage by the effective data ratio.
[0074] It also includes specialized data processing steps for specific research needs, including:
[0075] Reshape the non-negativity of NOx concentration downstream of SCR;
[0076] The jump and drift problems of GPS positioning are corrected by using OBD vehicle speed comparison based on wheel rotation calculation.
[0077] It includes: a storage format conversion module, used to convert the original row-based data file into a column-based storage format;
[0078] The timeline sorting module uses an adaptive bucket sorting algorithm to sort the converted data on the timeline;
[0079] Abnormal data cleaning module, used to clean abnormal values in the sorted data;
[0080] Data evaluation module, used to evaluate the mileage coverage and effective data ratio of cleaned data, and comprehensively evaluate the data representativeness;
[0081] Special processing module, used to perform special processing on data for specific research needs.
[0082] Theoretically, the time complexity of bucket sort is O(n), which is definitely better than the O(nlogn) of general sorting. In practice, based on some assumptions that are valid in most cases, the adaptive bucket sorting here can speed up the sorting of remote OBD data: in the worst case, based on the superadditivity of the function f(x) = xlogx, the time consumption of the adaptive bucket sorting part is less than The common quick sort is cnlogn, where c is a constant coefficient representing the time taken for a single operation. As long as the total number of frames in the data table is greater than 3600, the improved bucket sort can speed up the sorting, which corresponds to a time span of at least one hour in actual practice. In addition, the adaptive bucket sort algorithm is highly multi-thread friendly and can benefit from multi-core CPUs to further speed up in actual applications.
[0083] For adjacent data frames arranged in ascending order, subtract the timestamp of the previous item from the timestamp of the latter item to obtain the recording interval between the two frames, and classify them according to the interval length according to the table below. For each unique timestamp, we only retain a single data frame, that is, directly clean out the retransmitted data frame; short-term omissions of less than ten seconds generally occur at the same time as the engine is started, and the impact caused by it can be completely eliminated by subsequent further cleaning, so it is not processed in this step; long-term absences shown on the timeline may be due to long-term omissions caused by communication errors, or it may be a normal phenomenon under parking behavior, and it is quite common to have both. In order to eliminate the influence of factors such as cold starts, it is very necessary to divide a single trip. The trip segmentation referred to here is actually to detect whether there is a significant long-term parking event during the long vacancy. If so, the vacancy is set as the split point, and the two sections before and after are classified into different trips. The specific process is as follows Figure 2 shown.
[0084] The process uses two conditions to determine whether to segment: if the vacancy time is too long, it is considered that the vehicle state has changed significantly and the segmentation is marked; if the SCR outlet temperature at the starting point of the rear section is within the normal temperature range, it is considered that the SCR has cooled down after a long period of parking and the segmentation is marked. It is worth noting that after marking, the segmented trip can be filtered out, which is too short and has no high analysis value. The last step is to calculate the mileage coverage of the data table. The formula is as follows:
[0085]
[0086] Where n represents n long-term vacancies, time_abs_start i Represents the timestamp of the data before the start of the vacancy, time_abs_end i Represents the timestamp of the data after the end of the i-th gap, Mileage time_end and Mileage time_start Represent the timestamps of the earliest and latest row data respectively. Since the number of missed data frames is unknown, the mileage is used here to replace the number of frames, which will be used in the calculation of the subsequent data representative rate.
[0087] Abnormal data cleaning
[0088] The abnormal data of remote OBD comes from the abnormalities of the OBD system, such as sensor failure, abnormal data processing, improper software behavior, etc., which are reflected in the data itself, manifested as invalid return values specified by the standard, over-limit values that do not conform to the physical laws of the real world, and abnormal continuous identical values. This section targets these three types of abnormal data and conducts targeted cleaning in three steps, such as Figure 3 shown.
[0089] The cleaning basis of each feature is shown in Table 2. The primary basis for continuous repeated cleaning is data accuracy. When the accuracy is very low, continuous repetition is a very normal phenomenon. It is worth noting that the SCR upstream and downstream temperatures with higher accuracy have a very high correlation with the engine speed. When the engine speed does not change suddenly, its value tends to be stable. The frequency of the SCR upstream and downstream temperatures not changing in adjacent frames is quite high. Therefore, the features for continuous repeated cleaning are only vehicle speed, engine speed, intake volume, engine fuel flow, and SCR upstream and downstream NOx concentration.
[0090] Table 2 Cleaning basis for each feature
[0091]
[0092]
[0093] *: The features marked as "yes" in this column need to be cleaned repeatedly. These features have high data accuracy. If the same value within the normal range is continuously returned for a long time (more than 60 seconds in practice), it can be determined that this data is an abnormal continuous repetitive segment and needs to be cleaned. In particular, it is normal for the engine fuel flow and vehicle speed to be 0 for a long time. If these two features have a long 0 value segment, they will not be cleaned. In the specific implementation, the standard invalid value cleaning is based on equality judgment:
[0094] entry.isInvalid=(entry[feature]==invalid[feature])
[0095] Among them, invalid is a table that stores standard invalid return values, and isInvalid is a flag that indicates whether entry is judged as a standard invalid return value. The over-limit cleaning is based on the upper and lower limit judgment: entry.isOutofRange = (entry[feature] < lowerbound[feature])
[0096] ||(entry[feature]>upperbound[feature])
[0097] Among them, lowerbound and upperbound are tables that store the effective lower and upper bounds of the feature respectively, and isOutofRange is a flag that indicates whether the entry is judged to be out of limit.
[0098] According to the above embodiments, the beneficial effects of the present invention are mainly reflected in the following aspects:
[0099] Significantly improve data reading speed: The application of column storage format increases data reading speed several times, laying a solid foundation for subsequent processing.
[0100] Optimize the timeline sorting process: The adaptive bucket sorting algorithm greatly shortens the sorting time and supports multi-threaded processing to further improve efficiency.
[0101] Improve the cleaning effect of abnormal data: The three-step cleaning strategy is combined with the elimination of the engine not started state to effectively remove various abnormal data and improve data quality.
[0102] Introducing data representativeness evaluation: Comprehensively evaluate data representativeness through mileage coverage and effective data ratio to ensure the validity and reliability of preprocessing results.
[0103] Support special data processing: Provide special processing solutions for specific research needs to enhance the flexibility and practicality of preprocessing methods.
[0104] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.
Claims
1. A fast preprocessing method for OBD remote emission monitoring data, characterized in that: The following steps are involved: Convert the original row-based data files into column-based storage formats; Adopt adaptive bucket sorting algorithm to reshape the time sequence of converted column storage data, remove retransmitted frames according to whether the timestamps of adjacent frames are the same, and perform trip segmentation based on large timestamp jumps and low SCR outlet temperature; Perform abnormal data cleaning on the sorted data, including identifying and removing invalid values, over-limit values, and unreasonable continuous repeated values; Evaluate the mileage coverage and effective data ratio of the cleaned data, and comprehensively evaluate the data representativeness.
2. A fast preprocessing method for OBD remote emission monitoring data according to claim 1, characterized in that: The column storage format adopts Apache Parquet format to improve data reading speed and compression efficiency.
3. A fast preprocessing method for OBD remote emission monitoring data according to claim 1, characterized in that: The adaptive bucket sorting algorithm includes: Traverse the data to determine the timestamp range and divide the time axis within the coverage range into multiple buckets; Put the data frame into the corresponding bucket according to the timestamp; Quickly sort the data frame in each bucket; Merge the data frames in the buckets sequentially to get a data table in time order.
4. A fast preprocessing method for OBD remote emission monitoring data according to claim 1, characterized in that: The abnormal data cleaning step includes: Based on the standard invalid return value table, identify and clear standard invalid values; Based on the effective upper and lower limit tables, identify and eliminate unreasonable over-limit values; Identify and remove unreasonable continuous repeated values through a continuous repeated detection algorithm based on differential binarization; Before the three-stage cleaning, exclude the data of the engine not started state.
5. A fast preprocessing method for OBD remote emission monitoring data according to claim 1, characterized in that: The data representativeness evaluation step comprises: Calculate the mileage coverage of the cleaned data; The data representativeness is calculated by multiplying the weighted mileage coverage by the effective data ratio.
6. A fast preprocessing method for OBD remote emission monitoring data according to claim 1, characterized in that: It also includes specialized data processing steps for specific research needs, including: Reshape the non-negativity of NOx concentration downstream of SCR; The jump and drift problems of GPS positioning are corrected by using OBD vehicle speed comparison based on wheel rotation calculation.
7. A fast preprocessing method for OBD remote emission monitoring data according to claim 1, characterized in that: include: A storage format conversion module is used to convert the original row-based storage data file into a column-based storage format; The timeline sorting module uses an adaptive bucket sorting algorithm to sort the converted data on the timeline, removes retransmitted frames based on whether the timestamps of adjacent frames are identical, and performs trip segmentation based on large timestamp jumps and low SCR outlet temperatures; Abnormal data cleaning module, used to clean abnormal values in the sorted data; Data evaluation module, used to evaluate the mileage coverage and effective data ratio of cleaned data, and comprehensively evaluate the data representativeness; Special processing module, used to perform special processing on data for specific research needs.
Citation Information
Patent Citations
Method for calculating NOx emission of heavy-duty diesel vehicle by extracting combined short segments
CN112486962A
NOx data abnormity cleaning method remotely monitored by OBD (On-Board Diagnostic)
CN117708743A
Monitoring system for vehicle carbon emission in expressway
CN118465196A
