Data cleaning quality analysis method

Abstract

The embodiment of the invention discloses a data cleaning quality analysis method. The method comprises the following steps: acquiring to-be-cleaned original vehicle data; inputting the original vehicle data into a pre-trained data cleaning model to obtain cleaned target vehicle data; determining target quality index data of the target vehicle data about various quality indexes based on a plurality of preset quality indexes; and determining the cleaning quality of the target vehicle data based on target quality index data of the target vehicle data about various quality indexes. According to the embodiment of the invention, the situation that the data quality problem is passively discovered in the past is fundamentally changed, the data cleaning quality is actively and quickly evaluated, low-quality data can be intercepted in time before downstream business application, decision errors and business loss caused by dirty data are effectively avoided, and the service quality is improved. And the reliability and credibility of the whole data pipeline are improved.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and in particular to a data cleaning and quality analysis method. Background Technology

[0002] Existing vehicle data cleaning methods typically employ offline batch processing, lacking a real-time online monitoring mechanism for the quality of the cleaning results. This makes it difficult to detect data quality issues promptly, often only revealing defects in the data cleaning process when the data has already been applied to downstream business systems, such as report generation or model training, and anomalies occur. By this time, data quality problems may have already led to biased business decisions or degraded model performance, resulting in actual losses. Therefore, current technology suffers from the inability to proactively assess data cleaning quality in real time, thus failing to effectively prevent downstream business risks. Summary of the Invention

[0003] This application provides a data cleaning quality analysis method that solves the technical problem in the prior art that the data cleaning quality cannot be evaluated in real time and proactively, thus failing to effectively prevent downstream business risks. It realizes automated, multi-dimensional quality evaluation of data cleaning results, can promptly detect data defects in the cleaning process, effectively avoid decision bias and model performance degradation caused by low-quality data flowing into downstream business systems, and improve the reliability and security of data-driven business applications.

[0004] This application provides a data cleaning quality analysis method, the method comprising: Obtain the original vehicle data to be cleaned; The original vehicle data is input into a pre-trained data cleaning model to obtain cleaned target vehicle data; Based on a variety of preset quality indicators, the target quality indicator data of the target vehicle data is determined with respect to various quality indicators; Based on the target vehicle data and its target quality index data for various quality indicators, the cleaning quality of the target vehicle data is determined.

[0005] Furthermore, after acquiring the original vehicle data to be cleaned, the method further includes: Based on a variety of preset quality indicators, the original quality indicator data of the original vehicle data with respect to various quality indicators are determined; Based on the original quality index data of the original vehicle data with respect to various quality indicators, and the target quality index data of the target vehicle data with respect to various quality indicators, the cleaning quality of the target vehicle data is determined.

[0006] Furthermore, after determining the target quality index data of the target vehicle data with respect to various preset quality indicators, the method further includes: The target quality index data are classified according to various preset data dimensions; For each data dimension, the target quality indicator data is used to determine the quality change trend of the target quality indicator data according to the time sequence of the target quality indicator data. The quality change trends of the target quality index data corresponding to various data dimensions are visualized respectively.

[0007] Furthermore, after determining the target quality index data of the target vehicle data with respect to various preset quality indicators, the method further includes: Compare the target quality indicator data corresponding to each quality indicator with the historical quality indicator data to determine whether there are any anomalies in the target quality indicator data corresponding to each quality indicator. When there is an anomaly in the target quality indicator data corresponding to at least one quality indicator, identify the anomaly type and anomaly level; Generate target anomaly alarm information based on the anomaly type and the anomaly level; The target alarm method and target alarm device are determined based on the anomaly type and the anomaly level; The target anomaly alarm information is sent to the target alarm device according to the target alarm method to remind the user to handle the anomaly.

[0008] Furthermore, the historical quality indicator data refers to any one of the following quality indicator data: Quality index data corresponding to any historical collection period prior to the collection period corresponding to the original vehicle data. The quality index data corresponding to the vehicle data after cleaning the data cleaning model of the previous historical version; Quality index data for vehicle models other than the original vehicle data mentioned above.

[0009] Furthermore, if the target quality indicator data corresponding to at least one quality indicator is abnormal, the method further includes: The target quality index data that contains anomalies are stored as training samples; If the amount of data in the stored training samples meets the preset data volume condition, the data cleaning model is incrementally trained based on the stored training samples to obtain the updated data cleaning model. Alternatively, the data cleaning model is retrained based on the stored training samples, the original vehicle data within the preset historical time period, and the target vehicle data to obtain the updated data cleaning model.

[0010] Furthermore, when the preset update cycle corresponding to the data cleaning model is reached, the method further includes: The data cleaning model is incrementally trained based on the stored target quality index data containing anomalies to obtain an updated data cleaning model; or, the data cleaning model is retrained based on the stored target quality index data containing anomalies, the original vehicle data within a preset historical period, and the target vehicle data to obtain an updated data cleaning model.

[0011] Furthermore, after acquiring the original vehicle data to be cleaned, the method further includes: The original vehicle data is input into a pre-trained data cleaning model to generate a structured audit log. The structured audit log includes data tracking identifiers, original values ​​before data cleaning, target values ​​after data cleaning, cleaning rule identifiers, model decision scores, cleaning rules, and cleaning time.

[0012] Furthermore, after generating the structured audit log, the method also includes: Receive a query command containing the target query conditions sent by the terminal; If the structured audit log contains data that meets the target query conditions, the target data in the structured audit log that matches the target query conditions is fed back to the terminal.

[0013] Furthermore, the preset quality indicators include at least: completeness indicators, accuracy indicators, consistency indicators, and timeliness indicators; The integrity metrics include at least the percentage of non-empty fields and the packet loss rate; The accuracy indicators include at least the percentage of abnormal data; The consistency index includes at least the number of contradictory data entries for each vehicle within the same time interval; The timeliness indicator includes at least the time from the generation of the original vehicle data to the completion of the cleaning process.

[0014] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: This application utilizes a pre-trained model for data cleaning, which not only efficiently outputs high-quality target data but also ensures the consistency and intelligence of the cleaning process. Based on preset multi-dimensional quality indicators (such as completeness and accuracy), the cleaning results are quantitatively evaluated, generating objective and measurable target quality indicator data, making data quality transparent and visible. Finally, the cleaning quality is determined by comprehensively considering these indicator data, achieving accurate and automated acceptance of the data cleaning results. This application fundamentally changes the past situation of passively discovering data quality problems, enabling proactive and rapid evaluation of data cleaning quality. It can promptly intercept low-quality data before downstream business applications, effectively avoiding decision-making errors and business losses caused by dirty data, and improving the reliability and trustworthiness of the entire data pipeline. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating a data cleaning and quality analysis method provided in an embodiment of this application. Detailed Implementation

[0017] This application provides a data cleaning quality analysis method, which solves the technical problem in the prior art that the data cleaning quality cannot be evaluated in real time and proactively, thus failing to effectively prevent downstream business risks.

[0018] The technical solution of this application embodiment is to solve the above-mentioned technical problems, and the general idea is as follows: This application utilizes a pre-trained model for data cleaning, which not only efficiently outputs high-quality target data but also ensures the consistency and intelligence of the cleaning process. Based on preset multi-dimensional quality indicators (such as completeness and accuracy), the cleaning results are quantitatively evaluated, generating objective and measurable target quality indicator data, making data quality transparent and visible. Finally, the cleaning quality is determined by comprehensively considering these indicator data, achieving accurate and automated acceptance of the data cleaning results. This application fundamentally changes the past situation of passively discovering data quality problems, enabling proactive and rapid evaluation of data cleaning quality. It can promptly intercept low-quality data before downstream business applications, effectively avoiding decision-making errors and business losses caused by dirty data, and improving the reliability and trustworthiness of the entire data pipeline.

[0019] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0020] First, it should be clarified that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0021] This application provides a data cleaning quality analysis method, which includes steps S11-S14.

[0022] Step S11: Obtain the original vehicle data to be cleaned; Step S12: Input the original vehicle data into the pre-trained data cleaning model to obtain the cleaned target vehicle data; Step S13: Based on a variety of preset quality indicators, determine the target quality indicator data of the target vehicle data with respect to various quality indicators; Step S14: Based on the target quality index data of the target vehicle data regarding various quality indicators, determine the cleaning quality of the target vehicle data.

[0023] The data cleaning and quality analysis method provided in this application can be executed by the vehicle's onboard computing platform, specifically an onboard computer, a smart cockpit system, or a dedicated onboard data processing unit. This way, only the data of a single vehicle needs to be processed, which is beneficial for achieving localized real-time quality monitoring and rapid response.

[0024] Furthermore, the data cleaning and quality analysis method provided in this application embodiment can also be executed by a cloud server that interacts with multiple vehicles, specifically a cloud server cluster, edge computing node, or vehicle networking platform server. When executed by a cloud server that interacts with multiple vehicles, the cloud server can collect relevant data from multiple vehicles in real time and perform centralized and unified data processing and quality analysis on the relevant data of each vehicle, thereby enabling horizontal comparison and collaborative optimization of data quality at the fleet level and vehicle model level. The following descriptions in this application embodiment will only use a cloud server as an example.

[0025] Regarding step S11, obtain the original vehicle data to be cleaned.

[0026] Raw vehicle data can encompass multi-source heterogeneous data collected during vehicle operation by various onboard sensors, controllers, and onboard information systems. Specifically, this includes: real-time sensor readings such as vehicle position, speed, engine speed, and coolant temperature; fault codes and status parameters reported by onboard diagnostic systems; and more complex data streams generated by onboard infotainment systems and advanced driver assistance systems, such as multimedia interaction logs, camera images, and radar point clouds. These data collectively constitute a massive information source reflecting vehicle status, driver behavior, and the external environment.

[0027] The raw vehicle data is susceptible to various interferences during collection, transmission, and storage, leading to quality issues and necessitating cleaning before use. Specifically, sensor accuracy limitations, signal transmission interruptions, excessive vehicle network load, or electromagnetic interference can cause numerical anomalies, missing records, or data jumps. Inconsistent timestamps from different data sources and inconsistent data formats can cause data inconsistencies and conflicts. Furthermore, it may contain invalid or duplicate data due to communication errors. Directly using this "dirty data" (data before cleaning, i.e., raw vehicle data) will severely impact the accuracy and reliability of subsequent data analysis, vehicle condition assessment, intelligent connected applications, and autonomous driving model training.

[0028] The methods for acquiring raw vehicle data include: directly monitoring the bus and parsing messages from electronic control units and sensor nodes via in-vehicle networks (such as CAN, LIN, MOST, or in-vehicle Ethernet); transmitting data to a cloud data platform in real-time or near real-time using a vehicle telematics unit (T-Box) via wireless communication technologies such as cellular networks (4G / 5G); and, in specific scenarios, using diagnostic equipment for local data acquisition via physical interfaces (such as the OBD-II port). These methods collectively achieve multi-channel, continuous acquisition and aggregation of raw vehicle data.

[0029] In step S11, the acquired raw vehicle data can be organized and processed in the form of "micro-batches." A "micro-batch" is a data processing mode between traditional batch processing and real-time stream processing. It divides the continuously generated data stream into small data batches (i.e., micro-batches), and then performs subsequent cleaning, quality assessment, and other steps (S11-S14) on each micro-batch in a sequential and cyclical manner. Compared to processing massive amounts of data accumulated over several hours at once, micro-batch processing involves smaller data units and shorter processing cycles (typically on the order of seconds or minutes), thus achieving a near real-time data pipeline.

[0030] For example, the data platform can be configured to package all sensor readings, CAN bus messages, and other information received from the vehicle during this period into a micro-batch every 10 seconds; then, the system will immediately trigger an independent data acquisition, cleaning, quality analysis, and evaluation process for this new micro-batch containing the data from the most recent 10 seconds.

[0031] For the sake of simplicity, this embodiment will only use a single micro-batch of raw vehicle data as an example to illustrate the specific process of data cleaning and quality analysis. This micro-batch processing method greatly reduces the delay from data generation to the output of quality assessment results, making it possible to promptly detect and intervene in data quality issues.

[0032] Regarding step S12, the original vehicle data is input into a pre-trained data cleaning model to obtain cleaned target vehicle data.

[0033] Pre-trained data cleaning models can be trained using various mainstream machine learning or deep learning frameworks. For example, they can be built and trained using open-source frameworks such as TensorFlow, PyTorch, or Scikit-learn. For processing time-series vehicle data, models might be based on structures such as Long Short-Term Memory (LSTM) networks or autoencoders; while for identifying complex patterns or outliers, algorithms such as Isolation Forest or Generative Adversarial Networks (GANs) might be employed. Furthermore, this model can be integrated with rule-based automated data processing tools to form a hybrid data cleaning solution.

[0034] This application provides an exemplary training process for a data cleaning model. First, a large amount of historical vehicle data containing labeled quality issues (such as outliers and missing segments) is collected as a training set. Then, based on the specific cleaning objective (such as correcting abnormal sensor readings), a model architecture is selected and designed, for example, using an LSTM autoencoder to learn the temporal patterns of normal vehicle data. Next, the model is trained using the training set data, with the goal of enabling the model to accurately reconstruct normal data while generating significant reconstruction errors for abnormal data. After training, the model performance is evaluated and optimized using an independent validation set, ultimately solidifying the model capable of effectively identifying and repairing data anomalies into a deployable "pre-trained data cleaning model."

[0035] Data cleaning rules and content can include the following aspects: First, integrity processing, such as imputing missing values ​​(using the mean of previous and subsequent values, time-series prediction, etc.) or marking and deleting them; Second, rationality cleaning, that is, identifying and handling outliers that exceed reasonable physical ranges (such as negative engine speed); Third, consistency correction, resolving conflicts between different data sources caused by asynchronous timestamps or inconsistent units; Fourth, format standardization, converting data into a unified and standardized format; Fifth, smoothing and noise reduction, filtering out high-frequency noise caused by signal interference; Sixth, validity verification, removing obviously invalid or non-compliant data records.

[0036] The cleaned target vehicle data exhibits a fundamental improvement in quality, characterized by higher completeness, consistency, and accuracy. This provides a reliable data foundation for downstream data analysis, report generation, and AI model training. Secondly, high-quality data directly enhances the quality of data-driven decision-making and the reliability of business applications, such as making vehicle fault prediction more accurate and intelligent driving strategies safer. Finally, the automated and model-based cleansing process enables continuous and efficient monitoring and maintenance of data quality, effectively preventing the chain reactions and potential losses that could result from low-quality data flowing into business systems.

[0037] Furthermore, by inputting the original vehicle data into a pre-trained data cleaning model, a structured audit log can be generated. The structured audit log may include data tracking identifiers, original values ​​before data cleaning, target values ​​after data cleaning, cleaning rule identifiers, model decision scores, cleaning rules, and cleaning time.

[0038] Structured audit logs are standardized, machine-readable, and detailed records that systematically track and record key information throughout the entire process of how the data cleaning model processes each or every batch of raw vehicle data. Unlike traditional unstructured, plain text operation logs, they use predefined, fixed field formats to organize information, much like a well-structured data table. This allows the logs to be easily queried, analyzed, and automatically monitored, providing complete traceability and transparency for data cleaning operations.

[0039] Structured audit logs contain several core fields, each recording a specific aspect of the cleansing process. Regarding the data trace identifier, this is an identifier used to uniquely identify the original data record, typically a data record ID or a composite key combining the vehicle's VIN and a timestamp. For example, it could be "VIN123456789_20231030_120530_001", allowing precise location of the specific data record from a particular vehicle.

[0040] Regarding the original value before data cleaning and the target value after data cleaning, these two fields record the original value of a specific field before cleaning and the correction result after cleaning in a "before-after" comparison format. For example, the original value might be "Speed:350" (a value outside the reasonable range), while the target value is cleaned to "Speed:NULL" or "Speed:120" inferred according to the rules.

[0041] Regarding the cleaning rule identifier and the cleaning rule, the former is the unique code of the rule or model component that triggers this cleaning operation, such as "RULE_ACC_001" or "ML_ANOMALY_V1"; the latter is a human-readable description of the rule or operation, such as "physical range verification of speed value (0-300km / h)" or "outlier detection based on isolated forest".

[0042] Regarding the model decision score, when the cleaning operation is triggered by the machine learning model, this field records the confidence level or anomaly score of the model's decision. For example, for a velocity value judged as anomalous, the model might output an anomaly score as high as 0.95, quantifying the certainty of its judgment.

[0043] Regarding the cleaning time, this field precisely records the timestamp of the cleaning operation, such as "2023-10-30 12:05:35.123 UTC". This is crucial for performance monitoring, issue tracing, and timeliness analysis.

[0044] The core function of generating structured audit logs is to enable auditability, traceability, and continuous optimization of the data cleaning process. Firstly, it provides powerful problem diagnosis and root cause analysis capabilities. When downstream businesses discover data issues, they can quickly trace back to the cleaning stage to identify which rule or model was modified, the reason for the modification, and which original value was changed to the current result. Secondly, it serves the evaluation and optimization of model and rule effectiveness. By analyzing the correlation between "model decision scores" and subsequent business feedback, the accuracy of model judgments can be assessed, thereby optimizing model parameters or cleaning rules. Finally, these detailed records meet data governance and compliance requirements, providing a solid chain of evidence for data quality control and ensuring the standardization and transparency of the entire data processing flow.

[0045] After generating the structured audit log, the method further includes steps S121-S122.

[0046] Step S121: Receive a query instruction containing the target query conditions sent by the terminal; Step S122: If the structured audit log contains data that meets the target query conditions, the target data in the structured audit log that matches the target query conditions is fed back to the terminal.

[0047] Regarding step S121, the "terminal" refers to the client system used by the technical or business roles who need to access and review the details of the data cleaning process. Specifically, this could be a cloud platform management console used by data engineers for operation and maintenance monitoring, a data quality probing tool front-end used by data analysts, or an internal audit system logged in by compliance auditors. The "target query conditions" are search instructions initiated by these end users to precisely locate specific audit records. These can take various forms, such as: querying all data records modified by rule "RULE_ACC_001" within a specific time range (e.g., "2023-10-30 10:00 to 11:00"); querying all data cleaning operations for a specific vehicle (identified by VIN code "VIN123456789"); or querying high-confidence anomaly handling events with a model decision score higher than 0.9.

[0048] Regarding step S122, after receiving the query instruction, the system parses the "target query conditions" and converts them into a query statement for the "data table" of the structured audit log. Because the log is structured, each field (such as timestamp, vehicle VIN, and rule identifier) ​​has been standardized and indexed. The query engine (such as an SQL database or a dedicated log retrieval system) utilizes these indexes to quickly filter out the "target data" rows from the massive log records that fully satisfy all query conditions. Finally, the system assembles these matching records into a standard format (such as JSON or a list) and returns and displays it on the terminal interface that initiated the request.

[0049] Steps S121 and S122 transform the previously "black box" or complex script-based data cleaning process into a transparent, interactive, and user-friendly service. Through this mechanism, technical personnel can quickly respond to downstream data inquiries, accurately pinpoint rule conflicts or model misjudgments in the cleaning logic, thereby driving continuous optimization of the cleaning strategy. Simultaneously, it significantly improves the efficiency of problem investigation, reducing root cause analysis that might have taken hours to minutes, and provides business and auditing personnel with the ability to self-verify data reliability. Ultimately, it enhances the accuracy, reliability, and credibility of the entire data pipeline, meeting stringent data governance and compliance requirements.

[0050] Regarding step S13, based on a variety of preset quality indicators, the target quality indicator data of the target vehicle data is determined with respect to various quality indicators.

[0051] The preset quality indicators include at least: completeness indicators, accuracy indicators, consistency indicators, and timeliness indicators; the completeness indicators include at least the proportion of non-empty fields and the data packet loss rate; the accuracy indicators include at least the proportion of abnormal data; the consistency indicators include at least the number of contradictory data entries for each vehicle within the same time interval; and the timeliness indicators include at least the time from the generation of the original vehicle data to the completion of cleaning.

[0052] The integrity metric measures the completeness of data. Its core sub-metrics include "non-empty field percentage," which is the ratio of valid non-empty fields in a data record to the total number of fields, and "data packet loss rate," which is the ratio of the number of data packets lost during data transmission to the total number of packets that should be reported. For example, if 95 out of 100 CAN bus data packets that should be reported are successfully received, the loss rate is 5%. The accuracy metric assesses the degree to which data values ​​match the actual state. Its core sub-metric, "abnormal data percentage," can be calculated in two ways: one is the proportion of data records that are statistically deviated from the normal pattern, identified by a machine learning model; the other is the percentage of numerical data records that exceed a reasonable physical range (e.g., vehicle speed sensor reported values ​​greater than 300 km / h) out of the total number of records. Consistency metrics are used to examine logical inconsistencies within data and across data sources. A typical sub-metric is "the number of contradictory data entries for the same vehicle at the same time point." For example, at a certain timestamp, a vehicle reports a speed of 0 km / h, but its engine speed is as high as 5000 rpm. These two conflicting data points constitute a contradictory record. Timeliness metrics focus on the freshness of data and processing efficiency. Its main sub-metric, "the average delay from data generation to completion of cleaning," measures the average time it takes for data to be transmitted from its source (such as onboard sensors) until it is finally processed by the cleaning model. For example, this delay needs to be controlled within a few seconds to meet the requirements of real-time risk control.

[0053] Step S13 transforms the abstract concept of data quality into a series of calculable and measurable specific indicator data (i.e., target quality indicator data), thus freeing the effectiveness of data cleaning from subjective experience. This systematic evaluation accurately identifies potential weaknesses that may still exist after data cleaning (e.g., the cleaning process corrected formatting errors but failed to effectively reduce data latency), providing a clear, data-driven basis for subsequent optimization of cleaning models and rules. Step S13 achieves closed-loop monitoring and management of data cleaning quality, making the data quality status transparent and measurable. Ultimately, it ensures that the data flowing into downstream business systems (such as data analysis platforms and autonomous driving training models) is reliable, trustworthy, and meets timeliness requirements, fundamentally improving the robustness of data-driven business decisions and intelligent applications.

[0054] Regarding step S14, the cleaning quality of the target vehicle data is determined based on the target quality index data of various quality indicators of the target vehicle data.

[0055] The quality of data cleaning is objectively defined and judged by comparing the "target quality indicator data" calculated in step S13 with preset quality thresholds. Specifically, "good" cleaning quality typically means that all key quality indicators meet the expected standards. For example, the proportion of non-empty fields in the data integrity indicator is higher than 99.9%, the proportion of anomalous data in the accuracy indicator is lower than 0.1%, and all indicators are within their respective acceptable ranges. This means that the cleaned data is highly reliable and can be directly used downstream. Conversely, "bad" cleaning quality is manifested by one or more core indicators exceeding the tolerance limit. For example, the proportion of anomalous data in a micro-batch suddenly spikes to 5%, or the data packet loss rate reaches 15%. This clearly indicates deficiencies in the current data cleaning process or the emergence of new anomalous patterns in the data source, resulting in serious risks to the generated data. An immediate alert is needed to prevent its flow to subsequent business systems.

[0056] Step S14 involves a comprehensive assessment and interpretation of the quantitative indicators generated in step S13. This step is not simply a list of indicators, but rather a convergence analysis of all target quality indicator data based on preset business rules or intelligent decision-making models, thereby forming an actionable conclusion regarding the overall cleaning quality. For example, the system can be configured with a decision logic: the cleaning quality of this batch of data is judged as "excellent" or "pass" only when all four core indicators—completeness, accuracy, consistency, and timeliness—meet the standards; otherwise, the system will automatically determine the quality as "unqualified" based on specific unmet indicators (such as "timeliness exceeding the time limit") and trigger the corresponding early warning workflow. Step S14 elevates scattered, individual quality metrics into a holistic, driveable quality status signal, achieving automated and standardized acceptance of the data cleaning output quality.

[0057] In summary, this application's embodiments utilize a pre-trained model for data cleaning, not only efficiently outputting high-quality target data but also ensuring the consistency and intelligence of the cleaning process. Based on preset multi-dimensional quality indicators (such as completeness and accuracy), the cleaning results are quantitatively evaluated, generating objective and measurable target quality indicator data, making data quality transparent and visible. Finally, the cleaning quality is determined by comprehensively considering these indicator data, achieving accurate and automated acceptance of the data cleaning results. This application's embodiments fundamentally change the past situation of passively discovering data quality problems, enabling proactive and rapid evaluation of data cleaning quality. It can promptly intercept low-quality data before downstream business applications, effectively avoiding decision-making errors and business losses caused by dirty data, and improving the reliability and trustworthiness of the entire data pipeline.

[0058] Furthermore, based on the above solution, the embodiments of this application also provide the following optimization method.

[0059] After obtaining the original vehicle data to be cleaned, based on a variety of preset quality indicators, the original quality indicator data of the original vehicle data with respect to various quality indicators are determined. Based on the original quality index data of the original vehicle data with respect to various quality indicators, and the target quality index data of the target vehicle data with respect to various quality indicators, the cleaning quality of the target vehicle data is determined.

[0060] The process of determining the original quality index data of the original vehicle data is similar in method to the process of determining the target quality index data of the target vehicle data in step S13. Both are calculated based on the same preset quality index (such as completeness, accuracy, etc.), and the specific calculation process will not be described here.

[0061] The embodiments of this application first calculate the quality status of the original data before the cleaning operation, so as to clearly understand the specific problems and severity of the data before it is input into the cleaning model. This provides an indispensable basis for comparison for objectively and quantitatively measuring the actual effectiveness of the cleaning model.

[0062] Then, the final cleaning quality is determined by jointly analyzing the original quality indicator data and the target quality indicator data. Specifically, this process does not judge whether the target data "meets the standard" in isolation, but rather dynamically evaluates the quality improvement effect from "original" to "target". The system calculates the improvement value or improvement rate of key quality indicators. For example, if the proportion of abnormal data in the original data is 5%, and it decreases to 0.5% after cleaning, then the improvement rate is 90%. The final judgment of cleaning quality comprehensively considers the absolute quality level of the target data and its improvement relative to the original data.

[0063] As can be seen, the embodiments of this application achieve precise measurement of the effectiveness of data cleaning work, not only determining "whether the cleaned data is good or not," but also "whether the cleaning process is effective." This makes the evaluation results more accurately reflect the value of the cleaning model, provides clearer guidance for optimizing cleaning strategies (for example, focusing on optimizing indicators where improvement is not obvious), and also provides more convincing evidence of effectiveness for data governance, realizing the advancement from static result acceptance to dynamic performance evaluation.

[0064] Furthermore, after determining the target quality index data of the target vehicle data with respect to various quality indicators based on preset multiple quality indicators, the method further includes steps S21-S23.

[0065] Step S21: Classify the target quality index data according to various preset data dimensions; Step S22: For the target quality indicator data corresponding to each data dimension, determine the quality change trend of the target quality indicator data according to the time sequence of the target quality indicator data; Step S23: Visualize the quality change trends of the target quality indicator data corresponding to various data dimensions.

[0066] Regarding step S21, data dimensions refer to the different observation perspectives or grouping conditions used to classify and aggregate the target quality indicator data. These dimensions include at least: each vehicle (e.g., grouped by the vehicle's unique identifier VIN), each data type (e.g., distinguishing between different types such as CAN bus data, GPS positioning data, and sensor readings), and each administrative or geographical region (e.g., categorized by province, city, or region divided by road network). Step S21, after calculating the comprehensive target quality indicator data, does not treat it as a whole, but rather classifies and aggregates it according to these preset dimensions. For example, the system will calculate and summarize all data accuracy indicators for "Vehicle VIN_001", all completeness indicators for "GPS data type", and timeliness indicators for all vehicles in the "East China region". This step structures and organizes the mixed quality data, laying the foundation for subsequent refined trend analysis.

[0067] Regarding step S22, for each data dimension (such as a specific vehicle or a specific data type) classified in step S21, the system organizes the target quality indicator data (such as the percentage of abnormal data) according to time order (e.g., one data point per minute, hour, or day) to form a time series. Subsequently, the series is processed using time-series data analysis algorithms (such as moving average, trend line fitting, or abrupt change detection) to determine the quality change trend of the specific quality indicator under that dimension. For example, analysis may reveal that "the GPS data packet loss rate of vehicle VIN_002 has shown a slow upward trend over the past week," or "the consistency index of speed data for all vehicles showed a significant instantaneous deterioration after the system version upgrade."

[0068] Regarding step S23, visualization processing refers to the technical means of intuitively presenting abstract numerical data and analyzed trends through visual elements such as graphics and charts. Specific forms of the processing results include, but are not limited to: trend line charts (used to show changes in indicators over time), bar charts (used to compare quality differences between different dimensions), dashboards (used to intuitively display the current quality score or compliance status), and heatmaps (used to show the quality distribution across regional dimensions). Step S23 writes the quality change trend data corresponding to the various data dimensions obtained in step S22 into a dedicated time-series database for efficient storage. Subsequently, data visualization tools such as Grafana are used to read data from the time-series database and configure and build an interactive data quality monitoring dashboard. This dashboard can dynamically and in real-time display the quality change trends of each vehicle, data type, and region in rich chart formats, thereby transforming massive amounts of quality indicator data into easily understandable visual information, greatly improving monitoring efficiency and problem perception capabilities.

[0069] Furthermore, after determining the target quality index data of the target vehicle data with respect to various preset quality indicators, the method further includes steps S31-S35.

[0070] Step S31: Compare the target quality indicator data and historical quality indicator data corresponding to each quality indicator to determine whether there are any abnormalities in the target quality indicator data corresponding to each quality indicator. Step S32: When there is an anomaly in the target quality indicator data corresponding to at least one quality indicator, identify the anomaly type and anomaly level. Step S33: Generate target anomaly alarm information based on the anomaly type and the anomaly level; Step S34: Determine the target alarm method and target alarm device based on the anomaly type and the anomaly level; Step S35: Send the target abnormality alarm information to the target alarm device according to the target alarm method to remind the user to handle the abnormality.

[0071] The historical quality indicator data refers to any one of the following quality indicator data: Quality index data corresponding to any historical collection period prior to the collection period corresponding to the original vehicle data. The quality index data corresponding to the vehicle data after cleaning the data cleaning model of the previous historical version; Quality index data for vehicle models other than the original vehicle data mentioned above.

[0072] Regarding step S31, historical quality indicator data serves as benchmark data for comparative analysis with the current target quality indicator data. It can originate from three different contexts: first, quality indicator data calculated in any historical period prior to the current data collection period (e.g., the same time yesterday or the same day last week), used to identify periodic anomalies; second, quality indicator data obtained after processing the same or similar data by the previous historical version's data cleaning model, used to assess quality fluctuations resulting from model iteration upgrades; and third, quality indicator data generated by other vehicle models within the same time period, excluding the current vehicle model, used for cross-vehicle quality comparisons. Step S31 involves inputting these real-time calculated quality indicators along with the selected historical benchmark data into a preset threshold rule engine for comparison. This engine uses predefined rules (e.g., the current batch's "abnormal data percentage" increases sharply by more than 10 percentage points compared to the historical average, or the absolute value exceeds a 5% threshold) to determine if the current indicator deviates significantly, thereby triggering an abnormal state.

[0073] Regarding step S32, the anomaly type is classified according to the nature and anomaly pattern of the quality indicators, such as "sudden drop in data accuracy," "continuous deterioration in data integrity," or "regional data consistency conflict." The anomaly level is graded according to the severity of the anomaly and the scope of its impact on business, and can be divided into different levels such as "hint," "warning," and "critical." For example, a small fluctuation in the non-critical sensor data of a single vehicle can be classified as an "accuracy-hint" level anomaly; while a significant increase in the GPS data loss rate of an entire model series may be identified as a "integrity-critical" level anomaly. After the anomaly is determined in step S31, step S32 will comprehensively analyze which quality indicator(s) are abnormal, their anomaly pattern (such as sudden increase or gradual decrease), and the breadth of the anomaly's impact (such as a single vehicle or the entire fleet). Then, based on the preset classification and grading rule matrix, it will automatically match and output the specific anomaly type identifier and anomaly level code.

[0074] Regarding step S33, the target anomaly alarm information is a standardized message generated by the system to describe an abnormal event. It typically takes the form of a structured text message containing key information fields, such as "anomaly occurrence time," "anomaly indicator name," "affected data dimension (vehicle VIN / data type / region)," "anomaly type," "anomaly level," and "brief description." Step S33 selects or populates the corresponding template from a predefined alarm information template library based on the "anomaly type" and "anomaly level" output in step S32. For example, for a "critical" level "data packet loss rate anomaly," the system will automatically generate a target anomaly alarm message such as "[Critical Alarm] At {time}, the GPS data packet loss rate of vehicles in the East China region is as high as 15%, far exceeding the 5% threshold. Please investigate immediately!"

[0075] Regarding step S34, the target alarm method refers to the information push channel, including email, SMS, or messages sent via robot interfaces of office software such as DingTalk / WeChat. The target alarm device refers to the terminal or system receiving these alarm information, typically the mobile phones or duty computers held by the operations and maintenance team, or dedicated message groups configured by the data governance team. Step S34 queries the preset alarm policy routing table based on the identified "anomaly type" and "anomaly level." This routing table specifies the alarm methods and notification recipients for different levels and types of anomalies. For example, a "prompt" level anomaly may only be sent to the relevant data engineer via email; while a "critical" level anomaly will be simultaneously notified via SMS and DingTalk / WeChat robots, instantly notifying multiple devices of the operations and maintenance supervisor and the data governance team leader, ensuring that critical issues are responded to quickly.

[0076] Regarding step S35, after the target alarm method and target alarm device are determined in step S34, the corresponding message sending service (such as a mail server, SMS gateway, or API interface of a third-party IM platform) will be invoked to accurately and promptly deliver the structured target anomaly alarm information generated in step S33 to the target alarm device according to the specified method (such as immediate sending or non-repeated sending). This ensures that relevant operation and maintenance and data governance personnel can receive clear alarm notifications in the first instance, providing accurate information input for them to initiate subsequent problem investigation and handling processes.

[0077] As can be seen, the embodiments of this application achieve proactive, accurate, and efficient perception and response to data quality anomalies. By introducing multi-dimensional historical benchmarks for anomaly detection, the accuracy and foresight of alarms are improved, and false alarms are reduced. By defining anomaly types and levels and intelligently associating them with alarm methods and devices, precise routing and hierarchical push of alarm information are achieved, avoiding alarm storms and ensuring that critical issues are handled with high priority. Ultimately, this mechanism transforms the data quality management model from post-event remediation to in-process intervention and even pre-event warning, greatly shortening the discovery and response time of quality problems and effectively ensuring the stable operation of downstream business systems and the reliability of data products.

[0078] Furthermore, based on the aforementioned solution, the embodiments of this application also provide the following optimization solutions.

[0079] This application's embodiments define two conditions for triggering model training: one is abnormal triggering, which automatically starts when the system detects an anomaly in at least one quality indicator; the other is periodic triggering, which executes model training according to a preset time period (such as weekly or monthly). These two mechanisms complement each other, jointly ensuring that the data cleaning model can continuously adapt to data changes and maintain optimal performance.

[0080] Regarding the first type of exception triggering: If the target quality indicator data corresponding to at least one quality indicator is abnormal, the method further includes: The target quality index data that contains anomalies are stored as training samples; If the amount of data in the stored training samples meets the preset data volume condition, the data cleaning model is incrementally trained based on the stored training samples to obtain the updated data cleaning model. Alternatively, the data cleaning model is retrained based on the stored training samples, the original vehicle data within the preset historical time period, and the target vehicle data to obtain the updated data cleaning model.

[0081] In this embodiment, when the system detects an anomaly in at least one quality indicator, it automatically triggers a model optimization process. Specifically, the system stores these anomaly quality indicator data, which represent changes in the current data distribution or anomaly patterns, as training samples. When the accumulated number of samples meets preset conditions (e.g., reaching a certain number or covering a sufficient number of anomaly types), the system initiates a model update. There are two update methods: First, incremental training, which uses only newly collected anomaly samples to quickly fine-tune the existing model. This method consumes few computational resources, has a fast response speed, and enables the model to quickly adapt to the latest data changes, making it suitable for dealing with sudden or gradual new anomaly patterns. Second, retraining, which combines newly collected anomaly samples with large-scale raw and target vehicle data within a preset historical time period to comprehensively train the model from scratch. This method can more thoroughly learn data features, fundamentally improving the model's ability to handle complex or deep data quality problems and its generalization performance.

[0082] As can be seen, the embodiments of this application realize closed-loop optimization and adaptive evolution of the data cleaning model, directly transforming the problems discovered in the quality monitoring stage into fuel driving model iteration. Through incremental training, the system can react quickly to sudden data quality issues, maintaining the real-time effectiveness of the cleaning results; while through periodic retraining, it can ensure the continuous robustness and fundamental improvement of model performance. Ultimately, this mechanism enables the entire data cleaning system to continuously learn from actual operational feedback, significantly reducing the risk of model performance degradation due to changes in data patterns, reducing reliance on manual intervention, and thus ensuring and improving the upper limit of data cleaning quality in a long-term, automated manner.

[0083] Regarding the second type of periodic triggering: When the preset update cycle corresponding to the data cleaning model is reached, the method further includes: The data cleaning model is incrementally trained based on the stored target quality index data containing anomalies to obtain an updated data cleaning model; or, the data cleaning model is retrained based on the stored target quality index data containing anomalies, the original vehicle data within a preset historical period, and the target vehicle data to obtain an updated data cleaning model.

[0084] When the system reaches the preset fixed update cycle for the data cleaning model (e.g., once a month or once a quarter), it will automatically start the model retraining process regardless of whether any anomalies are recorded during that period. This training also provides two paths: First, incremental training, which uses all the accumulated abnormal quality indicator data within the cycle to fine-tune the model. This helps the system integrate various subtle data pattern changes discovered within a cycle, achieving smooth model evolution. Second, retraining, which, in addition to using accumulated abnormal data, also calls upon a larger amount of original vehicle data and cleaned target vehicle data from a preset historical period to perform comprehensive, zero-based training on the model. This effectively integrates data features from a longer period, fundamentally refreshing and improving the model's overall performance and generalization ability.

[0085] The periodic triggering mechanism enables systematic and preventative maintenance of the data cleaning model. It overcomes the blind spots that may exist when relying solely on anomaly triggers, ensuring that the model can be periodically "retrained" even in the absence of sudden severe anomalies, thus continuously adapting to the slow drift of the data environment and potential pattern changes. In this way, the system can proactively eliminate the natural degradation of model performance over time (i.e., technical debt), maintaining its advanced processing capabilities and robustness in the long term, ultimately constructing a highly automated data quality assurance system that combines real-time responsiveness with long-term stability.

[0086] Since the electronic device described in this embodiment is an electronic device used to implement the information processing method in the embodiments of this application, those skilled in the art can understand the specific implementation methods and various variations of the electronic device in this embodiment based on the information processing method described in the embodiments of this application. Therefore, how the electronic device implements the method in the embodiments of this application will not be described in detail here. Any electronic device used by those skilled in the art to implement the information processing method in the embodiments of this application falls within the scope of protection of this application.

[0087] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0089] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0090] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0091] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0092] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A data cleaning quality analysis method, characterized in that, The method includes: Obtain the original vehicle data to be cleaned; The original vehicle data is input into a pre-trained data cleaning model to obtain cleaned target vehicle data; Based on a variety of preset quality indicators, the target quality indicator data of the target vehicle data is determined with respect to various quality indicators; Based on the target vehicle data and its target quality index data for various quality indicators, the cleaning quality of the target vehicle data is determined.

2. The data cleaning quality analysis method as described in claim 1, characterized in that, After acquiring the original vehicle data to be cleaned, the method further includes: Based on a variety of preset quality indicators, the original quality indicator data of the original vehicle data with respect to various quality indicators are determined; Based on the original quality index data of the original vehicle data with respect to various quality indicators, and the target quality index data of the target vehicle data with respect to various quality indicators, the cleaning quality of the target vehicle data is determined.

3. The data cleaning quality analysis method as described in claim 1, characterized in that, After determining the target quality index data of the target vehicle data with respect to various preset quality indicators, the method further includes: The target quality index data are classified according to various preset data dimensions; For each data dimension, the target quality indicator data is used to determine the quality change trend of the target quality indicator data according to the time sequence of the target quality indicator data. The quality change trends of the target quality index data corresponding to various data dimensions are visualized respectively.

4. The data cleaning quality analysis method as described in claim 1, characterized in that, After determining the target quality index data of the target vehicle data with respect to various preset quality indicators, the method further includes: Compare the target quality indicator data corresponding to each quality indicator with the historical quality indicator data to determine whether there are any anomalies in the target quality indicator data corresponding to each quality indicator. When there is an anomaly in the target quality indicator data corresponding to at least one quality indicator, identify the anomaly type and anomaly level; Generate target anomaly alarm information based on the anomaly type and the anomaly level; The target alarm method and target alarm device are determined based on the anomaly type and the anomaly level; The target anomaly alarm information is sent to the target alarm device according to the target alarm method to remind the user to handle the anomaly.

5. The data cleaning quality analysis method as described in claim 4, characterized in that, The historical quality indicator data refers to any one of the following quality indicator data: Quality index data corresponding to any historical collection period prior to the collection period corresponding to the original vehicle data. The quality index data corresponding to the vehicle data after cleaning the data cleaning model of the previous historical version; Quality index data for vehicle models other than the original vehicle data mentioned above.

6. The data cleaning quality analysis method as described in claim 4, characterized in that, If the target quality indicator data corresponding to at least one quality indicator is abnormal, the method further includes: The target quality index data that contains anomalies are stored as training samples; If the amount of data in the stored training samples meets the preset data volume condition, the data cleaning model is incrementally trained based on the stored training samples to obtain the updated data cleaning model. Alternatively, the data cleaning model is retrained based on the stored training samples, the original vehicle data within the preset historical time period, and the target vehicle data to obtain the updated data cleaning model.

7. The data cleaning quality analysis method as described in claim 4, characterized in that, When the preset update cycle corresponding to the data cleaning model is reached, the method further includes: The data cleaning model is incrementally trained based on the stored target quality index data containing anomalies to obtain an updated data cleaning model; or, the data cleaning model is retrained based on the stored target quality index data containing anomalies, the original vehicle data within a preset historical period, and the target vehicle data to obtain an updated data cleaning model.

8. The data cleaning quality analysis method as described in claim 1, characterized in that, After acquiring the original vehicle data to be cleaned, the method further includes: The original vehicle data is input into a pre-trained data cleaning model to generate a structured audit log. The structured audit log includes data tracking identifiers, original values ​​before data cleaning, target values ​​after data cleaning, cleaning rule identifiers, model decision scores, cleaning rules, and cleaning time.

9. The data cleaning quality analysis method as described in claim 8, characterized in that, After generating structured audit logs, the method further includes: Receive a query command containing the target query conditions sent by the terminal; If the structured audit log contains data that meets the target query conditions, the target data in the structured audit log that matches the target query conditions is fed back to the terminal.

10. The data cleaning quality analysis method as described in claim 1, characterized in that, The preset quality indicators include at least: completeness indicators, accuracy indicators, consistency indicators, and timeliness indicators; The integrity metrics include at least the percentage of non-empty fields and the packet loss rate; The accuracy indicators include at least the percentage of abnormal data; The consistency index includes at least the number of contradictory data entries for each vehicle within the same time interval; The timeliness indicator includes at least the time from the generation of the original vehicle data to the completion of the cleaning process.

Images