A data quality evaluation method and system for traffic transportation public data resources

Abstract

This invention relates to the field of data quality assessment of public data resources in transportation, and discloses a method and system for data quality assessment of public data resources in transportation. It involves acquiring data from multiple sources, establishing a multi-dimensional quality evaluation index system for preliminary quality checks, and obtaining usable data. Behavioral analysis is performed on the usable data to identify potential anomalies, and machine learning models are used to classify the anomalies by pattern. This method introduces a heterogeneous data association analysis and causal inference mechanism based on preset association rules. It can obtain a set of association evidence from heterogeneous data sources with preset associations, and perform pattern alignment, entity parsing, and spatiotemporal normalization processing on this evidence. Then, it performs evidence aggregation analysis and causal inference to determine whether potential anomalies can be causally explained by external contextual factors. This mechanism effectively solves the problem of insufficient understanding of complex causal relationships and semantic associations between multi-source heterogeneous data in existing technologies.

Description

Technical Field

[0001] This application relates to the field of data quality assessment of public data resources in transportation, and more specifically, to a method and system for assessing the data quality of public data resources in transportation. Background Technology

[0002] As the transportation sector increasingly relies on digital information, massive amounts of public transportation data, such as road conditions, vehicle locations, and weather information, have become crucial for supporting intelligent transportation systems. This data is widely used in important scenarios such as traffic situation analysis, autonomous driving testing, and intelligent dispatching; therefore, the quality of the data directly affects the reliability and security of these application systems. However, during data collection and aggregation, due to differences in equipment, network instability, or incompatibility between systems, data loss, time-series errors, and even data content that contradicts reality or common sense often occur. If this low-quality data is used directly, it will seriously mislead traffic prediction and autonomous driving judgments, and may even lead to erroneous decisions.

[0003] Currently, traffic data quality management largely relies on manual inspection or simple numerical range judgments, making it difficult to effectively handle large-scale, real-time updated data. Existing methods often lack flexibility and in-depth analysis capabilities when facing the fusion of multi-source heterogeneous data. Traditional fixed rules are insufficient to cover the various complex dynamic changes in traffic operations, easily overlooking some subtle anomalies. For example, in a city traffic management center, the core data quality assessment system for public transportation data resources initially designed its "data consistency verification logic flow" for a single traffic management department within the city, primarily verifying data from the same source and with similar structure. This process can identify inconsistencies within a single data source, such as the simultaneous occurrence of abnormally high traffic flow peaks and extremely low average speeds.

[0004] However, with the advancement of urban smart transportation strategies, the system has begun to integrate cross-domain data from different departments such as public transportation, environmental monitoring, and road construction. These newly introduced data sources differ significantly from existing traffic data in terms of data models, time granularity, and geospatial reference standards. Existing "data consistency verification logic processes" are mainly based on simple numerical range judgments, enumerated value matching, or internal verification of single-source time series, which cannot effectively understand and handle the complex semantic relationships and cross-domain consistency between these new data sources. For example, it may be able to determine whether a vehicle speed value is within a reasonable range, but it cannot understand the potential indirect correlation between bus delays and decreased air quality on specific road sections.

[0005] In actual system operation, this lack of understanding of the causal relationships between cross-domain data leads to frequent misjudgments. For example, when congestion on a main road causes vehicle speeds to fall below a threshold and bus arrival times to be delayed, the existing system may isolate the bus delay as a "quality anomaly," ignoring the direct impact of external traffic conditions. Similarly, when an expressway is closed due to an accident, causing a sudden drop in traffic flow, the system may mark the traffic data as "missing" or "abnormal," rather than interpreting it as a normal state caused by an external event. These erroneous judgments result in the system frequently sending invalid alerts to data providers, consuming significant manpower for repeated verification, and causing departments to doubt the reliability of the assessment results, thus impacting their enthusiasm for data sharing.

[0006] Furthermore, the repeated execution of numerous invalid or inefficient verification tasks, coupled with the requirement to load and compare heterogeneous data from multiple sources with varying formats and semantics for each verification, results in prolonged high utilization of server CPU resources. This slows down overall assessment efficiency and delays the output of data quality assessment results. Genuine, deep-seated data quality issues are even more difficult to detect and address in a timely manner, as they are often buried under a large number of false positives. While operations personnel may attempt to mitigate the problem by adjusting local parameters, these adjustments often only address the symptoms and may even introduce new false positives, failing to address the core issue—the system's lack of intelligent understanding of the complex causal relationships and semantic connections between multi-source heterogeneous data.

[0007] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0008] This application discloses a data quality assessment method and system for public data resources in transportation, aiming to solve the problems that existing transportation data quality assessment methods lack flexibility and in-depth analysis capabilities when facing the fusion of multi-source heterogeneous data, making it difficult to effectively handle large-scale, real-time updated data, resulting in frequent misjudgments, low assessment efficiency, difficulty in discovering deep-seated data quality problems, and insufficient intelligent understanding of complex causal relationships and semantic associations between multi-source heterogeneous data.

[0009] The technical solution of this application is as follows:

[0010] Firstly, this application discloses a data quality assessment method for public data resources in the transportation sector, specifically including:

[0011] Data is acquired from public transportation data resources, which come from multiple data sources.

[0012] Establish a multi-dimensional quality evaluation index system, and perform quality checks on the acquired data based on this multi-dimensional quality evaluation index system to obtain usable data that meets the preset requirements;

[0013] Behavioral analysis of available data can identify potential anomalies where data behavior deviates from the norm.

[0014] A pre-defined machine learning model is used to identify anomaly patterns in potential anomalies, and anomaly pattern classification results are obtained.

[0015] Based on the preset association rules, for potential anomalies, related heterogeneous data is obtained from the preset associated heterogeneous data sources. The associated heterogeneous data is then processed by pattern alignment, entity parsing, and spatiotemporal normalization to generate a set of related evidence corresponding to the potential anomalies. Based on the set of related evidence, evidence aggregation analysis and causal reasoning are performed to determine whether the potential anomalies can be causally explained by external contextual factors.

[0016] Based on the results of the correlation analysis, potential anomalies are identified as data quality issues or fluctuations caused by external contextual factors;

[0017] For data identified as having data quality issues, generate processing suggestions including solutions and trigger the corresponding data repair mechanism;

[0018] The evaluation process is monitored in real time, and the multi-dimensional quality evaluation index system and processing suggestions are dynamically optimized.

[0019] Secondly, this application also discloses a data quality assessment system for public data resources in transportation, specifically including:

[0020] The data acquisition module is used to acquire data from public transportation data resources, which comes from multiple data sources.

[0021] The quality inspection module is used to establish a multi-dimensional quality evaluation index system, and to perform quality inspection on the acquired data based on this multi-dimensional quality evaluation index system to obtain usable data that meets the preset requirements.

[0022] The anomaly detection module is used to perform behavioral analysis on available data and identify potential anomalies where data behavior deviates from the normal pattern.

[0023] The pattern classification module is used to identify abnormal patterns of potential anomalies using a preset machine learning model, and obtain the abnormal pattern classification results.

[0024] The association analysis module is used to obtain associated heterogeneous data from preset association heterogeneous data sources for potential anomalies based on preset association rules, perform pattern alignment, entity parsing, and spatiotemporal normalization on the associated heterogeneous data, generate an association evidence set corresponding to the potential anomaly, and perform evidence aggregation analysis and causal reasoning based on the association evidence set to determine whether the potential anomaly can be causally explained by external contextual factors.

[0025] The anomaly detection module is used to identify potential anomalies as data quality issues or fluctuations caused by external contextual factors, based on the results of correlation analysis.

[0026] The suggested repair module is used to generate processing suggestions containing solutions for data that is determined to have data quality problems, and to trigger the corresponding data repair mechanism.

[0027] The monitoring and optimization module is used to monitor the evaluation process in real time and dynamically optimize the multi-dimensional quality evaluation index system, machine learning model, and processing suggestions.

[0028] This application discloses a data quality assessment method and system for public transportation data resources. It acquires data from multiple sources, establishes a multi-dimensional quality evaluation index system for preliminary quality checks, and obtains usable data. Based on this, behavioral analysis is performed on the usable data to identify potential anomalies, and machine learning models are used to classify these anomalies by pattern. Crucially, this method introduces a heterogeneous data association analysis and causal inference mechanism based on preset association rules. This mechanism can obtain a set of association evidence from heterogeneous data sources with preset associations, and perform pattern alignment, entity parsing, and spatiotemporal normalization on this evidence. Then, it performs evidence aggregation analysis and causal inference to determine whether potential anomalies can be causally explained by external contextual factors. This mechanism effectively solves the problem of insufficient understanding of complex causal relationships and semantic associations between multi-source heterogeneous data in existing technologies, avoiding misjudging fluctuations caused by external contexts as data quality problems. By distinguishing between data quality problems and external contextual fluctuations, this method can generate processing suggestions including solutions for genuine data quality problems and trigger corresponding data repair mechanisms. This avoids the problems of frequently sending invalid alarms and consuming a large amount of manpower for repeated verification in existing systems, thereby improving the enthusiasm for data sharing.

[0029] Furthermore, real-time monitoring of the evaluation process and dynamic optimization of the evaluation index system and processing suggestions have further improved the accuracy of the evaluation, reduced the false alarm rate and false negative rate, overcome the limitations of existing methods in effectively handling large-scale, real-time updated data, and solved the problems of high server resource utilization and low evaluation efficiency, enabling deep-seated data quality problems to be discovered and dealt with in a timely manner. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating a data quality assessment method for public data resources in transportation provided by an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of the structure of a data quality assessment system for public data resources in transportation provided by an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0033] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0034] A data quality assessment method for public data resources in transportation is proposed. Traditional existing data quality assessment methods for public data resources in transportation often lack flexibility and in-depth analysis capabilities when dealing with large-scale, real-time updated multi-source heterogeneous data. They are difficult to effectively identify and process complex dynamic changes, and are prone to missing inconspicuous anomalies, leading to frequent misjudgments and waste of resources. This makes it difficult to detect and deal with real deep-seated data quality problems in a timely manner.

[0035] In this regard, refer to Figure 1 , Figure 1 This is a flowchart illustrating a data quality assessment method for public transportation data resources provided in an embodiment of the present invention, including:

[0036] S11, Obtain data from public transportation data resources, the data coming from multiple data sources;

[0037] S12, establish a multi-dimensional quality evaluation index system, and perform quality checks on the acquired data based on the multi-dimensional quality evaluation index system to obtain usable data that meets preset requirements;

[0038] S13, Perform behavioral analysis on the available data to identify potential anomalies where the data behavior deviates from the normal pattern;

[0039] S14, an anomaly pattern is identified by a preset machine learning model to obtain an anomaly pattern classification result.

[0040] S15. Based on the preset association rules, for the potential anomaly, obtain associated heterogeneous data from the preset associated heterogeneous data source, perform pattern alignment, entity parsing, and spatiotemporal normalization processing on the associated heterogeneous data, generate an association evidence set corresponding to the potential anomaly, perform evidence aggregation analysis and causal reasoning based on the association evidence set, and determine whether the potential anomaly can be causally explained by external contextual factors.

[0041] S16, Based on the results of the correlation analysis, the potential anomalies are determined to be data quality problems or fluctuations caused by external contextual factors;

[0042] S17, For data determined to have the aforementioned data quality issues, generate processing suggestions containing solutions and trigger the corresponding data repair mechanism; monitor the evaluation process in real time and dynamically optimize the multi-dimensional quality evaluation index system and the processing suggestions.

[0043] This application aims to address the limitations of existing data quality assessment technologies by introducing a multi-dimensional quality evaluation index system, machine learning models for anomaly pattern recognition, causal reasoning based on association rules, and dynamic optimization mechanisms. It seeks to improve the accuracy and efficiency of assessment, reduce false alarms, and provide intelligent processing suggestions for data repair.

[0044] To better understand the method proposed in this application, some key terms involved will be explained first.

[0045] "Public data resources for transportation" refers to various data sets collected, managed, and released by government departments, public transportation operators, etc., to support the operation, management, and decision-making in the transportation sector. These data may include, but are not limited to, road traffic flow data, vehicle trajectory data, public transportation operation data, meteorological data, and road incident data.

[0046] A "multi-dimensional quality evaluation index system" refers to a set of indicators used to measure multiple different aspects of data quality, such as data completeness, consistency, accuracy, and timeliness. Each dimension may contain one or more specific metrics.

[0047] "Potential anomalies" refer to data points or sequences whose behavior deviates from the usual pattern and may be caused by data quality issues or external contextual factors.

[0048] "Preset machine learning model" refers to the algorithmic model used to identify and classify abnormal patterns in data during the data quality assessment process, such as isolated forest, autoencoder, gradient boosting model or graph neural network model.

[0049] "Heterogeneous data sources" refer to data sources with different data structures, formats, semantics, or storage methods, such as data from different departments or different sensors.

[0050] "Associated evidence set" refers to a set of data obtained from heterogeneous data sources and processed to explain potential anomalies, which can support or refute a certain hypothesis.

[0051] "External contextual factors" refer to reasons that may affect public transportation data, but are not due to problems with the quality of the data itself, such as weather changes, traffic accidents, road construction, etc.

[0052] A "data repair mechanism" refers to the process and tools used to correct and restore identified data quality issues.

[0053] The data quality assessment method for public transportation data resources proposed in this application is based on a series of collaborative steps to achieve a comprehensive and accurate assessment of data quality.

[0054] First, it's necessary to acquire data from public transportation data resources. This data may come from multiple sources, such as traffic monitoring cameras, GPS positioning devices, weather sensors, and public transportation card payment systems. Data acquisition can be achieved through various methods, including API calls, database synchronization, and file transfer. For example, a data collection service can be set up to periodically retrieve the latest traffic flow, vehicle locations, bus schedules, and real-time weather data from urban traffic management platforms, public transportation dispatch systems, and meteorological bureau data interfaces. After acquisition, this data typically undergoes preliminary format conversion and storage for subsequent processing.

[0055] Secondly, a multi-dimensional quality evaluation index system should be established, and quality checks should be performed on the acquired data based on this system to obtain usable data that meets preset requirements. This index system can include multiple dimensions such as completeness, consistency, accuracy, and timeliness. For example, completeness can measure the proportion of missing fields in data records; consistency can check whether the data conforms to preset logical rules, such as vehicle speed should not exceed road speed limits; accuracy can compare the deviation between the data and known benchmark values; and timeliness can assess the freshness of the data. In practice, corresponding weights and thresholds can be set for each index. For example, for traffic flow data, a threshold can be set; if the traffic flow data for a certain road segment is continuously missing for more than a certain period, its completeness is considered unqualified. Through these checks, data that clearly does not meet quality requirements can be initially screened out, resulting in relatively "clean" usable data.

[0056] Next, behavioral analysis is performed on the available data to identify potential anomalies where data behavior deviates from normal patterns. Behavioral analysis can be achieved through statistical methods, time series analysis, or pattern recognition techniques. For example, average vehicle speed, traffic flow, and other indicators for a specific road segment over a specific time period can be calculated and compared with historical data from the same period. If the current data significantly deviates from historical averages or trends, potential anomalies may exist. Alternatively, a sliding window can be used to monitor data in real time; when the data pattern within the window differs significantly from a preset normal pattern, it is marked as a potential anomaly. This analysis aims to discover data behaviors that do not conform to normal expectations, providing clues for further in-depth diagnosis.

[0057] Subsequently, a pre-defined machine learning model is used to identify anomaly patterns in potential anomalies, resulting in anomaly pattern classification. Machine learning models can learn normal patterns and known anomaly patterns from historical data, thereby classifying new potential anomalies. For example, a classification model can be trained to categorize potential anomalies into different types such as "sensor malfunction," "data transmission error," and "traffic congestion." This model could be an isolation forest model, which isolates anomalies by randomly selecting features and split points; it could be an autoencoder model, which identifies anomalies through reconstruction errors; or it could be a gradient boosting model or a graph neural network model, used to handle more complex spatiotemporal correlation data. By applying machine learning models, potential anomalies can be classified more finely, providing more accurate input for subsequent causal inference.

[0058] Furthermore, based on preset association rules, for potential anomalies, associated heterogeneous data is obtained from preset associated heterogeneous data sources. This associated heterogeneous data undergoes pattern alignment, entity parsing, and spatiotemporal normalization to generate an association evidence set corresponding to the potential anomaly. Based on this evidence set, evidence aggregation analysis and causal inference are performed to determine whether the potential anomaly can be causally explained by external contextual factors. For example, when traffic flow data for a certain road segment shows an abnormal decrease, the system can automatically query heterogeneous data sources for that road segment within the same time period, such as meteorological data (e.g., whether there was a rainstorm), road event data (e.g., whether there were traffic accidents or construction), and social media data (e.g., whether there were related discussions), according to preset association rules. The obtained heterogeneous data needs to undergo pattern alignment (e.g., unifying timestamps from different data sources), entity parsing (e.g., matching names referring to the same road segment in different data sources), and spatiotemporal normalization (e.g., unifying geographical location information under different coordinate systems), thereby generating an association evidence set containing multiple types of information. Subsequently, through evidence aggregation analysis and causal inference, such as using Bayesian networks or structural causal models, the causal relationship between this evidence and the potential anomaly is analyzed. If heavy rain or traffic accidents are found to be the direct cause of the decrease in traffic flow, it can be determined that the potential anomaly is a fluctuation caused by external contextual factors rather than a data quality issue.

[0059] Based on the results of the association analysis, potential anomalies are classified as either data quality issues or fluctuations caused by external contextual factors. If causal reasoning indicates that a potential anomaly is caused by external contextual factors, such as extreme weather or unforeseen events, then the anomaly will be labeled as "fluctuations caused by external contextual factors." Conversely, if no sufficient external contextual factor can be found to explain it, or if the evidence strongly points to a problem with the data itself, then the anomaly will be classified as "data quality issues." This step is crucial for distinguishing between problems with the data itself and the influence of the external environment, and helps to avoid false alarms.

[0060] For data identified as having quality issues, the system generates processing suggestions including solutions and triggers corresponding data repair mechanisms. For example, if a sensor is found to have missing data, the system can generate a suggestion to "inspect or replace the sensor" and trigger automatic interpolation repair or manual review. Processing suggestions can be customized based on the type, severity, and scope of the data quality issue. Data repair mechanisms can include automatic repair (such as interpolation, regression, and generative model repair), manual review, and re-collection. For example, for a small number of missing data points, interpolation algorithms can be used for automatic filling; for large-scale or complex data errors, manual intervention for review and correction, or even re-collection of data, may be necessary.

[0061] Finally, the evaluation process is monitored in real time, and the multi-dimensional quality evaluation index system and processing suggestions are dynamically optimized. During the operation of the data quality evaluation system, its performance is continuously monitored, including evaluation accuracy, false positive rate, and false negative rate. Based on this monitoring data, techniques such as reinforcement learning, multi-objective optimization, or Bayesian optimization can be used to dynamically adjust the index weights and thresholds in the multi-dimensional quality evaluation index system, the hyperparameters of the machine learning model, and the parameters for generating processing suggestions. For example, if the false positive rate of a certain index is found to be high, its weight or threshold can be adjusted appropriately; if a machine learning model performs poorly in identifying a specific type of anomaly, its hyperparameters can be optimized or new training data can be introduced. This dynamic optimization mechanism ensures that the evaluation system can adapt to the constantly changing data environment and business needs, continuously improving the accuracy and efficiency of the evaluation.

[0062] The data quality assessment method for public transportation data resources proposed in this application forms a complete and efficient closed loop for data quality assessment through the synergistic effect of the aforementioned technical features. First, multi-source data acquisition and multi-dimensional quality checks ensure the quality of the basic data for assessment. Subsequently, behavioral analysis and machine learning models identify and classify potential anomalies, improving the intelligence level of anomaly detection. Crucially, by introducing a causal reasoning mechanism based on association rules, this application can effectively distinguish between data quality problems and fluctuations caused by external contextual factors, avoiding misjudgments caused by a lack of understanding of complex causal relationships in traditional methods. For example, when traffic flow data shows anomalies, the system no longer simply marks it as "abnormal," but actively queries heterogeneous data such as weather and events to analyze whether external factors such as rainstorms or accidents have caused changes in traffic flow. If so, the anomaly is attributed to the external context, thus avoiding sending invalid alerts to the data provider. This causal explanation capability significantly reduces the false alarm rate and improves the reliability of the assessment results.

[0063] Furthermore, for data identified as having quality issues, the system can generate processing suggestions including solutions and trigger corresponding data repair mechanisms, achieving automation and intelligence from problem discovery to resolution. Finally, through real-time monitoring and dynamic optimization of the evaluation process, the entire evaluation system ensures continuous adaptation to the ever-changing data environment and business needs, constantly improving the accuracy and efficiency of the evaluation. This dynamic optimization mechanism enables the evaluation system to learn and evolve on its own. For example, when a high false alarm rate is detected for a certain indicator, the system automatically adjusts its weight or threshold, thus avoiding the tedious and inefficient process of frequently adjusting parameters manually.

[0064] In some embodiments described above, this application proposes acquiring associated heterogeneous data from pre-defined heterogeneous data sources based on preset association rules, performing pattern alignment, entity parsing, and spatiotemporal normalization to generate an association evidence set. This evidence set is then used for evidence aggregation analysis and causal inference to determine whether potential anomalies can be causally explained by external contextual factors. However, in practical applications, relying solely on preset association rules and acquired associated heterogeneous data may not adequately explain all complex and variable potential anomalies, especially when anomalies are caused by hidden, indirectly related external contextual factors. This could lead to misjudgment or insufficient explanation of the anomaly's cause. Failure to address these issues may reduce the accuracy of data quality assessment and affect the effectiveness of subsequent data repair mechanisms.

[0065] In this regard, this application further proposes that the steps for determining whether the aforementioned potential anomaly can be causally explained by external situational factors include:

[0066] If the preset association rules and the acquired heterogeneous association data cannot fully explain the potential anomaly, calculate the explanation gap between the potential anomaly and the acquired explanation;

[0067] Based on the aforementioned explanation gap, and combined with the geographical location, occurrence time, duration, and traffic flow characteristics of the potential anomaly, one or more hypotheses of potential hidden causes are matched and generated from a preset set of situational clues. The set of situational clues includes one or more of the following: local short-term showers, changes in road lighting, road construction debris, and corresponding traffic flow change patterns.

[0068] For the generated hypothesis, a query is initiated to an auxiliary data source to obtain evidence that supports or refutes the hypothesis. The auxiliary data source includes one or more of the following: high-resolution local meteorological data, vehicle behavior micro-feature data, road condition sensor data, and ambient light sensor data.

[0069] The obtained evidence is weighted and aggregated, and a confidence score for each hypothesis is calculated.

[0070] When the confidence score of a hypothesis exceeds a preset adoption threshold, the hypothesis is adopted as an explanation for the potential anomaly, and the potential anomaly is determined to be a fluctuation caused by external contextual factors.

[0071] Specifically, when the system finds that existing preset association rules and acquired heterogeneous association data are insufficient to provide a sufficient causal explanation for a potential anomaly, it first calculates the explanation gap between the potential anomaly and the acquired explanations. This explanation gap can be understood as the difference in information content or confidence level between the current explanation and the complete explanation. Its purpose is to quantify the inadequacy of existing explanations and provide a basis for subsequent in-depth analysis. For example, it can be calculated by comparing the deviation between the expected behavior of the potential anomaly and the actual observed behavior, or by assessing the coverage of the probability of anomaly occurrence by existing evidence.

[0072] Based on the calculated explanatory gap, the system combines contextual information such as the geographic location, occurrence time, duration, and traffic flow characteristics of potential anomalies to match and generate one or more hypotheses for potential hidden causes from a pre-defined set of contextual clues. The set of contextual clues is a predefined knowledge base containing various external contextual factors that may lead to traffic data anomalies, such as localized short-term showers, changes in road lighting, road construction debris, and specific traffic flow patterns. Through these contextual clues, potential, indirectly related external influencing factors can be systematically explored, thereby forming more comprehensive explanatory hypotheses.

[0073] In practical applications, for each generated hypothesis, the system queries auxiliary data sources to obtain evidence supporting or refuting that hypothesis. Auxiliary data sources are supplementary data sources beyond the initial heterogeneous data sources, containing more refined and targeted contextual information. For example, high-resolution local meteorological data can verify the short-duration shower hypothesis, vehicle behavior micro-feature data can reveal changes in driver behavior, road condition sensor data can confirm the impact of road construction debris, and ambient light sensor data can support the hypothesis of changes in road illumination. The acquisition of this evidence aims to provide an objective basis for verifying each hypothesis.

[0074] Furthermore, the acquired evidence is weighted and aggregated, and a confidence score is calculated for each hypothesis. The weighting of evidence can be based on factors such as the reliability of its source, its relevance to the hypothesis, and the timeliness of the evidence. Evidence aggregation involves integrating evidence from different auxiliary data sources to form a comprehensive support level for each hypothesis. The confidence score reflects the likelihood of each hypothesis being true, aiming to quantify the credibility of each potential hidden cause hypothesis.

[0075] Ultimately, when the confidence score of a hypothesis exceeds a preset adoption threshold, that hypothesis will be adopted as an explanation for the potential anomaly. This means the system considers the external contextual factor a reasonable cause of the potential anomaly, thus classifying the potential anomaly as a fluctuation caused by the external contextual factor, rather than a quality issue with the data itself. The adoption threshold needs to be adjusted based on the actual application scenario and the required tolerance for misjudgment.

[0076] This application's solution, by introducing the calculation of explanatory gaps, can identify the shortcomings of basic association analysis, thereby triggering a deeper causal reasoning mechanism. When the initial association rules and heterogeneous data cannot fully explain potential anomalies, the system is guided to actively explore more hidden external contextual factors by quantifying the explanatory gaps. Through the above technical solution, this application can effectively solve the problem of insufficient explanatory power of basic association analysis when facing complex or hidden external contextual factors. By introducing the quantification of explanatory gaps, hypothesis generation of contextual clue sets, and evidence querying and aggregation of auxiliary data sources, the comprehensiveness and accuracy of identifying potential anomaly causes are significantly improved. This solution enables the system to delve deeper into the true causes behind anomalies, avoiding misjudging fluctuations caused by external contextual factors as data quality problems due to insufficient information, thereby improving the accuracy of data quality assessment. In addition, by systematically exploring and verifying potential hidden causes, this application enhances the robustness of data quality assessment, making the assessment results more convincing and providing more accurate decision-making basis for subsequent data repair or contextual response.

[0077] In some preferred embodiments, suppose that on a specific section of an urban expressway, traffic flow data suddenly and persistently decreases significantly, and preliminary correlation analysis (e.g., querying historical traffic data and routine event records for that section) does not reveal any obvious anomalies or explanations. In this case, the system calculates the explanation gap between the current explanation and the actually observed decrease in traffic flow, finding a significant uncertainty. Based on this explanation gap, the system combines the geographical location of the section, the time of the anomaly (e.g., 2 PM to 3 PM), the duration (e.g., 30 minutes), and traffic flow characteristics (e.g., no significant decrease in vehicle speed but reduced traffic volume) to match and generate multiple hypotheses for potential hidden causes from a pre-defined set of contextual clues. For example, it might generate a hypothesis that "localized short-term showers caused drivers to avoid this section," or a hypothesis that "changes in road lighting caused abnormal sensor data collection," or even a hypothesis that "road construction debris caused vehicles to detour."

[0078] To address the hypothesis of "localized short-duration showers," the system will query auxiliary data sources (e.g., high-resolution local meteorological data platforms) to obtain precise rainfall and duration data for the road segment during the period of the anomaly. If the query results show that localized short-duration showers did indeed occur during that period, this evidence will support the hypothesis of "localized short-duration showers." Simultaneously, the system may also query vehicle behavior micro-feature data to observe whether any vehicles exhibit abnormal lane-changing or deceleration behavior near the road segment, further validating the hypothesis.

[0079] The acquired evidence is weighted and aggregated; for example, high-resolution meteorological data may be assigned a higher weight. Through calculation, if the confidence score of the "localized short-term showers" hypothesis exceeds a preset adoption threshold, the system will adopt the hypothesis and classify the traffic flow decline as a fluctuation caused by external situational factors (localized short-term showers) rather than a data quality issue. This avoids unnecessary adjustments to data that is actually normal but affected by external factors, improving the accuracy and efficiency of the assessment.

[0080] In some embodiments described above, this application proposes weighting and aggregating the acquired evidence and calculating a confidence score for each hypothesis. However, in practical applications, the data behavior of public transportation data resources may be influenced by various complex external contextual factors, leading to potential conflicts between acquired evidence or variations in the support of evidence depending on the specific traffic context. If these factors are not fully considered, simply weighting and aggregating the evidence may result in inaccurate interpretations of potential anomalies, thereby affecting the final determination of data quality issues or fluctuations caused by external contextual factors.

[0081] In response, this application further proposes the following steps for weighting and aggregating the acquired evidence and calculating the confidence score for each hypothesis:

[0082] Based on the geographical location, time of occurrence, and traffic flow characteristics of the potential anomalies, identify traffic situation parameters that affect the support of evidence;

[0083] Based on the identified traffic situation parameters, the weight values ​​of each type of evidence are adjusted using an objective weighting method;

[0084] Identify the degree of conflict between different pieces of evidence regarding the same hypothesis;

[0085] Based on one or more of the DS evidence theory or Bayesian inference, combined with one or more of the structural causal model, propensity score matching, and difference-in-differences method, the degree of evidence conflict identified is quantified, and the causal conflict and the relevance conflict in the degree of evidence conflict are distinguished.

[0086] Based on the adjusted evidence weights and the quantified degree of evidence conflict, a confidence score is calculated for each hypothesis, using either DS evidence theory or Bayesian inference.

[0087] Specifically, identifying traffic context parameters that influence the strength of evidence support means that the system analyzes the specific geographical location, time of occurrence, and traffic flow characteristics at the time of potential anomalies, such as traffic volume, speed, and congestion index, to determine how these contextual factors affect the strength of support for a specific hypothesis (such as "localized short-term showers caused the anomaly") from different types of evidence (such as meteorological data and sensor data). For example, on rainy days, meteorological data has a significantly enhanced explanatory power for traffic anomalies.

[0088] The objective weighting method, which adjusts the weight of each piece of evidence, can be understood as dynamically adjusting the importance of each piece of evidence in the aggregation process based on the identified traffic context parameters. Objective weighting methods, such as entropy weighting and CRITIC methods, can determine the weight based on the information content and variability of the evidence itself, avoiding subjective bias, and can finely adjust the weights by combining context parameters, so that evidence that is more explanatory in a specific context receives a higher weight.

[0089] In practical applications, identifying the degree of conflict between different pieces of evidence supporting the same hypothesis specifically refers to the system detecting contradictions or inconsistencies between evidence from different sources or of different types. For example, if road condition sensor data shows the road surface is dry, while local weather data shows it is raining, this constitutes a conflict of evidence.

[0090] Furthermore, based on one or more of Dempster-Shafer (DDS) evidence theory or Bayesian inference, and combined with one or more of structural causal models, propensity score matching, and difference-in-differences (DID), the degree of identified evidence conflict is quantified, and causal conflict is distinguished from correlation conflict. DDS provides a framework for handling uncertainty and imprecision, effectively integrating evidence from different sources and quantifying conflicts between them. Bayesian inference updates the posterior probability of the hypothesis through a probabilistic model. Combining structural causal models, propensity score matching, or DID allows for a deeper analysis of causal relationships between evidence, distinguishing between conflicts that are merely statistically correlated but lack causal connection (correlation conflicts) and conflicts where there is a genuine break or contradiction in the causal chain (causal conflicts), thus avoiding misjudging correlation as causation.

[0091] Therefore, based on the adjusted evidence weights and quantified degree of evidence conflict, a confidence score for each hypothesis is calculated using one or more of the DS evidence theory or Bayesian inference. This means that during evidence aggregation, not only are the original weights of the evidence considered, but they are also dynamically adjusted according to the context, and the conflicts between the evidence are quantified and differentiated, ultimately resulting in a more accurate and reliable confidence score that reflects the explanatory strength of each hypothesis for potential anomalies.

[0092] Specifically, this application selects at least one path, the DS evidence theory, and provides the entire core computational chain of causal reasoning in the order of basic probability assignment construction → Dempster combination → conflict coefficient K quantification → confidence level Bel, so as to make the computational chain explicit and executable.

[0093] (1) Construction of Basic Probability Assignment (BPA). All possible hidden cause hypotheses are used as elements in the identification framework. For each piece of evidence, the context weight of the evidence is obtained by combining the objective weighting method (entropy weighting method or CRITIC method) with traffic context parameters. Then, the comprehensive credibility of the evidence is obtained by reliability assessment and internal consistency check, and the support of the evidence for each hypothesis is given. The basic probability assignment on each hypothesis is the number obtained by multiplying these three. The sum of the assignments on all hypotheses and the identification framework as a whole is always equal to one. The remaining quality that cannot be directly attributed to any single hypothesis is uniformly assigned to the identification framework as a whole.

[0094] (2) Dempster combination. The basic probability assignments of all evidence are orthogonally merged pairwise. Each time the evidence is merged, the probabilities of two pieces of evidence supporting a certain subset are added together and normalized with a normalization coefficient (the value of which is equal to one minus the total conflict between the two pieces of evidence) so that the sum of the assignments on all non-empty subsets after the merge is still one, and the assignment on the empty set is zero. This process is repeated until all evidence is merged, resulting in a basic probability assignment that represents the combined effect of all evidence.

[0095] (3) Conflict coefficient K. The conflict between two pieces of evidence is represented by the conflict coefficient K, which is equal to the sum of the products of the two pieces of evidence when the probabilities are assigned to mutually exclusive subsets. The larger K is, the more contradictory the evidence is. In this application, when K is not less than 0.6 (preferably 0.6, with a range of 0.5 to 0.7), it is determined that there is a strong conflict between the two pieces of evidence. It is necessary to further distinguish whether the conflict is a causal conflict or is caused only by correlation in combination with the structural causal model, and to process the joint assignment accordingly.

[0096] (4) Confidence level Bel and final adoption criterion. The confidence level of each hypothesis is given by the Belief function, which is equal to the sum of the assignments in the joint assignment that fall on all subsets inside the hypothesis. The corresponding irrefutability is equal to the sum of the assignments in the joint assignment that intersect with the hypothesis. To facilitate cross-hypothesis comparison, the difference between the confidence level and the irrefutability is further used as the final score. When the difference is greater than 0.3 (preferably 0.3, with a range of 0.2 to 0.4) and the confidence level Bel is not less than 0.5, the corresponding hypothesis is judged as a reasonable explanation for the potential anomaly and is adopted. The above difference and Bel together constitute the confidence score of each hypothesis in this application, which fully covers the core calculation link of causal reasoning: explanation gap → evidence aggregation → conflict quantification → confidence score.

[0097] This application's solution introduces the identification of traffic context parameters, enabling the weighting of evidence to be dynamically adjusted based on the specific context of potential anomalies. This ensures that evidence more supportive of the anomaly explanation plays a greater role in different situations. Simultaneously, by identifying and quantifying the degree of conflict between different pieces of evidence, particularly distinguishing between causal and correlational conflicts, this application avoids misjudgments caused by simply aggregating contradictory evidence. The application of advanced analytical tools such as DS evidence theory, Bayesian inference, and structural causal models makes the evidence fusion process more scientific and rigorous, effectively handling uncertainty and inconsistency, thereby more accurately assessing the confidence level of each potential hidden cause hypothesis in complex and ever-changing traffic environments.

[0098] Through the aforementioned technical solutions, this application can significantly improve the accuracy and reliability of identifying potential anomalies. By dynamically adjusting evidence weights to adapt to changes in traffic scenarios and refining the handling of evidence conflicts, misjudgments caused by insufficient, redundant, or contradictory evidence can be effectively avoided, making the distinction between data quality issues and fluctuations caused by external contextual factors more accurate. This not only improves the intelligence level of data quality assessment but also provides a more solid and reliable decision-making basis for subsequent data repair or situational response.

[0099] In some preferred embodiments, a specific example is given below. Suppose that on a certain section of a highway, traffic flow data suddenly shows an abnormal decrease in vehicle speed, but no obvious traffic accidents or congestion are detected. The system will first identify this as a potential anomaly using the method described above.

[0100] Next, the system identifies traffic context parameters that may affect the strength of the evidence, based on the potential anomaly's geographical location (e.g., a mountainous section of road), time of occurrence (e.g., 5 a.m.), and traffic flow characteristics (e.g., low traffic volume). For example, mountainous sections of road may be at risk of fog or icy surfaces in the early morning, and these context parameters dynamically increase the evidentiary weight of meteorological data (such as visibility and temperature) and road condition sensor data (such as friction coefficient).

[0101] The system then retrieves evidence from auxiliary data sources. For example, if high-resolution local meteorological data shows light fog on the road section, but road condition sensor data indicates a dry road surface and a temperature above freezing, the system will identify a potential conflict between the meteorological data (light fog) and the road condition sensor data (dry road surface).

[0102] Furthermore, the system quantifies this conflict based on the DS evidence theory and a structural causal model. For example, by analyzing historical data, it is found that light fog usually does not lead to dry roads unless accompanied by other meteorological conditions. The system will distinguish that this may be a correlation conflict (light fog and dry roads are not directly contradictory causally, but may just be sensor error or local microclimate differences), rather than a strong causal conflict.

[0103] Finally, based on the adjusted evidence weights (with increased weights for meteorological data and road condition sensor data due to contextual parameters) and the quantified degree of evidence conflict, the system calculates a confidence score for the hypothesis that "light fog causes a decrease in vehicle speed." If this score exceeds a preset threshold, the hypothesis is adopted, and the abnormal vehicle speed is classified as a fluctuation caused by external contextual factors (light fog), rather than a data quality issue. In this way, the proposed solution can more precisely and accurately assess the true cause of potential anomalies.

[0104] In some embodiments of this application described above, when weighting and aggregating the acquired evidence to calculate the confidence score for each hypothesis, it is necessary to identify the degree of evidence conflict between different pieces of evidence regarding the same hypothesis. However, if the reliability and internal consistency of the evidence are not pre-assessed, and conflicts of different intensities and types are not distinguished, the accuracy of evidence aggregation may decrease, or even a hypothesis may be incorrectly adopted or rejected, thereby affecting the final determination of potential anomalies.

[0105] In this regard, this application further proposes a method for identifying the degree of conflict between different pieces of evidence regarding the same hypothesis, the steps of which include:

[0106] The reliability of each source of evidence is assessed to obtain an initial reliability level;

[0107] Based on the degree of deviation between the parameter value indicated by each piece of evidence and the preset range, an internal consistency check is performed on each piece of evidence to obtain the internal consistency check result;

[0108] Based on the results of the internal consistency check, adjust the current reliability level of each type of evidence;

[0109] Prioritize comparison of evidence with a higher current reliability level, and identify strong conflicts between such evidence with a higher current reliability level;

[0110] Compare evidence with a lower current reliability level with evidence with a higher current reliability level to identify questionable evidence;

[0111] The historical fluctuation patterns and recent abnormal records of the questionable evidence were queried to obtain the query results;

[0112] Based on the query results, a weak conflict is determined between the evidence with the lower current reliability level and the evidence with the higher current reliability level.

[0113] Specifically, conducting a reliability assessment of each source of acquired evidence involves assigning an initial reliability level to each piece of evidence based on pre-defined reliability standards according to the source attributes (e.g., official sensor data, third-party reports, crowdsourced data, etc.). For example, sensor data deployed by traffic management departments is typically assigned a higher initial reliability level, while user reports on social media may be assigned a lower initial reliability level. The purpose is to provide a basic trust level reference for subsequent identification of evidence conflicts.

[0114] Internal consistency checks are performed on each piece of evidence based on the degree of deviation of the parameter values ​​indicated by each type of evidence from the preset range. This involves checking the data quality of the evidence itself. For example, if a sensor reports a temperature value far exceeding the physically possible range, or experiences drastic and unreasonable fluctuations within a short period, its internal consistency is considered poor. The result of the internal consistency check can be a Boolean value (consistent / inconsistent) or a quantified deviation score. Its purpose is to discover inherent contradictions or anomalies within the evidence itself, providing a basis for adjusting its reliability level.

[0115] In practical applications, adjusting the current reliability level of each piece of evidence based on internal consistency check results means revising its initial reliability level according to its internal consistency performance. For example, if a piece of evidence with high initial reliability is found to have serious problems during an internal consistency check, its current reliability level will be downgraded; conversely, if a piece of evidence with low initial reliability exhibits extremely high consistency, its level may be moderately upgraded. The aim is to make the reliability assessment of evidence more dynamic and accurate, reflecting its current actual quality status.

[0116] Furthermore, prioritizing the comparison of evidence with a higher current level of reliability, and identifying strong conflicts among evidence with a higher current level of reliability, means that when dealing with conflicts of evidence, priority should be given to those pieces of evidence considered more credible. Strong conflicts typically refer to direct and irreconcilable contradictions between these highly reliable pieces of evidence. For example, two highly reliable sensors may simultaneously report traffic flow on the same road segment, but the values ​​differ significantly and exceed the reasonable margin of error. The aim is to quickly identify major contradictions in the core chain of evidence, which often point to deeper issues.

[0117] Furthermore, comparing evidence with a lower current reliability level with evidence with a higher current reliability level to identify questionable evidence involves comparing less reliable evidence with evidence already considered relatively reliable after addressing conflicts among highly reliable evidence. If there is a significant difference between the less reliable and highly reliable evidence, the less reliable evidence is marked as questionable. The purpose is to filter out evidence that may contain errors or be misleading, thus preventing it from negatively impacting the final judgment.

[0118] Specifically, querying historical fluctuation patterns and recent anomalies of questionable evidence to obtain query results refers to conducting a background investigation on evidence marked as questionable. For example, one can query the data source's data fluctuations over a past period, whether there have been any anomaly reports, and whether its data acquisition equipment has ever malfunctioned. The purpose is to provide more contextual information to determine the authenticity of questionable evidence.

[0119] Therefore, determining a weak conflict between evidence with a lower current reliability level and evidence with a higher current reliability level, based on the query results, refers to judging whether the difference between the questionable evidence and the highly reliable evidence is due to data quality issues, occasional fluctuations, or other reasons, based on the background investigation results. A weak conflict typically refers to a difference that exists but can be reconciled through some explanation (such as data delay, minor local differences, etc.), or whose impact is insufficient to completely refute the highly reliable evidence. Its purpose is to finely distinguish between evidentiary conflicts of different intensities, providing more accurate input for subsequent evidence aggregation and causal inference.

[0120] This application's solution effectively addresses the problem of inaccurate conflict identification caused by the diversity and varying quality of evidence sources during evidence aggregation by introducing a multi-stage, hierarchical evidence conflict identification mechanism. Through this technical solution, the application significantly improves the accuracy and robustness of evidence aggregation analysis in the assessment of public transportation data resources. Specifically, by conducting hierarchical reliability assessments and internal consistency checks on evidence, conflicts between different pieces of evidence can be identified and quantified more accurately, avoiding misjudgments caused by inconsistent evidence quality. This refined conflict identification mechanism allows for a more accurate reflection of the true support of evidence when calculating hypothesis confidence scores, thereby improving the accuracy of determining whether potential anomalies are data quality issues or fluctuations caused by external contextual factors. Furthermore, by distinguishing between strong and weak conflicts and conducting in-depth analysis of questionable evidence, this application can more comprehensively utilize all available evidence, reducing information loss and making the final causal inference results more convincing, providing a more reliable basis for data repair and system optimization.

[0121] In some preferred embodiments, a specific example is given below. Suppose that in a certain traffic segment, the system identifies a potential anomaly: the real-time vehicle speed data for this segment suddenly drops significantly, far below the historical average, but traffic flow does not increase significantly. To explain this anomaly, the system generates a hypothesis: "A localized, short-lived rain shower occurred in this segment, causing vehicles to slow down."

[0122] At this point, the system obtained multiple pieces of evidence from the auxiliary data source:

[0123] 1. Evidence A: Data from a high-resolution weather sensor near the road section shows that there were indeed short-term showers in the area during the period when the anomaly occurred. The sensor is maintained by an official agency and has a high initial reliability level.

[0124] 2. Evidence B: User reports from a third-party traffic app indicate that it was raining on this section of the road. The initial reliability level of this data source is moderate.

[0125] 3. Evidence C: Data from road condition sensors indicates that the road surface is slippery. These sensors were deployed by a private company and initially had a low level of reliability.

[0126] 4. Evidence D: Data from another official weather station, located far from the road section, shows no rainfall in the area. This weather station has a high initial reliability rating.

[0127] First, a reliability assessment is performed on each source of evidence to obtain an initial reliability level. For example, evidence A and D are initially rated as "high" reliability, evidence B as "medium" reliability, and evidence C as "low" reliability.

[0128] Secondly, an internal consistency check was performed on each type of evidence. Assuming that the parameter values ​​of evidence A and D are both within a reasonable range, their internal consistency is good. The user report in evidence B is clear and its internal consistency is good. Evidence C's road condition sensor data showed a brief malfunction before the anomaly occurred, and the internal consistency check results indicate fluctuations.

[0129] Based on the internal consistency check results, the current reliability level of each piece of evidence is adjusted. The current reliability levels of evidence A, B, and D remain unchanged or are slightly improved. Due to internal consistency issues, the current reliability level of evidence C is downgraded to "very low".

[0130] Next, evidence with a higher current reliability level is prioritized for comparison. There is a strong conflict between Evidence A (high reliability, showing rainfall) and Evidence D (high reliability, showing no rainfall). The system further analyzes the geographical scope, temporal precision, etc., between the two, finding that Evidence A is local high-resolution data, while Evidence D is regional data, which may have local differences.

[0131] Then, evidence with a lower current reliability level is compared with evidence with a higher current reliability level to identify questionable evidence. Evidence B (medium reliability, user reports rainfall) differs from Evidence D (high reliability, shows no rainfall), so Evidence B is marked as questionable. Evidence C (very low reliability, slippery road surface) is consistent with Evidence A (high reliability, shows rainfall), but its own reliability is extremely low, so it is also marked as questionable.

[0132] Finally, the historical fluctuation patterns and recent anomaly records of the questionable evidence were examined. For evidence B, the investigation revealed that user reports from the traffic app were generally highly timely for localized events, and there were no recent anomaly reports. For evidence C, the investigation found that the sensor had recently experienced multiple instances of data drift or brief malfunctions.

[0133] Based on the query results, the conflict type was determined. The conflict between evidence A and D was determined to be a weak conflict (local vs. regional) due to differences in geographical scope. The conflict between evidence B and D, considering their historical performance, was determined to be a weak conflict where evidence B supports a local event. The conflict between evidence C and evidence A, despite being identical in content, was significantly weakened in its support due to evidence C's extremely low reliability and anomalous records, and was determined to be a weak conflict caused by data quality issues.

[0134] Through this layered and refined conflict identification, the system can more accurately assess the confidence level of the hypothesis of "local short-term showers" and ultimately determine that the potential anomaly is a fluctuation caused by external contextual factors (local showers) rather than a data quality issue.

[0135] Specifically, in the steps of weighting and aggregating the acquired evidence and calculating the confidence score of each hypothesis, the step of identifying traffic context parameters that affect the support of evidence based on the geographical location, occurrence time, and traffic flow characteristics of the potential anomaly can be further refined.

[0136] The steps for identifying traffic context parameters that influence the strength of the evidence include:

[0137] Identify the geographical location, occurrence time, and traffic flow characteristics of the potential anomalies to obtain a set of traffic scenario parameters;

[0138] Query auxiliary data related to the set of traffic scenario parameters;

[0139] Analyze the interaction patterns of each parameter in the traffic scenario parameter set;

[0140] Based on the interaction pattern, contextual influence factors are generated using one or more of graph attention networks or interaction information entropy.

[0141] Based on the aforementioned contextual influence factors, traffic contextual parameters that influence the support of the evidence are identified.

[0142] Specifically, the first step is to identify and extract the geographical location, time of occurrence, and traffic flow characteristics of potential anomalies, thereby constructing a set of traffic context parameters containing this key information. For example, geographical location may include latitude and longitude coordinates or specific road segment markers, the time of occurrence may be accurate to the hour or even the minute, and traffic flow characteristics may cover traffic volume, vehicle speed, congestion index, etc.

[0143] Subsequently, the system will query preset auxiliary data sources for the set of traffic scenario parameters. These auxiliary data sources may include, but are not limited to, historical traffic data, real-time weather data, road event records, and holiday information, aiming to obtain supplementary information related to the current set of traffic scenario parameters to enrich the understanding of the background of potential anomalies.

[0144] Furthermore, an in-depth analysis of the interaction patterns among the parameters in the traffic context parameter set is conducted. For example, under specific geographical locations and times, traffic flow characteristics may be significantly affected by meteorological conditions (such as rainfall), or traffic flow patterns may exhibit regular changes during specific holidays. Analyzing these interaction patterns helps to reveal the intrinsic connections between different contextual factors.

[0145] Based on this, and according to the interaction patterns obtained from the analysis, this application uses a graph attention network to generate contextual influence factors. Specifically, the implementation is as follows: First, each traffic context parameter is used as a graph node. Directed graph edges are established based on the preset relationships between the parameters (including temporal relationships between different time periods on the same road segment, spatial relationships between adjacent road segments, and causal relationships between weather and traffic flow) to construct a traffic context graph. The traffic context graph is then input into the graph attention network, which consists of two stacked graph attention layers, each containing four parallel attention heads. The hidden layer dimension is 64, and the output dimension is 1. ReLU activation is used between layers, and the attention weights are calculated through a linear layer activated by LeakyReLU and normalized by Softmax. Through multi-round feature aggregation and attention weighting of the graph attention network, a scalar contextual influence factor score is obtained for each graph node. A higher score indicates a greater influence of the contextual parameter on the evidence support.

[0146] Finally, based on the generated contextual impact factors, traffic contextual parameters that significantly influence the strength of the evidence were identified. This means that not all contextual parameters are equally important in explaining potential anomalies; by quantifying the contextual impact factors, those parameters that are truly critical and can effectively support or refute anomaly hypotheses can be selected.

[0147] This application's solution systematically identifies and analyzes contextual parameters such as the geographical location, occurrence time, and traffic flow characteristics of potential anomalies, and further explores the interaction patterns between these parameters, enabling a more precise understanding of the background of anomaly occurrences. By introducing advanced technologies such as graph attention networks or interactive information entropy to generate contextual influence factors, the quantification and evaluation of complex contextual factors become possible. This effectively identifies traffic contextual parameters that have a key impact on the support of evidence, providing a more refined and objective basis for subsequent evidence weighting, avoiding simply treating all contextual parameters equally, and improving the accuracy and reliability of evidence evaluation.

[0148] Through the above technical solution, this application overcomes the potential blindness and one-sidedness of traditional methods in identifying context parameters, achieving refined identification and quantification of traffic context parameters. This not only helps to more accurately assess the degree of support of different pieces of evidence for potential anomaly hypotheses, but also improves the accuracy and robustness of the entire data quality assessment system in causal explanations of anomalies in complex traffic situations, thus providing a more solid foundation for the determination and remediation of data quality problems.

[0149] In some embodiments of this application described above, after performing behavioral analysis on available data and identifying potential anomalies that deviate from normal patterns, a preset machine learning model is needed to perform anomaly pattern recognition on the potential anomalies to obtain anomaly pattern classification results. Specifically, the step of using a preset machine learning model to perform anomaly pattern recognition on the potential anomalies and obtain anomaly pattern classification results may include the following:

[0150] Construct a training sample set containing known anomaly patterns, wherein the training sample set is labeled with one or more of the following tags: anomaly type, anomaly severity, and anomaly impact range;

[0151] A preset machine learning model is trained based on the training sample set. The preset machine learning model is one or more of the following: Isolation Forest model, Autoencoder model, Gradient Boosting model, and Graph Neural Network model.

[0152] The potential anomalies are input into the preset machine learning model to obtain anomaly pattern classification results, which include one or more of the following: anomaly type, anomaly severity, and anomaly impact range.

[0153] When the potential anomaly belongs to an anomaly pattern not covered by the training sample set, the potential anomaly is cold-started to be identified through one or more of transfer learning, meta-learning, and zero-shot learning to obtain the anomaly pattern classification result.

[0154] Specifically, before performing anomaly pattern recognition, a training sample set containing known anomaly patterns needs to be constructed. This training sample set is used to guide the machine learning model in learning and recognizing different anomaly patterns. Each sample in the training sample set should be labeled with detailed information, such as the type of anomaly (e.g., missing data, duplicate data, data deviation), the severity of the anomaly (e.g., minor, moderate, severe), and the potential scope of the anomaly's impact (e.g., specific region, specific time period, specific business process). This labeling information is crucial for the model to accurately learn and classify anomalies.

[0155] Furthermore, based on the constructed training sample set, a pre-defined machine learning model can be trained. This application selects at least the following two core algorithms and specifies their structure and parameters: (1) An autoencoder model for anomaly detection—the autoencoder model is a symmetrical encoder-decoder structure, including an input layer, a first fully connected encoder layer, a second fully connected encoder layer, a latent space layer, a first fully connected decoder layer, a second fully connected decoder layer, and an output layer connected in sequence. The input feature dimension of the input layer is 32, corresponding to standardized traffic flow indicators such as traffic volume, vehicle speed, occupancy rate, and travel time. The number of neurons in the first fully connected encoder layer, the second fully connected encoder layer, the latent space layer, the first fully connected decoder layer, and the second fully connected decoder layer are 64, 32, 16, 32, and 64, respectively. The output dimension of the output layer is 32. The ReLU activation function is used between each layer, and the loss function is the mean squared error reconstruction loss. The Adam optimizer (learning rate 1e-3, batch size 256, training epochs 100) is used to input the potential anomalies into the trained autoencoder model. The mean squared error reconstruction loss is used as the anomaly score. Samples with reconstruction errors greater than a preset threshold (the default value is 3 times the standard deviation of the mean reconstruction error of the training set) are judged as anomalies; (2) The gradient boosting model for anomaly pattern classification is implemented using XGBoost. It consists of 200 decision trees with a maximum depth of 6, a learning rate of 0.05, a subsampling ratio of 0.8, and a feature subsampling ratio of 0.8. The input features are 32-dimensional feature vectors (including autoencoder latent space representation, original statistical features, and contextual influence factors). The output is a three-class label for anomaly type, anomaly severity, and anomaly impact range.

[0156] Subsequently, the identified potential anomalies are input into a pre-trained machine learning model. Specifically, the potential anomalies are first standardized using z-scores, and then fed into a trained autoencoder model to obtain a 32-dimensional reconstruction vector and the corresponding reconstruction error. When the reconstruction error exceeds a preset threshold, it is determined to be an anomaly and input into the gradient boosting model to obtain the anomaly pattern classification result, which includes one or more of the following: anomaly type (e.g., sensor failure, data transmission error, traffic congestion, equipment aging, external interference), anomaly severity (mild / medium / high), and anomaly impact range (single point / road segment / region). These classification results provide important basis for subsequent data quality problem handling.

[0157] Furthermore, considering the possibility of encountering entirely new anomalous patterns not covered by the training sample set in practical applications, this application also proposes a cold-start identification mechanism. When a potential anomaly does not belong to a known anomalous pattern in the training sample set, it can be identified by introducing one or more techniques from transfer learning, meta-learning, or zero-shot learning. Transfer learning allows the model to apply knowledge learned in one domain to another related domain; meta-learning enables the model to quickly adapt to new tasks or patterns; and zero-shot learning allows the model to identify categories it has never seen before. Through these techniques, even when faced with unknown anomalous patterns, cold-start identification can be effectively performed, and corresponding anomalous pattern classification results can be obtained.

[0158] The solution proposed in this application constructs a training sample set containing rich label information and trains various selectable machine learning models based on this sample set, thereby enabling accurate anomaly pattern identification of potential anomalies in public transportation data resources. Through the above technical solution, this application can significantly improve the accuracy and coverage of anomaly pattern identification in the data quality assessment of public transportation data resources.

[0159] In some embodiments of this application, for data determined to have data quality issues, it is necessary to generate processing suggestions containing solutions and trigger a corresponding data repair mechanism. Specifically, the steps of generating processing suggestions containing solutions and triggering a corresponding data repair mechanism for data determined to have data quality issues include:

[0160] Based on the type of data quality problem, a preset repair strategy is matched based on one or more of case reasoning, rule matching, and machine learning classification. The preset repair strategy includes one or more of automatic repair strategy, manual review strategy, and re-collection strategy. The automatic repair strategy includes one or more of interpolation repair, regression repair, and generative model repair.

[0161] Generate treatment recommendations containing solutions based on the remediation strategy;

[0162] Based on the processing suggestions, one or more of the following operations are performed on the data corresponding to the data quality problem: automatic repair, manual review triggering, and re-collection, and the repair results are verified to meet the preset quality threshold.

[0163] Specifically, once the system identifies a data quality issue, it first categorizes it according to its specific type. For example, data quality issues might manifest as missing values, outliers, inconsistent data, or insufficient timeliness. For these different types of issues, the system employs an intelligent matching mechanism to determine the most appropriate remediation strategy. This matching mechanism can be based on case-based reasoning, recommending remediation solutions by comparing them with historical cases of handling similar data quality issues; it can also be based on pre-defined rules, such as triggering interpolation rules if a field is missing; or it can utilize a machine learning classification model, which, after training, can automatically classify data quality issues based on their characteristics and recommend corresponding remediation strategies.

[0164] The matched repair strategies are pre-defined and diverse to address data quality issues of varying complexity and impact. These strategies can include automatic repair strategies, manual review strategies, and re-acquisition strategies. Automatic repair strategies aim to automatically correct data using algorithms. For example, for missing data, interpolation (such as linear or polynomial interpolation) can be used to fill in missing values; for outlier data, regression repair can be used to predict and correct outliers by establishing regression relationships between data; or generative model repair can be used, utilizing deep learning and other generative models to generate reasonable values ​​that conform to the data distribution. Manual review strategies are suitable for complex, sensitive, or high-risk data problems requiring expert intervention for judgment and correction. Re-acquisition strategies are used when data is severely corrupted, unrepairable, or too costly to repair, involving re-acquiring data from the source.

[0165] Once a remediation strategy is determined, the system generates specific and actionable processing recommendations based on that strategy. These recommendations detail the remediation methods to be taken, the scope of data involved, and the expected remediation results. Subsequently, based on these processing recommendations, the system will perform corresponding remediation operations on the data corresponding to the data quality issues. This may include automatically performing interpolation, regression, or generative model repair, sending a manual review request to the data administrator, or triggering a re-collection process of the data source. After the remediation operations are completed, the system will also rigorously verify the remediation results to ensure that the remediated data meets the preset quality thresholds, thereby guaranteeing that the data is restored to a usable state.

[0166] The solution proposed in this application achieves accurate location and efficient processing of data quality problems by intelligently identifying and classifying the types of data quality problems and matching them with diverse preset repair strategies.

[0167] Through the aforementioned technical solutions, this application can flexibly select and apply the most suitable repair strategy based on the specific type of data quality problem, thereby improving the targeting and effectiveness of data repair. Diverse repair strategies, including automatic repair, manual review, and re-collection, enable the system to handle data quality problems of varying complexity and impact, avoiding the inefficiency or errors that may result from a "one-size-fits-all" approach. In particular, the introduction of automatic repair strategies, such as interpolation repair, regression repair, and generative model repair, greatly enhances the automation level and efficiency of data repair, reducing the cost of manual intervention. Simultaneously, the verification mechanism for repair results ensures that the repaired data meets preset quality requirements, effectively guaranteeing the accuracy, integrity, and availability of public transportation data resources, providing a high-quality foundation for subsequent data applications.

[0168] In some of the embodiments described above in this application, real-time monitoring of the evaluation process and dynamic optimization of the multi-dimensional quality evaluation index system and processing suggestions are proposed. However, in the implementation process, if a refined monitoring and adaptive adjustment mechanism is lacking, the performance of the evaluation system may decline over time, or it may be unable to effectively cope with new data patterns and abnormal situations, thereby affecting the accuracy and efficiency of data quality evaluation.

[0169] In response, this application further proposes specific steps for real-time monitoring of the evaluation process and dynamic optimization of the multi-dimensional quality evaluation index system and processing suggestions, so as to ensure the continuous and efficient operation and self-improvement of the evaluation system.

[0170] The steps mentioned above for real-time monitoring of the evaluation process and dynamic optimization of the multi-dimensional quality evaluation index system and processing suggestions include:

[0171] Continuously monitor the operational status of the quality assessment process to obtain monitoring data;

[0172] Calculate one or more evaluation indicators, such as accuracy rate, false alarm rate, and false negative rate, based on the monitoring data.

[0173] Based on the evaluation metrics, the weights and thresholds of the multi-dimensional quality evaluation metric system, the hyperparameters of the machine learning model, and the generation parameters of the processing suggestions are dynamically adjusted through one or more of reinforcement learning, multi-objective optimization, and Bayesian optimization, so as to maximize the evaluation accuracy and minimize the false positive rate and the false negative rate, and cross-domain knowledge transfer is performed on the machine learning model based on the evaluation metrics.

[0174] Specifically, continuous monitoring of the quality assessment process involves constantly tracking and recording the real-time performance, resource consumption, processing latency, data throughput, and output results of each module at each stage, including data acquisition, quality inspection, anomaly identification, pattern classification, correlation analysis, anomaly determination, and suggested remediation. This generates comprehensive monitoring data, which reflects the performance of the assessment system in actual operation.

[0175] The evaluation metrics, including accuracy, false positive rate, and false negative rate, are calculated based on monitoring data to quantify the system's performance in identifying data quality issues and external contextual factors. The accuracy rate measures the proportion of correctly identified data; the false positive rate indicates the proportion of normal data misclassified as problematic data; and the false negative rate indicates the proportion of problematic data that is not identified. These metrics are key parameters for measuring the effectiveness of the evaluation system.

[0176] Furthermore, based on the aforementioned evaluation metrics, one or more advanced optimization techniques, such as reinforcement learning, multi-objective optimization, or Bayesian optimization, can be employed to adaptively adjust the core parameters of the evaluation system. For example, the weights and thresholds of the multi-dimensional quality evaluation index system can be dynamically adjusted according to the actual evaluation results to better adapt to the quality requirements of different types of data; the hyperparameters of the machine learning model (such as learning rate, regularization coefficient, etc.) can be optimized to improve the accuracy and generalization ability of abnormal pattern recognition; the generation parameters of the processing suggestions (such as the priority of the repair strategy selection, repair threshold, etc.) can also be adjusted to ensure that the generated suggestions are more targeted and effective. The goal of these adjustments is to maximize the evaluation accuracy while minimizing the false positive rate and the false negative rate, thereby achieving an overall improvement in evaluation performance.

[0177] Furthermore, cross-domain knowledge transfer of machine learning models based on evaluation metrics refers to the process where, when an evaluation system performs well in a specific transportation data domain (such as highway data), the knowledge and patterns it has learned can be applied to other related but slightly different domains (such as urban road data or public transportation data) through techniques such as transfer learning. This accelerates model training in the new domain, improves its initial performance, and enhances the model's generalization ability and robustness.

[0178] This application's solution effectively addresses the inefficiencies and lack of adaptability inherent in dynamic optimization within basic solutions by introducing a refined real-time monitoring and adaptive optimization mechanism. Building upon this, the application utilizes advanced algorithms such as reinforcement learning, multi-objective optimization, or Bayesian optimization to use these evaluation metrics as feedback signals. This drives dynamic adjustments to the weights and thresholds of the multi-dimensional quality evaluation metric system, the hyperparameters of the machine learning model, and the parameters for generating processing suggestions. For example, when the false positive rate is high, the optimization algorithm may adjust the thresholds of certain quality metrics or the hyperparameters of the machine learning model to make it more cautious in judging anomalies; when the false negative rate is high, it may adjust the parameters to improve sensitivity to potential anomalies. This feedback-based adaptive adjustment mechanism enables the evaluation system to continuously learn and improve based on actual operating results, thereby avoiding performance bottlenecks and insufficient adaptability to new scenarios that may result from static configuration.

[0179] Furthermore, by performing cross-domain knowledge transfer on machine learning models based on evaluation metrics, the proposed solution can apply effective knowledge obtained from validated domains to new or data-scarce domains, significantly improving the initial performance and generalization ability of the model in new environments, reducing the time and resources required for training from scratch, thereby enhancing the robustness and applicability of the entire evaluation system.

[0180] Through the aforementioned technical solution, this application enables continuous and intelligent optimization of the quality assessment process for public transportation data resources. Specifically, the solution utilizes real-time monitoring and a feedback mechanism based on quantitative indicators, allowing the assessment system to dynamically adjust its internal parameters. This significantly improves the accuracy of data quality assessment and effectively reduces false positive and false negative rates. Furthermore, the introduction of reinforcement learning, multi-objective optimization, or Bayesian optimization techniques endows the system with strong adaptive capabilities, enabling it to proactively learn and adapt to constantly changing data characteristics and anomaly patterns, ensuring the long-term stability and efficiency of assessment performance. Particularly noteworthy is that through cross-domain knowledge transfer, the solution can rapidly apply existing assessment experience and model capabilities to new data scenarios, greatly enhancing the system's generalization ability and deployment efficiency. This allows it to provide high-quality and efficient data quality assessment services even when faced with diverse transportation data.

[0181] In some preferred embodiments, a specific example is given below. Suppose a city traffic management department has deployed the data quality assessment system of this application to monitor public transportation data from multiple data sources, such as intelligent traffic lights, bus GPS, and shared bicycle trajectories.

[0182] First, the system continuously monitors its operational status, such as the latency of the data processing module, the CPU utilization of the anomaly detection module, and the speed at which repair suggestions are generated, and collects this monitoring data. Simultaneously, the system periodically or after specific events calculates and evaluates the accuracy rate (e.g., the consistency between data quality issues identified by the system and those manually reviewed), the false alarm rate (e.g., the proportion of normal data misclassified as anomalies), and the false negative rate (e.g., the proportion of data quality issues that are not actually identified) based on this monitoring data. For example, if the system finds that traffic flow data in a certain area is frequently falsely reported as anomaly at night (increased false alarm rate), when in fact this may be a normal fluctuation due to lower traffic volume at night, the system will use a Bayesian optimization algorithm to automatically adjust the threshold of the "traffic flow data consistency" indicator in the multi-dimensional quality evaluation indicator system based on historical false alarm data and evaluation indicators, or adjust the hyperparameters of the machine learning model in the anomaly detection module to make its judgment on low traffic volume data more lenient at night, thereby reducing the false alarm rate.

[0183] For example, when the system processes new shared bicycle trajectory data, it finds that the accuracy of its anomaly pattern recognition is not high. In this case, the system can use transfer learning techniques to transfer some of the knowledge from an anomaly detection model trained on bus GPS data to the anomaly detection task for shared bicycle trajectory data. Specifically, some of the lower-level feature extraction layers of the bus GPS model can be frozen, and only the top-level classifier can be fine-tuned, thereby quickly improving the performance of shared bicycle trajectory anomaly pattern recognition and reducing the amount of data and time required for training from scratch.

[0184] Through continuous monitoring, indicator calculation, intelligent optimization, and knowledge transfer, this traffic data quality assessment system can continuously improve itself, ensuring that it maintains a high level of data quality assessment capability in a dynamically changing traffic environment.

[0185] Specifically, the steps of establishing a multi-dimensional quality evaluation index system, performing quality checks on the acquired data based on the multi-dimensional quality evaluation index system, and obtaining usable data that meets preset requirements can be further refined into the following process.

[0186] Establish a multi-dimensional quality evaluation index system that includes one or more dimensions such as completeness, consistency, accuracy, and timeliness. Completeness is measured by the ratio of the number of missing fields to the total number of fields; consistency is measured by the ratio of the number of records that violate preset logical constraints to the total number of records; accuracy is measured by the ratio of the absolute value of the deviation between the measured value and the benchmark value to the benchmark value; and timeliness is measured by the ratio of the deviation between the data timestamp and the current time to the preset time window.

[0187] Set weights and thresholds for each indicator in the multi-dimensional quality evaluation index system;

[0188] Based on the multi-dimensional quality evaluation index system, the weights and thresholds of each index, the acquired data is subjected to quality checks to obtain usable data that meets the preset requirements.

[0189] The multi-dimensional quality evaluation index system aims to comprehensively measure the quality of public data resources for transportation from multiple key perspectives.

[0190] Specifically, the completeness dimension focuses on whether there are missing values ​​in data records. It is measured by calculating the ratio of the number of missing fields to the total number of fields to quantify the completeness of the data; the smaller the ratio, the more complete the data. The consistency dimension focuses on whether the data conforms to preset business rules and logical constraints. It is measured by calculating the ratio of the number of records that violate preset logical constraints to the total number of records to assess data consistency; the smaller the ratio, the better the data consistency. The accuracy dimension measures how well the data matches real-world conditions. It is measured by calculating the ratio of the absolute value of the deviation between the measured value and the benchmark value to the benchmark value to reflect the accuracy of the data; the smaller the ratio, the more accurate the data. The timeliness dimension focuses on the freshness and timeliness of the data. It is measured by calculating the ratio of the deviation of the data timestamp from the current time to a preset time window to assess the timeliness of the data; the smaller the ratio, the more timely the data.

[0191] Furthermore, to enable the multi-dimensional quality evaluation index system to flexibly adapt to different business scenarios and data characteristics, it is necessary to set corresponding weights and thresholds for each index in the system. Weights reflect the relative importance of different quality dimensions in the overall evaluation. For example, in some scenarios, the timeliness of data may be more critical than its completeness; in this case, timeliness can be given a higher weight. Thresholds are used to set the pass / fail standards for each index. When the measurement result of an index exceeds a preset threshold, the data is considered to have a quality problem in that dimension.

[0192] Therefore, during actual quality checks, the system will conduct a comprehensive quality assessment of the acquired public transportation data resources based on the established multi-dimensional quality evaluation index system, the weights of each index, and the thresholds. Specifically, for each data record, the system will calculate its score in each dimension, such as completeness, consistency, accuracy, and timeliness, and then perform a weighted summation based on preset weights to obtain a comprehensive quality score (using linear weighting, i.e., the sum of the products of each dimension's score and its corresponding weight; the weights of each index are determined using the Analytic Hierarchy Process (AHP) or the entropy weight method). Subsequently, the comprehensive quality score will be compared with a preset overall quality threshold, or each dimension's score will be compared with its respective threshold, to determine whether the data meets the preset requirements. Only data that passes all quality checks and meets the preset standards will be deemed usable data and enter the subsequent processing flow.

[0193] This application's solution, by clearly defining the composition of a multi-dimensional quality evaluation indicator system and its specific measurement methods, ensures that data quality assessment is no longer a vague judgment based on a single dimension, but rather a quantitative analysis from multiple key perspectives such as completeness, consistency, accuracy, and timeliness. By setting weights and thresholds for each indicator, the importance of different quality dimensions in the overall assessment can be flexibly adjusted according to actual business needs and data characteristics, and standards for data qualification can be set. Therefore, during quality checks, the acquired data can be systematically and objectively screened based on these quantitative indicators and preset standards, efficiently identifying and filtering out unqualified data, ensuring that subsequent data processing and analysis use only high-quality, usable data.

[0194] refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of a data quality assessment system for public transportation data resources provided in an embodiment of the present invention, including:

[0195] The data acquisition module is used to acquire data from public transportation data resources, the data of which comes from multiple data sources;

[0196] The quality inspection module is used to establish a multi-dimensional quality evaluation index system, and to perform quality inspection on the acquired data based on the multi-dimensional quality evaluation index system to obtain usable data that meets preset requirements.

[0197] Anomaly detection module is used to perform behavioral analysis on the available data and identify potential anomalies in data behavior that deviate from the normal pattern.

[0198] The pattern classification module is used to perform anomaly pattern recognition on the potential anomalies using a preset machine learning model, and obtain anomaly pattern classification results.

[0199] The association analysis module is used to obtain associated heterogeneous data from a preset associated heterogeneous data source for the potential anomaly according to preset association rules, perform pattern alignment, entity parsing, and spatiotemporal normalization processing on the associated heterogeneous data, generate an association evidence set corresponding to the potential anomaly, and perform evidence aggregation analysis and causal reasoning based on the association evidence set to determine whether the potential anomaly can be causally explained by external contextual factors.

[0200] The anomaly detection module is used to determine the potential anomalies as data quality issues or fluctuations caused by external contextual factors based on the results of the correlation analysis.

[0201] The suggested repair module is used to generate processing suggestions containing solutions for data that is determined to have the aforementioned data quality problem, and to trigger the corresponding data repair mechanism;

[0202] The monitoring and optimization module is used to monitor the evaluation process in real time and dynamically optimize the multi-dimensional quality evaluation index system, the machine learning model, and the processing suggestions.

[0203] The data quality assessment system for public transportation data resources proposed in this application aims to address the shortcomings of traditional data quality assessment methods. These methods lack flexibility and in-depth analytical capabilities when dealing with large-scale, real-time updated, multi-source heterogeneous data. They struggle to effectively identify and handle complex dynamic changes, easily overlook subtle anomalies, leading to frequent misjudgments and resource waste, and making it difficult to promptly detect and address truly deep-seated data quality issues. This system forms a complete and efficient closed-loop data quality assessment system through the collaborative work of modules including data acquisition, quality inspection, anomaly identification, pattern classification, correlation analysis, anomaly determination, suggested repair, and monitoring optimization.

[0204] This system can achieve a comprehensive and accurate assessment of data quality, improve the accuracy and efficiency of the assessment, reduce false alarms, and provide intelligent processing suggestions for data repair. At the same time, it can deeply analyze the root causes and impact propagation paths of data quality problems.

[0205] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for assessing the data quality of public data resources in transportation, characterized in that, include: Data is acquired from public data resources related to transportation, and the data comes from multiple data sources. A multi-dimensional quality evaluation index system is established, and the acquired data is subjected to quality checks based on the multi-dimensional quality evaluation index system to obtain usable data that meets preset requirements. Behavioral analysis is performed on the available data to identify potential anomalies where the data behavior deviates from the normal pattern; An anomaly pattern is identified by a pre-defined machine learning model to obtain an anomaly pattern classification result. Based on preset association rules, for the potential anomaly, associated heterogeneous data is obtained from preset associated heterogeneous data sources. The associated heterogeneous data is then processed by pattern alignment, entity parsing, and spatiotemporal normalization to generate an association evidence set corresponding to the potential anomaly. Based on the association evidence set, evidence aggregation analysis and causal reasoning are performed to determine whether the potential anomaly can be causally explained by external contextual factors. Based on the results of the correlation analysis, the potential anomalies are identified as data quality issues or fluctuations caused by external contextual factors. For data identified as having the aforementioned data quality issues, a processing suggestion containing solutions is generated, and the corresponding data repair mechanism is triggered. The evaluation process is monitored in real time, and the multi-dimensional quality evaluation index system, the machine learning model, and the processing suggestions are dynamically optimized.

2. The method according to claim 1, characterized in that, The steps of obtaining associated heterogeneous data from a pre-defined heterogeneous data source based on preset association rules for the potential anomaly, performing pattern alignment, entity parsing, and spatiotemporal normalization on the associated heterogeneous data to generate an association evidence set corresponding to the potential anomaly, and performing evidence aggregation analysis and causal reasoning based on the association evidence set to determine whether the potential anomaly can be causally explained by external contextual factors include: If the preset association rules and the acquired heterogeneous association data cannot fully explain the potential anomaly, calculate the explanation gap between the potential anomaly and the acquired explanation; Based on the aforementioned explanation gap, and combined with the geographical location, occurrence time, duration, and traffic flow characteristics of the potential anomaly, one or more hypotheses of potential hidden causes are matched and generated from a preset set of situational clues. The set of situational clues includes one or more of the following: local short-term showers, changes in road lighting, road construction debris, and corresponding traffic flow change patterns. For the generated hypothesis, a query is initiated to an auxiliary data source to obtain evidence that supports or refutes the hypothesis. The auxiliary data source includes one or more of the following: high-resolution local meteorological data, vehicle behavior micro-feature data, road condition sensor data, and ambient light sensor data. The obtained evidence is weighted and aggregated, and a confidence score for each hypothesis is calculated. When the confidence score of a hypothesis exceeds a preset adoption threshold, ranging from 0.70 to 0.85, the hypothesis is adopted as an explanation for the potential anomaly, and the potential anomaly is determined to be a fluctuation caused by external contextual factors.

3. The method according to claim 2, characterized in that, The steps of weighting and aggregating the acquired evidence and calculating the confidence score for each hypothesis include: Based on the geographical location, time of occurrence, and traffic flow characteristics of the potential anomalies, identify traffic situation parameters that affect the support of evidence; Based on the identified traffic situation parameters, the weight values ​​of each piece of evidence are adjusted using an objective weighting method, which includes the entropy weighting method or the CRITIC method. Identify the degree of conflict between different pieces of evidence regarding the same hypothesis; Based on one or more of the DS evidence theory or Bayesian inference, combined with a structural causal model, the degree of evidence conflict identified is quantified, and causal conflict and relevance conflict are distinguished in the degree of evidence conflict. Based on the adjusted evidence weights and quantified evidence conflict levels, a confidence score for each hypothesis is calculated using one or more of the Dempster evidence theory or Bayesian inference. The confidence score calculation based on the Dempster evidence theory involves the following steps: a) Assigning a basic probability value to each piece of evidence for each hypothesis, the value of which is the product of the evidence's contextual weight, overall credibility, and support for that hypothesis, and assigning the remaining quality to the overall identification framework; b) Orthogonally merging the basic probability values ​​of all evidence pairwise according to the Dempster combination rule to obtain a combined basic probability value for all evidence; c) Calculating the conflict coefficient K between any two pieces of evidence, where a larger K indicates greater contradiction between the evidence, and normalizing the combined value using the value of K; d) Calculating the confidence score Bel for each hypothesis from the combined value, which is equal to the sum of the values ​​assigned to all subsets within the hypothesis, and obtaining the confidence score for each hypothesis.

4. The method according to claim 3, characterized in that, The step of identifying the degree of evidence conflict between different pieces of evidence regarding the same hypothesis includes: The reliability of each source of evidence is assessed to obtain an initial reliability level, which is divided into three levels: high, medium and low. High corresponds to officially deployed and calibrated sensor data, medium corresponds to third-party platform data, and low corresponds to crowdsourced or uncalibrated data. Based on the degree of deviation between the parameter value indicated by each piece of evidence and the preset range, an internal consistency check is performed on each piece of evidence to obtain the internal consistency check result; Based on the results of the internal consistency check, adjust the current reliability level of each type of evidence; Prioritize comparison of evidence with a higher current reliability level, and identify strong conflicts between such evidence with a higher current reliability level; Compare evidence with a lower current reliability level with evidence with a higher current reliability level to identify questionable evidence; The historical fluctuation patterns and recent abnormal records of the questionable evidence were queried to obtain the query results; Based on the query results, a weak conflict is determined between the evidence with the lower current reliability level and the evidence with the higher current reliability level.

5. The method according to claim 3, characterized in that, The step of identifying traffic situation parameters that affect the strength of evidence support based on the geographical location, time of occurrence, and traffic flow characteristics of the potential anomalies includes: Identify the geographical location, occurrence time, and traffic flow characteristics of the potential anomalies to obtain a set of traffic scenario parameters; Query auxiliary data related to the set of traffic scenario parameters; Analyze the interaction patterns of each parameter in the traffic scenario parameter set; Based on the interaction mode, each traffic scenario parameter is constructed as a graph node, and graph edges are established according to the preset correlation between each parameter to generate a traffic scenario graph. The traffic scenario graph is then input into a preset graph attention network, which includes two graph attention layers, each containing four attention heads, with a hidden layer dimension of 64 and an output dimension of 1. The ReLU activation function is used between each layer, and the features of each graph node are aggregated in multiple rounds according to the attention weights to obtain the scenario influence factor score of each graph node. Based on the contextual influence factor, traffic contextual parameters that affect the support of the evidence are identified.

6. The method according to claim 1, characterized in that, The step of using a preset machine learning model to identify anomaly patterns in the potential anomalies and obtaining anomaly pattern classification results includes: Construct a training sample set containing known anomaly patterns, wherein the training sample set is labeled with one or more of the following tags: anomaly type, anomaly severity, and anomaly impact range; A preset machine learning model is trained based on the training sample set; wherein, the preset machine learning model includes an autoencoder model for anomaly detection and a gradient boosting model for anomaly pattern classification; the autoencoder model includes an input layer, a first fully connected encoding layer, a second fully connected encoding layer, a latent space layer, a first fully connected decoding layer, a second fully connected decoding layer, and an output layer connected in sequence, wherein the input feature dimension of the input layer is 32, and the first fully connected encoding layer, the second fully connected encoding layer, the latent space layer, the first fully connected decoding layer, and the second fully connected decoding layer are connected in sequence. The number of neurons is 64, 32, 16, 32, and 64 respectively, and the output dimension of the output layer is 32. ReLU activation is used between layers, and the loss function is mean squared error reconstruction loss. The potential anomalies are input into the autoencoder model, and the mean squared error reconstruction loss is used as the anomaly score. The gradient boosting model includes 200 decision trees with a maximum depth of 6, a learning rate of 0.05, a subsampling ratio of 0.8, and a feature subsampling ratio of 0.

8. The input features are 32-dimensional feature vectors, and the output is a three-class label for anomaly type, anomaly severity, and anomaly impact range. The potential anomalies are input into the preset machine learning model to obtain anomaly pattern classification results, which include one or more of the following: anomaly type, anomaly severity, and anomaly impact range. When the potential anomaly belongs to an anomaly pattern not covered by the training sample set, the potential anomaly is cold-started to be identified through one or more of transfer learning, meta-learning, and zero-shot learning to obtain the anomaly pattern classification result.

7. The method according to claim 1, characterized in that, The steps of generating processing suggestions containing solutions for data identified as having data quality issues, and triggering the corresponding data repair mechanism, include: Based on the type of data quality problem, a preset repair strategy is matched based on one or more of case reasoning, rule matching, and machine learning classification. The preset repair strategy includes one or more of automatic repair strategy, manual review strategy, and re-collection strategy. The automatic repair strategy includes one or more of interpolation repair, regression repair, and generative model repair. Based on the repair strategy, generate processing recommendations containing solutions; Based on the processing suggestions, one or more of the following operations are performed on the data corresponding to the data quality problem: automatic repair, manual review triggering, and re-collection, and the repair results are verified to meet the preset quality threshold.

8. The method according to claim 1, characterized in that, The steps of real-time monitoring of the evaluation process and dynamic optimization of the multi-dimensional quality evaluation index system and the processing suggestions include: Continuously monitor the operational status of the quality assessment process to obtain monitoring data; Calculate one or more evaluation indicators, such as accuracy rate, false alarm rate, and false negative rate, based on the monitoring data. Based on the evaluation metrics, the weights and thresholds of the multi-dimensional quality evaluation metric system, the hyperparameters of the machine learning model, and the generation parameters of the processing suggestions are dynamically adjusted through one or more of reinforcement learning, multi-objective optimization, and Bayesian optimization, so as to maximize the evaluation accuracy and minimize the false positive rate and the false negative rate, and cross-domain knowledge transfer is performed on the machine learning model based on the evaluation metrics.

9. The method according to claim 1, characterized in that, The steps of establishing a multi-dimensional quality evaluation index system and performing quality checks on the acquired data based on the multi-dimensional quality evaluation index system to obtain usable data that meets preset requirements include: Establish a multi-dimensional quality evaluation index system that includes one or more dimensions such as completeness, consistency, accuracy, and timeliness. Completeness is measured by the ratio of the number of missing fields to the total number of fields; consistency is measured by the ratio of the number of records that violate preset logical constraints to the total number of records; accuracy is measured by the ratio of the absolute value of the deviation between the measured value and the benchmark value to the benchmark value; and timeliness is measured by the ratio of the deviation between the data timestamp and the current time to the preset time window. Set weights and thresholds for each indicator in the multi-dimensional quality evaluation index system; Based on the multi-dimensional quality evaluation index system, the weights and thresholds of each index, the acquired data is subjected to quality checks to obtain usable data that meets the preset requirements.

10. A data quality assessment system for public data resources in transportation, characterized in that, include: The data acquisition module is used to acquire data from public transportation data resources, the data of which comes from multiple data sources; The quality inspection module is used to establish a multi-dimensional quality evaluation index system, and to perform quality inspection on the acquired data based on the multi-dimensional quality evaluation index system to obtain usable data that meets preset requirements. Anomaly detection module is used to perform behavioral analysis on the available data and identify potential anomalies in data behavior that deviate from the normal pattern. The pattern classification module is used to perform anomaly pattern recognition on the potential anomalies using a preset machine learning model, and obtain anomaly pattern classification results. The association analysis module is used to obtain associated heterogeneous data from a preset associated heterogeneous data source for the potential anomaly according to preset association rules, perform pattern alignment, entity parsing, and spatiotemporal normalization processing on the associated heterogeneous data, generate an association evidence set corresponding to the potential anomaly, and perform evidence aggregation analysis and causal reasoning based on the association evidence set to determine whether the potential anomaly can be causally explained by external contextual factors. The anomaly detection module is used to determine the potential anomalies as data quality issues or fluctuations caused by external contextual factors based on the results of the correlation analysis. The suggested repair module is used to generate processing suggestions containing solutions for data that is determined to have the aforementioned data quality problem, and to trigger the corresponding data repair mechanism; The monitoring and optimization module is used to monitor the evaluation process in real time and dynamically optimize the multi-dimensional quality evaluation index system, the machine learning model, and the processing suggestions.

Images