Public data quality evaluation method and device

Abstract

The invention relates to the technical field of data processing, and discloses a public data quality evaluation method and device, and the key point of the technical scheme is that the method comprises the following steps: S1, in response to a data quality evaluation instruction, collecting public data, and carrying out the standardization processing to obtain standard data; s2, carrying out block chain evidence storage on the standard data; s3, performing quality detection on the data through an intelligent detection algorithm to obtain a detection result; s4, performing data quality evaluation according to the detection result to obtain an evaluation result; and S5, performing feedback adjustment on the intelligent detection algorithm according to an evaluation result, breaking through a traditional evaluation mode based on a single standard, comprehensively considering various data quality indexes, and providing more comprehensive data quality analysis.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a method and apparatus for assessing the quality of public data. Background Technology

[0002] In today's financial markets, asset securitization (ABS) has become an important financing tool, especially in public-private partnership (PPP) projects, real estate, energy, and infrastructure. ABS transforms illiquid assets into tradable financial products by packaging real assets (such as loans, bonds, and accounts receivable) into securities. In this process, the quality of the underlying assets directly impacts the credit rating, risk control, and investor confidence of the securities.

[0003] The role of public data in ABS financing is undeniable, especially publicly available data from governments and financial institutions. Reliable public data provides investors with valuable insights for ABS pricing, credit rating, and liquidity assessment. However, the quality of this public data remains a significant factor limiting its widespread application. Traditional public data primarily includes government-released statistical data, credit rating information, and corporate financial statements.

[0004] Currently, the quality assessment of public data mainly relies on traditional data management and processing techniques, such as data validation, format verification during ETL (Extract-Transform-Load) processes, and missing value imputation. While these techniques play a role in the data preprocessing stage, they cannot meet the increasingly complex needs of public data quality inspection, especially in environments involving cross-departmental, cross-domain, and massive amounts of data.

[0005] Defects and shortcomings of existing technology:

[0006] 1. Data Authenticity Issues: Existing data quality inspection methods typically rely on static rules or formatted checks, primarily focusing on whether the data structure conforms to predetermined specifications. However, these methods cannot effectively address the authenticity issue of public data. Due to the wide range of sources of public data, including government agencies, financial institutions, and non-governmental organizations, data authenticity often faces the following challenges:

[0007] (1) Data tampering and forgery: Due to the openness of public data, data may be maliciously tampered with or forged, especially in the financial field. The emergence of false data will directly affect the risk assessment and investment decisions of ABS products.

[0008] (2) Unclear data source: Without an effective traceability mechanism, the public cannot confirm whether the data is true and reliable, which affects the credibility of the data in the financial market.

[0009] Existing technologies are insufficient to solve this problem and cannot guarantee the traceability and tamper-proof nature of data.

[0010] 2. Data Consistency and Integrity Issues: Much public data originates from multiple different departments or institutions, with significant differences in data formats, standards, and update frequencies. This leads to data consistency issues. For example, duplicate, conflicting, or inconsistent records may exist between different datasets, directly impacting data quality assessment. Furthermore, data integrity issues cannot be ignored. Public data often contains missing or null values, making it unsuitable for direct use in high-risk financial transactions such as asset securitization.

[0011] Existing detection methods are mostly limited to format and standardization checks, lacking a comprehensive assessment of the logical consistency between data. This leads to many data quality problems, especially in the financial sector, failing to be detected and corrected in a timely manner.

[0012] 3. Insufficient Intelligence and Adaptability: Most current public data quality inspection methods are based on static rules, such as using SQL queries for data consistency checks or rule sets to detect data anomalies. These methods cannot adapt to rapidly changing data environments, nor can they learn and adapt to new data quality problems. In the modern financial environment, data sources are constantly increasing, and data types and formats are becoming increasingly diverse, making it impossible for static rules to effectively address complex data quality issues.

[0013] Furthermore, existing technologies generally lack intelligence, failing to automatically uncover potential quality issues from data, and also lacking dynamic adjustment and self-optimization capabilities. The process of public data quality inspection often relies on manual intervention, leading to inefficiency and high subjectivity.

[0014] 4. Lack of a quantitative data quality assessment system:

[0015] Current data quality assessment methods primarily rely on manual rules and experience, lacking a scientific quantitative evaluation system. In asset securitization, investors often cannot accurately understand the quality of public data, thus affecting their pricing and risk assessment of ABS products. Existing technologies for data quality assessment are mostly qualitative analyses, lacking objective scoring standards and quantitative indicators, resulting in a high degree of subjectivity in the data quality assessment process.

[0016] For ABS financing, the reliability of data directly impacts asset pricing, risk assessment, and credit rating. Without scientific and accurate quality assessment, investors will lack sufficient basis for their asset securitization decisions, increasing potential risks in the financial market. Summary of the Invention

[0017] The purpose of this invention is to provide a public data quality assessment method and apparatus that breaks through the traditional assessment method based on a single standard, and can comprehensively consider various data quality indicators to provide a more comprehensive data quality analysis.

[0018] The above-mentioned technical objective of the present invention is achieved through the following technical solution: a public data quality assessment method, comprising the following steps:

[0019] S1. In response to the data quality assessment instruction, collect public data and perform standardization processing to obtain standard data;

[0020] S2. Store standard data on the blockchain;

[0021] S3. The data is inspected using an intelligent detection algorithm to obtain the detection results;

[0022] S4. Based on the test results, conduct a data quality assessment to obtain the assessment results.

[0023] As a preferred embodiment of the present invention, in S1, the process of standardizing public data is as follows:

[0024] Fill in missing values ​​in public data;

[0025] Detect and remove outliers from public data;

[0026] Remove duplicate data from public data;

[0027] For public data, a unified conversion of data field formats is performed to obtain standard data.

[0028] As a preferred embodiment of the present invention, S2 includes:

[0029] Generate a hash value for each piece of standard data;

[0030] The hash value, collection time, and data source of each standard data item are stored in the blockchain through a smart contract. The smart contract is used to automatically detect whether the data conforms to the predetermined rules. If the data does not conform, it is rejected from being stored in the blockchain.

[0031] The data is verified using the hash value and timestamp of the blockchain evidence, and data that fails verification is discarded.

[0032] As a preferred embodiment of the present invention, S3 includes:

[0033] S31. Use intelligent detection algorithms to perform anomaly detection on the data, including:

[0034] The LSTM model is used to detect anomalies in time series data and capture the time dependencies in the data. The LSTM model stores historical information through memory units and predicts future data. Data with prediction errors greater than a set threshold is marked as anomalous data.

[0035] By using a Bayesian network, logical consistency checks are performed on the data, and data that fails the logical consistency check is marked as abnormal data.

[0036] S32. Perform integrity and consistency checks on the data;

[0037] Compare the hash values ​​of data from different data sources. If they are inconsistent, the data with different hash values ​​is marked as abnormal data.

[0038] Determine whether the proportion of missing values ​​in a certain data item exceeds the missing value threshold. If it does, then the data item is considered abnormal.

[0039] As a preferred embodiment of the present invention, S4 includes:

[0040] S41. Based on the test results, calculate the accuracy score, consistency score, and integrity score of the public data respectively;

[0041] The expression for accuracy scoring is: y i This is the actual value. Let N be the predicted value, and N be the number of data points. This represents the error value between the actual data and the predicted data obtained from the LSTM model.

[0042] The consistency score is calculated as follows: based on the hash value consistency result and the logical consistency result, the consistency score of the current data is determined by querying the preset consistency score rules. The consistency score rules are used to set the consistency score values ​​for the hash value consistency result and the logical consistency result.

[0043] The integrity score is calculated as follows: based on the proportion of missing values, the integrity score of the current data is determined by querying a preset integrity scoring rule, whereby the integrity scoring rule is used to set the integrity score value based on the proportion of missing values.

[0044] S42. Calculate the data quality score Q based on the accuracy score, consistency score, and completeness score.

[0045] Q = w1 * accuracy score + w2 * consistency score + w3 * completeness score; where w1 is the weight of the accuracy score, w2 is the weight of the consistency score, and w3 is the weight of the completeness score.

[0046] As a preferred embodiment of the present invention, after S4, S5 is also executed, and the content of S5 is: to adjust the intelligent detection algorithm based on the evaluation results.

[0047] A public data quality assessment device includes a processor and a memory, the memory storing a computer program executable by the processor, wherein the processor executes the computer program to implement the above-described method.

[0048] In summary, the present invention has the following beneficial effects: it is applied to the public data quality detection in asset securitization, aiming to improve the reliability, authenticity, consistency and integrity of the data.

[0049] This invention innovatively combines blockchain notarization technology and smart contracts to provide a secure and reliable trust guarantee mechanism for public data in asset securitization projects. By leveraging the immutability of blockchain technology and the automatic verification function of smart contracts, the authenticity and compliance of public data such as parking lot data and charging pile data are ensured. This technology overcomes the limitations of traditional data verification methods, enabling public data to have higher transparency and trustworthiness in asset securitization.

[0050] This invention innovatively combines LSTM and K-Means clustering analysis to provide an efficient intelligent anomaly detection method for public data quality assessment. By combining time series analysis with clustering algorithms, it can detect and identify potential abnormal behaviors in public data in real time. This method breaks through the dependence of traditional data anomaly detection methods on static data, improving the automated detection capability and accuracy of data quality.

[0051] This invention innovatively proposes a public data quality assessment method based on a multi-dimensional weighted scoring model. By comprehensively scoring multiple dimensions such as accuracy, consistency, and completeness of the data, and weighting the calculations based on the importance of different dimensions, an objective and quantitative quality assessment system for public data assets is provided. This method breaks through the traditional assessment approach based on a single standard, comprehensively considering various data quality indicators and providing a more comprehensive data quality analysis. Attached Figure Description

[0052] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0053] The present invention will be further described in detail below with reference to the accompanying drawings.

[0054] like Figure 1 As shown, the present invention provides a method for assessing the quality of public data, comprising the following steps:

[0055] S1. In response to the data quality assessment instruction, collect public data and perform standardization processing to obtain standard data;

[0056] Specifically, the invention collects public data: It adopts a multi-source data collection approach, supporting access to different types of public data sources, including: government open data, such as statistical data and policy documents; financial institution data, such as credit reports and historical loan data; corporate financial data, such as balance sheets and profit and loss statements; and market data, such as stock market data and macroeconomic data.

[0057] Data is collected via API interface or upload method and transmitted to the device for processing.

[0058] Specifically, the process of standardizing public data is as follows:

[0059] Missing values ​​in public data can be filled using machine learning models such as the KNN algorithm.

[0060] Outliers in public data are detected and removed; this is achieved using Z-scores and / or IQR, i.e., the quartile method.

[0061] To remove duplicate data from public data, a hash value matching algorithm is used to ensure data uniqueness and avoid duplicate data.

[0062] For public data, a standardized conversion of data field formats is performed to obtain standard data. The format of all data fields is unified to ensure that all types of data conform to consistent industry standards, such as the ISO-20022 financial standard.

[0063] S2. Store standard data on the blockchain;

[0064] S2 includes:

[0065] For each piece of standard data, a hash value is generated; for example, using algorithms such as SHA-256, a unique digital fingerprint is generated.

[0066] The hash value, collection time, and data source of each standard data item are stored in the blockchain through a smart contract. The smart contract is used to automatically detect whether the data conforms to the predetermined rules. If the data does not conform, it is rejected from being stored in the blockchain.

[0067] The data is verified using the hash value and timestamp of the blockchain-stored evidence. Data that fails verification is discarded to ensure that the data is tamper-proof and traceable.

[0068] S3. The data is inspected using an intelligent detection algorithm to obtain the detection results;

[0069] S3 includes:

[0070] S31. Use intelligent detection algorithms to perform anomaly detection on the data, including:

[0071] The LSTM model is used to detect anomalies in time series data and capture time dependencies, making it particularly suitable for time series analysis in financial data. The LSTM model stores historical information in memory cells and predicts future data, marking data with prediction errors exceeding a set threshold as outliers.

[0072] Specifically, the loss function of the LSTM model: Among them, y t This is the actual value. The value is the predicted value, and T is the time span.

[0073] The intelligent detection algorithm also includes the K-Means clustering algorithm, used for K-Means cluster analysis. For non-time-series data, such as corporate financial data, the K-Means clustering algorithm is used to perform cluster analysis and detect data points that differ significantly from the main cluster. The K-Means clustering formula is: Where, x i It is a data point, μ k N is the centroid of the cluster, and N is the number of data points. Data belonging to rare clusters are marked as anomalies.

[0074] In the data quality assessment of this invention, the K-Means clustering algorithm can identify data points that differ significantly from the main data group. In practical applications, such data is more likely to be anomalous. Therefore, the K-Means clustering algorithm can be used to identify potential anomalous data, thereby playing a pre-screening role.

[0075] In the method of this invention, k-Means plays an auxiliary role in localization. For example, K-Means clustering analysis is used to help identify data points that have significant differences in the full dataset. The clustering results serve as the pre-screening basis for subsequent accuracy or completeness scoring, which can clearly understand the range of potential abnormal data and guide whether the corresponding data should be included in the calculation or removed, thereby making the subsequent data quality assessment results more accurate.

[0076] Example: If a financial indicator data point is clustered into a marginal cluster, it is marked as data to be verified, and subsequent decisions on whether to remove or correct it will be made based on its actual deviation.

[0077] By using Bayesian networks, logical consistency checks are performed on the data, and data that fails the logical consistency check is marked as abnormal data; this ensures that the logical relationships between different data sources are reasonable. For example, it checks whether the "Total Assets" and "Total Liabilities" fields in the "Balance Sheet" conform to the predetermined financial logical relationship.

[0078] S32. Perform integrity and consistency checks on the data;

[0079] By comparing the hash values ​​of data from different data sources, data with different hash values ​​is marked as anomalous. This hash value comparison ensures data consistency across different sources. For example, if the "total assets" value recorded in two different databases is inconsistent, that data is marked as anomalous.

[0080] The system determines whether the proportion of missing values ​​in a given data item exceeds a missing value threshold, such as more than 30%. If it does, the data item is considered outlier. Specifically, if a field has a large number of missing values, its impact on asset securitization is assessed and a score is assigned.

[0081] S4. Based on the test results, conduct a data quality assessment to obtain the assessment results. This includes:

[0082] S41. Based on the test results, calculate the accuracy score, consistency score, and integrity score of the public data respectively;

[0083] The expression for accuracy scoring is: y i This is the actual value. Let N be the predicted value, and N be the number of data points. This represents the error value between the actual data and the predicted data obtained from the LSTM model.

[0084] K-Means clustering analysis was also used to assist in the evaluation. The clustering results were used to screen the data to be evaluated, and the accuracy score was calculated only for non-abnormal clusters to avoid interference from outliers.

[0085] Outlier Handling: If data is marked as outlier (error exceeds threshold or cluster outliers), then:

[0086] Exclusion: This data point is directly excluded and does not participate in the scoring calculation;

[0087] Reduce weighting: Reduce the weight of a component in a weighted scoring model.

[0088] The consistency score is calculated as follows: based on the hash value consistency result and the logical consistency result, the consistency score of the current data is determined by querying the preset consistency score rules. The consistency score rules are used to set the consistency score values ​​for the hash value consistency result and the logical consistency result. For example, if the data sources are inconsistent, the consistency score is deducted, or the consistency score is very low.

[0089] Abnormal data handling: Data with logical conflicts or inconsistent hash values ​​will have their consistency score deducted directly. The deduction value can be flexibly set according to the actual application requirements.

[0090] The integrity score is calculated as follows: based on the proportion of missing values, the integrity score of the current data is determined by querying a preset integrity scoring rule. This rule sets the integrity score value based on the proportion of missing values ​​and is used to evaluate the integrity of the data field. If there are many missing data points, the integrity score is low. If the proportion of missing data exceeds a certain percentage, such as 30%, the data will be considered low-quality data.

[0091] Outlier handling: Data points with missing values ​​exceeding the threshold are considered low-quality data, and their scores are reset to zero or reduced proportionally. The reduction percentage can be flexibly set according to actual application needs.

[0092] In this step, K-Means clustering analysis is also applied to dynamically adjust the integrity weights for potential outliers identified by clustering, such as clusters in sets of missing fields.

[0093] S42. Based on the accuracy score, consistency score, and completeness score, calculate the data quality score Q, i.e., the weighted scoring model: Q = w1 * accuracy score + w2 * consistency score + w3 * completeness score; where w1 is the weight of the accuracy score, w2 is the weight of the consistency score, and w3 is the weight of the completeness score. The weights of each score are dynamically adjusted based on factors such as the source of the data, its importance, or the proportion of outlier data.

[0094] S5. Adjust the intelligent detection algorithm based on the evaluation results.

[0095] The scoring model and anomaly detection algorithm are continuously optimized based on real-time data changes and historical data feedback. The process includes:

[0096] Historical data feedback: The weights of each dimension's score are optimized based on the scoring results of historical data.

[0097] Real-time data optimization: During system operation, the learning rate of the LSTM model and the number of clusters in K-Means are adjusted in real time based on the performance of new data.

[0098] Corresponding to the above method, the present invention also provides a public data quality assessment device, comprising: a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor implements the methods described in S1-S5 when executing the computer program.

[0099] This invention proposes a public data quality assessment method and apparatus, which is applied to the public data quality detection in asset securitization, aiming to improve the reliability, authenticity, consistency and completeness of the data.

[0100] 1. A public data credibility assurance mechanism based on blockchain notarization and smart contracts: This invention innovatively combines blockchain notarization technology and automatic compliance detection of smart contracts to provide a safe and reliable credibility assurance mechanism for public data in asset securitization projects.

[0101] By leveraging the immutability of blockchain technology and the automatic verification capabilities of smart contracts, the authenticity and compliance of public data such as parking lot data and charging pile data are ensured. This technology overcomes the limitations of traditional data verification methods, enabling public data to have greater transparency and trustworthiness in asset securitization.

[0102] Traditional data storage and verification mechanisms cannot guarantee the immutability and traceability of data. This method uses the hash notarization and timestamp functions of blockchain to ensure that the historical record of each piece of data is immutable and traceable.

[0103] In this invention, the smart contract performs automatic compliance checks during the data storage process to ensure that the data meets the requirements of asset securitization. Once non-compliance is detected, the data is automatically rolled back to prevent unqualified data from entering the market.

[0104] 2. Intelligent Anomaly Detection of Public Data Based on AI Deep Learning and Cluster Analysis: This invention innovatively combines LSTM and K-Means clustering analysis to provide an efficient intelligent anomaly detection method for public data quality assessment. By combining time series analysis with clustering algorithms, potential abnormal behaviors in public data can be detected and identified in real time, such as data input errors, transmission errors, or other unexpected anomalies. This method overcomes the dependence of traditional data anomaly detection methods on static data, improving the automated detection capability and accuracy of data quality.

[0105] Traditional anomaly detection methods, based on static rules, cannot effectively capture potential dynamic changes in data. This method analyzes time-series data using an LSTM model, capturing the temporal sequence and trends of the data to identify future anomalies in advance. Furthermore, this invention utilizes K-Means clustering to group the data and detect anomalous data that differs significantly from other data clusters, greatly improving the sensitivity and accuracy of anomaly detection.

[0106] 3. Quantitative Assessment of Public Data Quality Based on a Weighted Scoring Model: This invention innovatively proposes a public data quality assessment method based on a multi-dimensional weighted scoring model. By comprehensively scoring multiple dimensions such as accuracy, consistency, and completeness of the data, and combining the weighted calculations based on the importance of different dimensions, an objective and quantitative quality assessment system is provided for public data assets. This method breaks through the traditional assessment approach based on a single standard, and can comprehensively consider various data quality indicators, providing a more comprehensive data quality analysis.

[0107] Traditional data quality assessments often rely on subjective judgment, lacking systematicity and objectivity. This invention, however, introduces a multi-dimensional scoring mechanism to comprehensively evaluate data based on multiple dimensions such as accuracy, completeness, and consistency, employing a weighted model to reflect the importance of each dimension across different data sources. By adaptively adjusting the scoring weights, the scoring model can be dynamically optimized according to the different characteristics of the data source, improving the flexibility and real-time nature of data quality assessment.

[0108] The following are two implementation examples of the present invention.

[0109] Case 1: Government Data Bureau Parking Lot and Charging Pile Data Asset Financing ABS Project.

[0110] Background: A city's government data bureau possesses a large amount of public data assets, including parking lot data and charging station data, sourced from the government, transportation management departments, and public utilities companies. The government data bureau hopes to finance these data assets through asset securitization (ABS) to support future urban transportation infrastructure construction and green energy projects.

[0111] To ensure that the quality of the data meets the needs of investors and the standards of the financial market, the government data bureau has decided to adopt the public data quality assessment method based on blockchain and AI of this invention to conduct a comprehensive quality assessment of parking lot and charging pile data, ensuring the authenticity, consistency, accuracy and completeness of the data.

[0112] Step 1: Data acquisition and preprocessing.

[0113] Data collection: The government data bureau collects data from various departments, primarily including:

[0114] Parking data includes the number of parking spaces, parking lot location, parking duration, parking fees, and available parking spaces. Data is sourced from traffic management departments and parking lot operators.

[0115] Charging station data includes the number of charging stations, their locations, utilization rates, charging capacity, and charging costs. The data is sourced from charging station operators.

[0116] This data is collected via API interfaces and batch upload methods, and stored in devices or systems for subsequent processing.

[0117] Data preprocessing involves the following steps after data acquisition:

[0118] Missing value handling: Suppose that usage data for some charging stations is missing. Use the KNN algorithm (based on similar data points) to impute missing values, or use a regression algorithm to predict missing values ​​based on data from other charging stations.

[0119] Outlier Detection: For charging station charging data, it is assumed that the charging volume of some charging stations is abnormally high. The Z-score method is used to detect outliers, and data deviating from a preset threshold (e.g., Z-score > 3) are marked as outliers.

[0120] For example, if a certain charging station has a monthly charging capacity of 5,000 kilowatts, while other charging stations have a monthly charging capacity of 2,000 kilowatts, this may indicate a data error or an anomaly.

[0121] Deduplication: Suppose some parking lot data is duplicated, possibly due to repeated uploads from different sources. Deduplication is performed using hash values ​​to ensure each data entry is unique.

[0122] Data standardization: Assuming that parking fees are not standardized in units, with some parking lots charging in "yuan / hour" and others in "yuan / trip", consistency is ensured by standardizing all fee data to yuan / hour.

[0123] Step 2: Blockchain evidence storage.

[0124] Data hashing: To ensure the immutability and traceability of data, all collected data (such as the number of available parking spaces in a parking lot, the usage time of charging piles, etc.) will be processed using SHA-256 hashing.

[0125] Suppose the data for a certain charging station is as follows:

[0126] Charging station number: ABC123;

[0127] Charging capacity: 250 kilowatts;

[0128] Usage time: 50 hours;

[0129] The data is hashed using the SHA-256 algorithm to generate a hash value: H(Charging Station Data) = SHA-256(ABC123,250,50); this hash value will become the unique identifier of the data.

[0130] Smart contracts store all data, including hash values, source information, and collection time, on the blockchain. The smart contract will perform the following operations:

[0131] Data verification: Smart contracts automatically check data format, data consistency, etc., to ensure that the data conforms to preset rules.

[0132] For example, if the parking fee format in the parking lot data is incorrect, the smart contract will refuse to store the data.

[0133] Automatic rollback: When non-compliant data is detected, the smart contract will trigger an automatic rollback to ensure that non-compliant data is not stored on the blockchain.

[0134] Data traceability: Data stored on the blockchain possesses complete traceability, allowing for the tracking of the source, modification records, and collection history of each piece of data. For example, if the data of a charging station is tampered with, the hash value on the blockchain will change, allowing for immediate detection of the problem.

[0135] Step 3: AI intelligent detection module.

[0136] Intelligent detection algorithms utilize deep learning and traditional machine learning methods to detect anomalies in parking lot and charging pile data.

[0137] LSTM Time Series Data Anomaly Detection: LSTM, or Long Short-Term Memory network, is suitable for detecting abnormal patterns in time series data. For example, the monthly charging volume of charging piles exhibits seasonal variations; LSTM can capture these trends and predict future charging demand.

[0138] K-Means clustering analysis: K-Means clustering is performed on all parking lot and charging pile data to detect outliers that differ significantly from other data points. For example, if the utilization rate of a charging pile is lower than normal, it may be due to data errors or hardware problems.

[0139] Bayesian network analysis: This analyzes the logical relationships between data to ensure the rationality of parking lot and charging pile data. For example, if the usage time of some charging piles is abnormally high, it may be related to the parking lot usage in that area. Bayesian networks can automatically infer and analyze the logical relationships.

[0140] Cross-data source consistency check: Compare whether data from different data sources are consistent, such as parking lot vacancy data provided by different departments or operators. If conflicts exist, the data is marked as inconsistent.

[0141] Missing data detection: The "number of parking spaces" field in the parking lot data is checked. If more than 30% of the data is missing, it will be marked as low-quality data and a warning will be triggered.

[0142] Step 4: Data quality scoring.

[0143] Data quality scores are evaluated from the following dimensions:

[0144] Accuracy: The accuracy of the data is assessed by comparing it with historical standard data or external authoritative data.

[0145] Consistency: Data consistency is assessed by checking the consistency between different data sources. If the data sources are inconsistent, the score will be lower.

[0146] Completeness: Whether the detection data is complete. A higher proportion of missing values ​​results in a lower completeness score.

[0147] Based on the test results, a data quality assessment is conducted to obtain the assessment results. The final data quality score is calculated using a weighted model.

[0148] Step 5: Feedback optimization mechanism.

[0149] Historical data feedback involves analyzing historical data scoring results to adjust the scoring model. For example, if the accuracy score for parking lot data is low, it may be necessary to re-evaluate the quality of the data source or optimize the data collection methods.

[0150] Real-time data optimization: Based on changes in real-time data, the scoring rules and anomaly detection models are automatically adjusted to ensure the real-time optimization of the scoring mechanism.

[0151] Through the blockchain and AI-based public data quality assessment method of this invention, government data bureaus can ensure the quality of their parking lot and charging pile data. This comprehensively quality-assessed data will be transformed into ABS financing projects, providing investors with more transparent and secure data support and ensuring the smooth progress of the financing process.

[0152] Case 2: Data quality assessment of real estate ABS financing based on public data.

[0153] Scenario: Suppose a financial institution plans to package a group of real estate loan assets into securitized products, such as mortgage loans and secured loans, and issue them to the market for financing. To ensure that the data quality of these underlying assets meets the needs of investors, the financial institution needs to conduct a comprehensive quality assessment of the data, including accuracy, completeness, consistency, and authenticity. Therefore, the institution decides to adopt the public data quality assessment method of this invention.

[0154] Step 1: Data acquisition and preprocessing.

[0155] Data collection:

[0156] First, financial institutions import real estate loan data from various sources into the device:

[0157] Real estate loan data includes loan amount, borrower information, repayment history, etc.

[0158] Credit score data: Data from third-party credit rating agencies.

[0159] Macroeconomic data: Real estate market data from government statistics bureaus, such as housing price indices and loan interest rates.

[0160] This data is collected via API or batch upload and then enters the preprocessing stage.

[0161] Data preprocessing:

[0162] Missing value handling: Suppose that the borrower's age is missing in some loan data. The KNN algorithm is used to fill in these missing values ​​based on the age data of similar borrowers.

[0163] Outlier detection: Suppose a loan amount is significantly higher than the market average. The Z-score method is used to detect and label this loan as outlier data.

[0164] Deduplication: Remove duplicate loan records for the same borrower by matching hash values.

[0165] Data standardization: Min-Max standardization is performed on data such as housing prices, loan amounts, and interest rates to ensure that values ​​in different units can be compared.

[0166] Step 2: Blockchain evidence storage.

[0167] Data hashing: Each piece of collected data (such as loan amount, borrower's age, property location, etc.) is hashed using SHA-256.

[0168] Suppose a loan transaction has the following details: Loan amount = 500,000 yuan, Borrower age = 35 years old, Property location = Beijing.

[0169] Generate a hash value for this data using the SHA-256 algorithm:

[0170] H(Loan Data) = SHA-256(500000,35,Beijing)

[0171] The generated hash value will uniquely identify the data.

[0172] Smart contract storage stores information such as the hash value of each piece of data, the data source (such as banks, governments, credit rating agencies), and the collection time in the blockchain.

[0173] The smart contract will check the integrity and compliance of the data (e.g., verifying the format of the loan amount, whether the borrower's credit score meets the standards, etc.). If the data does not conform to the preset format or rules, the smart contract will refuse to store the data and automatically issue a warning.

[0174] Step 3: AI intelligent detection module.

[0175] Anomaly detection uses AI algorithms to detect anomalies in data:

[0176] LSTM Time Series Data Anomaly Detection: An LSTM model is used to predict the repayment history of each loan. For example, if the predicted and actual repayment history of a loan of 500,000 yuan differs significantly, it indicates abnormal repayment behavior.

[0177] K-Means clustering analysis: All loan data, such as loan amount and borrower age, are clustered using the K-Means clustering algorithm. If some data points fail to cluster successfully or belong to a rare cluster, these data points are marked as potential anomalies.

[0178] Bayesian network analysis examines the logical relationships between loan data. For example, if a borrower's age and loan amount do not conform to normal lending patterns—such as a 35-year-old applying for a loan of 500,000 yuan, while historical data shows the average loan amount for a 35-year-old is 100,000 yuan—the Bayesian network will detect this discrepancy and mark it as an anomaly.

[0179] Data integrity and consistency verification:

[0180] Cross-data source consistency checks: These checks whether identical data fields, such as housing price data, from different data sources are consistent. For example, if there are significant differences between a housing price index provided by the government and data provided by a credit rating agency, it would be flagged as a consistency problem.

[0181] Missing data validation: For each loan record, check for missing values. For example, some loans may lack borrowers' credit scores; if the missing percentage exceeds 30%, it is marked as low-quality data with poor integrity.

[0182] Step 4: Data quality scoring.

[0183] Data quality scores will be evaluated from the following dimensions:

[0184] Accuracy: The accuracy of data is calculated by comparing it with historical or standard data. For example, the accuracy of loan amounts would be assessed by comparing them with the market average loan amount.

[0185] Consistency: If there are data conflicts across sources, such as inconsistent house price indices, the consistency score will be low.

[0186] Completeness: The proportion of missing data affects the completeness of the data. If more than 30% of a field is missing, the data score will be lowered.

[0187] The final quality score is calculated using a weighted scoring model.

[0188] Accuracy score, consistency score, completeness score

[0189] Step 5: Feedback optimization mechanism.

[0190] Historical data feedback involves periodically analyzing the quality scores of historical data to identify potential problems in the scoring model. For example, if the model showing the relationship between loan amount and borrower's age is found to be inaccurate, the model parameters can be adjusted.

[0191] Real-time data optimization adjusts the anomaly detection and scoring models in real time based on newly collected data. For example, when new economic data is released, the scoring criteria for the housing price index will be automatically updated, and the comparison criteria for historical data will be adjusted.

[0192] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for assessing the quality of public data, characterized by: Includes the following steps: S1. In response to the data quality assessment instruction, collect public data and perform standardization processing to obtain standard data; S2. Store standard data on the blockchain; S3. The data is inspected using an intelligent detection algorithm to obtain the detection results; S4. Based on the test results, conduct a data quality assessment to obtain the assessment results.

2. The public data quality assessment method according to claim 1, characterized in that: In S1, the process of standardizing public data is as follows: Fill in missing values ​​in public data; Detect and remove outliers from public data; Remove duplicate data from public data; For public data, a unified conversion of data field formats is performed to obtain standard data.

3. The public data quality assessment method according to claim 1, characterized in that: S2 includes: Generate a hash value for each piece of standard data; The hash value, collection time, and data source of each standard data item are stored in the blockchain through a smart contract. The smart contract is used to detect whether the data conforms to the predetermined rules and refuse to store non-compliant data. The data is verified using the hash value and timestamp of the blockchain-stored evidence, and data that fails verification is discarded.

4. The public data quality assessment method according to claim 1, characterized in that: S3 includes: Anomaly detection is performed on time series data using an LSTM model. Historical information is stored in memory units, and future data is predicted. Data with prediction errors greater than a set value is recorded as anomalies. By using a Bayesian network, logical consistency checks are performed on the data, and data that fails the check is marked as an anomaly. Compare the hash values ​​of data from different data sources; if they are inconsistent, record the data as an anomaly. Determine whether the proportion of missing values ​​in the data exceeds the missing value threshold. If it does, then record the data as an anomaly.

5. A public data quality assessment method according to claim 4, characterized in that: S4 includes: S41. Based on the test results, calculate the accuracy score, consistency score, and integrity score of the public data respectively; The expression for accuracy scoring is: y i This is the actual value. Here, N represents the predicted value, and N is the number of data points. This represents the error value between the actual data and the predicted data obtained from the LSTM model. The consistency score is calculated as follows: based on the hash value consistency check results and the logical consistency check results, the consistency score of the current data is determined by querying the preset scoring rules. The integrity score is calculated by determining the integrity score of the current data based on the proportion of missing values ​​and by querying preset scoring rules. S42. Calculate the data quality score Q based on the accuracy score, consistency score, and completeness score. Q = w1 * accuracy score + w2 * consistency score + w3 * completeness score; w1 is the weight of the accuracy score, w2 is the weight of the consistency score, and w3 is the weight of the completeness score.

6. The public data quality assessment method according to claim 1, characterized in that: After S4, S5 is executed, which involves adjusting the intelligent detection algorithm based on the evaluation results.

7. A public data quality assessment device, characterized in that: include: A processor and a memory, the memory storing a computer program executable by the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-6.

Images