Automatic optimization method for data transaction quality evaluation under multi-source data fusion

Abstract

The invention relates to a data transaction quality evaluation automatic optimization method under multi-source data fusion, and relates to the technical field of data transaction management, and the method comprises the following steps: data collection and preprocessing: collecting multi-source data including data meta-information, third-party verification data and transaction historical data; a data quality evaluation model: constructing a data quality evaluation index system, and obtaining the data quality evaluation model by using the decision tree; and automatically optimizing and executing dynamic optimization, outputting a data quality score according to the evaluation model, and automatically executing a data cleaning, supplementing or encrypting optimization strategy based on a score result. The method is beneficial to improving data quality, reducing manual intervention, improving efficiency and reducing cost.

Description

Technical Field

[0001] The present invention relates to the technical field of data transaction management, and in particular to an automatic optimization method for data transaction quality assessment under multi-source data fusion. Background Art

[0002] In today's digital age, data has become a critical asset, and data transactions are widely used across industries. However, ensuring the quality of transaction data remains a critical issue that needs to be addressed. Existing methods for assessing data transaction quality suffer from numerous limitations. Firstly, existing methods primarily rely on data provided by both parties to the transaction. This single data source results in a lack of comprehensiveness and objectivity, making it difficult to accurately reflect the true level of data in the industry. This can lead to misjudgments of data quality by both parties, thus creating potential transaction risks. Secondly, existing methods focus on data accuracy and completeness, but pay insufficient attention to other key dimensions such as data security and compliance. In data transactions, data security directly impacts the interests of all parties involved, while compliance is a fundamental requirement for ensuring legal transactions. Ignoring these dimensions can lead to serious problems such as data leaks, privacy violations, and violations of relevant laws and regulations.

[0003] Furthermore, existing data transaction quality assessment methods mostly rely on static evaluation systems with pre-defined fixed indicators and thresholds. This system struggles to adapt to the dynamic nature of data and the diverse nature of data types. In real-world applications, data accuracy and completeness may change over time, and different data may have different characteristics and requirements. Therefore, fixed evaluation indicators and thresholds are difficult to meet practical needs. Even more worryingly, existing methods lack automatic optimization capabilities. Even when data quality issues can be identified, they often rely on manual intervention to optimize the data. This approach is not only inefficient and error-prone, but also makes it difficult to respond to data quality issues in a timely manner, seriously impacting the reliability and stability of data transactions.

[0004] Existing data transaction quality assessment methods have the following main defects:

[0005] (1) Relying on a single data source, the evaluation results are one-sided and potential deviations or anomalies in the data cannot be detected. This makes the transaction parties lack comprehensive and accurate information when making decisions, which may increase transaction risks.

[0006] (2) In terms of evaluation dimensions, the focus is on data accuracy and completeness, while insufficient attention is paid to important dimensions such as data security and compliance. This may lead to data leakage, privacy infringement, and violation of laws and regulations during data transactions, causing serious losses to all parties involved in the transaction;

[0007] (3) The evaluation system lacks flexibility. It uses a static evaluation system with pre-set fixed indicators and thresholds, which cannot adapt to the dynamic changes and diversification of data. This makes the evaluation results lag behind the actual changes in data quality and cannot reflect the real situation of the data in a timely and accurate manner.

[0008] (4) Insufficient optimization capabilities. Lack of automatic optimization capabilities. Even when data quality issues are discovered, manual intervention and optimization are often relied upon, which is inefficient and prone to errors. This results in data quality issues not being resolved in a timely manner, affecting the smooth progress of data transactions and the realization of data value. Summary of the Invention

[0009] In order to address the defects and shortcomings of existing data transaction quality assessment methods, the purpose of the present invention is to provide a method that can comprehensively, accurately and dynamically assess data transaction quality and has automatic optimization capabilities to adapt to the current complex and changing data transaction environment and meet the industry's demand for high-quality data.

[0010] In order to achieve the above objectives, the technical solution of the present invention provides a method for automatically optimizing data transaction quality assessment under multi-source data fusion, comprising the following steps:

[0011] Data collection and preprocessing, collecting multi-source data, including data metadata, third-party verification data, and transaction history data;

[0012] Data quality assessment model, build a data quality assessment indicator system, and use the decision tree to obtain the data quality assessment model;

[0013] Automatic optimization performs dynamic optimization, outputs data quality scores according to the evaluation model, and automatically executes data cleaning, supplementation or encryption optimization strategies based on the scoring results.

[0014] Preferably, the data collection and preprocessing steps include:

[0015] Data metadata collection: obtaining raw data from data providers, including structured data, unstructured data, and metadata, and standardizing the format of heterogeneous data;

[0016] Third-party verification data access, calling authoritative data sources to verify data authenticity, establishing a verification rule library, and building verification evidence;

[0017] Transaction history data integration, extracting key information from historical transaction records, including data buyer feedback scores, transaction dispute records, and data repeat transaction rates.

[0018] Preferably, in the data metadata collection step, an Oracle / MySQL relational database is connected via JDBC / ODBC, an incremental pull strategy is adopted, and NLP pipeline processing is used for text data.

[0019] Preferably, in the third-party verification data access step, the data fingerprint calculation formula is adopted: data fingerprint = SHA3-256 (original data + timestamp + verification result).

[0020] Preferably, in the transaction history data integration step, the attenuation degree is calculated according to the data value attenuation algorithm: the repeated transaction attenuation factor α=1-e^(-λt), where the λ value of scientific and technological data is 0.15, the λ value of financial data is 0.33, and the λ value of industrial data is 0.08, and t is the data survival period.

[0021] Preferably, the steps of constructing a data quality assessment indicator system include:

[0022] Determine whether the data has been verified by a third-party authority;

[0023] Determine whether the data provider's historical transaction evaluation is good;

[0024] Determine whether key data fields are missing;

[0025] Determine whether the data update time meets the expected frequency;

[0026] Determine whether data is stored and transmitted encrypted.

[0027] Preferably, the automatic optimization step of performing dynamic optimization includes:

[0028] Quality scoring and response strategies, and respond accordingly based on the scoring results;

[0029] Data cleaning uses a dynamic rule engine, interpolation to complete missing time series data, and invalid data is eliminated based on the rule engine;

[0030] Data supplementation uses GAT entity alignment and automatic request strategies to send API requests to data providers, complete missing fields, and automatically crawl public data sources to supplement key information;

[0031] Privacy encryption uses CP-ABE encryption to encrypt sensitive fields and dynamically detect data usage scenarios. If there is no permission to access the original data, desensitization is triggered.

[0032] Preferably, in the data cleaning step, the anomaly score calculation formula S=α·D(X)+(1-α)·[the number of rules that X does not conform to the rule base] / N_rules is adopted, where α=0.7.

[0033] Preferably, in the data supplementation step, a graph attention network is used to construct an entity relationship graph, and the similarity calculation formula SimGAT(ei,ej)=σ(W·[hi||hj])>0.85.

[0034] Preferably, in the privacy encryption step, the encryption decision matrix determines the encryption scheme based on the data sensitivity and usage scenario. When low sensitivity is used for internal analysis, the AES-256+HMAC encryption scheme is used; when high sensitivity is shared across organizations, the CP-ABE encryption scheme is used.

[0035] The method for automatically optimizing data transaction quality assessment under multi-source data fusion provided by the present invention exhibits the following significant beneficial technical effects compared to the existing technology:

[0036] By integrating multi-source data, including metadata, third-party verification data, and historical transaction data, the perspectives and information sources for data evaluation are greatly enriched. This multi-source data fusion approach effectively overcomes the one-sidedness of traditional evaluation methods that rely solely on a single data source. It enables more comprehensive and accurate assessments of data transaction quality, can more keenly detect potential deviations or anomalies in the data, and provides more authentic and reliable data quality information to both parties of the transaction, thereby reducing transaction risks and enhancing the credibility of data transactions.

[0037] The evaluation system constructed by this invention has achieved a major breakthrough in the evaluation dimension, comprehensively covering multiple key dimensions such as data accuracy, integrity, timeliness, security, and compliance. Unlike existing methods that focus on data accuracy and integrity, this invention fully recognizes the importance of data security and compliance in data transactions and incorporates them into the core indicators of the evaluation system. This not only helps to protect the interests of all parties in the data transaction process and ensure the legality and compliance of data transactions, but also effectively avoids risks such as privacy leakage and data tampering caused by data security issues, creating a more secure and reliable environment for data transactions.

[0038] This invention innovatively constructs an automatic optimization mechanism that automatically optimizes data transaction quality based on evaluation results. This mechanism successfully addresses the inefficiency and error-prone nature of existing methods, which rely on manual optimization intervention after data quality issues are discovered. This automated data optimization process not only enables timely and efficient remediation of data quality issues and improves data quality, but also significantly reduces manual intervention costs, improves the efficiency and stability of data transactions, and ensures smooth data transactions and maximizes data value. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flowchart of automatic optimization of data transaction quality assessment according to the present invention;

[0040] Figure 2 This is a flow chart of data collection and preprocessing of the present invention;

[0041] Figure 3This is a structural diagram of the decision tree evaluation model of the present invention;

[0042] Figure 4 Schematic diagram of the automatic optimization mechanism of the present invention. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0044] To achieve data transaction quality assessment and dynamic optimization through multi-source data fusion, this paper proposes an automatic optimization method for data transaction quality assessment. This method collects and integrates multi-source data, including data metadata provided by both parties (e.g., data source, update frequency), third-party verification data (e.g., certification by authoritative institutions), and transaction history data (e.g., historical ratings, complaint records). It constructs an assessment system encompassing metrics such as data accuracy, completeness, timeliness, and security, and uses a decision tree to complete quality assessment. Based on the real-time assessment results, optimization strategies such as data cleaning, supplementation, and encryption are automatically triggered to automatically optimize the data. The automatic optimization process for data transaction quality assessment in this paper primarily includes the following three steps:

[0045] (1) Collect multi-source data, including data metadata, third-party verification data, and transaction history data.

[0046] (2) Construct a data quality assessment indicator system and use the decision tree to obtain a data quality assessment model.

[0047] (3) Build an automatic optimization mechanism to automatically execute optimization strategies such as data cleaning, supplementation, or encryption based on the scoring results.

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings.

[0049] like Figure 1 As shown, the present invention provides a data transaction quality assessment and dynamic optimization method under multi-source data fusion, comprising the following steps:

[0050] Step S1: Data collection and preprocessing. Collect multi-source data, including data metadata, third-party verification data, and transaction history data;

[0051] Step S2: Data quality assessment model. Construct a data quality assessment indicator system and use the decision tree to obtain a data quality assessment model.

[0052] Step S3: Automatically optimize and execute dynamic optimization strategies. Output data quality scores based on the evaluation model, and automatically execute optimization strategies such as data cleaning, supplementation, or encryption based on the scoring results.

[0053] like Figure 2 As shown, the present invention provides a data acquisition and preprocessing method, wherein step S1 is specifically implemented as follows:

[0054] Step S1.1: Data metadata collection: Obtain raw data from the data provider, including structured and unstructured data, metadata (data generation time, field definition, and update frequency), and standardize the format of heterogeneous data (such as converting JSON to CSV and unifying timestamps to UTC format).

[0055] Connect to a relational database such as Oracle or MySQL via JDBC / ODBC and use an incremental pull strategy (e.g. based on the last_updated timestamp field):

[0056] SELECT transaction_id, data_seller_id, TO_CHAR(create_time, 'YYYY-MM-DD HH24:MI:SS') AS utc_time, JSON_EXTRACT(data_pa yload, '$.sensor_values[0]') AS primary_metric FROM rawtransactions WHERE update_time > '${last_extract_time}'

[0057] The text data is processed using the NLP pipeline.

[0058] In this step, the raw data obtained from the data provider covers structured data, unstructured data, and metadata. This comprehensive data collection method ensures data diversity and integrity. By connecting to multiple relational databases through JDBC / ODBC and adopting an incremental pull strategy, it is possible to efficiently extract incrementally updated data from common databases such as Oracle and MySQL, avoiding the waste of resources caused by repeated pulling of full data. At the same time, using NLP pipeline processing on text data can effectively parse and understand text content and extract valuable information, further enriching the scope and depth of data collection and providing a comprehensive and accurate data foundation for subsequent data quality assessment and optimization.

[0059] After data collection, heterogeneous data is formatted and standardized, such as converting JSON to CSV and unifying timestamps to UTC. This standardization eliminates format differences between different data sources, allowing data from multiple channels and in different raw formats to be subsequently processed and analyzed within a unified framework. This improves data processing efficiency and reduces errors and compatibility issues caused by format inconsistencies. It also provides standardized data input for the subsequent construction of data quality assessment models and the execution of automated optimization strategies, ensuring the smooth execution of the entire data transaction quality assessment and dynamic optimization process, and improving the overall performance and reliability of the system.

[0060] Step S1.2: The third party verifies data access, calls the authoritative data source to verify the authenticity of the data, establishes a verification rule library, and builds verification evidence. The calculation formula for data fingerprint is:

[0061] Data fingerprint = SHA3-256 (original data + timestamp + verification result)

[0062] Example of evidence structure:

[0063] TXT #1639 | 2025-04-20T08:15:23Z

[0064] ├─ Tx1: Data fingerprint 0x8a3d...c7f2

[0065] ├─ Verify signature: Digital Signature_V2

[0066] Verification result: Score=92.7 / 100 (Passed);

[0067] In this step, the original data, timestamp, and verification result are concatenated to form a unified input string. This fusion ensures that the data fingerprint simultaneously incorporates the data's content, time, and quality characteristics, making the generated fingerprint more comprehensively unique. This fused string is used as input and calculated using the SHA3-256 hash algorithm. SHA3-256 performs a series of complex mathematical transformations on the input data, including initialization, absorption, and extrusion steps, ultimately outputting a fixed-length 256-bit hash value, the data fingerprint.

[0068] Due to the characteristics of the SHA3-256 algorithm, even slight differences in the input data will produce completely different output values ​​after the hash calculation. Therefore, any tampering with the original data, timestamp, or verification results will result in a significant change in the data fingerprint, enabling rapid and accurate detection of data modifications. Furthermore, the hash algorithm's security features, such as irreversibility and collision resistance, make it difficult to use the data fingerprint to deduce the original data or construct other data with the same fingerprint.

[0069] Step S1.3: Integrate transaction history data and extract key information from historical transaction records, such as data buyer feedback scores (such as accuracy and practicality), transaction dispute records (such as data tampering and privacy leakage incidents), and data repeat transaction rates (reflecting the degree of data value decay).

[0070] When extracting data, the attenuation degree is calculated according to the data value attenuation algorithm:

[0071] Repeat transaction decay factor α=1-e^(-λt),

[0072] Where λ = 0.15 (technological data) / 0.33 (financial data) / 0.08 (industrial data), and t = data survival period (unit: quarter)

[0073] The feature weights were calculated using the hierarchical analysis method, and the results were: accuracy (63%) > timeliness (26%) > completeness (11%).

[0074] In this step, the transaction history data integration process comprehensively extracts key information such as buyer feedback scores, transaction dispute records, and data repeat transaction rates. This information reflects the data quality status from multiple dimensions. For example, buyer feedback scores directly reflect the accuracy and practicality of the data in real-world applications, transaction dispute records reveal quality issues such as possible data tampering or privacy leaks, and the data repeat transaction rate reflects the decline in data value over time. Furthermore, a data value decay algorithm is used to calculate the repeat transaction decay factor α. Different values ​​λ are set based on the characteristics of different data types (such as scientific data, financial data, and industrial data). Combined with the data lifecycle t, this accurately quantifies the degree of data value decay, making data quality assessment more dynamic and accurate, avoiding the limitations of traditional static assessment methods. This precise assessment provides a reliable basis for subsequent data transaction decisions, helping to select high-quality data resources and reduce transaction risks.

[0075] Using the Analytic Hierarchy Process (AHP) to determine feature weights, the weight distribution is clearly in the order of accuracy (63%) > timeliness (26%) > completeness (11%). This result aligns with practical data quality assessment requirements, highlighting the centrality of accuracy while also taking into account the impact of timeliness and completeness. This weight distribution method provides a scientific and rational framework for data quality assessment, enabling the assessment results to more objectively reflect the actual value of the data. This feature weight distribution has important guiding significance in data management practice. It guides data managers to invest more energy and resources in ensuring data accuracy, such as strengthening data review and verification mechanisms. It also reminds managers to pay attention to data timeliness and completeness, and to take appropriate management measures based on the weights of different features, such as optimizing data update processes to improve timeliness and improving data collection scope to enhance completeness. This will achieve comprehensive improvements in data quality and provide strong support for the healthy development of the data trading market.

[0076] like Figure 3 As shown in Figure 2, the specific implementation steps of the data quality assessment model are as follows:

[0077] Step S2.1: Determine whether the data has been verified by a third-party authority

[0078] Determine whether the data has been verified by a third-party authority. If the data has been verified by a third-party authority, proceed to step S2.2 to further verify the historical transaction evaluation of the data provider. Conversely, if the data has not been verified by a third-party authority, proceed to step S2.6 to determine whether the data has completed self-checking.

[0079] In this step, data reliability is initially screened by determining whether it has been verified by a third-party authority. This judgment mechanism ensures that only authoritatively verified data can enter the subsequent, more detailed historical transaction evaluation and verification process, effectively filtering out low-quality or unreliable data that has not undergone authoritative verification. For data that has not undergone third-party authoritative verification, further determination is made to determine whether it has completed self-inspection, which allows for basic quality control of the data again, preventing completely unverified data from directly entering the transaction process. This classification process not only ensures the rigor of data quality assessments but also improves the efficiency of the evaluation process, ensuring that only data that meets certain quality standards can continue to participate in subsequent evaluations and transactions, thereby enhancing the credibility and stability of the entire data trading system.

[0080] Step S2.2: Determine whether the data provider's historical transaction evaluation is good

[0081] Determine whether the data provider's historical transaction evaluation is good. If so, execute step S2.3 to check whether key data fields are missing; if not, output "Failed reliability verification, recommended to adjust and re-evaluate", indicating that the data provider's historical transaction evaluation is poor, the reliability of the current data is questionable, and the data needs to be adjusted and optimized, and then re-evaluated after adjustment.

[0082] The sample code for obtaining historical data is as follows:

[0083] scores = get_scores(provider_id)

[0084] disputes = get_records(provider_id)

[0085] transactions = get_volume(provider_id)

[0086] The sample code for calculating core indicators is as follows:

[0087] score = (scores['accuracy']*0.6 + scores['timeliness']*0.3 + scores['completeness']*0.1)

[0088] rate = len(disputes) / trans['total'] if trans['total']>0 else 1

[0089] factor = 1 - (transactions['current'] / trans['peak'])

[0090] The sample code for decision tree judgment is as follows:

[0091] if (score >= 4.5 and rate <= 0.02 and factor <= 0.25):

[0092] return {"status": "approved", "next_step": "field"}

[0093] else:

[0094] reasons = []

[0095] if score < 4.5: reasons.append("Insufficient overall score")

[0096] if rate > 0.02: reasons.append("Dispute rate exceeds standard")

[0097] if factor > 0.25: reasons.append("Severe value decay")

[0098] return {"status": "rejected", "reasons": reasons}

[0099] In this step, by evaluating the data provider's historical transaction reviews, potentially high-quality data can be effectively screened. This judgment step plays a critical role in the data quality assessment process. If a data provider's historical transaction reviews are positive, indicating that the data they have provided in the past is of high reliability and quality, the system can reasonably assume that the current data is also of good quality, allowing it to proceed to the subsequent key data field missing check (step S2.4). Conversely, if the historical transaction reviews are poor, the system will promptly output a warning message, preventing unreliable data from continuing the assessment process. This not only saves resources and time in subsequent assessment steps, but also effectively reduces the risk of low-quality data entering the trading pool, maintaining the quality and credibility of the entire data trading system. Furthermore, by prompting data providers to make adjustments and optimizations, this step provides a feedback mechanism for improving data quality, helping data providers improve their data quality and, in the long term, enhancing the data quality of the entire data trading market.

[0100] Step S2.3: Determine whether key data fields are missing

[0101] Determine whether the key data fields are missing. If so, execute step S2.4 to check whether the data update time meets the expected frequency. If not, directly output "Failed integrity verification and recommended adjustment and re-evaluation", indicating that the key data fields are complete, but there are other integrity issues. The data needs to be adjusted and optimized, and then re-evaluated after adjustment.

[0102] The following is a sample code for writing SQL queries to count the percentage of NULL values ​​in each field:

[0103] SELECT COUNT(*) AS total_rows, COUNT(field_A) AS non_null_count,(COUNT(*) - COUNT(field_A)) / COUNT(*) * 100 AS missing_rate FROM your_table;

[0104] The judgment criteria and execution logic are as follows:

[0105] If the key field missing rate > threshold:

[0106] If the time fields are missing consecutively:

[0107] Go to step S2.5 (check data update frequency)

[0108] else:

[0109] Output "Failed integrity verification, recommended adjustments and re-evaluation"

[0110] else:

[0111] Prompt other integrity issues;

[0112] In this step, by checking whether key data fields are missing, data integrity issues can be discovered in a timely manner. If key fields are missing, further checking whether the data update time meets the expected frequency will help determine whether the data is in an effective maintenance and update state, which provides a more accurate basis for subsequent data quality assessments. If key fields are not missing, but the system directly outputs a prompt message requiring data adjustment and optimization, this not only prevents data with questionable integrity from entering subsequent processes, but also provides data providers with clear improvement directions to improve data quality. This hierarchical verification mechanism effectively protects data integrity, improves the accuracy and reliability of the entire data quality assessment, and ensures that the data entering the transaction process is complete in terms of key information, thereby enhancing the credibility and success rate of data transactions.

[0113] Step S2.4: Determine whether the data update time meets the expected frequency

[0114] Determine whether the data update time meets the expected frequency. If so, proceed to step S2.5 to verify whether the data is encrypted for storage and transmission. If not, directly output "Failed integrity verification, recommended adjustments and re-evaluation", indicating that the data is not updated in a timely manner and there are integrity issues. Adjustment and optimization of the data are required, and re-evaluation is required after adjustments.

[0115] The sample code for checking whether the maximum time interval meets expectations is as follows:

[0116] SELECT

[0117] MAX(time_gap) AS max_interval,

[0118] AVG(time_gap) AS avg_interval

[0119] FROM (

[0120] SELECT

[0121] TIMESTAMPDIFF(HOUR, LAG(update_time) OVER (ORDER BY update_time),update_time) AS time_gap

[0122] FROM data_table

[0123] WHERE update_time BETWEEN '2025-01-01' AND '2025-01-30'

[0124] ) t;

[0125] The sample code for calculating and analyzing time series is as follows:

[0126] df = pd.read_csv('data.csv', parse_dates=['time'])

[0127] # Convert to hour intervals

[0128] df['time_gap'] = df['time'].diff().dt.seconds() / 3600

[0129] # Exceeding 1.5 times is considered abnormal

[0130] intervals = df[df['time_gap'] > expected_interval * 1.5];

[0131] In this step, by determining whether the data update time meets the expected frequency, we can ensure the data's good timeliness, thereby protecting the data's practicality and reference value. If the data update meets the expected frequency, we further verify whether the data is encrypted for storage and transmission to ensure data security and integrity. This progressive verification mechanism can effectively prevent problems such as untimely data updates or data corruption during storage and transmission, thereby improving data quality and reliability, reducing potential risks in data transactions, and ensuring that only data that meets integrity requirements can continue to participate in subsequent processes. At the same time, timely output of prompt information for data that does not meet the requirements also provides data providers with clear improvement directions, helping to improve the overall quality of the data.

[0132] Step S2.5: Determine whether the data is encrypted for storage and transmission

[0133] Determine whether the data is encrypted during storage and transmission. If so, the output is "Can be traded," indicating that encryption measures have been taken during the data storage and transmission process, making it safe and reliable and allowing for trading. Conversely, if the data is not encrypted during storage and transmission, the output is "Can be traded after executing the optimization strategy," indicating that encryption optimization is required for data storage and transmission. After executing the corresponding optimization strategy, the data will meet security requirements and allow for trading.

[0134] This step ensures data security and privacy during storage and transmission by determining whether the data is encrypted for storage and transmission. If the data is encrypted, the transaction is allowed directly, improving transaction efficiency. If the data is not encrypted, encryption optimization is required before the transaction can proceed. This encourages data providers to improve data quality, ensures the credibility and reliability of data transactions, reduces transaction risks, and strengthens the standardization and compliance of the data transaction process.

[0135] Step S2.6: Determine whether the self-test is completed

[0136] Determine whether the data has completed self-checking. If so, proceed to step S2.7 to further verify the data provider's historical transaction evaluation. Conversely, if the self-checking has not been completed, directly output "Failed to pass accuracy check, transaction not allowed", indicating that the data has not completed the basic self-checking process, its accuracy cannot be guaranteed, and the transaction is not allowed;

[0137] This step determines whether the data has completed self-checking, providing a fundamental level of control over its accuracy. Data that has completed self-checking can enter the subsequent verification process to ensure its accuracy meets certain requirements. Data that has not completed self-checking is directly blocked and not allowed to be traded. This prevents data of questionable accuracy from entering the transaction process in the first place, reducing transaction risks and ensuring the quality and reliability of data transactions.

[0138] Step S2.7: Determine whether the data provider's historical transaction evaluation is good

[0139] Determine whether the data provider's historical transaction evaluation is good. If so, proceed to step S2.8 to check whether key data fields are missing. If not, directly output "Failed reliability verification, recommended adjustment and re-evaluation", indicating that the data provider's historical transaction evaluation is poor, the reliability of the current data is questionable, and the data needs to be adjusted and optimized before re-evaluation.

[0140] Step S2.8: Determine whether key data fields are missing

[0141] Determine whether key data fields are missing. If so, proceed to step S2.9 to check whether the data update time meets the expected frequency. If not, directly output "Failed integrity verification, recommended adjustments and re-evaluation", indicating that the key data fields are complete, but other integrity issues exist. The data needs to be adjusted and optimized, and then re-evaluated after adjustments.

[0142] Step S2.9: Determine whether the data update time meets the expected frequency

[0143] Determine whether the data update time meets the expected frequency. If so, proceed to step S2.10 to verify whether the data is encrypted for storage and transmission. If not, directly output "Failed integrity verification, recommended adjustments and re-evaluation", indicating that the data is not updated in a timely manner and there are integrity issues. Adjustment and optimization of the data are required, and re-evaluation is required after adjustments.

[0144] Step S2.10: Determine whether the data is encrypted for storage and transmission

[0145] Determine whether the data is encrypted during storage and transmission. If so, the output is "Can be traded," indicating that encryption measures have been taken during the data storage and transmission process, making it safe and reliable and allowing for trading. Conversely, if the data is not encrypted during storage and transmission, the output is "Can be traded after executing the optimization strategy," indicating that encryption optimization is required for data storage and transmission. After executing the corresponding optimization strategy, the data will meet security requirements and allow for trading.

[0146] like Figure 4 As shown, the specific steps of the automatic optimization mechanism method are as follows:

[0147] Step S3.1: Quality scoring and response strategy

[0148] Respond accordingly based on the scoring results: if the score is ≥85, the system will directly enter the trading pool and be recommended to the buyer first; if the score is 70≤ and the score is <85, automatic cleaning will be triggered, such as filling missing values ​​and deduplication; if the score is <70, the transaction will be blocked and the provider will be notified to supplement the data or re-encrypt it.

[0149] In this step, data is categorized and processed based on the scoring results through quality scoring and response strategies, achieving accurate assessment and efficient management of data quality. Data with a score ≥85 enters the trading pool directly and is prioritized, ensuring the rapid circulation of high-quality data. Data with a score of 70 ≤ and <85 triggers automatic cleaning, promptly remediating quality issues and improving data availability. Data with a score <70 is blocked from trading and notified for supplementation or encryption, effectively preventing low-quality data from entering the market. This step improves the efficiency and quality of data transactions, reduces transaction risks, reduces manual intervention, and enhances the automation level of data quality management.

[0150] Step S3.2: Data cleaning

[0151] Traditional solutions are based on fixed rule threshold cleaning. This invention adopts a dynamic rule engine, uses interpolation to complete missing time series data, and eliminates invalid data (such as records whose IP addresses do not belong to the service area) based on the rule engine.

[0152] The rule base contains preset cleaning rules, and the rule type structure is defined as:

[0153] PRIORITY: float # dynamic weight (0-1)

[0154] CONDITION: Lambda # Exception judgment function

[0155] ACTION: [Standardize, Interpolate, Delete] # Execute the operation

[0156] In the anomaly detection stage, the original data stream X∈R^(n×d) is input and the anomaly score is calculated as:

[0157] S = α·D(X)+(1-α)·[Number of rules that X does not meet] / N_rules, where:

[0158] S represents the anomaly score, which is a quantitative assessment of the overall anomaly of the input data X, reflecting the degree to which the data does not conform to the rules of the rule base and the anomaly of the data itself;

[0159] α is a weight coefficient used to balance the impact of the degree of abnormality D(X) of the data itself and the number of rules that do not conform to the rule base on the abnormality score. Here, α=0.7 is a value determined by cross-validation, indicating that the degree of abnormality of the data itself accounts for a larger proportion in the calculation of the abnormality score;

[0160] D(X) represents the degree of abnormality of the input data X. It is the value obtained by analyzing the data through a certain anomaly detection algorithm, reflecting the abnormality of the data in the feature space.

[0161] (1-α) represents another weight coefficient, which is used to balance the impact of the number of violations on the anomaly score. Its value is 1-α=0.3, which reflects the relative importance of the number of violations in the anomaly score.

[0162] [Number of X that does not match the rules in the rule base] refers to the number of input data X that do not match the predefined rules in the rule base, which directly reflects the abnormality of the data in terms of rule matching;

[0163] N_rules represents the total number of rules in the rule base and is used to normalize the number of non-compliant rules so that the scores of non-compliant parts are comparable under different numbers of rules.

[0164] If S>T_t, multi-level cleaning is started, and LSTM prediction is used to fill in the numeric fields:

[0165] xt′=fLSTM(xt-10,...,xt-1)+ε·N(0,σ²), where:

[0166] xt′ represents the value of the numeric field after LSTM prediction and filling at time step t, which is an estimation and supplement of missing data;

[0167] fLSTM(), which represents the prediction function of the LSTM (Long Short-Term Memory) model, is used to predict the current missing value based on the historical data sequence;

[0168] xt-10,...,xt-1, represents the historical data sequence from time step t-10 to time step t-1, which serves as the input of the LSTM model to capture the time series characteristics and trends of the data in order to predict the current missing values;

[0169] ε·N(0,σ²) represents the noise term added to the LSTM predictions; ε is the noise coefficient, which is used to control the amplitude of the noise; N(0,σ²) represents Gaussian noise with mean 0 and variance σ². Adding noise can increase the robustness and generalization ability of the model, avoid overfitting, and simulate random fluctuations that may exist in the data.

[0170] When three consecutive records are abnormal, the Kafka transaction rollback mechanism is triggered.

[0171] In this step, traditional data cleaning solutions are usually based on fixed rule thresholds, which are difficult to adapt to complex and changing data environments. The present invention adopts a dynamic rule engine, which can flexibly adjust the cleaning strategy according to the actual situation of the data. By using interpolation to complete the missing time series data, the problem of missing data is effectively solved to ensure the integrity of the data. At the same time, based on the rule engine to eliminate invalid data, such as records where the IP address does not belong to the service area, it can accurately identify and remove data that does not meet the requirements, thereby improving data quality. In addition, the preset cleaning rules in the rule base have a clear structural definition, including dynamic weights, anomaly judgment functions and execution operations, making the cleaning process more transparent and controllable, and can be flexibly configured according to different data characteristics and needs, thereby enhancing the adaptability and versatility of the cleaning solution.

[0172] During the anomaly detection phase, a quantitative assessment mechanism is introduced by inputting the raw data stream and calculating anomaly scores, making the determination of data anomalies more objective and accurate. The formula comprehensively considers the degree of anomaly in the data itself and the number of violations within the rule base. The parameter α, determined through cross-validation, further improves the accuracy of the anomaly score calculation. When the anomaly score exceeds the set threshold, a multi-level cleaning mechanism is activated. An LSTM predictive filling method is used for numeric fields, leveraging historical information from sequence data to intelligently predict and fill missing values. Compared to traditional simple filling methods, this method better restores the true state of the data and improves data accuracy and reliability. Furthermore, when multiple consecutive records are anomaly, a Kafka transaction rollback mechanism is triggered to promptly correct batch data anomalies and prevent further spread of anomalies and impact subsequent processing. This demonstrates the enhanced intelligence and automation of the entire data cleaning process.

[0173] The combination of a dynamic rules engine and a multi-level cleaning mechanism makes the data cleaning process more efficient. On the one hand, the dynamic rules engine can quickly match and process different types of data, reducing unnecessary cleaning operations and wasted resources. On the other hand, the multi-level cleaning mechanism can take appropriate cleaning measures based on the degree of data anomaly, avoiding uniform cleaning intensity for all data and improving cleaning efficiency. When processing missing time series data, the LSTM predictive filling method can quickly and accurately fill in missing data based on learned data patterns, achieving higher computational efficiency and better processing results than traditional methods. Furthermore, the Kafka transaction rollback mechanism can quickly respond to and correct issues when handling batch anomalies, reducing the time and cost of manual intervention and data repair, ensuring the smoothness of the data cleaning process and the steady improvement of overall performance.

[0174] Step S3.3: Data supplementation

[0175] The existing technology relies on manual specification of external data sources. The present invention adopts GAT entity alignment and automatic request strategy to send API requests to data providers, complete missing fields, and automatically crawl public data sources to supplement key information.

[0176] Use the Graph Attention Network (GAT) to build an entity relationship graph, where the nodes come from entities in different data sources and the alignment conditions are:

[0177] SimGAT(ei,ej)=σ(W·[hi||hj])>0.85, where:

[0178] SimGAT(ei,ej) means calculating the similarity between node ei and node ej in the graph attention network (GAT), which is used to measure the semantic or feature relevance of the two nodes;

[0179] σ() represents the activation function, usually the sigmoid function, which is used here to map the result of the weighted concatenation vector to the (0, 1) interval, thereby converting the similarity into a probability value form, which facilitates the subsequent judgment of whether the two nodes are similar enough;

[0180] W is a learnable weight matrix used to perform a linear transformation on the concatenated node feature vectors, thereby learning the importance of different feature dimensions for similarity judgment. It is a parameter learned by the model through data during training.

[0181] [hi||hj] represents the concatenation of the feature vector hi of node ei and the feature vector hj of node ej, integrating the two independent feature vectors into a joint vector as the input of the subsequent linear transformation, so that the model can comprehensively consider the characteristics of the two nodes to judge the similarity between them;

[0182] >0.85 means setting a threshold of 0.85. When the calculated similarity SimGAT(ei,ej) is greater than 0.85, it is considered that nodes ei and ej have high similarity in semantics or features, meeting the conditions for similarity judgment. This threshold can be adjusted according to specific task requirements and data distribution.

[0183] Trigger API query actions based on the type of missing data and available data sources:

[0184] R=β·Completeness+(1-β)·(1 / Costtime), where:

[0185] R represents a comprehensive score or weighted value used to measure the quality and efficiency of a strategy, operation, or data. It takes into account both completeness and time cost. The specific meaning depends on the application scenario of the formula. For example, in the data supplementation scenario, it can represent the comprehensive evaluation score of the data supplementation operation.

[0186] β is a dynamic adjustment coefficient used to balance the relative importance of completeness and 1 / costtime in the comprehensive score. Its value range is usually between [0, 1]. For example, when β is large, it means that more emphasis is placed on the completeness factor, and when β is small, the time cost factor has a greater weight. In this step, β is set to 0.6;

[0187] Completeness, which represents completeness, is usually a value between 0 and 1 and is used to measure the completeness of data, information, or tasks. In data quality scenarios, it represents the completeness and degree of missingness of data records, such as missing data fields. In task execution scenarios, it can represent the completeness of task completion.

[0188] 1-β, representing the weight coefficient of 1 / Costtime, complements β and jointly determines that the sum of the weights of the two factors in the comprehensive score is 1, ensuring that the weight distribution of the comprehensive score is reasonable and the sum is 1;

[0189] Costtime represents cost time, which is usually a value greater than 0, representing the time cost required to complete a certain operation, task, or obtain data. Its reciprocal 1 / Costtime reflects the efficiency or time value per unit time. The lower the time cost and the higher the efficiency, the greater the score of this part. For example, in the data acquisition scenario, if it takes less time to supplement missing data through a certain data source, then its 1 / Costtime value is larger, and its contribution to R in the comprehensive score is also greater.

[0190] In the missing supplement identification stage, the cleaned data X' is input and the SVM multi-classifier is used to identify the missing type as missing at random (MCAR), missing at dependent field (MAR) or missing not at random (MNAR). Different strategies are activated according to the missing type, such as calling an external API.

[0191] This process not only improves the accuracy of data supplementation, but also enhances the integrity and reliability of the data, thereby effectively reducing analytical biases and decision-making errors caused by missing data, providing high-quality data support for subsequent data analysis and application, and ensuring the accuracy and effectiveness of data-driven decision-making.

[0192] Step S3.4: Privacy Encryption

[0193] The industry conventionally uses static AES encryption. This invention adopts CP-ABE encryption to encrypt sensitive fields (such as mobile phone numbers and ID cards), dynamically detect data usage scenarios, and trigger desensitization if there is no permission to access the original data (such as retaining the first 6 digits of the ID card number).

[0194] The encryption decision matrix determines the encryption scheme based on data sensitivity and usage scenarios. For low-sensitivity data used for internal analysis, AES-256+HMAC is used; for high-sensitivity data shared across organizations, CP-ABE is used.

[0195] First, input the data to be encrypted D, and use the classifier to identify the sensitivity based on features including data source, field type, etc. When dynamic encryption is executed, CP-ABE encryption is applied to the structured data fields, through access policies such as:

[0196] (Department:IT) AND (Position Level>=5)

[0197] Generate a key:

[0198] MK=(α,β)∈G_T×Z_p, where:

[0199] MK, or Master Key, is a key parameter in cryptographic systems, used to generate other encryption keys or perform encryption and decryption operations. In advanced encryption schemes such as Attribute-Based Encryption (ABE), the Master Key controls the key generation and encryption process for the entire system and is a core element of system security.

[0200] α and β are two parameters that make up the master key. They are usually random numbers generated within a specific mathematical structure or values ​​that meet specific conditions. Their specific values ​​and generation methods depend on the encryption algorithm used and the security parameter settings. In some encryption schemes, α and β may be used in different key generation or encryption transformation processes, respectively, to ensure the security and correctness of encryption.

[0201] G_T is a specific multiplicative group, commonly used in elliptic curve cryptography (ECC) or pairing-based cryptography. In pairing-based cryptography, G_T is often the target group for pairing, used to store the results of pairing operations. The elements in this group satisfy specific algebraic structures and operation rules, and the group order (i.e., the number of elements in the group) is usually a large prime number to ensure cryptographic security. Common groups in pairing cryptography include G1, G2, and G_T. G_T is used to store pairing results, and its elements can be used for operations such as encryption and signing.

[0202] Z_p stands for the Integer Ring Modulo p, where p is a large prime number. The elements of Z_p are integers from 0 to p-1, and all operations in the ring (such as addition, subtraction, and multiplication) are modular operations. In cryptography, Z_p is often used to generate keys, perform encryption and decryption operations, and implement various cryptographic protocols because modular operations have good mathematical properties and security characteristics. For example, in the Diffie-Hellman key exchange and related encryption algorithms, Z_p is a common fundamental mathematical structure used to ensure the security of the key generation and exchange process.

[0203] × represents the Cartesian product, which here means combining the group G_T and the integer ring Z_p to form a Cartesian product space. An element in the Cartesian product space G_T×Z_p is an ordered pair consisting of an element in G_T and an element in Z_p, that is, an element of the form (g,z), where g∈G_T and z∈Z_p. The master key MK=(α,β) is an element in this Cartesian product space, indicating that α comes from G_T and β comes from Z_p, and together they constitute the complete form of the master key.

[0204] Apply encryption to associated unstructured data:

[0205] c = gm·rnmodn² (m: plaintext, r∈Zn∗), where:

[0206] c represents the encrypted ciphertext, which is the result of the encryption algorithm and is used for transmission over an insecure channel. Only the recipient with the corresponding key can decrypt and restore the original plaintext.

[0207] g is usually a fixed generator. In cryptography, a generator is an element in a specific group that can generate all elements in the group through its exponentiation. In the formula, g is a public parameter used in the encryption process;

[0208] m, represents plaintext, that is, the original data or message that needs to be encrypted;

[0209] r is a random number belonging to Zn∗; here Zn∗ is the multiplicative group of integers modulo n, consisting of integers that are relatively prime to n. The selection of the random number r is crucial to the security of the encryption, as it ensures that each encryption of the same plaintext produces a different ciphertext, thereby increasing the semantic security of the encryption scheme.

[0210] n is the modulus. In cryptography, the modulus defines the size of the space used for operations. In cryptographic systems based on modular operations, such as the Paillier cipher, n is usually the product of two large prime numbers. Its size directly affects the security of the encryption; the larger n is, the more difficult it is to crack.

[0211] modn² stands for modulo n², meaning the result of the encryption process is modulo n², limiting the result to the range of 0 to n²-1. Modular arithmetic is a fundamental operation in cryptography that ensures data circulates within a finite space and is used to construct algebraic structures of finite groups or rings.

[0212] Build automated optimization methods with autonomous decision-making capabilities.

[0213] In this step, CP-ABE encryption dynamically protects sensitive data. Compared to traditional static AES encryption, it automatically adjusts the encryption scheme based on data sensitivity and usage scenario. AES-256+HMAC is used for low-sensitivity internal data to ensure efficiency, while CP-ABE is used for highly sensitive cross-organizational shared data to enhance security. Leveraging CP-ABE's key generation mechanism, combined with encryption formulas such as MK=(α,β)∈G_T×Z_p and c=gm·rnmodn², precise control over data encryption and decryption is achieved. Furthermore, data usage scenarios are dynamically monitored, and desensitization is automatically triggered when unauthorized access is denied, such as retaining the first six digits of an ID card. This flexible encryption approach protects data privacy while ensuring data availability and security in different scenarios, enhancing the security, flexibility, and intelligence of data management.

[0214] Through the above method, the present invention constructs an automatic optimization method with autonomous decision-making capabilities, realizes accurate evaluation of data transaction quality, automatically optimizes data quality issues, thereby improving data quality, reducing manual intervention, improving efficiency and reducing costs.

[0215] By integrating multi-source data, the present invention constructs an evaluation system with multi-dimensional indicators and an automatic optimization mechanism, effectively solving the problems of traditional data transaction quality evaluation methods such as single dimension, poor adaptability, and high manual dependence, and realizes accurate evaluation and automatic optimization of data transaction quality, greatly improving the quality, efficiency and security of data transactions, reducing transaction risks and costs, and bringing an innovative and practical technical solution to the field of data transaction management technology, which is expected to promote the healthy development of the data transaction industry.

[0216] The above content fully presents the specific implementation methods of the present invention, covering the entire process from data acquisition, preprocessing, quality assessment to automatic optimization. Each step elaborates on the methods, technical means and specific execution logic adopted to ensure the integrity and logic of the content.

[0217] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An automatic optimization method for data transaction quality assessment under multi-source data fusion, characterized by: The following steps are involved: Data collection and preprocessing, collecting multi-source data, including data metadata, third-party verification data, and transaction history data; Data quality assessment model, build a data quality assessment indicator system, and use the decision tree to obtain the data quality assessment model; Automatic optimization performs dynamic optimization, outputs data quality scores according to the evaluation model, and automatically executes data cleaning, supplementation or encryption optimization strategies based on the scoring results.

2. The automatic optimization method for data transaction quality assessment under multi-source data fusion according to claim 1 is characterized in that: Data collection and preprocessing steps include: Data metadata collection: obtaining raw data from data providers, including structured data, unstructured data, and metadata, and standardizing the format of heterogeneous data; Third-party verification data access, calling authoritative data sources to verify data authenticity, establishing a verification rule library, and building verification evidence; Transaction history data integration, extracting key information from historical transaction records, including data buyer feedback scores, transaction dispute records, and data repeat transaction rates.

3. The automatic optimization method for data transaction quality assessment under multi-source data fusion according to claim 2 is characterized in that: In the data metadata collection step, we connect to the Oracle / MySQL relational database through JDBC / ODBC, adopt an incremental pull strategy, and use the NLP pipeline to process the text data.

4. The automatic optimization method for data transaction quality assessment under multi-source data fusion according to claim 3 is characterized in that: In the third-party verification data access step, the data fingerprint calculation formula is used: data fingerprint = SHA3-256 (original data + timestamp + verification result).

5. The automatic optimization method for data transaction quality assessment under multi-source data fusion according to claim 4 is characterized in that: In the transaction history data integration step, the attenuation degree is calculated according to the data value attenuation algorithm: the repeated transaction attenuation factor α=1-e^(-λt), where the λ value for scientific and technological data is 0.15, the λ value for financial data is 0.33, and the λ value for industrial data is 0.08, and t is the data survival period.

6. The automatic optimization method for data transaction quality assessment under multi-source data fusion according to claim 1 is characterized in that: The steps to build a data quality assessment indicator system include: Determine whether the data has been verified by a third-party authority; Determine whether the data provider's historical transaction evaluation is good; Determine whether key data fields are missing; Determine whether the data update time meets the expected frequency; Determine whether data is stored and transmitted encrypted.

7. The automatic optimization method for data transaction quality assessment under multi-source data fusion according to claim 1 is characterized in that: Automatic optimization performs dynamic optimization steps including: Quality scoring and response strategies, and respond accordingly based on the scoring results; Data cleaning uses a dynamic rule engine, interpolation to complete missing time series data, and invalid data is eliminated based on the rule engine; Data supplementation uses GAT entity alignment and automatic request strategies to send API requests to data providers, complete missing fields, and automatically crawl public data sources to supplement key information; Privacy encryption uses CP-ABE encryption to encrypt sensitive fields and dynamically detect data usage scenarios. If there is no permission to access the original data, desensitization is triggered.

8. The automatic optimization method for data transaction quality assessment under multi-source data fusion according to claim 1 is characterized in that: In the data cleaning step, the anomaly score calculation formula S = α·D(X)+(1-α)·[the number of rules that X does not conform to the rule base] / N_rules is used, where α = 0.

7.

9. The automatic optimization method for data transaction quality assessment under multi-source data fusion according to claim 1 is characterized in that: In the data supplementation step, the graph attention network is used to construct the entity relationship graph, and the similarity calculation formula SimGAT(ei,ej)=σ(W·[hi||hj])>0.

85.

10. The automatic optimization method for data transaction quality assessment under multi-source data fusion according to claim 1 is characterized in that: In the privacy encryption step, the encryption decision matrix determines the encryption scheme based on data sensitivity and usage scenarios. When low-sensitivity data is used for internal analysis, the AES-256+HMAC encryption scheme is used; when high-sensitivity data is shared across organizations, the CP-ABE encryption scheme is used.

Images