A data cleaning system based on data transaction

Abstract

The application relates to the technical field of data processing, and discloses a data cleaning system based on data transaction, which comprises a federal portrait and demand perception module, a privacy cleaning operator library module and a collaborative cleaning configuration module. The federal portrait and demand perception module is used for constructing a data quality portrait through federal learning and generating a quality report and a demand vector based on transaction demand. The privacy cleaning operator library module is used for providing configurable cleaning operators with built-in differential privacy mechanisms, responding to a strategy optimization module, and delivering privacy parameters. The collaborative cleaning configuration module is used for generating collaborative cleaning parameters through a secure multi-party computing protocol, so that the strategy optimization module can perfect a cleaning strategy. The multi-target strategy optimization module is used for combining the quality report, the demand vector and the collaborative parameters to generate a Pareto optimal cleaning strategy which takes into account utility, cost and privacy. The application generates a Pareto optimal strategy through a multi-target optimization model, achieves automatic strategy adaptation and multi-party safe cooperation, and improves system privacy protection, strategy optimization efficiency and collaborative efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a data cleaning system based on data transactions. Background Technology

[0002] Data cleaning systems are technical systems that optimize the quality, adjust the structure, and unify the semantics of raw data during data transactions. Their main functions include identifying and removing erroneous, duplicate, and missing values, unifying data formats, standardizing fields, and performing noise reduction and correction on the data. Based on data transaction scenarios, these systems particularly emphasize the credibility and usability of data before circulation, sharing, and commercialization, ensuring that data assets have sufficient value and compliance when transferred between buyers and sellers.

[0003] Existing data cleaning systems in data trading scenarios mostly adopt a centralized processing architecture. Data sellers need to transmit raw data to a central platform, where quality optimization is performed through preset cleaning algorithms. Privacy protection uses fixed-parameter noise addition or generalization techniques for unified desensitization. Strategy generation is based on a single optimization objective, generating solutions through preset algorithm logic. Finally, the results are verified through manual verification or comparison with fixed indicators.

[0004] However, current data cleaning systems still face the following problems in data transaction applications: First, rule-driven cleaning methods heavily rely on manually defined rules. When dealing with heterogeneous data from different industries, rules need to be configured separately for each type of data. This not only leads to poor system adaptability and high maintenance costs, but also makes it difficult to cope with sudden changes in data structure. Second, although machine learning has been introduced to improve cleaning effectiveness, the training of such models depends on a large number of high-quality labeled samples. However, in actual transaction environments, data sources are diverse, labels are scarce, or even impossible to label, resulting in insufficient model generalization ability and unstable cleaning effects. Third, data transactions involve providers and receivers, and the two parties often have differences in data field naming, format, and semantics. Existing systems have limited automated support for semantic alignment and still require manual intervention, which seriously affects processing efficiency. In addition, to ensure data security and privacy, many cleaning processes need to be completed locally, making centralized processing difficult. Especially in high-frequency real-time transaction scenarios, where the data volume is large and processing time is tight, system performance can easily become a bottleneck, leading to cleaning delays and affecting the timeliness of transaction decisions. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a data cleaning system based on data transactions, aiming to solve the problems of poor adaptability and high operation and maintenance costs of rule-driven cleaning, insufficient generalization of machine learning models due to scarce labels, lack of automated semantic alignment capabilities among multiple parties in data transactions, and performance bottlenecks of local cleaning in high-frequency scenarios.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a data cleaning system based on data transactions, comprising: The Federated Profiling and Demand Awareness module is used to build data quality profiles through federated learning and generate quality reports and demand vectors based on transaction requirements. The privacy cleaning operator library module provides configurable cleaning operators with built-in differential privacy mechanisms, responds to policy optimization module calls, and passes privacy parameters. The collaborative cleaning configuration module is used to generate collaborative cleaning parameters through a secure multi-party computation protocol, which are then used by the strategy optimization module to improve the cleaning strategy. The multi-objective strategy optimization module is used to combine quality reports, demand vectors, and collaborative parameters to generate Pareto-optimal cleaning strategies that take into account utility, cost, and privacy. The net worth proof and verification module is used to generate verifiable digital credentials based on the cleansing strategy and cleansing results, record them in the distributed ledger, and interact with smart contracts to drive transactions.

[0007] Preferably, the federal profiling and demand awareness module includes: Federated learning-driven distributed quality modeling units are used to receive updates of local data quality models trained locally by each data seller based on their original data, and generate a global data quality profile through a federated aggregation algorithm. The transaction market demand dynamic sensing unit is used to collect and quantify the preferences of data quality dimensions from data trading platform interfaces or potential data buyers, forming a multi-dimensional demand weight vector. The quality report and requirement vector generation unit is used to integrate the global data quality profile and requirement weight vector, generate data quality reports and requirement vectors, and output them to the multi-objective strategy optimization module.

[0008] Preferably, the privacy cleaning operator library module includes: A privacy budget allocation unit is used to dynamically allocate a differential privacy budget to each cleaning operator based on the data sensitivity classification results. The noise injection operator unit is used to add noise that satisfies the differential privacy mechanism to the statistics or intermediate results involved in the data cleaning operation based on the allocated differential privacy budget during the data cleaning process. The privacy cost tracking unit records the differential privacy budget consumed during the execution of each cleaning operator and updates the remaining global privacy budget.

[0009] Preferably, in the noise injection operator unit, the amount of noise added to the numerical statistics is... Calculated using the following formula: ; In the formula, For noise level, For query functions Sensitivity on adjacent datasets, The privacy budget allocation unit allocates a differential privacy budget to the current cleaning operator.

[0010] Preferably, the collaborative cleaning configuration module includes: A multi-party security parameter negotiation unit is used to generate collaboratively cleaned privacy-preserving parameters among data sellers through a secure multi-party computation protocol; The parameter consistency verification unit is used to verify the compliance of the parameters submitted by each participant based on zero-knowledge proof. The joint computing middleware unit is used to encapsulate secure multi-party computation protocols and provide a collaborative cleaning parameter interface to the policy optimization module.

[0011] Preferably, in the multi-party security parameter negotiation unit, the privacy protection parameter... Calculated using the following formula: ; In the formula, For privacy protection parameters, This represents the parameter negotiation function defined in the secure multi-party computation protocol. to Private parameters input by each participant.

[0012] Preferably, the multi-objective strategy optimization module includes: The utility-privacy joint modeling unit is used to construct an optimization objective function that includes data utility metrics, privacy protection strength metrics, and computational cost metrics. The constraint parsing unit is used to parse the constraints in the data buyer demand vector and transform them into the feasible domain boundary of the optimization problem. The distributed solution unit is used to solve multi-objective optimization problems in a distributed manner based on the alternating direction multiplier method, and outputs a Pareto optimal cleaning strategy set.

[0013] Preferably, in the utility-privacy joint modeling unit, the objective function is optimized. Defined by the following formula: ; In the formula, To optimize the objective function, Score the data utility. Rate the level of privacy protection. To calculate the expense score, These are weighting coefficients that are dynamically adjusted based on the demand vector.

[0014] Preferably, the net asset value verification module includes: The net value calculation unit is used to calculate the data utility retention rate based on the difference in characteristic distribution between the cleaned data and the original data. A proof generation unit is used to generate a digital signature proof of the data utility retention rate based on an asymmetric encryption algorithm. The verification execution unit is used to verify the authenticity of the digital signature proof through a blockchain smart contract and to determine whether the data utility retention rate has reached a preset threshold.

[0015] Preferably, the data utility retention rate in the net asset value calculation unit is calculated using the following formula: ; In the formula, Indicates data utility retention rate. Data distribution after cleaning Compared with the original data distribution Between divergence, This is the maximum allowable divergence value set based on the data buyer's needs.

[0016] This invention provides a data cleaning system based on data transactions. It has the following beneficial effects: 1. This invention introduces a federated learning mechanism to construct a quality profile model for multi-source heterogeneous data while ensuring that data remains local and privacy is protected. At the same time, it combines dynamically perceived transaction needs to generate real-time quality assessment reports and personalized demand vectors, thereby achieving comprehensive modeling and intelligent matching of data quality. Compared with traditional centralized quality assessment methods, this solution helps to avoid the risk of user data exposure and improves dynamic perception and response to personalized data transaction needs.

[0017] 2. This invention designs a configurable cleaning operator library with embedded differential privacy mechanism and introduces a privacy budget dynamic allocation strategy based on sensitivity analysis, which helps to improve the privacy protection capability in the data cleaning process. Compared with the traditional extensive method that relies on static noise addition or manual desensitization, this solution not only has higher controllability and adaptability, but also effectively solves the problems of single privacy protection methods, lack of quantification, and opaque execution process.

[0018] 3. This invention constructs a multi-objective optimization model that integrates data utility, privacy protection strength, and computational resource consumption. It can automatically generate Pareto optimal solution sets for cleaning strategies that meet the needs of different scenarios. Unlike previous single-objective cleaning mechanisms that focus on one dimension, this model achieves effective trade-offs and collaborative optimization of multiple indicators, avoiding the problem of strategy imbalance caused by excessive sacrifice of privacy or computational efficiency.

[0019] 4. This invention introduces a net worth proof mechanism and digital certificate system based on distributed ledger technology to ensure the measurability and verifiability of the utility of cleaned data, and supports automated data transaction execution processes driven by smart contracts. Compared with traditional practices that rely on manual review or lack standardized value assessment, this invention helps to improve the transparency and credibility of the transaction process and helps to solve the problems of difficulty in determining the value of cleaned data and lack of trust support for transactions. Attached Figure Description

[0020] Figure 1 This is a system architecture diagram of the present invention; Figure 2 This is a schematic diagram of the federal profiling and demand perception module of the present invention; Figure 3 This is a schematic diagram of the privacy cleaning operator library module of the present invention; Figure 4 This is a schematic diagram of the collaborative cleaning configuration module of the present invention; Figure 5 This is a schematic diagram of the multi-objective strategy optimization module of the present invention; Figure 6 This is a schematic diagram of the net asset value verification module of the present invention. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Please see the appendix Figure 1 - Appendix Figure 6 This invention provides a data cleaning system based on data transactions, including a federated profiling and demand perception module, a privacy cleaning operator library module, a collaborative cleaning configuration module, a multi-objective strategy optimization module, and a net worth proof and verification module.

[0023] The following is a detailed description of each module in the system of this invention.

[0024] The Federated Profiling and Demand Awareness module is used to build data quality profiles through federated learning and generate quality reports and demand vectors based on transaction requirements. Specifically, the federal profiling and demand perception module includes: The federated learning-driven distributed quality modeling unit, the transaction market demand dynamic perception unit, and the quality report and demand vector generation unit are connected sequentially through an internal data bus and control logic to collaboratively complete the assessment of data quality and the accurate perception of transaction demands.

[0025] The federated learning-driven distributed quality modeling unit's main function is to aggregate the local data quality models generated locally by various data sellers to form a global data quality profile.

[0026] First, it receives local data quality model parameters from multiple data sellers, which are trained and updated locally based on their original data. These local model parameters, such as model weights or gradients, are calculated without the data leaving the local area.

[0027] Subsequently, a federated aggregation algorithm is used to weighted aggregate the collected local model parameters. This aggregation process aims to generate a global data quality profile that represents the overall data quality characteristics of all participating parties. This profile reflects a comprehensive evaluation result across multiple dimensions, including data completeness, consistency, and accuracy. The federated aggregation algorithm can be represented as follows: ; In the formula, These are the aggregated global model parameters. The total number of data sellers participating in federated learning. For the first The number of data samples used by each data seller to train a local model. The total data sample size of all participants. For the first Local model parameters uploaded by the seller; The aggregated global data quality profile is then passed to the quality report and requirement vector generation unit.

[0028] The core task of the dynamic sensing unit for demand in the data trading market is to capture and quantify the specific requirements of potential data buyers for data quality in the data trading market.

[0029] Information on potential data buyers’ preferences for various dimensions of data quality can be collected from data trading platforms through pre-defined application programming interfaces (APIs), or through structured questionnaires, direct input, and other methods.

[0030] The collected preference information may include minimum requirements for data accuracy, expected levels of data integrity, and sensitivity to data timeliness. These qualitative or quantitative preference information are analyzed and normalized to ultimately form a multi-dimensional demand weight vector.

[0031] This demand weight vector explicitly expresses the data buyer's evaluation of the relative importance of different data quality dimensions. The demand weight vector can be represented as: ; In the formula, This is the demand weight vector. The elements of the demand weight vector represent the preference weights for each data quality dimension. This represents the total number of data quality dimensions. Indicates vector transpose; This requirement weight vector is then sent to the quality report and requirement vector generation unit.

[0032] The Quality Report and Demand Vector Generation Unit, as the final output of this module, is responsible for integrating the global data quality profile and demand weight vector generated by the aforementioned units.

[0033] First, the evaluation results of each data quality dimension in the global data quality profile are aligned and matched with the weights of each dimension in the demand weight vector.

[0034] Based on this alignment result, the weighted scores of each quality indicator are further calculated and compared with the benchmark set by the buyer. Finally, a structured data quality report is generated, which intuitively shows the extent to which the current dataset meets the specific needs of potential buyers.

[0035] Simultaneously, this unit will solidify the integrated demand information into a standard demand vector, the output of which can be represented as: ; In the formula, The final output demand vector, For integration functions, To create a profile of overall data quality, This is the demand weight vector; The generated data quality report and requirement vector are output together to the multi-objective strategy optimization module, serving as key inputs for formulating its cleaning strategy.

[0036] This implementation method, through the federated profiling and demand awareness module, enables a comprehensive assessment of the quality of multi-source data while protecting data privacy, and can dynamically perceive market transaction demands, providing reliable data support and directional guidance for subsequent data cleaning and transaction decisions.

[0037] The privacy cleaning operator library module provides configurable cleaning operators with built-in differential privacy mechanisms, responds to policy optimization module calls, and passes privacy parameters. Specifically, the privacy cleaning operator library module includes: The privacy budget allocation unit, noise injection operator unit, and privacy loss tracking unit are tightly coupled with the data flow through their internal control flow and interact with other modules of the system through standardized interfaces to work together to achieve differential privacy protection throughout the data processing process.

[0038] The privacy budget allocation unit's core function is to dynamically allocate differential privacy budgets to each specific cleaning operator executed subsequently, based on the data's sensitivity level and a preset overall privacy budget.

[0039] First, obtain the sensitivity classification report of the dataset to be processed from the system configuration module or the federated profiling and demand awareness module. This report can be automatically generated based on data attributes (such as whether it is a personally identifiable information (PII), data content or predefined business rules, or can be manually set by the data administrator.

[0040] Subsequently, based on the obtained data sensitivity classification results, the characteristics of the specific cleaning task (such as query type, expected data utility loss), and the overall privacy budget constraints, a specific differential privacy budget value is calculated and allocated for the cleaning operator to be invoked using a preset allocation strategy.

[0041] Allocation strategies may include: The uniform allocation, on-demand allocation, or adaptive allocation mechanism based on historical utility feedback aims to maximize data utility while satisfying privacy constraints. The allocated budget value is securely passed to the noise injection operator unit and the privacy loss tracking unit.

[0042] The noise injection operator unit is responsible for adding random noise that satisfies the differential privacy mechanism to the statistics or intermediate results involved in the data cleaning operation, according to the differential privacy budget specified by the privacy budget allocation unit, during the actual data cleaning operation.

[0043] It encapsulates a variety of differential privacy cleaning operators for different data types (including numeric, categorical, and text types) and different cleaning objectives (such as outlier handling, missing value imputation, data aggregation, and data generalization).

[0044] When cleaning numerical statistics, the first step is to determine the global or local sensitivity of the corresponding query function (such as summation, mean, or count) on the statistic. The sensitivity is calculated based on the query function itself and the characteristics of the dataset.

[0045] Subsequently, based on the allocated differential privacy budget and the calculated sensitivity, the required amount of noise to be added is calculated using the following formula. Scale parameters: ; In the formula, For noise level, For query functions Sensitivity on adjacent datasets, The privacy budget allocation unit allocates a differential privacy budget to the current cleaning operator.

[0046] Subsequently from Noise is sampled from the Laplace distribution with scale parameter , and the noise is added to the original statistical results to form the desensitized output data.

[0047] The privacy loss tracking unit's main task is to accurately record and track the differential privacy budget consumed during the execution of each cleaning operator globally, ensuring that the overall privacy leakage risk is controllable.

[0048] Each time a noise injection operator unit completes a differential privacy operation, it reports to the unit the specific differential privacy budget value consumed by that operation.

[0049] Maintain a global privacy budget account through a secure, persistent privacy ledger, accumulate the total amount of privacy budget consumed in real time, and update the remaining available privacy budget.

[0050] This accumulation process strictly follows the serial or parallel combinatorial theorem of differential privacy, as well as the advanced combinatorial theorem where applicable, to accurately calculate the overall privacy loss.

[0051] If the remaining budget is insufficient to support subsequent operations, an alert can be triggered or the process can be aborted to prevent exceeding the total budget. The tracking results are not only used for real-time monitoring, but also for subsequent privacy audits and compliance verification.

[0052] This implementation method ensures data privacy and security throughout the entire data cleaning lifecycle by meticulously sensing data sensitivity, employing intelligent privacy budget allocation strategies, calibrating noise injection mechanisms for different scenarios, and conducting comprehensive and rigorous privacy loss tracking audits. At the same time, it strives to balance data utility, providing core technical support for building a trustworthy data trading environment.

[0053] The collaborative cleaning configuration module is used to generate collaborative cleaning parameters through a secure multi-party computation protocol, which are then used by the strategy optimization module to improve the cleaning strategy. Specifically, the collaborative cleaning configuration module includes: The system consists of a multi-party security parameter negotiation unit, a parameter consistency verification unit, and a joint computing middleware unit. These three units are connected sequentially through internal secure communication protocols and logical control to collaboratively complete the secure generation and configuration of parameters required for collaborative cleaning.

[0054] The multi-party security parameter negotiation unit's main function is to enable multiple data sellers (i.e., participants) to jointly calculate one or more shared parameters for collaborative cleaning tasks without directly exposing their own private data or inputs.

[0055] First, it receives private parameters or locally processed data summaries input from multiple data sellers, such as local coefficient values ​​for configuring the joint statistical model or local private thresholds for federated anomaly detection, denoted as: ; In the formula, For the first A probability value, For the first A probability value, For the first A probability value, The total number of probabilities; Subsequently, a specific Secure Multi-Party Computation (SMC) protocol is used to perform this negotiation process. Depending on the complexity and security requirements of the required negotiation parameters, protocols such as secret-sharing (e.g., the Shamir secret-sharing mechanism), homomorphic encryption, or obfuscated circuits can be selected.

[0056] The person in charge of coordinating the necessary information exchange and calculation steps among the participating parties in accordance with the selected SMC protocol, the specific calculation process for parameter negotiation can be represented as follows: ; In the formula, For privacy protection parameters, This represents the parameter negotiation function defined in the secure multi-party computation protocol. to Private parameters input by each participant; This parameter is the result of multi-party consultation and is used for subsequent collaborative cleaning operations.

[0057] The core function of the parameter consistency verification unit is to ensure that, during the multi-party security parameter negotiation process, the parameters or intermediate results submitted by each participant meet the preset compliance requirements and are actually calculated.

[0058] Zero-knowledge proof (ZKP) techniques are used to verify the private parameters of the input. It must meet certain publicly agreed specifications (such as value range constraints) or prove that it strictly follows the computation steps of the SMC protocol.

[0059] The party responsible for validating the submitted proofs may indicate non-compliance or potential calculation errors if any party's proof fails to be verified.

[0060] The output is the verification status of the contributions of each participant. Only when the parameters or behaviors of all relevant parties have passed the consistency verification is the aggregate parameter generated by the multi-party security parameter negotiation unit considered trustworthy and ready to be passed to the joint computing middleware unit.

[0061] The joint computing middleware unit's main function is to act as a bridge between this module and the multi-objective policy optimization module.

[0062] First, the multi-objective strategy optimization module receives trusted aggregated collaborative parameters confirmed by the parameter consistency verification unit. Through the standardized application programming interface (API) provided by this middleware unit, the module can request collaborative cleaning parameters of a specific type or purpose without caring how these parameters are securely generated through the complex SMC protocol.

[0063] This unit is also responsible for managing the lifecycle of secure computing sessions, including session establishment, coordination among participants, and maintenance of secure communication channels.

[0064] Finally, the joint computing middleware unit provides the necessary collaborative cleaning parameters to the multi-objective strategy optimization module on demand. These parameters enable the multi-objective strategy optimization module to fully consider and utilize the benefits brought by multi-party collaboration in its subsequent strategy formulation process, thereby improving and optimizing the data cleaning strategy.

[0065] This implementation method, through a collaborative cleaning configuration module, enables the secure generation and efficient configuration of key parameters in multi-party collaborative data cleaning scenarios. At the same time, it effectively protects the original data privacy of each participant, providing important technical support and parameter foundation for subsequent refined multi-objective strategy optimization and the construction of a trustworthy data trading environment.

[0066] The multi-objective strategy optimization module is used to combine quality reports, demand vectors, and collaborative parameters to generate Pareto-optimal cleaning strategies that take into account utility, cost, and privacy. Specifically, the multi-objective strategy optimization module includes: The system consists of a utility-privacy joint modeling unit, a constraint parsing unit, and a distributed solution unit; these three units are sequentially connected through internal data links to form a logical closed loop.

[0067] The utility-privacy joint modeling unit includes a model building section, a scoring calculation section, and an objective function generation section. The model building section establishes a ternary scoring system for data cleaning strategies and calculates data utility scores for each strategy (SS). Privacy strength score Calculate the cost score The scoring system is defined by the following three functions: Data utility scoring function: ; In the formula, Data cleaning strategy Overall data utility score This represents the total number of quality dimensions involved in the data quality assessment. For the first The weights of each quality dimension in the demand vector For strategy In the Quality improvement value in each quality dimension; Privacy protection strength function: ; In the formula, For strategy Privacy strength score, This represents the cumulative privacy budget consumed by all differential privacy operators in the strategy. A positive constant term used to prevent division by zero; Calculate the cost scoring function: ; In the formula, For strategy Computational cost score, The number of operator categories participating in the computational cost evaluation. For the first Cost weights corresponding to class operators For strategy The Middle The average execution cost of the class operator; The above scoring results are used in the objective function generation section to construct a multi-objective optimization function system. For ease of solving, the objective functions can be aggregated into scalar optimization objective functions of the following form: ; In the formula, To optimize the objective function, Score the data utility. Rate the level of privacy protection. To calculate the expense score, These are weighting coefficients that are dynamically adjusted based on the demand vector.

[0068] The constraint parsing unit comprises a semantic extraction unit, an expression transformation unit, and a boundary mapping unit. The semantic extraction unit extracts the constraint content from the requirement vector generated by the federated profiling and requirement awareness modules. The expression transformation unit converts these semantic constraints into formal mathematical expressions, such as: Lower bound constraint on utility: ; In the formula, Data cleaning strategy Overall data utility score The minimum data utility threshold set for users; Upper limit constraints on privacy budgets (derived from the lower bound of privacy protection): ; In the formula, For strategy Privacy strength score, The minimum privacy threshold set for users, For strategy The cumulative privacy budget consumed in China These are positive constant terms used for boundary control; Upper limit constraint on execution cost: ; In the formula, For strategy Computational cost score, The maximum computing overhead limit set for the user; The above inequalities constitute the feasible solution domain SfeasibleSfeasible, which is uniformly encapsulated into constraint vector form by the boundary mapping department and transmitted to the distributed solution unit.

[0069] The distributed solution unit includes a problem decomposition unit, a local solution unit, and a synchronization coordination unit. The problem decomposition unit transforms the original multi-objective optimization problem into multiple sub-problems, as shown below: ; In the formula, Candidate data cleaning strategy, The set of feasible strategies that satisfy all constraints. For the first A single objective function for the policy The rating results The index is the objective function, with values ​​of 1, 2, and 3, corresponding to data utility, privacy strength, and computational cost, respectively. The local solver uses the ADMM algorithm to process each objective function subproblem independently. Each subproblem solves its corresponding objective independently and calculates the Lagrange dual variables. The synchronization and coordination unit iteratively adjusts the shared variables among all subproblems in the global dimension and updates the dual multipliers.

[0070] In each iteration, the ADMM algorithm performs the following steps: Minimize local subproblems; Aggregation of intermediate variables; Dual variable update.

[0071] The iterative process continues until any of the following termination conditions are met: Reach the preset maximum number of iterations; The Pareto boundary converges between two adjacent cycles; Obtain the non-dominated solution set that satisfies the target number.

[0072] Finally, the non-dominated solution set output by this unit is represented as follows: ; In the formula, For the first A non-dominated data cleaning strategy This represents the final number of Pareto optimal policies. This then constitutes the Pareto optimal solution set, each solution The strategy set, which cannot be strictly dominated by other solutions in at least one target direction, is uniformly encapsulated and output to the net asset value proof and verification module.

[0073] This implementation method achieves comprehensive optimization of the data cleaning strategy under multiple objectives through the coordinated operation of the above three units. It can take into account the balance requirements of data utility, privacy protection and system performance in different transaction scenarios, and has good adaptability and scalability.

[0074] The net worth proof and verification module is used to generate verifiable digital credentials based on the cleansing strategy and cleansing results, record them in the distributed ledger, and interact with smart contracts to drive transactions. Specifically, the net asset value verification module includes: The system comprises a net asset value calculation unit, a proof generation unit, and a verification execution unit. The net asset value calculation unit connects the original data storage unit before cleaning and the result caching unit after cleaning. It is used to read the original data and the cleaned data and establish a utility retention rate evaluation model.

[0075] First, the difference in data distribution before and after cleaning is calculated using the following model: ; In the formula, For the degree of data distribution difference, For the distribution of cleaned data, For the original data distribution, This is a function used to measure the difference in probability distributions; Subsequently, based on the maximum permissible variability, the data utility retention rate is calculated as follows: ; In the formula, Indicates data utility retention rate. Data distribution after cleaning Compared with the original data distribution Between divergence, This is the maximum allowable divergence value set based on the data buyer's needs.

[0076] The proof generation unit includes a signature algorithm unit and a key management unit. The signature algorithm unit is used for... The asymmetric cryptographic signature operation is performed, and the key management unit is used to manage the private key used for signing.

[0077] The signing process is as follows: ; In the formula, For the generated digital signature, Represents the private key The signature function, User data that has been filtered and retained; The verification execution unit connects the blockchain smart contract interface and the data transaction control logic unit. It is used to verify the signature result and determine whether the utility retention requirement is met. The verification logic is as follows: ; In the formula, For public key The signature verification function, Sign the user's signature result. User data that has been filtered and retained. The Boolean value for verifying the result; Returns a boolean value indicating whether the signature verification passed.

[0078] After successful verification, perform the following threshold judgment operation: ; In the formula, The utility value of the filtered and retained user data. The minimum utility threshold; If the conditions are met, the verification execution unit calls the on-chain smart contract logic, triggering the data cleaning delivery and digital certificate on-chain process, and completing traceable rights settlement and verification.

[0079] This implementation method ensures the rapid deployment and execution of the net asset value (NAV) verification mechanism in different trading scenarios, possesses good adaptability and scalability, provides a standardized and automated data processing flow, and improves the system's security, compliance, and operational efficiency.

[0080] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data cleaning system based on data transactions, characterized in that, include: The Federated Profiling and Demand Awareness module is used to build data quality profiles through federated learning and generate quality reports and demand vectors based on transaction requirements. The privacy cleaning operator library module provides configurable cleaning operators with built-in differential privacy mechanisms, responds to policy optimization module calls, and passes privacy parameters. The collaborative cleaning configuration module is used to generate collaborative cleaning parameters through a secure multi-party computation protocol, which are then used by the strategy optimization module to improve the cleaning strategy. The multi-objective strategy optimization module is used to combine quality reports, demand vectors, and collaborative parameters to generate Pareto-optimal cleaning strategies that take into account utility, cost, and privacy. The net worth proof and verification module is used to generate verifiable digital credentials based on the cleansing strategy and cleansing results, record them in the distributed ledger, and interact with smart contracts to drive transactions.

2. The data cleaning system based on data transactions according to claim 1, characterized in that, The federal profiling and demand perception module includes: Federated learning-driven distributed quality modeling units are used to receive updates of local data quality models trained locally by each data seller based on their original data, and generate a global data quality profile through a federated aggregation algorithm. The transaction market demand dynamic sensing unit is used to collect and quantify the preferences of data quality dimensions from data trading platform interfaces or potential data buyers, forming a multi-dimensional demand weight vector. The quality report and requirement vector generation unit is used to integrate the global data quality profile and requirement weight vector, generate data quality reports and requirement vectors, and output them to the multi-objective strategy optimization module.

3. The data cleaning system based on data transactions according to claim 1, characterized in that, The privacy cleaning operator library module includes: A privacy budget allocation unit is used to dynamically allocate a differential privacy budget to each cleaning operator based on the data sensitivity classification results. The noise injection operator unit is used to add noise that satisfies the differential privacy mechanism to the statistics or intermediate results involved in the data cleaning operation based on the allocated differential privacy budget during the data cleaning process. The privacy cost tracking unit records the differential privacy budget consumed during the execution of each cleaning operator and updates the remaining global privacy budget.

4. The data cleaning system based on data transactions according to claim 3, characterized in that, The noise injection operator unit adds noise to the numerical statistics. Calculated using the following formula: ； In the formula, For noise level, For query functions Sensitivity on adjacent datasets, The privacy budget allocation unit allocates a differential privacy budget to the current cleaning operator.

5. A data cleaning system based on data transactions according to claim 1, characterized in that, The collaborative cleaning configuration module includes: A multi-party security parameter negotiation unit is used to generate collaboratively cleaned privacy-preserving parameters among data sellers through a secure multi-party computation protocol; The parameter consistency verification unit is used to verify the compliance of the parameters submitted by each participant based on zero-knowledge proof. The joint computing middleware unit is used to encapsulate secure multi-party computation protocols and provide a collaborative cleaning parameter interface to the policy optimization module.

6. A data cleaning system based on data transactions according to claim 5, characterized in that, In the multi-party security parameter negotiation unit, the privacy protection parameters Calculated using the following formula: ； In the formula, For privacy protection parameters, This represents the parameter negotiation function defined in the secure multi-party computation protocol. to Private parameters input by each participant.

7. A data cleaning system based on data transactions according to claim 1, characterized in that, The multi-objective strategy optimization module includes: The utility-privacy joint modeling unit is used to construct an optimization objective function that includes data utility metrics, privacy protection strength metrics, and computational cost metrics. The constraint parsing unit is used to parse the constraints in the data buyer demand vector and transform them into the feasible domain boundary of the optimization problem. The distributed solution unit is used to solve multi-objective optimization problems in a distributed manner based on the alternating direction multiplier method, and outputs a Pareto optimal cleaning strategy set.

8. A data cleaning system based on data transactions according to claim 7, characterized in that, In the utility-privacy joint modeling unit, the objective function is optimized. Defined by the following formula: ； In the formula, To optimize the objective function, Score the data utility. Rate the level of privacy protection. To calculate the expense score, These are weighting coefficients that are dynamically adjusted based on the demand vector.

9. A data cleaning system based on data transactions according to claim 1, characterized in that, The net asset value verification module includes: The net value calculation unit is used to calculate the data utility retention rate based on the difference in characteristic distribution between the cleaned data and the original data. A proof generation unit is used to generate a digital signature proof of the data utility retention rate based on an asymmetric encryption algorithm. The verification execution unit is used to verify the authenticity of the digital signature proof through a blockchain smart contract and to determine whether the data utility retention rate has reached a preset threshold.

10. A data cleaning system based on data transactions according to claim 9, characterized in that, The data utility retention rate in the net asset value calculation unit is calculated using the following formula: ； In the formula, Indicates data utility retention rate. Data distribution after cleaning Compared with the original data distribution Between divergence, This is the maximum allowable divergence value set based on the data buyer's needs.

Images