A universal data quality assessment method and device

Abstract

The present invention provides a general data quality assessment method and device, including: performing feature vectorization processing on the original data set to form feature synthesis data; performing data quality assessment model training and feature perturbation processing respectively according to the feature synthesis data, and sending the model training parameters and the perturbed data to the server respectively; receiving the training update parameters and data mean sent by the server, wherein the training update parameters are updated to the data quality assessment model to optimize the model; obtaining the data quality prediction value according to the data quality assessment model, and obtaining the data relative quality estimation value according to the data mean processing. The client processes the original data, and the output is the perturbed data to protect the privacy of the data; the server receives the perturbed data, completes the construction of the data quality assessment model and the calculation of the mean of the data quality feature vector; the client obtains the server-side synthesis model and mean, and calculates the publicly available quality prediction value and relative quality estimation value.

Description

Technical Field

[0001] The present invention relates to the field of big data analysis and data security technology, and in particular to a general data quality assessment method and device. Background Art

[0002] At present, most data quality assessment methods rely on a trusted third party, requiring the data owner to transfer the data to the trusted third party, who then completes the data quality assessment on the plaintext data. There are two main data quality assessment methods in the prior art: 1. Extraction of intrinsic features of the dataset, which is mainly based on data quality features of constraint rules; 2. The contribution of the dataset to the computing task, which is mainly for machine learning / deep learning tasks, to evaluate whether the gradient of the dataset before and after training is closer to the local optimal point.

[0003] The first method mentioned above is a pre-evaluation algorithm. The data owner can complete it locally without a third party. However, this quality evaluation method is very simple and can only reflect the basic quality problems of the data, such as data missing and data errors. The second method mentioned above is a post-evaluation algorithm. The data quality can only be evaluated after the calculation task is completed; but in many cases, data quality is the premise for determining whether the data requester uses the data. A feasible solution is to use the characteristics of the historical data set and the contribution evaluation training model, so that the contribution value of the new data set can be predicted based on the characteristics of the new data set. However, the data set owned by a single data owner is relatively limited, and it is difficult to achieve a good training effect. As a result, it is necessary to rely on a third party to collect various data sets and their contributions for model training. However, third parties are not all trustworthy. When data leaves the data owner's local location and is sent to a third party, it may cause uncontrollable events such as data being copied, disseminated, and leaked by untrustworthy third parties. In addition to the above two methods, in the data market, the relative quality of data needs to be used to reflect the market situation to help data requesters quickly evaluate the quality of the current data set compared with other data sets, so as to decide whether to use the data set. However, the calculation of the relative quality of data also depends on the collection of data sets, and there is a problem of untrustworthy third parties. Summary of the invention

[0004] The present invention provides a universal data quality assessment method and device, which are used to solve the defect that data quality assessment in the prior art relies on a third party, and to achieve data quality assessment while protecting the privacy and security of customer information.

[0005] The present invention provides a general data quality assessment method, which is applied to a client, wherein the client communicates with a server, and the method comprises:

[0006] Perform feature vectorization on the original data set to form feature synthetic data;

[0007] Performing data quality assessment model training and feature perturbation processing respectively according to the feature synthesis data, and sending the training parameters of the data quality assessment model and the data after feature perturbation processing to the server respectively;

[0008] Receiving the training update parameters and the data mean sent by the server, wherein the training update parameters are updated into the data quality assessment model to optimize the model;

[0009] A data quality prediction value is obtained according to the data quality assessment model, and a data relative quality estimation value is obtained according to the data mean processing.

[0010] According to a general data quality assessment method provided by the present invention, the process of performing feature vectorization processing on the original data set to form feature synthetic data includes:

[0011] Performing quality assessment on the data in the original data set according to the constraint rules to obtain the first eigenvector that meets the standard quality;

[0012] Perform cluster analysis on the data in the original data set to infer the similarity data between the data;

[0013] The first feature vector is synthesized with the similarity data to form a second feature vector.

[0014] According to a general data quality assessment method provided by the present invention, the quality of data in the original data set is assessed according to the constraint rules to obtain a first eigenvector that meets the standard quality, including: evaluating the data in the original data set with m data items according to the data quality measurement standard of the constraint rules to obtain a first eigenvector with a length of n-1 that meets the standard quality of the constraint rules.

[0015] According to a general data quality assessment method provided by the present invention, cluster analysis is performed on the data in the original data set to infer similarity data between the data, including:

[0016] Perform feature vectorization on the original data set with m data, and generate an m*k feature matrix formed by m k-dimensional feature vectors;

[0017] An unsupervised clustering method is used to perform clustering inference on the feature matrix to obtain similarity data between the data.

[0018] According to a general data quality assessment method provided by the present invention, the synthesizing the first feature vector and the similarity data to form a second feature vector includes: synthesizing the first feature vector and the similarity data into an n-dimensional vector as the second feature vector.

[0019] According to a general data quality assessment method provided by the present invention, the data quality assessment model training and feature perturbation processing are performed according to the feature synthesis data, and the training parameters of the data quality assessment model and the data after feature perturbation processing are sent to the server respectively, including:

[0020] According to the horizontal federated logistic regression method, the second feature vector is used as training data, and a contribution label for model training is calculated based on the training data to obtain the data quality assessment model, and the model training parameters are sent to the server;

[0021] The second feature vector is normalized, a random response mechanism is applied to perturb the normalized feature, and the perturbed data is sent to the server.

[0022] According to a general data quality assessment method provided by the present invention, the data quality prediction value is obtained according to the data quality assessment model, and the data relative quality estimation value is obtained according to the data mean processing, including:

[0023] Updating the training update parameters sent by the server to the data quality assessment model to obtain an optimized data quality assessment model, and inputting the new data set into the optimized data quality assessment model to obtain a publicly available data quality prediction value of the new data set;

[0024] The publicly available relative data quality estimation value of the new data set is obtained according to the average relative error between the feature vector of the new data set and the feature of the data mean.

[0025] The present invention also provides a general data quality assessment method, which is applied to a server, wherein the server communicates with a plurality of clients, and the method comprises:

[0026] Receiving the training parameters of the data quality assessment model and the data after feature perturbation processing sent by each of the clients; wherein each client applies the same random response mechanism to perform feature perturbation processing;

[0027] Applying a weighted average method to process the training parameters sent by each of the clients to obtain training update parameters, and sending the training update parameters to each of the clients respectively;

[0028] The relationship between the sum of the disturbance values ​​and the sum of the original values ​​is derived according to the disturbance mechanism parameters, the mean of the original values ​​is inferred, and the inferred mean of the original values ​​is used as the data mean and sent to each of the clients.

[0029] The present invention also provides a general data quality assessment device, which is applied to a client, and the client communicates with a server, and the device comprises:

[0030] A feature vectorization unit, used for performing feature vectorization processing on the original data set to form feature synthesis data;

[0031] A feature synthesis data processing unit, used to perform data quality assessment model training and feature perturbation processing according to the feature synthesis data, and send the training parameters of the data quality assessment model and the data after feature perturbation processing to the server respectively;

[0032] A receiving server data unit is used to receive training update parameters and data mean sent by the server, wherein the training update parameters are updated to the data quality assessment model to optimize the model;

[0033] The quality assessment unit obtains a data quality prediction value according to the data quality assessment model, and obtains a data relative quality estimation value according to the data mean processing.

[0034] The present invention also provides a general data quality assessment device, which is applied to a server, and the server communicates with multiple clients, and the device includes:

[0035] A client data receiving unit is used to receive the training parameters of the data quality assessment model and the data after feature perturbation processing sent by each of the clients; wherein each client applies the same random response mechanism to perform feature perturbation processing;

[0036] A training parameter processing unit, used for processing the training parameters sent by each of the clients using a weighted average method to obtain training update parameters, and sending the training update parameters to each of the clients respectively;

[0037] The original value mean processing unit derives the relationship between the sum of the disturbance values ​​and the sum of the original values ​​according to the disturbance mechanism parameters, infers the mean of the original values, takes the inferred original value mean as the data mean and sends it to each of the clients.

[0038] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned general data quality assessment methods are implemented.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned general data quality assessment methods are implemented.

[0040] The general data quality assessment method and device provided by the present invention, the client processes the original data, and outputs the processed data after disturbance to protect the privacy of the data; the server receives the disturbed data, completes the construction of the data quality assessment model and the mean calculation of the data quality feature vector; the client obtains the server-side synthesis model and mean, and calculates the publicly available quality prediction value and relative quality estimation value. The method of the present invention integrates the two quality assessment results of quality prediction value and relative quality estimation value, and comprehensively displays the data quality, so that the data requester can fully understand the data without seeing the data; by combining the transmission of only parameters without the original data and the data disturbance mechanism, data assessment can be completed without relying on a trusted third party, reducing the risk of privacy leakage. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a principle block diagram of the general data quality assessment system provided by the present invention;

[0043] Figure 2 It is a flow chart of a general data quality assessment method from the client's perspective provided by the present invention;

[0044] Figure 3 yes Figure 2 Flow chart of the method of step 210;

[0045] Figure 4 yes Figure 3 The method flow chart of step 212;

[0046] Figure 5 yes Figure 2 The method flow chart of step 220;

[0047] Figure 6 yes Figure 2 The method flow chart of step 220;

[0048] Figure 7 It is a flow chart of a general data quality assessment method from the perspective of the server provided by the present invention;

[0049] Figure 8 It is a schematic diagram of the structure of a universal data quality assessment device from the client's perspective provided by the present invention;

[0050] Fig. 9It is a schematic diagram of the structure of a general data quality assessment device from the perspective of a server provided by the present invention;

[0051] Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] like Figure 1 The embodiment of the present invention provides a general data quality assessment system, including a client and a server.

[0054] The client is used to perform local model training and feature perturbation processing on the data of the original data set after feature vectorization processing, and send the parameters of the local model training and the perturbed data to the server, and process the training update parameters and data mean returned by the server to obtain two data quality assessment parameters, namely, data quality prediction value and relative quality estimation value;

[0055] The server is used to receive the local model training parameters and disturbance feature data sent by the client, obtain the training update parameters after weighted averaging the local model training parameters of each client, and infer the mean of the original value, that is, the data mean, based on the disturbance feature, and return the training update parameters and the data mean to the client respectively.

[0056] In the embodiment of the present invention, the client is the data owner and can directly access the data. The server is an untrusted third party and needs to be prevented from obtaining any information that can infer the content of the data. The quantitative relationship between the server and the client is one-to-many, and one server collects data from multiple clients. In the embodiment of the present invention, it is assumed that the participating parties are semi-honest, that is, honest but curious, which means that the participating parties will perform calculations according to the pre-agreed algorithm, but will try to use the obtained information to infer more information and data.

[0057] like Figure 2 The general data quality assessment method applied to the client in the embodiment of the present invention includes:

[0058] Step 210: performing feature vectorization processing on the original data set to form feature synthesis data;

[0059] like Figure 3The figure shows a specific flow chart of step 210, which specifically includes:

[0060] Step 211: Performing quality assessment on the data in the original data set according to the constraint rules to obtain a first feature vector that meets the standard quality;

[0061] This step specifically includes: evaluating and processing the data in the original data set with m data according to the data quality measurement standard of the constraint rules, and obtaining the first eigenvector with a length of n-1 that meets the standard quality of the constraint rules. The constraint rules in the embodiment of the present invention refer to the basic quality measurement standards designed for the correctness, integrity, consistency, etc. of the data set of size m, such as the number of empty attributes, the number of inconsistent data, the number of value type errors, the number of value range errors, etc., which are counted automatically by the code to form a eigenvector with a length of n-1. In this step, each data in the original data set of m data eventually forms the first eigenvector with a length of n-1 dimensions. This step is the process of analyzing the data quality in the original data set and quantizing the features to generate the eigenvector. Each dimension of the eigenvector generated by the original data set is obtained by the constraint rule statistics, so in this step, only a quality assessment statistic is made for each data.

[0062] Step 212: Perform cluster analysis on the data in the original data set to infer similarity data between the data;

[0063] like Figure 4 The figure shows a specific flow chart of step 212, which specifically includes:

[0064] Step 2121: perform feature vectorization on the original data set with m data, and generate an m*k feature matrix formed by m k-dimensional feature vectors;

[0065] In a specific example, if the computing task, i.e. the evaluation content, is to analyze the domain name data, the domain name is converted into a feature vector according to the needs of the domain name data analysis. That is, this step is to convert the original data set into the form of a feature matrix.

[0066] Step 2122: Apply an unsupervised clustering method to cluster the feature matrix to obtain the data similarity data. In this step, a sufficiently small threshold T is set, and an unsupervised clustering algorithm, such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), is used to cluster the m k-dimensional feature vectors, and the similarity of the data is inferred based on the number of clusters.

[0067] Step 213: synthesize the first feature vector and the similarity data to form a second feature vector.

[0068] This step specifically includes: synthesizing the first feature vector and similarity data into an n-dimensional vector as the second feature vector, so the second feature vector is the output feature synthesis data of step 210. In this step, the output after measurement in step 212 is a feature, which is one dimension of the synthesized n-dimensional feature vector.

[0069] In the embodiment of the present invention, step 211 and step 212 are relatively independent processes.

[0070] Step 220: Performing data quality assessment model training and feature perturbation processing respectively according to the feature synthesis data, and sending the training parameters of the data quality assessment model and the data after feature perturbation processing to the server respectively;

[0071] In the embodiment of the present invention,

[0072] like Figure 5 The figure shows a specific flow chart of step 220, which specifically includes:

[0073] Step 221: according to the horizontal federated logistic regression method, the second feature vector is used as training data, and the contribution label to model training is calculated based on the training data to obtain the data quality assessment model, and the model training parameters are sent to the server;

[0074] The input in this step is the output of step 210, and the data label represents the contribution of the training data to the computing task, that is, the quality assessment model training calculation. In the specific example of training the domain name analysis model, that is, the contribution of the gradient change of the data contribution when training the domain name analysis model. Specifically, the model training process of the computing task is to continuously approach the local optimal point through the gradient descent method. When a piece of data is input into the computing model, the gradient will change. A gradient decrease indicates that the data contributes a lot, otherwise it indicates a small contribution; the more the gradient decreases, the greater the contribution. In the embodiment of the present invention, the data label is known before training.

[0075] Preferably, in this step, before transmitting the training model parameters to the server, in order to ensure the privacy of the calculation process, the model parameters are perturbed using a differential privacy algorithm and then sent to the server.

[0076] Step 222: normalize the second feature vector, perturb the normalized feature using a random response mechanism, and send the perturbed data to the server.

[0077] The input in this step is the output of step 210. First, the features are normalized, and then the features are perturbed using a random response mechanism. This needs to satisfy ε-local differential privacy, that is, the probability that any two values ​​are perturbed to the same output is no greater than e ε, and send the perturbed value to the server.

[0078] Step 230: receiving the training update parameters and the data mean sent by the server, wherein the training update parameters are updated into the data quality assessment model to optimize the model;

[0079] In this step, the server integrates the training parameters of each client to form training update parameters, and returns them to the local model of each client for the next round of iteration.

[0080] Step 240: Obtain a data quality prediction value according to the data quality assessment model, and obtain a data relative quality estimation value according to the data mean processing.

[0081] like Figure 6 The figure shows a specific flow chart of step 240, which specifically includes:

[0082] Step 241: Update the training update parameters sent by the server to the data quality assessment model to obtain an optimized data quality assessment model, input the new data set into the optimized data quality assessment model to obtain a publicly available data quality prediction value of the new data set that does not disclose any original data; in this step, the numerical range of the data quality prediction value is [0,1]. In this step, based on the model parameters processed by the server to make the model converge, the client loads the parameters locally for quality prediction of the new data set.

[0083] Step 242: Obtain a publicly available data relative quality estimate of the new data set based on the average relative error between the feature vector of the new data set and the feature of the data mean. The relative quality estimate obtained in this step is a value that can be directly publicly output without leaking any privacy information, and the value range is [0,1].

[0084] In the embodiment of the present invention, the client completes the processing and handling of the original data set locally to ensure that the output of the client is the disturbed data to protect the privacy of the data; the server receives the disturbed data, and completes the construction of the data quality assessment model and the mean calculation of the data quality feature vector based on the processed data; the client obtains the model and mean synthesized by the server, and calculates the publicly available quality prediction value and relative quality score.

[0085] In the embodiment of the present invention, 1) multiple data quality assessment methods are integrated to fully display the data quality so that the data requester can fully understand the data without seeing the data; 2) the federated learning algorithm is combined to realize the prediction of the contribution of the data set with privacy protection that does not rely on a trusted third party; 3) the differential privacy algorithm is combined to realize the relative quality assessment with privacy protection that does not rely on a trusted third party. The core idea of ​​the federated learning algorithm is to transfer parameters without transferring data. Each data owner trains the model locally and updates the parameters to the third party for integration as the parameters for this round of model update; after the third party integrates, the parameters are distributed to each data owner for updating the local model. Iterate until convergence. The core idea of ​​differential privacy is to use random variables that obey a certain distribution to perturb the original data, so that the original data is indistinguishable, and the relationship between the expectation of a function of the perturbed value and the expectation of the original value function can be inferred based on the knowledge of probability theory, so as to realize the calculation of the original value function value based on the perturbation value function.

[0086] like Figure 7 The embodiment of the present invention further provides a general data quality assessment method, which is applied to a server, wherein the server communicates with a plurality of clients, and the method comprises:

[0087] Step 710: receiving the training parameters of the data quality assessment model and the data after feature perturbation processing sent by each of the clients; wherein each client applies the same random response mechanism to perform feature perturbation processing;

[0088] In the embodiment of the present invention, in order to protect the privacy of the client data, the client performs disturbance processing before sending the data to the server.

[0089] Step 720: Apply a weighted average method to process the training parameters sent by each of the clients to obtain training update parameters, and send them to each of the clients respectively;

[0090] The input in this step is the parameters of the local model training from each client; the parameters of the synthetic model are obtained using a weighted average algorithm, and the parameters of the synthetic model are sent to each client as training update parameters for optimizing the local model of each client.

[0091] Step 730: derive the relationship between the sum of the disturbance values ​​and the sum of the original values ​​according to the disturbance mechanism parameters, infer the mean of the original values, use the inferred mean of the original values ​​as the data mean and send it to each of the clients.

[0092] In the embodiment of the present invention, step 720 and step 730 are independent processing procedures, which return the processed training update parameters and data mean to the client respectively, and the client uses the returned values ​​to perform quality prediction and relative quality calculation of the new data set respectively.

[0093] like Figure 8 The embodiment of the present invention further provides a general data quality assessment device, which is applied to a client, and the client communicates with a server, and the device includes:

[0094] The data feature vectorization unit 810 is used to perform feature vectorization processing on the original data set to form feature synthesis data;

[0095] In the embodiment of the present invention, the data feature quantization unit 810 includes two subunits, namely a data quality quantization subunit and a data similarity measurement subunit.

[0096] The data quality vectorization subunit is used to vectorize the data quality based on the constraint rules, where the constraint rules are designed for the basic quality measurement standards such as correctness, completeness, and consistency of the data set of size m, such as the number of empty attributes, the number of inconsistent data, the number of value type errors, the number of value range errors, etc., which are automatically counted by the code to form a feature vector of length n-1;

[0097] The data similarity measurement subunit is used to vectorize m pieces of original data according to the computing task, generate an m*k feature matrix, and then apply an unsupervised clustering algorithm to cluster the m k-dimensional feature vectors, and infer the similarity of the data based on the number of clusters.

[0098] Finally, the results of the data quality quantization subunit and the data similarity measurement subunit are combined into feature synthesis data.

[0099] A feature synthesis data processing unit 820 is used to perform data quality assessment model training and feature perturbation processing according to the feature synthesis data, and send the training parameters of the data quality assessment model and the data after feature perturbation processing to the server respectively;

[0100] In an embodiment of the present invention, each client constructs and trains a local data quality assessment model based on the feature synthesis data obtained from data processing, and sends the model training parameters after local training to the server. However, in order to ensure the privacy of the calculation process, the model training parameters are perturbed by a differential privacy algorithm and then sent to the server.

[0101] In the embodiment of the present invention, the feature synthesis data processing unit 820 processes the feature synthesis data by normalizing the features and then perturbing the features using a random response mechanism. This needs to satisfy ε-local differential privacy, that is, the probability that any two values ​​are perturbed to the same output is no greater than e ε , and send the perturbed value to the server.

[0102] A server data receiving unit 830 is used to receive training update parameters and data mean values ​​sent by the server, wherein the training update parameters are updated into the data quality assessment model to optimize the model;

[0103] The quality assessment unit 840 obtains a data quality prediction value according to the data quality assessment model, and obtains a data relative quality estimation value according to the data mean processing.

[0104] In the embodiment of the present invention, the client loads the training update parameters locally for quality prediction of the new data set, and outputs a publicly available prediction value that does not disclose any original data.

[0105] The principle of calculating the relative quality of a data set is to calculate the average relative error between the feature vector and the data mean of the local data set as the relative quality estimate of the data set; this value does not disclose any privacy information and can be directly output publicly.

[0106] like Fig. 9 The embodiment of the present invention further provides a general data quality assessment device, which is applied to a server, and the server communicates with multiple clients, and the device includes:

[0107] The client data receiving unit 910 is used to receive the training parameters of the data quality assessment model and the data after feature perturbation processing sent by each of the clients; wherein each client applies the same random response mechanism to perform feature perturbation processing;

[0108] A training parameter processing unit 920 is used to process the training parameters sent by each of the clients using a weighted average method to obtain training update parameters, and send them to each of the clients respectively;

[0109] In the embodiment of the present invention, the training parameter processing unit 920 applies a weighted average algorithm to obtain the parameters of the synthesis model as the parameters for this round of update, and then returns the updated parameters to the local model of the client for the next round of iteration.

[0110] The original value mean processing unit 930 derives the relationship between the sum of the disturbance values ​​and the sum of the original values ​​according to the disturbance mechanism parameters, infers the mean of the original values, takes the inferred original value mean as the data mean and sends it to each of the clients.

[0111] In the embodiment of the present invention, each client applies the same random response mechanism to perform feature vector perturbation processing, and derives the relationship between the sum of the perturbation values ​​and the sum of the original values ​​according to the perturbation mechanism parameters, thereby inferring the data mean of the original values; the data mean is returned to the client for relative quality calculation.

[0112] Combine the following Fig.10A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention is described as follows: Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute a general data quality assessment method, which includes: performing feature vectorization processing on the original data set to form feature synthesis data; performing data quality assessment model training and feature perturbation processing respectively according to the feature synthesis data, and sending the training parameters of the data quality assessment model and the data after feature perturbation processing to the server respectively; receiving the training update parameters and data mean sent by the server, wherein the training update parameters are updated to the data quality assessment model to optimize the model; obtaining a data quality prediction value according to the data quality assessment model, and obtaining a data relative quality estimation value according to the data mean processing.

[0113] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0114] On the other hand, an embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the general data quality assessment method provided by the above methods, which includes: performing feature vectorization processing on the original data set to form feature synthesis data; performing data quality assessment model training and feature perturbation processing according to the feature synthesis data, and sending the training parameters of the data quality assessment model and the data after feature perturbation processing to the server respectively; receiving the training update parameters and data mean sent by the server, wherein the training update parameters are updated to the data quality assessment model to optimize the model; obtaining a data quality prediction value according to the data quality assessment model, and obtaining a data relative quality estimation value according to the data mean processing.

[0115] On the other hand, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned general data quality assessment methods: performing feature vectorization processing on the original data set to form feature synthesis data; performing data quality assessment model training and feature perturbation processing according to the feature synthesis data, and sending the training parameters of the data quality assessment model and the data after feature perturbation processing to the server respectively; receiving the training update parameters and data mean sent by the server, wherein the training update parameters are updated to the data quality assessment model to optimize the model; obtaining a data quality prediction value according to the data quality assessment model, and obtaining a data relative quality estimation value according to the data mean processing.

[0116] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A general data quality assessment method, characterized in that: Applied to a client, the client communicates with a server, and the method includes: Perform feature vectorization on the original data set to form feature synthetic data; Performing data quality assessment model training and feature perturbation processing respectively according to the feature synthesis data, and sending the training parameters of the data quality assessment model and the data after feature perturbation processing to the server respectively, wherein the quantitative relationship between the server and the client is one-to-many, and one server collects data from multiple clients; Receiving the training update parameters and the data mean sent by the server, wherein the training update parameters are updated into the data quality assessment model to optimize the model; Obtain a data quality prediction value according to the data quality assessment model, and obtain a data relative quality estimation value according to the data mean processing; Applied to a server, the server communicates with a plurality of clients, the method comprising: Receive the training parameters of the data quality assessment model and the data after feature perturbation processing sent by each of the clients; wherein each client applies the same random response mechanism to perform feature perturbation processing, the quantitative relationship between the server and the client is one-to-many, and one server collects data from multiple clients; Applying a weighted average method to process the training parameters sent by each of the clients to obtain training update parameters, and sending the training update parameters to each of the clients respectively; The relationship between the sum of the disturbance values ​​and the sum of the original values ​​is derived according to the disturbance mechanism parameters, the mean of the original values ​​is inferred, and the inferred mean of the original values ​​is used as the data mean and sent to each of the clients.

2. The general data quality assessment method according to claim 1, characterized in that: The process of performing feature vectorization processing on the original data set to form feature synthesis data includes: Performing quality assessment on the data in the original data set according to the constraint rules to obtain the first eigenvector that meets the standard quality; Perform cluster analysis on the data in the original data set to infer the similarity data between the data; The first feature vector is synthesized with the similarity data to form a second feature vector.

3. The general data quality assessment method according to claim 2, characterized in that: The method of performing quality assessment on the data in the original data set according to the constraint rules to obtain a first eigenvector that meets the standard quality includes: evaluating the data in the original data set with m data items according to the data quality measurement standard of the constraint rules to obtain a first eigenvector with a length of n-1 that meets the standard quality of the constraint rules.

4. The general data quality assessment method according to claim 3, characterized in that: The cluster analysis is performed on the data in the original data set to infer the similarity data between the data, including: Perform feature vectorization on the original data set with m data, and generate an m*k feature matrix formed by m k-dimensional feature vectors; An unsupervised clustering method is used to perform clustering inference on the feature matrix to obtain similarity data between the data.

5. The general data quality assessment method according to claim 4, characterized in that: The synthesizing the first feature vector and the similarity data to form a second feature vector includes: synthesizing the first feature vector and the similarity data into an n-dimensional vector as the second feature vector.

6. The general data quality assessment method according to claim 5, characterized in that: The data quality assessment model training and feature perturbation processing are performed according to the feature synthesis data, and the training parameters of the data quality assessment model and the data after the feature perturbation processing are sent to the server, respectively, including: According to the horizontal federated logistic regression method, the second feature vector is used as training data, and a contribution label for model training is calculated based on the training data to obtain the data quality assessment model, and the model training parameters are sent to the server; The second feature vector is normalized, a random response mechanism is applied to perturb the normalized feature, and the perturbed data is sent to the server.

7. The general data quality assessment method according to claim 6, characterized in that: The step of obtaining a data quality prediction value according to the data quality assessment model and obtaining a data relative quality estimation value according to the data mean processing includes: Updating the training update parameters sent by the server to the data quality assessment model to obtain an optimized data quality assessment model, and inputting the new data set into the optimized data quality assessment model to obtain a publicly available data quality prediction value of the new data set; A publicly available data relative quality estimation value of the new data set is obtained according to an average value of relative errors between the feature vector of the new data set and the feature of the data mean.

8. A general data quality assessment device, characterized in that: Applied to a client, the client communicates with a server, and the device comprises: A feature vectorization unit, used for performing feature vectorization processing on the original data set to form feature synthesis data; A feature synthesis data processing unit, used to perform data quality assessment model training and feature perturbation processing according to the feature synthesis data, and send the training parameters of the data quality assessment model and the data after feature perturbation processing to the server respectively; A receiving server data unit, used for receiving training update parameters and data mean sent by the server, wherein the training update parameters are updated into the data quality assessment model to optimize the model, wherein the quantitative relationship between the server and the client is one-to-many, and one server collects data from multiple clients; A quality assessment unit, which obtains a data quality prediction value according to the data quality assessment model and obtains a data relative quality estimation value according to the data mean processing; Applied to a server, the server communicates with multiple clients, and the device includes: A client data receiving unit is used to receive the training parameters of the data quality assessment model and the data after feature perturbation processing sent by each of the clients; wherein each client applies the same random response mechanism to perform feature perturbation processing, the quantitative relationship between the server and the client is one-to-many, and one server collects data from multiple clients; A training parameter processing unit, used for processing the training parameters sent by each of the clients using a weighted average method to obtain training update parameters, and sending the training update parameters to each of the clients respectively; The original value mean processing unit derives the relationship between the sum of the disturbance values ​​and the sum of the original values ​​according to the disturbance mechanism parameters, infers the mean of the original values, takes the inferred original value mean as the data mean and sends it to each of the clients.

Images