A data verification method, electronic device and storage medium

Abstract

This invention relates to the field of computer data management technology, and in particular to a data verification method, electronic device, and storage medium, comprising: obtaining a mapping table t of a dataset to be verified (data); obtaining a data analysis model corresponding to data based on t; obtaining an operator sequence x contained in the data analysis model, wherein the number of operators in x is m; and obtaining the (m-1)th operator x. m‑1 tag type b m‑1 b m‑1 The data can be either a statistical operator or a population package operator. Different methods are used to verify statistical report data and population package data respectively. The present invention determines the data type based on the operator type of the previous operator of the output operator, and uses different processing methods for different data types to protect data privacy, thereby reducing resource waste.

Description

Technical Field

[0001] This invention relates to the field of computer data management technology, and in particular to a data verification method, electronic device, and storage medium. Background Technology

[0002] When data is collected from multiple parties for statistical calculations, due to the security and privacy protection issues of the multi-party data, it is necessary to perform joint calculations on the multi-party data and then output the results to the user. Before outputting the results to the user, the results need to be verified. Verification of the results can ensure the privacy of the original data provided by each party and also ensure the correctness of the data obtained by the user.

[0003] CN107844515A discloses a data compliance inspection method and apparatus. First, it automatically filters data files that meet predetermined conditions, reads fields from the data files, then filters out the fields to be inspected from all read fields, preprocesses the fields to be inspected, and stores the processed fields to be inspected in a first list. Finally, it performs a compliance check on each field to be inspected in the first list according to predetermined rules to obtain whether each field to be inspected conforms to the predetermined rules, thus obtaining the inspection result. However, the above-mentioned prior art also has the following technical problems:

[0004] First, existing technologies do not consider the different data types of the dataset to be verified and use the same data verification method for all data types, which will result in a waste of hardware resources for datasets with high data compliance during the verification process.

[0005] Second, existing technologies use preset file extensions as filtering conditions, that is, they filter out files with the same extension as the preset extensions, extract the fields to be checked from the filtered files, and use the same preset rules to perform compliance checks on different types of fields in the fields to be checked one by one, which consumes a lot of computing resources. Summary of the Invention

[0006] To address the aforementioned technical problems, the present invention adopts the following technical solution: a data verification method, the method comprising the following steps:

[0007] S100, obtain the dataset to be verified, where data is statistical report data or population package data output by the data analysis model, and data includes M fields; the data analysis model includes an operator sequence; the operator sequence is a set of operators configured by the data analysis model according to the execution order.

[0008] S200, obtain the mapping table t of data; t includes the mapping relationship between data and the unique identifier ID of the data analysis model.

[0009] S300, obtain the data analysis model corresponding to the dataset data based on the mapping table t.

[0010] S400, Obtain the operator sequence x = {x1, x2, ..., x} contained in the data analysis model. p ,…,x m-1 ,x m}, where x p Let x be the p-th operator in the data analysis model execution process, where p ranges from 1 to m, and m is the number of operators in x. p With label type b p Tag type b p These are statistical operators or population package operators.

[0011] S500, obtain the (m-1)th operator x m-1 tag type b m-1 If the tag type is b m-1 If the operator is a statistical operator, then `data` belongs to the statistical report category, and step S600 is executed; otherwise, b m-1 For the crowd package operator, where data belongs to the crowd package data, execute step S700.

[0012] S600 uses regular expressions to match each field of data. If all M fields match successfully, the data validation passes; if any field fails to match, the data validation fails.

[0013] The S700 outputs data directly after desensitizing it.

[0014] The present invention has at least the following beneficial effects:

[0015] First, this invention uses the label type of the operator before the output operator in the operator sequence to determine whether the dataset belongs to statistical report data or population package data; and uses regular expressions to examine statistical report data one by one, and directly desensitizes population package data before outputting it; compared with the existing technology that performs field-by-field verification without distinguishing data types, this invention divides the data into two categories and uses different processing methods for different data types, which improves matching efficiency and reduces the waste of hardware computing resources.

[0016] Secondly, the mapping relationship between the dataset to be verified and the data analysis model can be directly obtained from the mapping table, thereby obtaining the sequence of operators executed by the dataset to be verified in the data analysis model. This invention obtains the sequence of operators executed sequentially by the data analysis model of the output dataset through the mapping table, and the data type of data can be determined from the operator sequence, making the determination of the data type of data more accurate and improving the accuracy of data verification. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart for data verification provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 , Figure 1 This invention illustrates a data verification method provided by an embodiment of the present invention, the method comprising the following steps:

[0021] S100, obtain the dataset to be verified, where data is statistical report data or population package data output by the data analysis model, and data includes M fields; the data analysis model includes an operator sequence; the operator sequence is a set of operators configured by the data analysis model according to the execution order.

[0022] The data analysis model is an analysis model built on a low-code modeling platform by dragging and dropping data input operators, various analysis operators, and output operators, and connecting the relationships between the operators; it is used to perform comprehensive calculations on data provided by multiple parties using various operators and output the execution results; optionally, the data analysis model is an attribution analysis model or a user profile analysis model.

[0023] It should be noted that data validation checks whether the data format of the execution result "data" of the data analysis model is correct and performs privacy protection processing on the data; optionally, the data format is the length and type of the data.

[0024] Among them, statistical reports include statistical data, which is generally obtained by performing statistical operations on source data.

[0025] Among them, the audience package is a group of users that are classified and packaged according to preset rules. Generally, users are classified according to their mobile phone number or bank card number. Audience package data can be subjected to logical operations such as intersection and union.

[0026] The fields are data items in the data, and they have different types. Optional fields can be decimal, integer, or string.

[0027] The operator sequence, executed in the following order, includes: input operators, operators performing data operations, and output operators. Optionally, for statistical report data, the operators performing data operations are summation operators, averaging operators, and difference operators. Optionally, for population package data, the operators performing data operations are intersection operators and union operators.

[0028] S200, obtain the mapping table t of data; t includes the mapping relationship between data and the unique identifier ID of the data analysis model.

[0029] The mapping table includes: the mapping relationship between the verification terminal ID and the task ID, and the mapping relationship between the task ID and the unique identifier ID of the data analysis model; wherein, the verification terminal ID is the address of the verification terminal that is pre-configured to verify the dataset data to be verified generated after the data analysis model executes the task; each data analysis model executes multiple tasks, and each task has a unique identifier ID.

[0030] Specifically, the mapping table t reflects the mapping relationship between the data analysis model and the execution result of the data analysis model. The mapping table t is used to obtain the data analysis model corresponding to the dataset to be verified, and then to obtain the set of operators used in the execution of the data analysis model. The data type of data is determined by the operator type, and different data verification methods are used for different data types. That is, the data type of data is determined based on the operators in the data analysis model, making the data type judgment more accurate and improving the accuracy of data verification.

[0031] S300, obtain the data analysis model corresponding to the dataset data based on the mapping table t.

[0032] The steps for obtaining the data analysis model corresponding to the dataset data based on the mapping table t include:

[0033] S320: Obtain the task ID of data based on the verification terminal ID where data is located.

[0034] S340, obtain the unique identifier ID of the data analysis model based on the task ID.

[0035] S360 obtains the data analysis model based on the unique identifier ID of the data analysis model.

[0036] In this process, the mapping table t obtains the task ID of data based on the verification terminal ID where data is located, thereby obtaining the unique identity ID of the data analysis model, and thus the data analysis model of the output data can be obtained. Therefore, the mapping table establishes a mapping relationship between the dataset data and the corresponding data analysis model. This method of obtaining the data analysis model of the output execution result data based on the mapping relationship is more accurate and convenient.

[0037] S400, Obtain the operator sequence x = {x1, x2, ..., x} contained in the data analysis model. p ,…,x m-1 ,x m}, where x p Let x be the p-th operator in the data analysis model execution process, where p ranges from 1 to m, and m is the number of operators in x. p With label type b p Tag type b p These are statistical operators or population package operators.

[0038] Wherein, the m-th operator x in the operator sequence x m For the output operator, x m-1 This outputs the operator from the last operation performed by the previous data analysis model.

[0039] Optionally, statistical operators include summation operators, average operators, and difference operators.

[0040] Optionally, the group-based operators are the intersection operator and the union operator.

[0041] S500, obtain the (m-1)th operator x m-1 tag type b m-1 If the tag type is b m-1 If the operator is a statistical operator, then `data` belongs to the statistical report category, and step S600 is executed; otherwise, b m-1 For the crowd package operator, where data belongs to the crowd package data, execute step S700.

[0042] S600 uses regular expressions to match each field of data; if all M fields match successfully, the data validation passes; if any field fails to match, the data validation fails.

[0043] When using regular expressions to match a specific field q in data, the following steps are performed:

[0044] S620, retrieves the field type of q. Optional field types include integer, decimal, and string.

[0045] S640, retrieve the pre-configured regular expression for this field type.

[0046] Regular expressions are used to describe and match a series of strings that satisfy a certain syntax rule using a single string. They are usually used to search for and replace data that conforms to a certain rule. For example, a regular expression is: / \d{2}-\d{5} / , which can match a string consisting of 2 digits, a hyphen, and 5 digits.

[0047] S660 matches q based on the obtained regular expression.

[0048] Specifically, each field in the `data` object has a corresponding regular expression; a regular expression can match and validate multiple fields of the same type. Validation passes when a field matches the regular expression; otherwise, validation fails.

[0049] Furthermore, when data validation fails, the following steps are executed:

[0050] S620, retrieve all fields f that failed to match.

[0051] S640: Randomly select any number of fields from f for re-validation. If all fields are successfully validated during the re-validation process, output the data; otherwise, manually validate all fields in f one by one, process the fields that fail to match, and then output the data.

[0052] Optionally, further verification can be performed by manually extracting any number of fields from f and validating the data for those fields.

[0053] Specifically, if all randomly selected fields are successfully verified during the re-verification process, the data can be output; if a field fails to be verified during manual re-verification, the field that failed to match the regular expression is re-verified, thus achieving double verification of non-compliant fields that may appear in the dataset. This can avoid dataset verification failures caused by errors in the regular expression matching verification process and improve the accuracy of data verification.

[0054] The S700 outputs data directly after desensitizing it.

[0055] Anonymization, or data masking, involves removing the identity of data related to user security or commercially sensitive information without violating regulations. For example, personal information such as phone numbers and bank card numbers requires anonymization to protect sensitive and private data. Anonymization is a database security technology that can significantly protect user-provided data, preventing others from obtaining the one-to-one mapping relationships between data points.

[0056] Optionally, the data anonymization process can be performed using differential privacy or K-anonymity algorithms.

[0057] Specifically, data anonymization using differential privacy involves adding noise and removing data. For example, consider a validated dataset `data1` and a dataset to be validated `data2`. `data1` has already been sent to the user. `data1` contains 100 user phone numbers, and `data2` contains 101 user phone numbers. Specifically, the 101 records in `data2` contain all 100 records from `data1`. Therefore, a user can obtain newly added phone numbers from `data2` based on the phone numbers in `data1`. To protect data privacy, noise and removal need to be added to `data2` before it is sent to the user. Specifically, adding noise and removing data involves randomly selecting `n` data points from the existing sample library and replacing the `n` data points in `data2` that overlap with those in `data1`. This method prevents users from finding a single, uniquely added phone number when comparing the data in `data2` with the data in `data1`. Differential privacy ensures that the user can only obtain global statistical information, not pinpoint a specific record, thus protecting data privacy and security.

[0058] Specifically, the K-anonymity algorithm is used for data anonymization. For example, in the dataset to be verified, data = {data3, data4}, data3 shows information about four males and one female, while data4 shows the names of four males and one female. Users might try to obtain information about the female among the five individuals based on data. Therefore, the K-anonymity algorithm either adds several female name records to data4 or directly generalizes the name data in data. Optionally, generalization can involve hiding certain characters in all names. The K-anonymity algorithm prevents users from obtaining the one-to-one mapping between data, minimizing data distortion. While preserving the usability of the output data and satisfying basic or complex data analysis and mining needs, it ensures that the privacy attributes of any record in the output data cannot be linked to an individual. Privacy attributes include phone number, age, and salary, etc.

[0059] In summary, this embodiment performs data verification before the execution result of the data analysis model is output to the user. The data verification method is as follows: First, the data analysis model ID corresponding to the dataset to be verified, data, is obtained according to the mapping table. Then, the label type of the operator sequence preceding the output operator is obtained by obtaining the operator sequence in the data analysis model. Based on the label type, the data type of data is determined to be either statistical report data or population package data. For statistical report data, regular expressions are used for data verification. For population package data, after desensitization processing, it is directly output. This invention uses the mapping table to obtain which data analysis model's execution result data belongs to, and then determines the data type based on the operator sequence in the data analysis model, which can ensure the correctness of the obtained data type. Compared with the prior art that performs field-by-field verification without distinguishing data types, this invention divides the data into two categories and uses different processing methods for different data types, which improves matching efficiency and reduces the waste of computing resources.

[0060] Embodiments of the present invention also provide a non-transitory computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiments, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiments.

[0061] Embodiments of the present invention also provide an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0062] Embodiments of the present invention also provide a computer program product including program code, which, when the program product is run on an electronic device, causes the electronic device to perform the steps of the methods described above in various exemplary embodiments of the present invention.

[0063] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the above examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. A data verification method, characterized in that, The method includes the following steps: S100, Obtain the dataset to be verified, where data is statistical report data or population package data output by the data analysis model, and data includes M fields; the data analysis model includes an operator sequence; the operator sequence is a set of operators configured by the data analysis model according to the execution order; S200, obtain the mapping table t of data; t includes the mapping relationship between data and the unique identifier ID of the data analysis model; S300, Obtain the data analysis model corresponding to the dataset data based on the mapping table t; S400, Obtain the sequence of operators x={x1,x2,…,x} contained in the data analysis model. p ,…,x m-1 ,x m }, where x p Let x be the p-th operator in the data analysis model execution process, where p ranges from 1 to m, and m is the number of operators in x. p With label type b p Tag type b p For statistical operators or population package operators; S500, obtain the (m-1)th operator x m-1 tag type b m-1 If the tag type is b m-1 If the operator is a statistical operator, then `data` belongs to the statistical report category, and step S600 is executed; otherwise, b m-1 For the crowd package operator, where data belongs to the crowd package data, execute step S700; S600 uses regular expressions to match each field of data. If all M fields match successfully, the data validation passes; if any field fails to match, the data validation fails. The S700 outputs data directly after desensitizing it.

2. The method according to claim 1, characterized in that, The mapping table includes: the mapping relationship between the verification terminal ID and the task ID, and the mapping relationship between the task ID and the unique identifier ID of the data analysis model; wherein, the verification terminal ID is the address of the verification terminal that is pre-configured to verify the dataset data to be verified generated after the data analysis model executes the task; each data analysis model executes multiple tasks, and each task has a unique identifier ID.

3. The method according to claim 2, characterized in that, The steps in S300 to obtain the data analysis model corresponding to the dataset data based on the mapping table t include: S320, the task ID of the data is obtained based on the verification terminal ID where the data is located; S340, obtain the unique identifier ID of the data analysis model based on the task ID; S360 obtains the data analysis model based on the unique identifier ID of the data analysis model.

4. The method according to claim 1, characterized in that, The m-th operator x in the operator sequence x m For the output operator, x m-1 This outputs the operator from the last operation performed by the previous data analysis model.

5. The method according to claim 1, characterized in that, The statistical operators include summation operators, average operators, and difference operators.

6. The method according to claim 1, characterized in that, The group-based operators include intersection operators and union operators.

7. The method according to claim 1, characterized in that, When the data verification in S600 fails, the following steps are performed: S620, retrieve all fields f that failed to match; S640: Randomly select any number of fields from f for re-validation. If all fields are successfully validated during the re-validation process, output the data; otherwise, manually validate all fields in f one by one, process the fields that fail to match, and then output the data.

8. The method according to claim 1, characterized in that, The data anonymization process involves using differential privacy or K-anonymity algorithms to process the data.

9. A non-transitory computer-readable storage medium storing at least one instruction or at least one program segment, said at least one instruction or said at least one program segment being loaded and executed by a processor to implement the method of any one of claims 1-8.

10. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 9.

Images