Data construction method and system based on privacy set cross big data test

Through the big data test data construction method and system for interchange of privacy sets, the problem of the generation of test data in the existing technology is not realistic enough and not suitable for federal scenarios, and the data generation and intersection calculation of various types and scales is realized, which improves the The accuracy and applicability of model training.

CN118916913AInactive Publication Date: 2024-11-08LANXIANG ZHILIAN (HANGZHOU) TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411069974.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In the prior art, in constructing a big data test scenario based on privacy set interception, the generated test data is not true enough, especially when processing multiple data types and large-scale data, and the data is not satisfied when model training in federated scenarios. Convergence, resulting in poor model accuracy.

Method used

A big data test data construction method and system based on privacy set interception is adopted, and a lexical analysis is performed by inputting command line parameters, constructing two-dimensional data with multiple rows and columns, and performing privacy set interception processing, generating and satisfying multiple types. and scale test data, supporting the intersection calculation of plaintext and ciphertext data, and is suitable for federal modeling scenarios.

Benefits of technology

It realizes the generation of real test data in the privacy collection intercourse scenario, supports multiple data types and scales, improves the diversity and applicability of data, and improves the model training accuracy in federated scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916913A_ABST
    Figure CN118916913A_ABST
Patent Text Reader

Abstract

The invention discloses a data construction method and system based on a privacy set cross-over big data test, and relates to the data construction method based on the privacy set cross-over big data test, which comprises the following steps: S1, inputting command line parameters, and carrying out lexical analysis on the input command line parameters, obtaining specification information of multi-row and multi-column two-dimensional data needing to be constructed; s2, judging whether a directory corresponding to the specification information has a write permission or not, and if yes, executing a step S3; s3, constructing each piece of corresponding data according to fragment parameters in the command line parameters, and writing the data into a file to obtain two-dimensional data; and S4, carrying out privacy set intersection processing on the two-dimensional data obtained by the two parties to obtain an intersection result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data construction method and system based on privacy set intersection big data testing. Background Art

[0002] Private Set Intersection (PSI) is a specific application scenario in the field of privacy computing. PSI allows multiple parties holding their own private data sets to calculate the intersection of their data without leaking any information other than the intersection. In actual application scenarios, the private data sets held by each party will first be converted into ciphertext data before calculating their data intersection with multiple parties.

[0003] In the big data test scenario based on the intersection of privacy sets, the test data needs to meet the diversity requirements. It often requires two-dimensional test data in plain text and cipher text, and the two-dimensional test data in plain text and cipher text needs to support multiple types, such as 16-bit, 32-bit, and 64-bit strings encrypted by md5, as well as data in the formats of integer, floating point, string, Chinese, Boolean, and time. The test data also needs to meet a certain level, such as millions, tens of millions, and hundreds of millions. The column dimension needs to support about ten dimensions in ordinary scenarios, and hundreds of dimensions in extreme scenarios, and even thousands of dimensions. It is obviously unreasonable to construct this data manually.

[0004] The existing technology constructs two-dimensional data with multiple rows and columns through random numbers, character concatenation, and the md5 values ​​of natural numbers. The generated test data is not realistic enough compared with the actual business scenarios while meeting the test requirements. Secondly, there are still certain deficiencies in constructing abnormal test data. For example, there must be certain missing values ​​or abnormal values ​​in the integer column data. Secondly, if the test data constructed by this tool is used to expand the model training in the federated scenario, the data does not meet the convergence requirements and the trained model has poor accuracy.

[0005] In response to the above technical problems, the present invention proposes a data construction method and system based on privacy set intersection big data testing. Summary of the invention

[0006] The purpose of the present invention is to address the defects of the prior art and provide a data construction method and system based on privacy set intersection big data testing.

[0007] In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0008] A data construction method based on privacy set intersection big data testing, comprising:

[0009] S1. Input command line parameters, and perform lexical analysis on the input command line parameters to obtain specification information of multi-row and multi-column two-dimensional data to be constructed;

[0010] S2. Determine whether there is write permission in the directory corresponding to the specification information. If so, execute step S3;

[0011] S3. construct each corresponding piece of data according to the slicing parameters in the command line parameters, and write it into the file to obtain two-dimensional data;

[0012] S4. Perform privacy set intersection processing on the two-dimensional data obtained by both parties to obtain the intersection result.

[0013] Furthermore, the specification information of the multi-row and multi-column two-dimensional data in step S1 includes data volume, feature columns, type of each column, number of shards, whether to add additional target columns, id column type, and file name.

[0014] Furthermore, in step S4, the id column types of the two parties involved are consistent.

[0015] Accordingly, a data construction system based on privacy set intersection big data testing is also provided, including:

[0016] An analysis module is used to input command line parameters and perform lexical analysis on the input command line parameters to obtain specification information of multi-row and multi-column two-dimensional data to be constructed;

[0017] A judgment module is used to judge whether there is write permission in the directory corresponding to the specification information;

[0018] A construction module is used to construct each corresponding piece of data according to the slicing parameters in the command line parameters, and write it into a file to obtain two-dimensional data;

[0019] The processing module is used to perform privacy set intersection processing on the two-dimensional data obtained by both parties to obtain the intersection result.

[0020] Furthermore, the specification information of the multi-row and multi-column two-dimensional data in the analysis module includes data volume, feature columns, type of each column, number of shards, whether to add additional target columns, id column type, and file name.

[0021] Furthermore, the id column types of the two parties involved in the processing module are consistent.

[0022] Compared with the prior art, in the business test scenario of hidden set intersection, the initiator and the participant can generate corresponding test data through this data construction tool to verify the scenario of intersection, the scenario of non-intersection, and the scenario of partial intersection. In addition, this tool can also be extended to the generation of modeling test data in the federated modeling scenario. Before generating data, data with and without target columns are generated according to which party owns the target column. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of a data construction method based on privacy set intersection big data testing provided in Example 1;

[0024] Figure 2 This is a schematic diagram of various parameters of the command line provided in Example 1;. DETAILED DESCRIPTION

[0025] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0026] The purpose of the present invention is to address the defects of the prior art and provide a data construction method and system based on privacy set intersection big data testing.

[0027] Embodiment 1

[0028] This embodiment provides a data construction method based on privacy set intersection big data test, including:

[0029] S1. Input command line parameters, and perform lexical analysis on the input command line parameters to obtain specification information of multi-row and multi-column two-dimensional data to be constructed;

[0030] S2. Determine whether there is write permission in the directory corresponding to the specification information. If so, execute step S3;

[0031] S3. construct each corresponding piece of data according to the slicing parameters in the command line parameters, and write it into the file to obtain two-dimensional data;

[0032] S4. Perform privacy set intersection processing on the two-dimensional data obtained by both parties to obtain the intersection result.

[0033] This embodiment is a command line tool for data construction. The source code is written in golang, runs across platforms, supports more complex command input, and constructs corresponding multi-row and multi-column two-dimensional data according to the input commands. It can be targeted at initiators and participants; among them, the initiator is the demander of the privacy set exchange service, and the demander is the supplier of the privacy set exchange service.

[0034] In step S1, command line parameters are input, and lexical analysis is performed on the input command line parameters to obtain specification information of multi-row and multi-column two-dimensional data to be constructed.

[0035] After entering the command, the lexical analysis will be performed based on the command, and then the specification information of the two-dimensional data to be constructed will be confirmed. The specification information includes the amount of data, feature columns, type of each column, number of shards, whether to add additional target columns, type of id column and file name, etc. For example, the command entered is gdata-r 1000-c6-t "int(2):1,float(2|3):1,string(5):1,ch(3):1,time:1,bool:1" -i m32-m ps 100-fdata. The lexical analysis is: construct a two-dimensional csv data file according to the parameters passed in by the command, as follows: Generate a data file with 1000 rows. The first column constructed is the id column. The default type is integer 1234..., -c 6 specifies 6 columns of features in addition to the id column. The -t parameter specifies the types as integer (2 digits), floating point (2 digits for integers and 3 digits for decimals), string (5 digits), Chinese (3 characters), time (year-month-day format), and Boolean (0 or 1), etc. -i specifies the type of the id column as a 32-bit encrypted string. -mp specifies that there is no target column. The target column is supported by modeling scenario extensions. -ml specifies that there is a target column. The generated data will have an additional target column with a column header of y, and the value is a Boolean value. -s 100 specifies that the shard size is 100. The purpose of setting this parameter is to prevent the program from crashing due to the memory being quickly filled with oom when constructing billion-level big data. -f data specifies that the generated file name with the .csv suffix is ​​data.

[0036] The specification information of each data in this embodiment is constructed by combining the existing library with random functions and certain rules. For example, the integer is generated by randomly taking out the input digits from the constant 123456789 defined; the floating point number is generated in the same way as the integer, except that it is performed twice and the decimal point is spliced; the string is the same way, except that the constant that the string depends on is larger in range, covering all the characters on the keyboard; the 32-bit encrypted string is generated by the golang library function md5.sum() for the id column 1234... The md5 value corresponding to the value 1234... is generated by the golang library function sha256.Sum256() for the id column 1234... The Sum256 value corresponding to the value 1234... is generated; Chinese is also generated by randomly taking the corresponding digits from the constant Chinese string; the time is generated by parsing the year, month, and day from the current timestamp, and the Boolean is generated by randomly taking 1 digit from the value 01. Other types can also be expanded and supported in the future without affecting the original main process.

[0037] In this embodiment, the random function and the certain rules are as follows: the random function generally generates an integer randomly, but in this embodiment, the integer can be multiplied by 0.01 so that it becomes a decimal, which is equivalent to that in this embodiment, a decimal can be obtained randomly. Similarly, the random integer is made the subscript of a string, so in this embodiment, a random character or string can be obtained according to the random subscript.

[0038] In step S2, it is determined whether there is write permission in the directory corresponding to the specification information, and if so, step S3 is executed.

[0039] After determining the specific specifications of the multi-row and multi-column two-dimensional data to be constructed, further determine whether the current directory has write permission. If so, it means that the data is ready to be constructed and written; otherwise, the program is terminated and an error prompt is returned.

[0040] In this embodiment, the file directories of different platforms are somewhat different, such as the file directory of Linux platform is the file directory of Linux, the file directory of Mac platform is the file directory of Mac, and the file directory of Windows platform is the file directory of Windows.

[0041] The purpose of judging whether there is write permission is to create a file under the current directory and write the constructed data into the file. If there is no write permission, the file creation will fail, and it will be prompted that the file cannot be created without write permission. This embodiment takes the Linux platform as an example, and the write permission of this folder can be checked through the ls -al command.

[0042] In step S3, each corresponding piece of data is constructed according to the slice parameters in the command line parameters and written into a file to obtain two-dimensional data.

[0043] This embodiment constructs a corresponding piece of data to be written into the file according to the shard size specified by the shard parameter -s entered in the command line. The constructed data is written one piece at a time. During the operation, a log of the completion of writing each piece of data will be printed, and the progress of data construction can be confirmed based on the log.

[0044] In this embodiment, constructing a corresponding piece of data is specifically as follows: constructing a two-dimensional csv data file through three traversals, the first traversal is to calculate the total number of pieces to be constructed based on the total amount of constructed data obtained by lexical analysis and the default size of 1 million per piece, and then traverse one piece at a time, the second traversal is 1 million rows per piece, and each row is written in a loop, and the third traversal is to traverse each column in each row according to the total number of columns obtained by lexical analysis, and write the type of each column obtained by lexical analysis and the randomly generated data of the corresponding type into the created csv file.

[0045] After compiling the tool into an executable program, set the above parameters after the executable program and run it to construct the corresponding multi-row and multi-column two-dimensional data. The data will be written to the corresponding .csv suffix file. For example, if the name of the compiled executable program is gdata, the command to construct the data is: gdata -r 1000 -c 6 -t "int(2):1,float(2|3):1,string(5):1,ch(3):1,time:1,bool:1" -i m32 -m ps 100 -f data. The generated data is a 1000-row data. The first column constructed is the id column. The default type is integer 1234..., -c 6 specifies 6 columns of features in addition to the id column. The -t parameter specifies the types as integer (2 digits), floating point (2 digits for integers and 3 digits for decimals), string (5 digits), Chinese (3 characters), time (year-month-day format), and Boolean (0 or 1), etc. -i specifies the type of the id column as a 32-bit encrypted string. -mp specifies that there is no target column. The target column is supported by modeling scenario extensions. -ml specifies that there is a target column. The generated data will have an additional target column with a column header of y, and the value is a Boolean value. -s 100 specifies that the shard size is 100. The purpose of setting this parameter is to prevent the program from crashing due to the memory being quickly filled with oom when constructing billion-level big data. -f data specifies that the generated file name with the .csv suffix is ​​data.

[0046] In step S4, the two-dimensional data obtained by the two parties are subjected to privacy set intersection processing to obtain an intersection result.

[0047] The initiator and the participants all generate data through the same data construction tool. As long as the type of the id column is consistent, the data can be 100% hit when intersecting the privacy sets, because the default type of the id column is a positive integer 1234..., and its other types are all 1234... corresponding to the MD5 encryption type, but the number of encryption bits is different.

[0048] Privacy set intersection is the core business in the privacy computing industry, and privacy set intersection requires at least two parties to conduct a business. However, when testing and verifying the core business during the testing phase, two two-dimensional csv data files are required to verify whether the core business is available. At this time, a data construction tool is generated. The data construction tool is only generated to provide corresponding test data for testing the business. The purpose of privacy intersection is to obtain the intersection of two data. The result of privacy set intersection is the intersection data on the business side. For example, Bank A wants to know whether user Zhang San on its side has also opened an account at Bank B. Through privacy set intersection, the corresponding result set of the intersection is obtained.

[0049] This embodiment has the following beneficial effects:

[0050] (1) Use command line parameters to pass in the number of rows and columns and column types of constructed data.

[0051] (2) Construct a combination of data structures with different data types in each column.

[0052] (3) The data is constructed by dynamically specifying the number of shards to ensure that the program runs on big data without being constrained by machine configuration.

[0053] Embodiment 2

[0054] This embodiment provides a data construction system based on privacy set intersection big data testing, including:

[0055] An analysis module is used to input command line parameters and perform lexical analysis on the input command line parameters to obtain specification information of multi-row and multi-column two-dimensional data to be constructed;

[0056] A judgment module is used to judge whether there is write permission in the directory corresponding to the specification information;

[0057] A construction module is used to construct each corresponding piece of data according to the slicing parameters in the command line parameters, and write it into a file to obtain two-dimensional data;

[0058] The processing module is used to perform privacy set intersection processing on the two-dimensional data obtained by both parties to obtain the intersection result.

[0059] Furthermore, the specification information of the multi-row and multi-column two-dimensional data in the analysis module includes data volume, feature columns, type of each column, number of shards, whether to add additional target columns, id column type, and file name.

[0060] Furthermore, the id column types of the two parties involved in the processing module are consistent.

[0061] It should be noted that the data construction system based on privacy set intersection big data testing provided in this embodiment is similar to that in Embodiment 1 and will not be elaborated herein.

[0062] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data construction method based on privacy set intersection big data test, characterized in that: include: S1. Input command line parameters, and perform lexical analysis on the input command line parameters to obtain specification information of multi-row and multi-column two-dimensional data to be constructed; S2. Determine whether there is write permission in the directory corresponding to the specification information. If so, execute step S3; S3. construct each corresponding piece of data according to the slicing parameters in the command line parameters, and write it into the file to obtain two-dimensional data; S4. Perform privacy set intersection processing on the two-dimensional data obtained by both parties to obtain the intersection result.

2. According to the data construction method of claim 1, it is characterized in that: The specification information of the multi-row and multi-column two-dimensional data in step S1 includes data volume, feature columns, type of each column, number of slices, whether to add additional target columns, id column type, and file name.

3. According to the data construction method of claim 1, it is characterized in that: In step S4, the id column types of the two parties involved are consistent.

4. A data construction system based on privacy set intersection big data testing, characterized in that: include: An analysis module is used to input command line parameters and perform lexical analysis on the input command line parameters to obtain specification information of multi-row and multi-column two-dimensional data to be constructed; A judgment module is used to judge whether there is write permission in the directory corresponding to the specification information; A construction module is used to construct each corresponding piece of data according to the slicing parameters in the command line parameters, and write it into a file to obtain two-dimensional data; The processing module is used to perform privacy set intersection processing on the two-dimensional data obtained by both parties to obtain the intersection result.

5. According to claim 4, a data construction system based on privacy set intersection big data testing is characterized in that: The specification information of the multi-row and multi-column two-dimensional data in the analysis module includes the data volume, feature columns, the type of each column, the number of slices, whether to add additional target columns, the ID column type, and the file name.

6. According to claim 4, a data construction system based on privacy set intersection big data testing is characterized in that: The id column types of the two parties involved in the processing module are consistent.

Citation Information

Patent Citations

  • Data encryption method and device based on privacy set intersection

    CN114448702A

  • Privacy set intersection solving method based on hash value segmentation alternate comparison

    CN116318701A

  • Privacy intersection method and device applied to multi-participant scene, medium and equipment

    CN117371035A

  • Data processing method and device for security computing, medium and electronic equipment

    CN118070308A

  • Techniques for private and secure collection and processing of data for generalized analysis

    US20230009067A1