A custom bucketing method and system based on data security

Abstract

The present invention provides a custom bucketing method and system based on data security, the method comprising storing a bucketing data structure, a data type and a bucketing function into a custom bucketing database; configuring a bucketing strategy for an input table and an output table; performing a first inspection on the input table and the output table, and if qualified, submitting the bucketing task to a Spark cluster for execution; obtaining data records of a data set corresponding to the input table, and matching the data records with the custom bucketing database; determining a bucketing object list based on a bucketing function and performing a second inspection on the bucketing object list based on the input table; if qualified, bucketing the input table based on the bucketing object list and the bucketing function, and writing the bucketing result set into the output table to obtain bucketed data. The present invention can be suitable for generalizing some personal privacy information and improving the generalization effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of data security and privacy protection, and specifically relates to a custom bucketing method and system based on data security. Background Art

[0002] In today's digital age, data has become an important asset for enterprises and organizations, but it also brings severe challenges to data security and privacy protection. With the rapid growth of data volume and the continuous enrichment of data application scenarios, it is crucial to effectively de-identify sensitive data.

[0003] De-identification technology is a method used to process data so that it cannot directly or easily identify a specific individual. Common de-identification algorithms include masking, hashing, encryption, randomization, and generalization. By transforming sensitive data, it balances privacy protection and data availability, providing possibilities for data analysis and research while protecting personal privacy.

[0004] The generalization technique in the de-identification method refers to converting specific values ​​into broader categories and ranges, such as converting specific ages into age groups and generalizing specific addresses into cities or regions. However, there are still some shortcomings in the existing technology: 1. There is a lack of specific generalization methods for numerical data; 2. There is a lack of field bucketing methods for large-scale data tables. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a custom bucketing method and system based on data security, which are used to solve the technical problems in the prior art.

[0006] On the one hand, the present invention provides the following technical solution, a custom bucketing method based on data security, comprising:

[0007] Define the data structure, data type and bucket function of the bucket, and store the data structure, data type and bucket function of the bucket in a custom bucket database;

[0008] Acquire an original data table and use the original data table as an input table, and configure a bucketing strategy for the input table and the output table to obtain configuration information;

[0009] Perform a first inspection on the input table and the output table, and output a first inspection result. If the first inspection result is that the inspection is qualified, create a bucket task and create a Spark task based on the bucket task, and submit the bucket task to the Spark cluster for execution using Spark.submit based on the Spark task;

[0010] Reading the input table as a PCollection object on the Spark cluster, obtaining data records of the data set corresponding to the input table from the PCollection object, matching the data records with the custom bucket database, and assigning bucket information in the corresponding configuration information to each data record;

[0011] Determine a bucket object list based on the bucket information in the configuration information and perform a second test on the bucket object list based on the input table, and output a second test result;

[0012] If the second inspection result is qualified, the input table is bucketed based on the bucketing object list and the bucketing function to output a bucketing result set, and the bucketing result set is written into the output table to obtain bucketed data.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: the present application proposes a customized bucketing method, which supports generalization of numerical types, supports customization of the upper and lower bounds of each bucket, and supports the output of customized bucketing results. The method is suitable for generalizing some personal privacy information, such as common salary, age, height, weight, etc. At the same time, users can customize the bucketing strategy, configure the information of each field of the table in the strategy, including the input name, input type, applied algorithm, algorithm parameters and output name and output type corresponding to the field, based on the big data computing framework, supports the processing of data tables at the billion level, buckets some fields in the original input table and writes them into the output table, and obtains the output table processed by the bucketing algorithm.

[0014] Preferably, in the step of defining the data structure, data type and bucket function of bucketing, and storing the data structure, data type and bucket function of bucketing in a custom bucketing database, the data structure of bucketing includes a bucket upper bound, a bucket lower bound and a bucket result replacement value, the data type of bucketing includes short type data, int type data, long type data, float type data and double type, and the bucketing functions of bucketing include bucketShortToShort function, bucketShortToString function, bucketIntToInt function, bucketIntToString function, bucketLongToLong function, bucketLongToString function, bucketFloatToFloat function, bucketFloatToString function, bucketDoubleToDouble function and bucketDoubleToString function.

[0015] Preferably, the step of obtaining an original data table and using the original data table as an input table, and performing bucket strategy configuration on the input table and the output table to obtain configuration information includes:

[0016] Configure data table configuration information for the input table and the output table respectively, wherein the data table configuration information includes a database address, a database connection user name, a database connection password, a data table name, and a primary key name of the data table;

[0017] The original field strategy information is specified for the fields in the input table and the output table, wherein the field strategy information includes input field type, input field name, algorithm name, algorithm parameters, output field type, and output field name.

[0018] Preferably, the step of performing a first inspection on the input table and the output table and outputting a first inspection result comprises:

[0019] Querying metadata information of the input table and the output table through SQL statements, wherein the metadata information includes field names and field types of the input table and the output table;

[0020] Based on the metadata information, check whether the number of fields in the configuration information is the same as the number of fields in the input table, and whether the field input type in the configuration information is the same as the field type of the field with the same name in the input table, and output a first field check result;

[0021] Check whether the input field type in the configuration information is the same as the input parameter field type of the algorithm configured by the field, and whether the output field type in the configuration information is consistent with the type of the function return value of the algorithm configured by the field, and output the second field test result;

[0022] Check whether the number of fields in the configuration information is the same as the number of fields in the output table, whether the output field type in the configuration information is the same as the field type of the field with the same name in the output table, and output a third field check result;

[0023] If the first field inspection result, the second field inspection result and the third field inspection result are all qualified, then the first inspection result is qualified; if the first field inspection result or the second field inspection result or the third field inspection result is unqualified, then the first inspection result is unqualified.

[0024] Preferably, the steps of reading the input table as a PCollection object on the Spark cluster, obtaining data records of the data set corresponding to the input table from the PCollection object, matching the data records with the custom bucket database, and assigning bucket information in the corresponding configuration information to each data record include:

[0025] Creating a data processing Pipeline object in the Spark cluster according to the data table configuration information of the input table in the configuration information;

[0026] Call the corresponding data source to read the input table through the Pipeline object according to the type of the database of the input table, pass in the configuration information of the input table and configure the primary key as the partition field;

[0027] Read the partitions of the input table using the primary key and the Pipeline object, and read the entire input table into a PCollection object;

[0028] Obtaining data records of the data set corresponding to the input table from the PCollection object, and matching the data records with the configuration information to obtain matching information;

[0029] The matching information is matched with the custom bucket database and the bucket information corresponding to each matching information is called.

[0030] Preferably, the step of determining the bucket object list based on the bucket information in the configuration information and performing a second test on the bucket object list based on the input table, and outputting the second test result comprises:

[0031] Arrange each bucket in the bucket object list in ascending order, traverse the upper and lower bounds of each bucket in the bucket object list, check whether the upper bound of each bucket is greater than the lower bound, and output a first bucket test result;

[0032] Randomly select a bucket in the bucket object list as a reference bucket, check whether the lower bound of the reference bucket is greater than the upper bound of the previous bucket adjacent to it, and check whether the upper bound of the reference bucket is less than the lower bound of the next bucket adjacent to it, and output a second bucket test result;

[0033] Check whether the data in the input table is greater than the lower limit of the first bucket and less than the upper limit of the last bucket, and output the third bucket test result;

[0034] If the first bucket inspection result, the second bucket inspection result and the third bucket inspection result are all qualified, the first inspection result is qualified; if the first bucket inspection result, the second bucket inspection result or the third bucket inspection result is unqualified, the first inspection result is unqualified.

[0035] Preferably, the step of bucketing the input table based on the bucket object list and the bucket function to output a bucket result set, and writing the bucket result set into the output table to obtain bucket data includes:

[0036] Obtain the number of buckets in the bucket object list, sort them in ascending order, and subscript each bucket in sequence;

[0037] Calculate the intermediate subscript based on the subscript of each bucket, and determine whether the data in the input table belongs to the bucket corresponding to the intermediate subscript;

[0038] If the data in the input table belongs to the bucket corresponding to the middle subscript, then the bucket result replacement value of the bucket corresponding to the middle subscript is output;

[0039] If the data in the input table does not belong to the bucket corresponding to the middle subscript, it is determined that the data in the input table is greater than the upper limit of the bucket corresponding to the middle subscript or less than the lower limit of the bucket corresponding to the middle subscript;

[0040] If the data in the input table is larger than the upper limit of the bucket corresponding to the middle subscript, then sequential bucket searches are performed on the right side of the bucket corresponding to the middle subscript until the corresponding bucket is found and the corresponding bucket result replacement value is output;

[0041] If the data in the input table is smaller than the lower limit of the bucket corresponding to the middle subscript, then sequentially perform bucket searches on the left side of the bucket corresponding to the middle subscript until the corresponding bucket is found and the corresponding bucket result replacement value is output;

[0042] Store the bucket result replacement values ​​corresponding to all the data in the input table into the bucket result set, create a Schema object and concatenate the output field type and output field name in the configuration information into the Schema object in sequence, use ParDo and configure the Schema object into the bucket result set through the withOutputSchema method to obtain an output result set with a Schema object, write the output result set into the output table according to the output data source type of the output result set to obtain bucketed data.

[0043] In a second aspect, the present invention provides the following technical solution: a custom bucketing system based on data security, the system comprising:

[0044] A definition module is used to define a bucket data structure, a data type, and a bucket function, and store the bucket data structure, data type, and bucket function in a custom bucket database;

[0045] A configuration module, used to obtain an original data table and use the original data table as an input table, and perform bucketing strategy configuration on the input table and the output table to obtain configuration information;

[0046] an execution module, configured to perform a first inspection on the input table and the output table, output a first inspection result, and if the first inspection result is qualified, create a bucket task and create a Spark task based on the bucket task, and submit the bucket task to the Spark cluster for execution using Spark.submit based on the Spark task;

[0047] A matching module is used to read the input table as a PCollection object on the Spark cluster, obtain data records of the data set corresponding to the input table from the PCollection object, match the data records with the custom bucket database, and assign bucket information in the corresponding configuration information to each data record;

[0048] a verification module, configured to determine a bucket object list based on the bucket information in the configuration information, perform a second verification on the bucket object list based on the input table, and output a second verification result;

[0049] A bucketing module is used to bucket the input table based on the bucketing object list and the bucketing function if the second inspection result is qualified, so as to output a bucketing result set, and write the bucketing result set into the output table to obtain bucketed data.

[0050] In a third aspect, the present invention provides the following technical solution: a computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned custom bucketing method based on data security when executing the computer program.

[0051] In a fourth aspect, the present invention provides the following technical solution: a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned custom bucketing method based on data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A flowchart of a custom bucketing method based on data security provided in the first embodiment of the present invention;

[0054] Figure 2 A structural block diagram of a custom bucketing system based on data security provided in the second embodiment of the present invention;

[0055] Figure 3 A schematic diagram of the hardware structure of a computer provided in another embodiment of the present invention.

[0056] The embodiments of the present invention will be further described below with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the embodiments of the present invention, and should not be construed as limiting the present invention.

[0058] Embodiment 1

[0059] In the first embodiment of the present invention, Figure 1 As shown in the figure, a custom bucketing method based on data security includes:

[0060] S1. Define the data structure, data type and bucket function of the bucket, and store the data structure, data type and bucket function of the bucket in a custom bucket database;

[0061] Wherein, in the step S1, the data structure of the bucket includes a bucket upper bound, a bucket lower bound and a bucket result replacement value, the data type of the bucket includes short type data, int type data, long type data, float type data and double type, and the bucket functions of the bucket include bucketShortToShort function, bucketShortToString function, bucketIntToInt function, bucketIntToString function, bucketLongToLong function, bucketLongToString function, bucketFloatToFloat function, bucketFloatToString function, bucketDoubleToDouble function and bucketDoubleToString function;

[0062] Specifically, for bucketing of short type data, it supports bucketing of short type original data, the upper and lower bounds of the bucket are both short type data, and the replacement value of the bucketing result can be string and short type; for bucketing of int type data, it supports bucketing of int type original data, the upper and lower bounds of the bucket are both int type data, and the replacement value of the bucketing result can be string and int type; for bucketing of long type data, it supports bucketing of long type original data, the upper and lower bounds of the bucket are both long type data, and the replacement value of the bucketing result can be string and long type; for bucketing of float type data, it supports bucketing of float type original data, the upper and lower bounds of the bucket are both float type data, and the replacement value of the bucketing result can be string and float type, and for bucketing of double type, it supports bucketing of double type original data, the upper and lower bounds of the bucket are both double type data, and the replacement value of the bucketing result can be string and double type.

[0063] Specifically, for the bucketShortToShort function, it means that it is a function that buckets short type data and the return value of the function is also of short type; for the bucketShortToString function, it means that it is a function that buckets short type data and the return value of the function is also of string type; for the bucketIntToInt function, it means that it is a function that buckets int type data and the return value of the function is also of int type; for the bucketIntToString function, it means that it is a function that buckets int type data and the return value of the function is also of string type; for the bucketLongToLong function, it means that it is a function that buckets long type data and the return value of the function is also of long type; for the bucketLongToLong function, it means that it is a function that buckets long type data and the return value of the function is also of long type; For the bucketToString function, it means that it is a function that buckets long type data and the return value of the function is also of string type; for the bucketFloatToFloat function, it means that it is a function that buckets float type data and the return value of the function is also of float type; for the bucketFloatToString function, it means that it is a function that buckets float type data and the return value of the function is also of string type; for the bucketDoubleToDouble function, it means that it is a function that buckets double type data and the return value of the function is also of double type; for the bucketDoubleToString function, it means that it is a function that buckets double type data and the return value of the function is also of string type.

[0064] S2, obtaining an original data table and using the original data table as an input table, and performing bucketing strategy configuration on the input table and the output table to obtain configuration information;

[0065] Wherein, the step S2 comprises:

[0066] S21. Configure data table configuration information for the input table and the output table respectively, wherein the data table configuration information includes a database address, a database connection user name, a database connection password, a data table name, and a primary key name of the data table.

[0067] S22, specifying original field strategy information for the fields in the input table and the output table, wherein the field strategy information includes input field type, input field name, algorithm name, algorithm parameters, output field type, and output field name;

[0068] Specifically, the configuration information includes two parts: one is the table library information of the input table and the output table, and the other is the algorithm configuration of the input table and the fields and field mapping information of the output table. As for the algorithm name and algorithm parameters, they are specifically the name of the corresponding bucketing function and the parameters of the bucketing function. The information in this step is pre-configured and can be corrected through the allocation process in subsequent steps.

[0069] S3, performing a first inspection on the input table and the output table, and outputting a first inspection result. If the first inspection result is qualified, creating a bucket task and creating a Spark task based on the bucket task, and submitting the bucket task to the Spark cluster for execution using Spark.submit based on the Spark task;

[0070] Specifically, a Spark task is a task based on a big data distributed computing framework, and Spark.submit represents a tool for submitting a Spark task to a cluster.

[0071] Wherein, the step S3 comprises:

[0072] S31. Query metadata information of the input table and the output table through SQL statements, where the metadata information includes field names and field types of the input table and the output table.

[0073] S32, based on the metadata information, checking whether the number of fields in the configuration information is the same as the number of fields in the input table, and whether the field input type in the configuration information is the same as the field type of the field with the same name in the input table, and outputting a first field check result;

[0074] Specifically, if the number of fields in the configuration information is the same as the number of fields in the input table, and the field input type in the configuration information is the same as the field type of the field with the same name in the input table, then the first field inspection result is qualified inspection; if the number of fields in the configuration information is different from the number of fields in the input table, and the field input type in the configuration information is different from the field type of the field with the same name in the input table, then the first field inspection result is unqualified inspection.

[0075] S33, checking whether the input field type in the configuration information is the same as the input parameter field type of the algorithm configured by the field, and whether the output field type in the configuration information is consistent with the type of the function return value of the algorithm configured by the field, and outputting the second field test result;

[0076] Specifically, if the input field type in the configuration information is the same as the input parameter field type of the algorithm configured by the field, and the output field type in the configuration information is consistent with the type of the function return value of the algorithm configured by the field, then the second field inspection result is a qualified inspection; if the input field type in the configuration information is different from the input parameter field type of the algorithm configured by the field, and the output field type in the configuration information is inconsistent with the type of the function return value of the algorithm configured by the field, then the second field inspection result is a failed inspection.

[0077] S34, checking whether the number of fields in the configuration information is the same as the number of fields in the output table, whether the output field type in the configuration information is the same as the field type of the field with the same name in the output table, and outputting a third field check result;

[0078] Specifically, if the number of fields in the configuration information is the same as the number of fields in the output table, and the output field type in the configuration information is the same as the field type of the field with the same name in the output table, then the third field inspection result is qualified; if the number of fields in the configuration information is different from the number of fields in the output table, and the output field type in the configuration information is different from the field type of the field with the same name in the output table, then the third field inspection result is unqualified.

[0079] S35. If the first field inspection result, the second field inspection result and the third field inspection result are all qualified, the first inspection result is qualified; if the first field inspection result or the second field inspection result or the third field inspection result is unqualified, the first inspection result is unqualified.

[0080] S4, reading the input table as a PCollection object on the Spark cluster, obtaining data records of the data set corresponding to the input table from the PCollection object, matching the data records with the custom bucket database, and assigning bucket information in the corresponding configuration information to each data record;

[0081] Specifically, the corresponding bucketing information here includes a bucketing function and a corresponding bucketing data structure.

[0082] Wherein, the step S4 comprises:

[0083] S41, creating a Pipeline object for data processing in the Spark cluster according to the data table configuration information of the input table in the configuration information;

[0084] Specifically, the Spark cluster is a big data distributed computing framework, and the Pipeline object represents an object class in the Spark cluster for processing massive data.

[0085] S42: Call the corresponding data source to read the input table through the Pipeline object according to the type of the database of the input table, pass in the configuration information of the input table and configure the primary key as the partition field.

[0086] S43, performing partition reading of the input table by using the primary key and the Pipeline object, and reading the entire input table as a PCollection object;

[0087] Specifically, the PCollection object is one of the core concepts for data processing in Apache Beam. It can be regarded as a distributed data collection that can contain a large number of data elements. Apache Beam represents an open source data processing framework from Google.

[0088] S44, obtaining data records of the data set corresponding to the input table from the PCollection object, and matching the data records with the configuration information to obtain matching information;

[0089] Specifically, the matching information here includes the field type, field name, etc. between the data record and the corresponding input table.

[0090] S45: Match the matching information with the custom bucket database and call the bucket information corresponding to each matching information.

[0091] S5, determining a bucket object list based on the bucket information in the configuration information, performing a second test on the bucket object list based on the input table, and outputting a second test result;

[0092] Wherein, the step S5 comprises:

[0093] S51, arranging each bucket in the bucket object list in ascending order, traversing the upper and lower bounds of each bucket in the bucket object list, checking whether the upper bound of each bucket is greater than the lower bound, and outputting a first bucket test result;

[0094] Specifically, if the upper limit of each bucket is greater than the lower limit of the bucket, the first bucket inspection result is qualified; if the upper limit of each bucket is not greater than the lower limit of the bucket, the first bucket inspection result is unqualified.

[0095] S52, randomly selecting a bucket in the bucket object list as a reference bucket, checking whether the lower bound of the reference bucket is greater than the upper bound of the previous bucket adjacent to it, and checking whether the upper bound of the reference bucket is less than the lower bound of the next bucket adjacent to it, and outputting a second bucket test result;

[0096] Specifically, if the lower bucket limit of the benchmark bucket is greater than the upper bucket limit of the previous bucket adjacent to it and the upper bucket limit of the benchmark bucket is less than the lower bucket limit of the next bucket adjacent to it, then the second bucket inspection result is a qualified inspection; if the lower bucket limit of the benchmark bucket is not greater than the upper bucket limit of the previous bucket adjacent to it or the upper bucket limit of the benchmark bucket is not less than the lower bucket limit of the next bucket adjacent to it, then the second bucket inspection result is an unqualified inspection.

[0097] S53, checking whether the data in the input table is greater than the lower limit of the first bucket and less than the upper limit of the last bucket, and outputting the third bucket test result;

[0098] Specifically, if the data in the input table is greater than the lower bound of the first bucket and less than the upper bound of the last bucket, the third bucket test result is qualified; if the data in the input table is not greater than the lower bound of the first bucket or not less than the upper bound of the last bucket, the third bucket test result is unqualified.

[0099] S54. If the first bucket inspection result, the second bucket inspection result and the third bucket inspection result are all qualified, the first inspection result is qualified; if the first bucket inspection result, the second bucket inspection result or the third bucket inspection result is unqualified, the first inspection result is unqualified.

[0100] S6. If the second inspection result is qualified, bucketing the input table based on the bucketing object list and the bucketing function to output a bucketing result set, and writing the bucketing result set into the output table to obtain bucketed data.

[0101] Wherein, the step S6 comprises:

[0102] S61, obtaining the number of buckets in the bucket object list, arranging them in ascending order, and subscripting each bucket in sequence.

[0103] S62: Calculate the intermediate subscript based on the subscript of each bucket, and determine whether the data in the input table belongs to the bucket corresponding to the intermediate subscript.

[0104] S63. If the data in the input table belongs to the bucket corresponding to the middle subscript, output the bucket result replacement value of the bucket corresponding to the middle subscript.

[0105] S64. If the data in the input table does not belong to the bucket corresponding to the middle subscript, determine whether the data in the input table is greater than the upper limit of the bucket corresponding to the middle subscript or less than the lower limit of the bucket corresponding to the middle subscript.

[0106] S65, if the data in the input table is larger than the upper limit of the bucket corresponding to the middle subscript, then sequentially perform bucket searches on the right side of the bucket corresponding to the middle subscript until the corresponding bucket is found and the corresponding bucket result replacement value is output;

[0107] Specifically, if the data in the input table is greater than the upper limit of the bucket corresponding to the middle subscript, a judgment is made on the bucket on the right side of the bucket corresponding to the middle subscript. If the bucket meets the requirements, the corresponding bucket result replacement value is output. If the bucket does not meet the requirements, the search continues on the right side.

[0108] S66. If the data in the input table is smaller than the lower limit of the bucket corresponding to the middle subscript, sequentially perform bucket searches on the left side of the bucket corresponding to the middle subscript until the corresponding bucket is found and the corresponding bucket result replacement value is output;

[0109] Specifically, if the data in the input table is smaller than the lower limit of the bucket corresponding to the middle subscript, a judgment is made on the bucket on the left side of the bucket corresponding to the middle subscript. If the bucket meets the requirements, the corresponding bucket result replacement value is output. If the bucket does not meet the requirements, the search continues on the left side.

[0110] S67, storing the bucket result replacement values ​​corresponding to all the data in the input table into the bucket result set, creating a Schema object and sequentially splicing the output field type and output field name in the configuration information into the Schema object, using ParDo and configuring the Schema object into the bucket result set through the withOutputSchema method to obtain an output result set with a Schema object, and writing the output result set into the output table according to the output data source type of the output result set to obtain bucket data;

[0111] Specifically, the Schema object represents the definition and description of the data structure. It specifies which fields are contained in the data table, the data type, length, constraints and other information of each field. ParDo represents an operation for parallel data processing in Apache Beam, which allows a function to be applied to each element in the PCollection object to achieve parallel conversion or processing of data. The withOutputSchema method is used to define the field name, data type and other information of the output data. For example, if Mysql uses JdbcIO to write data to the output table, if the HiveIO.write method is used to write data to the specified Hive table, Mysql is a relational database, and JdbcIO provides a way to read and write relational databases in the Apache Beam data processing pipeline. Through JdbcIO, SQL statements can be used to read data from or write data to the database. The HiveIO.write method represents an operation for writing data to a Hive table. The Hive table is a data warehouse tool that can map structured data files to a database table.

[0112] The custom bucketing method based on data security provided in the first embodiment of the present invention supports generalization of numerical types, supports customization of the upper and lower bounds of each bucket, and supports output of custom bucketing results. The method is suitable for generalizing some personal privacy information, such as common salary, age, height, weight, etc. At the same time, users can customize the bucketing strategy, and configure the information of each field of the table in the strategy, including the input name, input type, applied algorithm, algorithm parameters and output name and output type corresponding to the field. Based on the big data computing framework, it supports the processing of data tables at the billion level, and writes some fields in the original input table into the output table after bucketing to obtain the output table processed by the bucketing algorithm.

[0113] Embodiment 2

[0114] like Figure 2 As shown, in the second embodiment of the present invention, a custom bucketing system based on data security is provided, and the system includes:

[0115] Definition module 1, used to define the data structure, data type and bucket function of the bucket, and store the data structure, data type and bucket function of the bucket in a custom bucket database;

[0116] Configuration module 2, used to obtain an original data table and use the original data table as an input table, and configure a bucketing strategy for the input table and the output table to obtain configuration information;

[0117] Execution module 3 is used to perform a first inspection on the input table and the output table, output a first inspection result, and if the first inspection result is qualified, create a bucket task and create a Spark task based on the bucket task, and submit the bucket task to the Spark cluster for execution based on the Spark task and using Spark.submit;

[0118] A matching module 4 is used to read the input table as a PCollection object on the Spark cluster, obtain data records of the data set corresponding to the input table from the PCollection object, match the data records with the custom bucket database, and assign bucket information in the corresponding configuration information to each data record;

[0119] A verification module 5, configured to determine a bucket object list based on the bucket information in the configuration information and perform a second verification on the bucket object list based on the input table, and output a second verification result;

[0120] The bucketing module 6 is used to bucket the input table based on the bucketing object list and the bucketing function to output a bucketing result set if the second inspection result is qualified, and write the bucketing result set into the output table to obtain bucketed data.

[0121] The configuration module 2 includes:

[0122] A first configuration submodule, configured to configure data table configuration information for the input table and the output table respectively, wherein the data table configuration information includes a database address, a database connection user name, a database connection password, a data table name, and a primary key name of the data table;

[0123] The second configuration submodule is used to specify original field strategy information for the fields in the input table and the output table, wherein the field strategy information includes input field type, input field name, algorithm name, algorithm parameters, output field type, and output field name.

[0124] The execution module 3 includes:

[0125] A query submodule, used to query the metadata information of the input table and the output table through SQL statements, wherein the metadata information includes the field names and field types of the input table and the output table;

[0126] A first field verification submodule is used to verify whether the number of fields in the configuration information is the same as the number of fields in the input table, whether the field input type in the configuration information is the same as the field type of the field with the same name in the input table, based on the metadata information, and output a first field verification result;

[0127] A second field inspection submodule is used to inspect whether the input field type in the configuration information is the same as the input parameter field type of the algorithm configured by the field, and whether the output field type in the configuration information is consistent with the type of the function return value of the algorithm configured by the field, and output a second field inspection result;

[0128] A third field verification submodule is used to verify whether the number of fields in the configuration information is the same as the number of fields in the output table, whether the output field type in the configuration information is the same as the field type of the field with the same name in the output table, and output a third field verification result;

[0129] The fourth field inspection submodule is used to determine that if the first field inspection result, the second field inspection result and the third field inspection result are all qualified, then the first inspection result is qualified; if the first field inspection result or the second field inspection result or the third field inspection result is unqualified, then the first inspection result is unqualified.

[0130] The matching module 4 comprises:

[0131] Create a submodule, which is used to create a Pipeline object for data processing in the Spark cluster according to the data table configuration information of the input table in the configuration information;

[0132] A calling submodule is used to call the corresponding data source to read the input table through the Pipeline object and according to the type of the database of the input table, pass in the configuration information of the input table and configure the primary key as the partition field;

[0133] An object reading submodule, used to read the partitions of the input table through the primary key and the Pipeline object, and read the entire input table as a PCollection object;

[0134] A matching submodule, configured to obtain data records of the data set corresponding to the input table from the PCollection object, and match the data records with the configuration information to obtain matching information;

[0135] The function calling submodule is used to match the matching information with the custom bucketing database and call the bucketing information corresponding to each matching information.

[0136] The inspection module 5 comprises:

[0137] A first bucket test submodule, used to sort each bucket in the bucket object list in ascending order, traverse the bucket upper bound and bucket lower bound of each bucket in the bucket object list, check whether the bucket upper bound of each bucket is greater than the bucket lower bound, and output a first bucket test result;

[0138] A second bucket test submodule is used to select a bucket in the bucket object list as a reference bucket, check whether the lower bound of the reference bucket is greater than the upper bound of the previous bucket adjacent to it, and check whether the upper bound of the reference bucket is less than the lower bound of the next bucket adjacent to it, and output a second bucket test result;

[0139] A third bucket check submodule is used to check whether the data in the input table is greater than the lower limit of the first bucket and less than the upper limit of the last bucket, and output the third bucket check result;

[0140] The fourth bucket inspection submodule is used to determine that if the first bucket inspection result, the second bucket inspection result and the third bucket inspection result are all qualified, then the first inspection result is qualified; if the first bucket inspection result or the second bucket inspection result or the third bucket inspection result is unqualified, then the first inspection result is unqualified.

[0141] The bucketing module 6 includes:

[0142] A labeling submodule, used to obtain the number of buckets in the bucket object list, arrange them in ascending order, and label each bucket in sequence;

[0143] A calculation submodule, used to calculate the intermediate subscript based on the subscript of each bucket, and determine whether the data in the input table belongs to the bucket corresponding to the intermediate subscript;

[0144] A first output submodule, configured to output a bucket result replacement value of the bucket corresponding to the middle subscript if the data in the input table belongs to the bucket corresponding to the middle subscript;

[0145] A judgment submodule, configured to judge whether the data in the input table is greater than an upper limit of the bucket corresponding to the middle subscript or less than a lower limit of the bucket corresponding to the middle subscript if the data in the input table does not belong to the bucket corresponding to the middle subscript;

[0146] The second output submodule is used for, if the data in the input table is greater than the upper limit of the bucket corresponding to the middle subscript, sequentially searching the buckets on the right side of the bucket corresponding to the middle subscript until the corresponding bucket is found and outputting the corresponding bucket result replacement value;

[0147] A third output submodule is used for, if the data in the input table is smaller than the lower limit of the bucket corresponding to the middle subscript, sequentially searching for buckets on the left side of the bucket corresponding to the middle subscript until the corresponding bucket is retrieved and the corresponding bucket result replacement value is output;

[0148] The fourth output submodule is used to store the bucket result replacement values ​​corresponding to all the data in the input table into the bucket result set, create a Schema object and sequentially concatenate the output field type and output field name in the configuration information into the Schema object, use ParDo and configure the Schema object into the bucket result set through the withOutputSchema method to obtain an output result set with a Schema object, and write the output result set into the output table according to the output data source type of the output result set to obtain bucketed data.

[0149] In some other embodiments of the present invention, the embodiments of the present invention provide the following technical solution: a computer, comprising a memory 102, a processor 101, and a computer program stored in the memory 102 and executable on the processor 101, wherein the processor 101 implements the custom bucketing method based on data security as described above when executing the computer program.

[0150] Specifically, the processor 101 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.

[0151] Among them, the memory 102 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 102 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 102 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 102 may be inside or outside the data processing device. In a specific embodiment, the memory 102 is a non-volatile memory. In a specific embodiment, the memory 102 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0152] The memory 102 may be used to store or cache various data files that need to be processed and / or used for communication, as well as possible computer program instructions executed by the processor 101 .

[0153] The processor 101 implements the above-mentioned custom bucketing method based on data security by reading and executing computer program instructions stored in the memory 102.

[0154] In some embodiments, the computer may further include a communication interface 103 and a bus 100. Figure 3 As shown, the processor 101, the memory 102, and the communication interface 103 are connected via a bus 100 and communicate with each other.

[0155] The communication interface 103 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present invention. The communication interface 103 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0156] The bus 100 includes hardware, software or both, and couples the components of the computer device to each other. The bus 100 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 100 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 100 may include one or more buses. Although embodiments of the present invention describe and illustrate a particular bus, the present invention contemplates any suitable bus or interconnect.

[0157] The computer can execute the custom bucketing method based on data security of the present invention based on the acquisition of the custom bucketing system based on data security, thereby realizing the custom bucketing based on data security.

[0158] In some further embodiments of the present invention, in combination with the above-mentioned custom bucketing method based on data security, the embodiments of the present invention provide the following technical solution: a storage medium having a computer program stored thereon, and the computer program implements the above-mentioned custom bucketing method based on data security when executed by a processor.

[0159] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0160] More specific examples of readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0161] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0162] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A custom bucketing method based on data security, characterized in that: include: Define the data structure, data type and bucket function of the bucket, and store the data structure, data type and bucket function of the bucket in a custom bucket database; Acquire an original data table and use the original data table as an input table, and configure a bucketing strategy for the input table and the output table to obtain configuration information; Perform a first inspection on the input table and the output table, and output a first inspection result. If the first inspection result is that the inspection is qualified, create a bucket task and create a Spark task based on the bucket task, and submit the bucket task to the Spark cluster for execution using Spark.submit based on the Spark task; Reading the input table as a PCollection object on the Spark cluster, obtaining data records of the data set corresponding to the input table from the PCollection object, matching the data records with the custom bucket database, and assigning bucket information in the corresponding configuration information to each data record; Determine a bucket object list based on the bucket information in the configuration information and perform a second test on the bucket object list based on the input table, and output a second test result; If the second inspection result is qualified, bucketing the input table based on the bucketing object list and the bucketing function to output a bucketing result set, and writing the bucketing result set into the output table to obtain bucketed data; In the step of defining the data structure, data type and bucket function of bucketing, and storing the data structure, data type and bucket function of bucketing in a custom bucketing database, the data structure of bucketing includes a bucket upper bound, a bucket lower bound and a bucket result replacement value, the data type of bucketing includes short type data, int type data, long type data, float type data and double type, and the bucketing functions of bucketing include bucketShortToShort function, bucketShortToString function, bucketIntToInt function, bucketIntToString function, bucketLongToLong function, bucketLongToString function, bucketFloatToFloat function, bucketFloatToString function, bucketDoubleToDouble function and bucketDoubleToString function; The step of obtaining the original data table and using the original data table as the input table, and performing bucket strategy configuration on the input table and the output table to obtain configuration information includes: Configure data table configuration information for the input table and the output table respectively, wherein the data table configuration information includes a database address, a database connection user name, a database connection password, a data table name, and a primary key name of the data table; Specifying original field strategy information for the fields in the input table and the output table, the field strategy information including input field type, input field name, algorithm name, algorithm parameters, output field type, and output field name; The step of performing a first inspection on the input table and the output table and outputting a first inspection result comprises: Querying metadata information of the input table and the output table through SQL statements, wherein the metadata information includes field names and field types of the input table and the output table; Based on the metadata information, check whether the number of fields in the configuration information is the same as the number of fields in the input table, and whether the field input type in the configuration information is the same as the field type of the field with the same name in the input table, and output a first field check result; Check whether the input field type in the configuration information is the same as the input parameter field type of the algorithm configured by the field, and whether the output field type in the configuration information is consistent with the type of the function return value of the algorithm configured by the field, and output the second field test result; Check whether the number of fields in the configuration information is the same as the number of fields in the output table, whether the output field type in the configuration information is the same as the field type of the field with the same name in the output table, and output a third field check result; If the first field inspection result, the second field inspection result and the third field inspection result are all inspection-qualified, the first inspection result is inspection-qualified; if the first field inspection result, the second field inspection result or the third field inspection result is inspection-unqualified, the first inspection result is inspection-unqualified; The step of determining a bucket object list based on the bucket information in the configuration information and performing a second test on the bucket object list based on the input table, and outputting a second test result comprises: Arrange each bucket in the bucket object list in ascending order, traverse the upper and lower bounds of each bucket in the bucket object list, check whether the upper bound of each bucket is greater than the lower bound, and output a first bucket test result; Randomly select a bucket in the bucket object list as a reference bucket, check whether the lower bound of the reference bucket is greater than the upper bound of the previous bucket adjacent to it, and check whether the upper bound of the reference bucket is less than the lower bound of the next bucket adjacent to it, and output a second bucket test result; Check whether the data in the input table is greater than the lower limit of the first bucket and less than the upper limit of the last bucket, and output the third bucket test result; If the first bucket inspection result, the second bucket inspection result and the third bucket inspection result are all qualified, the first inspection result is qualified; if the first bucket inspection result, the second bucket inspection result or the third bucket inspection result is unqualified, the first inspection result is unqualified.

2. The custom bucketing method based on data security according to claim 1 is characterized in that: The step of reading the input table as a PCollection object on the Spark cluster, obtaining data records of the data set corresponding to the input table from the PCollection object, matching the data records with the custom bucket database, and assigning bucket information in the corresponding configuration information to each data record comprises: Creating a data processing Pipeline object in the Spark cluster according to the data table configuration information of the input table in the configuration information; Call the corresponding data source to read the input table through the Pipeline object according to the type of the database of the input table, pass in the configuration information of the input table and configure the primary key as the partition field; Read the partition of the input table using the primary key and the Pipeline object, and read the entire input table into a PCollection object; Obtaining data records of the data set corresponding to the input table from the PCollection object, and matching the data records with the configuration information to obtain matching information; The matching information is matched with the custom bucket database and the bucket information corresponding to each matching information is called.

3. The custom bucketing method based on data security according to claim 1 is characterized in that: The step of bucketing the input table based on the bucket object list and the bucket function to output a bucket result set, and writing the bucket result set into the output table to obtain bucket data includes: Obtain the number of buckets in the bucket object list, sort them in ascending order, and subscript each bucket in sequence; Calculate the intermediate subscript based on the subscript of each bucket, and determine whether the data in the input table belongs to the bucket corresponding to the intermediate subscript; If the data in the input table belongs to the bucket corresponding to the middle subscript, then the bucket result replacement value of the bucket corresponding to the middle subscript is output; If the data in the input table does not belong to the bucket corresponding to the middle subscript, it is determined that the data in the input table is greater than the upper limit of the bucket corresponding to the middle subscript or less than the lower limit of the bucket corresponding to the middle subscript; If the data in the input table is larger than the upper limit of the bucket corresponding to the middle subscript, then sequential bucket searches are performed on the right side of the bucket corresponding to the middle subscript until the corresponding bucket is found and the corresponding bucket result replacement value is output; If the data in the input table is smaller than the lower limit of the bucket corresponding to the middle subscript, then sequentially perform bucket searches on the left side of the bucket corresponding to the middle subscript until the corresponding bucket is found and the corresponding bucket result replacement value is output; Store the bucket result replacement values ​​corresponding to all the data in the input table into the bucket result set, create a Schema object and concatenate the output field type and output field name in the configuration information into the Schema object in sequence, use ParDo and configure the Schema object into the bucket result set through the withOutputSchema method to obtain an output result set with a Schema object, write the output result set into the output table according to the output data source type of the output result set to obtain bucketed data.

4. A custom bucketing system based on data security, the system adopts the custom bucketing method based on data security as claimed in claim 1, characterized in that: The system comprises: A definition module is used to define a bucket data structure, a data type, and a bucket function, and store the bucket data structure, data type, and bucket function in a custom bucket database; A configuration module, used to obtain an original data table and use the original data table as an input table, and perform bucketing strategy configuration on the input table and the output table to obtain configuration information; An execution module, configured to perform a first inspection on the input table and the output table, output a first inspection result, and if the first inspection result is qualified, create a bucket task and create a Spark task based on the bucket task, and submit the bucket task to the Spark cluster for execution using Spark.submit based on the Spark task; A matching module is used to read the input table as a PCollection object on the Spark cluster, obtain data records of the data set corresponding to the input table from the PCollection object, match the data records with the custom bucket database, and assign bucket information in the corresponding configuration information to each data record; a verification module, configured to determine a bucket object list based on the bucket information in the configuration information, perform a second verification on the bucket object list based on the input table, and output a second verification result; A bucketing module is used to bucket the input table based on the bucketing object list and the bucketing function if the second inspection result is qualified, so as to output a bucketing result set, and write the bucketing result set into the output table to obtain bucketed data.

5. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the custom bucketing method based on data security as described in any one of claims 1 to 3 is implemented.

6. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the custom bucketing method based on data security as described in any one of claims 1 to 3.

Images