Method and apparatus for identifying target type data in mass data

Abstract

The application discloses a kind of mass data in identifying target type data method, device and electronic equipment, the method includes: according to the sampling rule corresponding to data table type in data warehouse from data table n field respectively extract n sample data;The i sample data is input to the M interface of field identification model, and the identification result of the i sample data on M preset field type is obtained;According to the identification result of the i sample data on M preset field type, determine the probability that the i field is judged as M preset field type;According to the probability, target type field is identified.The application determines the probability that the i field is judged as M preset field type from the whole according to target type field, so that the identification result can more reflect the probability of field sample value as a whole, not just field sample value, so that the target type field identification result is more accurate, and the protection of sensitive data security is strengthened, effectively avoiding sensitive data leakage.

Description

Technical Field

[0001] This invention relates to the field of data security processing technology, and more specifically, to a method, apparatus, electronic device, and computer-readable medium for identifying target type data in massive amounts of data. Background Technology

[0002] With the rapid development of the internet, the value and accessibility of massive databases like enterprise data warehouses have greatly increased. However, this has also brought significant challenges to data security. Therefore, accurately identifying valuable target data types within a data warehouse has become crucial. In a data warehouse, data is primarily stored in tables by field; thus, accurately identifying target field types is essential.

[0003] Currently, identifying target type fields requires the application system to retrieve the data table through a JDBC database connection pool and determine whether a sampled value of a single field in the data table matches a predetermined target type field. If a match is found, the field is identified as the target type field. This approach only focuses on whether the sampled value of a single field matches the predetermined target type field, ignoring the proportion of the matched field value in the total sample. This affects the accuracy of the identification results and poses a security risk of sensitive data leakage. Summary of the Invention

[0004] In view of this, the main objective of the present invention is to provide a method, apparatus, electronic device and computer-readable medium for identifying target type data in massive amounts of data, in order to at least partially solve at least one of the above-mentioned technical problems.

[0005] To address the aforementioned technical problems, the first aspect of this invention proposes a method for identifying target type data from massive datasets, the method comprising:

[0006] Based on the sampling rules corresponding to the data table types in the data warehouse, n sample data are extracted from each of the n fields of the data table;

[0007] The i-th sample data is input into M interfaces of the field recognition model to obtain the recognition results of the i-th sample data on M preset field types; where: each interface of the field recognition model corresponds to the recognition of a preset field type;

[0008] The probability of classifying the i-th field as one of the M preset field types is determined based on the recognition results of the i-th sample data on the M preset field types;

[0009] The target type field is identified based on the probability.

[0010] Where M, n, and i are all natural numbers greater than zero, and i is less than or equal to n.

[0011] According to a preferred embodiment of the present invention, the sample data is in JSON format; the step of inputting the i-th sample data into M interfaces of the field recognition model to obtain the recognition results of the i-th sample data on M preset field types includes:

[0012] The i-th sample data is decomposed according to the JSON structure to obtain multiple primary key values ​​of the i-th sample data;

[0013] Each primary key value is input into M interfaces of the field recognition model to obtain the recognition results of each primary key value in M ​​preset field types;

[0014] The identification result of the i-th sample data in the M preset field types is determined based on the identification result of each primary key value in the M preset field types.

[0015] According to a preferred embodiment of the present invention, the probability q of the i-th field being determined as the j-th preset field type is... ij Obtain it using the following formula:

[0016]

[0017] Where: N 1ij N is the number of samples for which the recognition result of the i-th sample data on the j-th preset field type is the first recognition result, N is the total number of the i-th sample data, and j is a natural number greater than zero and less than or equal to M.

[0018] According to a preferred embodiment of the present invention, the method further includes:

[0019] Configure the probability threshold for each preset field type;

[0020] Determine whether the probability of the target type field being identified as the j-th preset field type is greater than the probability threshold of the j-th preset field type;

[0021] If it is greater than the specified j-th preset field type, mark the type of the target type field.

[0022] The target type field is desensitized according to the type.

[0023] According to a preferred embodiment of the present invention, the data table includes: a partitioned table and a non-partitioned table; for the non-partitioned table, n sample data are extracted from n fields of the data table respectively using a first sampling rule; for the partitioned table, n sample data are extracted from n fields of each partition in sequence according to the partition order.

[0024] According to a preferred embodiment of the present invention, the method further includes:

[0025] Extract the target type data table based on the target type field, and configure a multi-level approval mechanism for the target type data table;

[0026] And / or:

[0027] Perform data anonymization on the target type fields.

[0028] According to a preferred embodiment of the present invention, the preset field type includes at least one of the following: name type, identity information type, contact information type, and account information type.

[0029] To address the aforementioned technical problems, a second aspect of the present invention provides an apparatus for identifying target type data from massive amounts of data, the apparatus comprising:

[0030] The sampling module is used to extract n sample data from n fields of a data table according to the sampling rules corresponding to the data table type in the data warehouse.

[0031] The first recognition module is used to input the i-th sample data into M interfaces of the field recognition model to obtain the recognition results of the i-th sample data on M preset field types; wherein: each interface of the field recognition model corresponds to the recognition of a preset field type;

[0032] The determination module is used to determine the probability that the i-th field is identified as one of the M preset field types based on the recognition results of the i-th sample data on the M preset field types;

[0033] The second identification module is used for the probability identification target type field;

[0034] Where M and n are natural numbers, and i is less than or equal to n.

[0035] According to a preferred embodiment of the present invention, the sample data is in JSON format; the first identification module includes:

[0036] The decomposition module is used to decompose the i-th sample data according to the JSON structure to obtain multiple primary key values ​​of the i-th sample data;

[0037] The input module is used to input each primary key value into M interfaces of the field recognition model to obtain the recognition results of each primary key value in M ​​preset field types;

[0038] The sub-determination module is used to determine the recognition result of the i-th sample data on the M preset field types based on the recognition result of each primary key value on the M preset field types.

[0039] According to a preferred embodiment of the present invention, the determining module obtains the probability q of the i-th field being determined as the j-th preset field type using the following formula. ij :

[0040]

[0041] Where: N 1ij N is the number of samples for which the recognition result of the i-th sample data on the j-th preset field type is the first recognition result, N is the total number of the i-th sample data, and j is a natural number greater than zero and less than or equal to M.

[0042] According to a preferred embodiment of the present invention, the apparatus further includes:

[0043] The configuration module is used to configure the probability threshold for each preset field type;

[0044] The judgment module is used to determine whether the probability of the target type field being identified as the j-th preset field type is greater than the probability threshold of the j-th preset field type;

[0045] The marking module is used to mark the type of the target type field according to the j-th preset field type if the value is greater than the specified value.

[0046] The desensitization module is used to desensitize the target type field according to the type.

[0047] According to a preferred embodiment of the present invention, the data table includes: a partitioned table and a non-partitioned table; the sampling module extracts n sample data from n fields of the data table for the non-partitioned table using a first sampling rule; and extracts n sample data from n fields of each partition in the partitioned table according to the partition order.

[0048] According to a preferred embodiment of the present invention, the apparatus further includes:

[0049] The extraction module is used to extract a target type data table based on the target type field and configure a multi-level approval mechanism for the target type data table;

[0050] And / or:

[0051] The processing module is used to desensitize target type fields.

[0052] According to a preferred embodiment of the present invention, the preset field type includes at least one of the following: name type, identity information type, contact information type, and account information type.

[0053] To solve the above-mentioned technical problems, a third aspect of the present invention provides an electronic device, comprising:

[0054] Processor; and

[0055] A memory that stores computer-executable instructions, which, when executed, cause the processor to perform the methods described above.

[0056] To address the aforementioned technical problems, a fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs that, when executed by a processor, implement the above-described method.

[0057] This invention extracts n sample data from n fields of a data table according to sampling rules corresponding to the data table types in the data warehouse; inputs the i-th sample data into M interfaces of the field recognition model to obtain the recognition results of the i-th sample data on M preset field types; further, based on the recognition results of the i-th sample data on the M preset field types, it determines the probability that the i-th field is identified as one of the M preset field types; thus, based on the probability that the i-th field is identified as one of the M preset field types, it identifies the target type field as a whole, making the recognition result more reflective of the overall probability of the field sample values, rather than just the field sample values, making the target type field recognition result more accurate, and effectively avoiding the situation of sensitive data leakage caused by the inaccurate determination of the target type field in related technologies. Attached Figure Description

[0058] To make the technical problems solved by this invention, the technical means adopted, and the technical effects achieved clearer, specific embodiments of this invention will be described in detail below with reference to the accompanying drawings. However, it should be noted that the drawings described below are merely drawings of exemplary embodiments of this invention. Those skilled in the art can obtain drawings of other embodiments based on these drawings without any creative effort.

[0059] Figure 1 This is a flowchart illustrating a method for identifying target type data from massive datasets according to an embodiment of the present invention.

[0060] Figure 2 This is a schematic diagram of the structural framework of a device for identifying target type data in massive data according to an embodiment of the present invention;

[0061] Figure 3 This is a structural block diagram of an exemplary embodiment of an electronic device according to the present invention;

[0062] Figure 4 This is a schematic diagram of an embodiment of a computer-readable medium according to the present invention. Detailed Implementation

[0063] Exemplary embodiments of the invention will now be described more fully with reference to the accompanying drawings. While these exemplary embodiments can be implemented in various specific ways, they should not be construed as limiting the invention to the embodiments set forth herein. Rather, these exemplary embodiments are provided to make the content of the invention more complete and to facilitate a full communication of the inventive concept to those skilled in the art.

[0064] Subject to the inventive concept, the structures, performance, effects or other features described in a particular embodiment may be combined in any suitable manner with one or more other embodiments.

[0065] In the description of specific embodiments, detailed descriptions of structures, performance, effects, or other features are provided to enable those skilled in the art to fully understand the embodiments. However, this does not preclude those skilled in the art from implementing the present invention with technical solutions that do not contain the aforementioned structures, performance, effects, or other features under specific circumstances.

[0066] The flowcharts in the accompanying drawings are merely illustrative examples and do not imply that the solution of this invention must include all the content, operations, and steps shown in the flowcharts, nor do they imply that the execution must be performed in the order shown in the diagrams. For example, some operations / steps in the flowcharts can be decomposed, some operations / steps can be combined or partially combined, etc. Without departing from the inventive spirit of this invention, the execution order shown in the flowcharts can be changed according to the actual situation.

[0067] The box in the attached diagram Figure 1 Generally, these refer to functional entities, and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0068] The same reference numerals in the accompanying drawings denote the same or similar elements, components, or parts, and therefore, repeated descriptions of the same or similar elements, components, or parts may be omitted below. It should also be understood that although terms such as first, second, third, etc., indicating numbers may be used herein to describe various devices, elements, components, or parts, these devices, elements, components, or parts should not be limited by these terms. That is, these terms are only used to distinguish one from another. For example, a first device may also be referred to as a second device, without departing from the essential technical solution of the invention. Furthermore, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0069] Please see Figure 1 , Figure 1This invention provides a method for identifying target type data from massive datasets, such as... Figure 1 As shown, the method includes:

[0070] S1. Based on the sampling rules corresponding to the data table types in the data warehouse, extract n sample data from each of the n fields of the data table;

[0071] This embodiment primarily focuses on identifying target type fields for massive amounts of data in data warehouses and large databases. Taking Hive as an example, data tables in Hive can include partitioned tables and non-partitioned tables. Partitioned tables divide a large table into different levels of data directories for storage, with each data directory corresponding to a partition. For example, the first-level data directory partitions the table into front, middle, and back partitions, and the second-level data directory further partitions these partitions. In this step, we first determine whether the data table is partitioned or non-partitioned. For non-partitioned tables, we use a first sampling rule to extract n sample data from each of the n fields of the data table. The first sampling rule can be a preset sampling rule, such as the limit sampling rule. For partitioned tables, due to business continuity and business security planning, the initial data in the data table may be plaintext, while subsequent data becomes ciphertext, but the previous table data remains unchanged. Therefore, it is necessary to sample data from different periods of the data table. The partition order can reflect the sequential order of the table data. Therefore, we can extract n sample data from each of the n fields of each partition according to the partition order. The partitioning order can be determined based on the level of the data target. For example, multiple partitions can be extracted from the front, middle, and back areas according to the first-level data directory. Then, from the extracted partitions, n sample data points can be extracted from each of the n fields according to the second-level data directory. Where n is a natural number greater than zero, and i = 1, 2, ..., n.

[0072] In the specific extraction process, one sample data is extracted for each field. The sample data extracted from the i-th field is denoted as the i-th sample data. Here, the sample data can be different field values ​​within the field.

[0073] Furthermore, in existing technologies, application systems obtain data tables through JDBC database connection pools. However, the data read by the application system via JDBC is limited and cannot meet the requirements for identifying target type fields in massive datasets. Before this step, all data tables in the Hive source database can be obtained through a computing engine suitable for large datasets, thus satisfying the computational needs of massive datasets. Further, the big data engine can determine whether each data table has already identified sensitive data, and for data tables that have not yet identified sensitive data, steps S1 to S4 of this invention are executed. To facilitate the determination, identified data tables can be stored in an identified table. Simply querying the identified table to see if the current data table exists allows determination of whether the current data table has already been identified as sensitive data. The computing engine can be a Spark engine, a Flink engine, etc.

[0074] S2. Input the i-th sample data into the M interfaces of the field recognition model to obtain the recognition results of the i-th sample data on the M preset field types;

[0075] In this embodiment, the field recognition model may include multiple interfaces for input data. Each interface corresponds to recognizing a preset field type. That is, each interface can identify whether a field is the preset field type corresponding to the interface based on the input sample data. If yes, it outputs "1"; otherwise, it outputs "0". For example, the field recognition model may consist of multiple machine models, each of which can be considered an interface corresponding to a preset field type. The field recognition model may also consist of multiple recognition rules, each of which can be considered an interface corresponding to a preset field type.

[0076] The preset field type is a pre-defined type of the target type field to be identified, where M is a natural number greater than zero, and M can be determined based on the number of target type fields to be identified. For example, the preset field type may include at least one of the following: name type, identity information type, contact information type, and account information type. In one example, the field recognition model includes four interfaces, corresponding to the recognition of four target type fields: name (e.g., personal name, company name, etc.), identity information (e.g., ID card, corporate taxpayer identification number, etc.), contact information (e.g., mobile phone number, email address, etc.), and account information (e.g., bank card account, membership account, etc.).

[0077] In one example, the i-th field is the name field in the data table, and the i-th sample is: Zhang San, Li Si, Wang Wu, Information Network and Technology. The field recognition model includes a name determination interface for identifying fields of name type and a mobile phone number determination interface for identifying fields of contact information type. In this step, inputting Zhang San, Li Si, Wang Wu, and Information Network and Technology into the name determination interface of the field recognition model yields the name type recognition result for the i-th sample: 1, 1, 1, 0; inputting Zhang San, Li Si, Wang Wu, and Information Network and Technology into the mobile phone number determination interface of the field recognition model yields the contact information type recognition result for the i-th sample: 0, 0, 0, 0.

[0078] In practice, sample data can be transmitted in various formats, such as XML and JSON. Different formats require different processing methods in this step. Therefore, before proceeding to this step, it is necessary to determine the transmission format of the sample data and process it accordingly.

[0079] JSON format, as a common data transmission format, is widely used in intermediate data processing. However, existing target type field recognition processes require a predefined JSON structure for JSON sample data, which cannot meet the needs of processing arbitrary JSON data. In this embodiment, if the sample data is in JSON format, sending the i-th sample data to multiple interfaces of the field recognition model to obtain the recognition results of the i-th sample data on M preset field types includes:

[0080] S21. Decompose the i-th sample data according to the JSON structure to obtain multiple primary key values ​​of the i-th sample data;

[0081] The JSON structure is enclosed in curly braces "{}", which contain zero or more comma-separated key-value pairs. During the decomposition process, the primary key value corresponding to each key within the curly braces is extracted. For example, based on the hierarchical structure, the two primary key values, value1 and value2, for the positions "abc" and "abd" in the JSON are extracted. For instance, given the table `table_info` with the field `custum`, the following three sample data entries are extracted:

[0082] {"a":{"b":{"c":"Zhang San","d":"Information and Technology"}}}

[0083] {"a":{"b":{"c":Li Si","d":"Zhao Liu"}}}

[0084] {"a":{"b":{"c":"Computer Networks","d":"Networks and Media"}}}

[0085] After decomposing the JSON structure, the two primary key values ​​for "abc" and "abd" in the JSON locations of the three sample data entries are as follows:

[0086] abc Zhang San abd Information and Technology

[0087] abc Li Si abd Zhao Liu

[0088] abc Computer Networks abd Networks and Media

[0089] S22. Input each primary key value into the M interfaces of the field recognition model to obtain the recognition results of each primary key value in the M preset field types;

[0090] For example, by sequentially inputting the six primary key values ​​of the above three sample data into the name type determination interface of the field recognition model, the recognition results of the two primary key values ​​of the three sample data in the name type are as follows:

[0091] abc 1 abd 0

[0092] abc 1 abd 1

[0093] abc 0 abd 0

[0094] S23. Determine the recognition result of the i-th sample data on the M preset field types based on the recognition result of each primary key value on the M preset field types.

[0095] For example, it can be determined whether the sum of the recognition results of all primary key values ​​in each sample data in the preset field type is greater than a threshold (e.g., 1). If it is greater, the recognition result of the corresponding sample data in the preset field type is determined to be 1; otherwise, it is 0. Accordingly, the recognition results of the two primary key values ​​of the above 3 sample data in the name type in step S22 can be obtained as follows: 1, 1, 0.

[0096] Through the above steps S21 to S23, the processing of JSON data in any format is achieved.

[0097] S3. Determine the probability that the i-th field is identified as one of the M preset field types based on the recognition results of the i-th sample data on the M preset field types;

[0098] In one example, the probability of each field being classified as the j-th preset field type is determined, and then the probability q of the i-th field being classified as the j-th preset field type is obtained.ij Obtain it using the following formula:

[0099]

[0100] Where: N 1ij The first recognition result is the number of samples for which the recognition result of the i-th sample data on the j-th preset field type is the first recognition result. N is the total number of the i-th sample data, and j is a natural number greater than zero and less than or equal to M. For example, the first recognition result means that the j-th interface output result is "1".

[0101] For example, in one example, the data table `table_info` has a `name` field. We extract three values ​​from the `name` field of `table_info`: Zhang San, Li Si, and Information Network and Technology. These are then input into the name type determination interface, mobile phone number type determination interface, ID number determination interface, and bank card number determination interface of the field recognition model, respectively. The following recognition results are obtained:

[0102] Name type recognition results: 1, 1, 0

[0103] Mobile number type recognition result: 0, 0, 0

[0104] ID card number recognition result: 0, 0, 0

[0105] Bank card number recognition result: 0, 0, 0

[0106] According to the formula The probability of identifying the 'name' field as a name is 2 / 3; the probability of identifying it as a mobile phone number is 0 / 3; the probability of identifying it as an ID card number is 0 / 3; and the probability of identifying it as a bank card number is 0 / 3. For ease of application, this result can be stored in Table 1, where 'table' is the table name, 'type' is the preset field type, 'field' is the table field to be identified, and 'score' is the probability that the field to be identified is determined to be the corresponding preset field type.

[0107]

[0108]

[0109] Table 1 shows the probability of the field 'name' in four preset field types.

[0110] In another example, to synthesize the recognition results of the field across various preset field types and improve the accuracy of target type field recognition, the method is to... The probability q of determining the i-th field as the j-th preset field type ijNext, this step can summarize the probabilities of the i-th field being classified as each preset field type to obtain the probabilities of the i-th field being classified as one of the M preset field types. For example, the probability Pi of the i-th field being classified as one of the M preset field types is:

[0111]

[0112] According to the formula The probability that the name field is determined to be one of the four preset field types is 15%.

[0113] S4. Identify the target type field based on the probability;

[0114] The identification method in this step corresponds to the method used in step S3 to determine the probability that the i-th field is classified as one of the M preset field types.

[0115] For example: In one example, step S3 is performed by... The probability q of determining the i-th field as the j-th preset field type ij This step allows you to configure the probability threshold Q for each preset field type. j The probability q of determining if the i-th field is of the j-th preset field type. ij Is the probability threshold Q of the j-th preset field type greater than? j If it is greater than , then mark the i-th field as the target type field.

[0116] For example: In another example, step S3 is performed by... The probability q of determining the i-th field as the j-th preset field type ij Afterwards, according to The probability Pi of classifying the i-th field as one of the M preset field types is obtained. This step can then configure a probability threshold; when the probability of classifying the i-th field as one of the M preset field types is greater than this threshold, the i-th field is identified as the target type field.

[0117] Furthermore, after identifying the target type field, it can be pushed to the application system, which can then process it as needed. For example, if pushed to the query system, the query system can first de-identify the target type field before providing the query function. For instance, de-identification can be achieved by using encryption algorithms (such as asymmetric encryption and hash algorithms) and / or masking the data.

[0118] Optionally, the present invention can also identify the type of the target type field by setting a probability threshold for the field type, and perform corresponding desensitization processing on the target type field according to the type of the target type field. The present invention may further include:

[0119] S51. Configure the probability threshold for each preset field type;

[0120] For example, the probability threshold for name type is 60%, and the probability threshold for mobile phone number type is 70%.

[0121] S52. Determine whether the probability of the target type field being identified as the j-th preset field type is greater than the probability threshold of the j-th preset field type;

[0122] S53. If it is greater than the j-th preset field type, mark the type of the target type field according to the j-th preset field type.

[0123] For example, if the probability of the target type field being identified as a name is 0 and the probability of it being identified as a mobile phone number is 80%, then the target type field is marked as a mobile phone number.

[0124] S54. Perform desensitization processing on the target type field according to the type.

[0125] For example, the desensitization processing method corresponding to the type of each target type field can be pre-configured. The desensitization processing method may include: encryption algorithm and / or masking method. Taking the masking method as an example, the number of bits in the type mask of each target type field can be configured, such as: name type mask: Zhang*; telephone type mask: 1521111****; ID card type mask: 3507**19100101**4*; bank card type mask: 622****12345123456.

[0126] In a real-world query system, Hive data is queried using big data engines such as Spark, Hive, and Presto, and the results are returned to the query system for display. The query system periodically refreshes the result set of previously identified target type fields. If a user queries a target type field in a corresponding table, the data is anonymized according to the anonymization method corresponding to the target type field before the processed result is returned to the user.

[0127] After identifying the target type field, the target type field can be pushed to, for example, a business approval system. The method may further include:

[0128] S501. Extract the target type data table based on the target type field;

[0129] For example, the business approval system determines whether a data table is a target type data table based on the target type fields in the same data table. In this embodiment, the higher the probability that the target type fields in the data table are determined to be M preset field types, the higher the probability that the data table is a target type data table. A threshold-sensitive probability can be used to determine whether the probability that all target type fields in the data table are determined to be M preset field types is greater than the threshold-sensitive probability. If so, the data table is a target type data table; otherwise, the data table is not a target type data table.

[0130] S502. Configure a multi-level approval mechanism for the target type data table;

[0131] By employing a multi-tiered approval mechanism, data security can be protected by preventing unauthorized users from viewing target data tables.

[0132] It should be noted that, regarding the name category, the data table contains country names, place names, and general proper noun categories, which have a high degree of repetition and are easily confused with names. To prevent proper nouns such as country names and place names from being identified as names, the field marked as a name type in step S53 can obtain sample data for this field from the cached sample data. The sample data for this field is first deduplicated, then divided by the total amount of sample data in the data table to obtain the first probability. The probability of this field being identified as a name type in step S3 is then weighted and averaged with the first probability to obtain the final probability of this field being identified as a name type. This reduces interference caused by proper nouns when identifying the name field.

[0133] Figure 2 This invention relates to a device for identifying target type data from massive datasets, such as... Figure 2 As shown, the device includes:

[0134] Sampling module 21 is used to extract n sample data from n fields of a data table according to the sampling rules corresponding to the data table type in the data warehouse.

[0135] The first recognition module 22 is used to input the i-th sample data into the M interfaces of the field recognition model to obtain the recognition results of the i-th sample data on the M preset field types; wherein: each interface of the field recognition model corresponds to the recognition of a preset field type;

[0136] The determination module 23 is used to determine the probability that the i-th field is identified as one of the M preset field types based on the recognition results of the i-th sample data on the M preset field types;

[0137] The second identification module 24 is used to identify the target type field based on the probability that the i-th field is determined to be of the type of M preset fields;

[0138] Where M, N, and i are all natural numbers greater than zero, and i is less than or equal to n.

[0139] In one implementation, the sample data is in JSON format; the first identification module 22 includes:

[0140] The decomposition module is used to decompose the i-th sample data according to the JSON structure to obtain multiple primary key values ​​of the i-th sample data;

[0141] The input module is used to input each primary key value into M interfaces of the field recognition model to obtain the recognition results of each primary key value in M ​​preset field types;

[0142] The sub-determination module is used to determine the recognition result of the i-th sample data on the M preset field types based on the recognition result of each primary key value on the M preset field types.

[0143] The determining module 23 obtains the probability q of determining the i-th field as the j-th preset field type using the following formula. ij :

[0144]

[0145] Where: N 1ij N is the number of samples for which the recognition result of the i-th sample data on the j-th preset field type is the first recognition result, N is the total number of the i-th sample data, and j is a natural number greater than zero and less than or equal to M;

[0146] The probability Pi that the i-th field is determined to be of type M preset fields by the determining module 23 is:

[0147]

[0148] Furthermore, the device also includes:

[0149] The configuration module is used to configure the probability threshold for each preset field type;

[0150] The judgment module is used to determine whether the probability of the target type field being identified as the j-th preset field type is greater than the probability threshold of the j-th preset field type;

[0151] The marking module is used to mark the type of the target type field according to the j-th preset field type if the value is greater than the specified value.

[0152] The desensitization module is used to desensitize the target type field according to the type.

[0153] In one example, the data table includes: a partitioned table and a non-partitioned table; the sampling module 21 extracts n sample data from each of the n fields of the data table using a first sampling rule for the non-partitioned table; and extracts n sample data from each of the n fields of each partition in the partitioned table according to the partition order.

[0154] Furthermore, the device also includes:

[0155] The extraction module is used to extract a target type data table based on the target type field and configure a multi-level approval mechanism for the target type data table;

[0156] And / or:

[0157] The processing module is used to desensitize target type fields.

[0158] In this embodiment, the preset field types include at least one of the following: name type, identity information type, contact information type, and account information type.

[0159] Those skilled in the art will understand that the modules in the above-described device embodiments can be distributed throughout the device as described, or they can be modified accordingly and distributed in one or more devices different from the above embodiments. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0160] The following describes embodiments of the electronic device of the present invention, which can be considered as implementations of the physical form of the methods and apparatus embodiments of the present invention described above. Details described in the embodiments of the electronic device of the present invention should be considered as supplements to the methods or apparatus embodiments described above; details not disclosed in the embodiments of the electronic device of the present invention can be implemented with reference to the methods or apparatus embodiments described above.

[0161] Figure 3 This is a structural block diagram of an exemplary embodiment of an electronic device according to the present invention. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0162] like Figure 3 As shown, the electronic device 300 of this exemplary embodiment is manifested in the form of a general data processing device. The components of the electronic device 300 may include, but are not limited to: at least one processing unit 310, at least one storage unit 320, a bus 330 connecting different electronic device components (including storage unit 320 and processing unit 310), a display unit 340, etc.

[0163] The storage unit 320 stores a computer-readable program, which may be source code or read-only program code. The program can be executed by the processing unit 310, causing the processing unit 310 to perform the steps of various embodiments of the present invention. For example, the processing unit 310 can perform actions such as... Figure 1 The steps are shown.

[0164] The storage unit 320 may include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) unit 3201 and / or a cache memory unit 3202, and may further include a read-only memory unit (ROM) unit 3203. The storage unit 320 may also include a program / utility 3204 having a set (at least one) of program modules 3205, such program modules 3205 including, but not limited to: operating electronic devices, one or more application programs, other program modules, and program data; each or some combination of these examples may include an implementation of a network environment.

[0165] Bus 330 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0166] Electronic device 300 can also communicate with one or more external devices 100 (e.g., keyboard, display, network device, Bluetooth device, etc.), enabling users to interact with electronic device 300 via these external devices 100, and / or enabling electronic device 300 to communicate with one or more other data processing devices (e.g., router, modem, etc.). This communication can be made via input / output (I / O) interface 350, and also via network adapter 360 to one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet). Network adapter 360 can communicate with other modules of electronic device 300 via bus 330. It should be understood that, although... Figure 3 As not shown, other hardware and / or software modules may be used in electronic device 300, including but not limited to: microcode, device drivers, redundancy processing units, external disk drive arrays, RAID electronic devices, tape drives, and data backup storage electronic devices.

[0167] Figure 4 This is a schematic diagram of a computer-readable medium embodiment of the present invention. Figure 4As shown, the computer program can be stored on one or more computer-readable media. A computer-readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electronic device, apparatus, or device that is electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. When the computer program is executed by one or more data processing devices, it enables the computer-readable medium to implement the above-described method of the present invention, namely: extracting n sample data from n fields of a data table according to sampling rules corresponding to the data table types in the data warehouse; inputting the i-th sample data to M interfaces of a field recognition model to obtain the recognition results of the i-th sample data on M preset field types; wherein: each interface of the field recognition model corresponds to recognizing a preset field type; determining the probability that the i-th field is identified as one of the M preset field types based on the recognition results of the i-th sample data on the M preset field types; recognizing the target type field based on the probability; wherein: M, n, and i are all natural numbers, and i is less than or equal to n.

[0168] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described in this invention can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) or on a network, including several instructions to cause a data processing device (such as a personal computer, server, or network device, etc.) to execute the methods described above according to this invention.

[0169] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in conjunction with an electronic device, apparatus, or device that executes instructions. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0170] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as "C" or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0171] In summary, the present invention can be implemented by methods, apparatus, electronic devices, or computer-readable media that execute computer programs. In practice, some or all of the functions of the present invention can be implemented using general-purpose data processing devices such as microprocessors or digital signal processors (DSPs).

[0172] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying target type data from massive datasets, characterized in that, The method includes: Based on the sampling rules corresponding to the data table types in the data warehouse, extract n sample data in JSON format from each of the n fields of the data table; The i-th sample data is decomposed according to the JSON structure to obtain multiple primary key values ​​of the i-th sample data; each primary key value is input into M interfaces of the field recognition model to obtain the recognition result of each primary key value on M preset field types; the recognition result of the i-th sample data on M preset field types is determined based on the recognition result of each primary key value on M preset field types; where: each interface of the field recognition model corresponds to the recognition of one preset field type; The probability of classifying the i-th field as the j-th preset field type is determined based on the recognition results of the i-th sample data on the M preset field types; The target type field is identified based on the probability. Configure a probability threshold for each preset field type; determine whether the probability of the target type field being identified as the j-th preset field type is greater than the probability threshold of the j-th preset field type; if the probability of the target type field being identified as the j-th preset field type is greater than the probability threshold of the j-th preset field type, then mark the type of the target type field according to the j-th preset field type; perform desensitization processing on the target type field according to the type of the target type field; Where M, n, and i are all natural numbers greater than zero, and i is less than or equal to n, and j is a natural number greater than zero and less than or equal to M.

2. The method according to claim 1, characterized in that, The probability that the i-th field is determined to be of the j-th preset field type Obtain it using the following formula: ； Where: N 1ij N is the number of samples for which the recognition result of the i-th sample data on the j-th preset field type is the first recognition result, N is the total number of the i-th sample data, and j is a natural number greater than zero and less than or equal to M.

3. The method according to claim 1, characterized in that, The data table includes: a partitioned table and a non-partitioned table; for the non-partitioned table, n sample data are extracted from each of the n fields of the data table using a first sampling rule; for the partitioned table, n sample data are extracted from each of the n fields of each partition in the order of partitioning.

4. The method according to claim 1, characterized in that, The method further includes: Extract the target type data table based on the target type field, and configure a multi-level approval mechanism for the target type data table.

5. A device for identifying target type data from massive amounts of data, characterized in that, The device includes: The sampling module is used to extract n JSON format sample data from n fields of a data table according to the sampling rules corresponding to the data table type in the data warehouse. The first recognition module is used to decompose the i-th sample data according to the JSON structure to obtain multiple primary key values ​​of the i-th sample data; input each primary key value into M interfaces of the field recognition model to obtain the recognition result of each primary key value on M preset field types; and determine the recognition result of the i-th sample data on M preset field types based on the recognition result of each primary key value on the M preset field types; wherein: each interface of the field recognition model corresponds to recognizing one preset field type; The determination module is used to determine the probability that the i-th field is identified as the j-th preset field type based on the recognition results of the i-th sample data on M preset field types; The second identification module is used to identify the target type field based on the probability. The configuration module is used to configure the probability threshold for each preset field type; The judgment module is used to determine whether the probability of the target type field being identified as the j-th preset field type is greater than the probability threshold of the j-th preset field type; The marking module is used to mark the type of the target type field according to the j-th preset field type if the probability of the target type field being determined to be the j-th preset field type is greater than the probability threshold of the j-th preset field type; The desensitization module is used to desensitize the target type field according to its type. Where M, n, and i are all natural numbers greater than zero, and i is less than or equal to n, and j is a natural number greater than zero and less than or equal to M.

6. The apparatus according to claim 5, characterized in that, The determining module obtains the probability that the i-th field is determined to be of the j-th preset field type using the following formula. : ； Where: N 1ij N is the number of samples for which the recognition result of the i-th sample data on the j-th preset field type is the first recognition result, N is the total number of the i-th sample data, and j is a natural number greater than zero and less than or equal to M.

7. The apparatus according to claim 5, characterized in that, The data table includes: a partitioned table and a non-partitioned table; the sampling module extracts n sample data from each of the n fields of the non-partitioned table using a first sampling rule; and extracts n sample data from each of the n fields of each partition in the partitioned table according to the partition order.

8. The apparatus according to claim 5, characterized in that, The device further includes: The extraction module is used to extract a target type data table based on the target type field and configure a multi-level approval mechanism for the target type data table.

9. An electronic device, comprising: processor; as well as A memory storing computer-executable instructions, which, when executed, cause the processor to perform the method according to any one of claims 1-4.

10. A computer-readable storage medium, wherein, The computer-readable storage medium stores one or more programs that, when executed by a processor, implement the method of any one of claims 1-4.

Images