Personal information identification method and computer-readable storage medium

Abstract

The present invention discloses a personal information identification method and computer-readable storage medium. The method comprises: obtaining database metadata and the data type and data characteristic value of each field in the database; verifying each field based on the preset data type, data value range, data length, and data rules of the personal information data item, as well as the data type and data characteristic value of each field, and determining the candidate personal information data item corresponding to each field; calculating the name matching degree, entity recognition matching degree, and numerical fingerprint matching degree of each field and its corresponding candidate personal information data item, and calculating the final matching degree of each field and its candidate personal information data item; if the final matching degree between a field and a candidate personal information data item is the largest and greater than a preset threshold, then the candidate personal information data item is used as the target personal information data item. The present invention can automatically identify personal information data items involved in the database.

Description

Technical Field

[0001] The present invention relates to the field of data identification technology, and in particular to a personal information identification method and a computer-readable storage medium. Background Art

[0002] With the development of the internet and the evolution of big data, personal information has become sensitive online data, a crucial component of cybersecurity protection. Regulators need information technology to support data security reviews, ensuring they are comprehensive, in-depth, efficient, and accurate.

[0003] Currently, relevant technologies are already under development. On the client side, dynamic detection technology is available for applications that collect personal information. On the server side, technologies are already available for data component identification, data component security vulnerability detection, data content identification, and data asset display. These technologies can identify various cloud components, big data components, relational database components, encrypted containers, and more; detect database permission vulnerabilities, application vulnerabilities, and data leakage vulnerabilities; identify personal data through regular expressions and keywords; and display data assets based on dimensions such as data category and data quantity.

[0004] However, regulated entities may hide or conceal the storage of personal information for certain purposes, so personal information identification is a critical step. Currently, the personal information identification technology used in data audits is relatively simple, not mature or comprehensive enough. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a personal information identification method and a computer-readable storage medium, which can automatically identify personal information data items involved in a database.

[0006] In order to solve the above technical problems, the present invention adopts a technical solution: a personal information identification method, comprising:

[0007] Obtain database metadata, and obtain the data type and data characteristic value of each field in the database;

[0008] Based on the preset data type, data value range, data length, and data rules of the personal information data item, as well as the data type and data characteristic value of each field, each field is verified to determine the candidate personal information data item corresponding to each field;

[0009] Calculate the name matching degree, entity recognition matching degree, and numerical fingerprint matching degree for each field and its corresponding candidate personal information data item, and calculate the final matching degree between the name field and its candidate personal information data item based on the name matching degree, entity recognition matching degree, and numerical fingerprint matching degree;

[0010] If the final matching degree between a field and a candidate personal information data item is the largest and greater than a preset threshold, the candidate personal information data item is used as the target personal information data item for matching the field.

[0011] The present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned method when the program is executed by a processor.

[0012] The beneficial effects of the present invention are: performing data exploration through two aspects, namely, database metadata and traversal data distribution characteristics; verifying data through data type, data value range, data length, and data rules, thereby quickly determining possible matching personal information data items; calculating the matching degree between a field and its candidate personal information data items through three dimensions: field name, data entity identification, and data fingerprint; and determining the target personal information data item based on the final matching degree, thereby achieving automatic intelligent detection of personal information involved in the database and ensuring the accuracy of the detection. The present invention utilizes the metadata structure and data value range characteristics, continuously and step by step, combines data verification and matching degree calculation, and achieves intelligent matching of data items, thereby automatically detecting stored personal information items, providing support for data security supervision and management. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of a personal information identification method of the present invention;

[0014] Figure 2 Schematic diagram of the process of the method according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0015] To illustrate the technical content, achieved objectives and effects of the present invention in detail, the following is a detailed description in conjunction with the embodiments and accompanying drawings.

[0016] See also Figure 1 ,A method for identifying personal information, including: obtaining database metadata, and obtaining data types and data feature values ​​of each field in the database;

[0017] Based on the preset data type, data value range, data length, and data rules of the personal information data item, as well as the data type and data characteristic value of each field, each field is verified to determine the candidate personal information data item corresponding to each field;

[0018] Calculate the name matching degree, entity identification matching degree, and numerical fingerprint matching degree for each field and its corresponding candidate personal information data item, and calculate the final matching degree for each field and its corresponding candidate personal information data item based on the name matching degree, entity identification matching degree, and numerical fingerprint matching degree;

[0019] If the final matching degree between a field and a candidate personal information data item is the largest and greater than a preset threshold, the candidate personal information data item is used as the target personal information data item for matching the field.

[0020] As can be seen from the above description, the beneficial effects of the present invention are that it can realize automatic intelligent detection of personal information involved in the database and ensure the efficiency and accuracy of the detection.

[0021] Furthermore, the data types include text type, number type, date and time type, short text type, enumeration type and binary type.

[0022] Furthermore, the data type and data characteristic value of each field in the database are specifically obtained as follows:

[0023] If the data type of a field is text type, get the minimum and maximum value of the text length;

[0024] If the data type of a field is a numeric type or a date and time type, the maximum value, minimum value, average value and variance of the data are obtained;

[0025] If the data type of a field is a short text type or an enumeration type, then the value range list of the value is obtained;

[0026] If the data type of a field is binary, get the minimum and maximum length of the binary data.

[0027] As can be seen from the above description, different data feature values ​​are obtained for fields of different data types to facilitate subsequent data verification and to quickly determine possible matching personal information data items.

[0028] Furthermore, the name matching degree, entity recognition matching degree, and numerical fingerprint matching degree of each field and its corresponding candidate personal information data item are calculated respectively, and the final matching degree of each field and its corresponding candidate personal information data item is calculated based on the name matching degree, entity recognition matching degree, and numerical fingerprint matching degree. Specifically, the following is the calculation result:

[0029] Calculate the degree of match between the field name and field description of each field and the name of each candidate personal information data item, to obtain the degree of match between each field and the name of each candidate personal information data item;

[0030] Using the conditional random field, calculate the matching degree between the data value of the short text field and each corresponding candidate personal information data item, and obtain the entity recognition matching degree between the short text field and each candidate personal information data item;

[0031] Calculate the matching degree between the numerical fingerprint of the field with a numeric data type or a date and time data type and the numerical fingerprint of the corresponding candidate personal information data item, and obtain the numerical fingerprint matching degree between the field with a numeric data type or a date and time data type and each candidate personal information data item;

[0032] Calculate the coverage of the data set of the field whose data type is the typical type of enumerated word and the dictionary value set of each corresponding candidate personal information data item, and obtain the numerical fingerprint matching degree between the field whose data type is the typical type of enumerated word and each candidate personal information data item;

[0033] The final matching degree between each field and each candidate personal information data item is calculated based on the name matching degree, entity recognition matching degree and numerical fingerprint matching degree between each field and each candidate personal information data item.

[0034] From the above description, it can be seen that calculating the matching degree through the three dimensions of field name, data entity identification, and data fingerprint can improve the accuracy of the matching degree calculation, thereby improving the recognition accuracy of subsequent target personal information data items.

[0035] Furthermore, the degree of matching between the field name and field description of each field and the data item name of each candidate personal information data item corresponding thereto is calculated respectively, and the degree of matching between each field and the name of each candidate personal information data item is obtained as follows:

[0036] Calculate the word vector of the field name of a field to obtain the first word vector;

[0037] Calculating a word vector of the field description of the field to obtain a second word vector;

[0038] Calculate a word vector for the data item name of a candidate personal information data item corresponding to the field to obtain a third word vector;

[0039] Calculating the cosine similarity between the field name of the field and the data item name of the candidate personal information data item based on the first word vector and the third word vector to obtain a first cosine similarity;

[0040] Calculating the cosine similarity between the field description of the field and the data item name of the candidate personal information data item based on the second word vector and the third word vector to obtain a second cosine similarity;

[0041] The name matching degree between the field and the candidate personal information data item is calculated based on the first cosine similarity, the second cosine similarity, and the preset first weight coefficient and second weight coefficient.

[0042] From the above description, we can see that by converting text into word vectors and calculating the name matching degree based on cosine similarity, the accuracy of the name matching calculation can be improved.

[0043] Furthermore, the conditional random field is used to calculate the matching degree between the data value of the field with the short text data type and each corresponding candidate personal information data item, and the entity recognition matching degree between the field with the short text data type and each corresponding candidate personal information data item is obtained as follows:

[0044] Get a field whose data type is short text type;

[0045] Calculate, by using a preset conditional random field, a prediction score of a candidate personal information data item in the field corresponding to the data value of the field;

[0046] Based on the prediction score and a preset third weight coefficient, the entity recognition matching degree between the field and the candidate personal information data item is calculated.

[0047] As can be seen from the above description, by identifying entities in text through named entity recognition, it is possible to further determine matching personal information data items for short text data.

[0048] Furthermore, the degree of matching between the numerical fingerprint of the field whose data type is a numeric type or a date and time type and the numerical fingerprint of each corresponding candidate personal information data item is calculated respectively, and the degree of matching between the numerical fingerprint of the field whose data type is a numeric type or a date and time type and each corresponding candidate personal information data item is obtained as follows:

[0049] Get a field whose data type is numeric or date and time type;

[0050] Obtaining six-dimensional characteristic values ​​according to the value of the field to obtain a numerical fingerprint of the field, wherein the six-dimensional characteristic values ​​include a minimum value, a first quartile, a median, a third quartile, a maximum value, and a variance;

[0051] Obtaining a six-dimensional feature value of a candidate personal information data item corresponding to the field, and obtaining a numerical fingerprint of the candidate personal information data item;

[0052] The Euclidean distance is calculated based on the numerical fingerprint of the field and the numerical fingerprint of the candidate personal information data item, and the matching degree of the numerical fingerprint of the field and the candidate personal information data item is calculated based on the Euclidean distance and a preset fourth weight coefficient.

[0053] As can be seen from the above description, a numerical fingerprint is generated based on the characteristics of the numerical distribution, and the degree of match between the two is measured by the distance between the numerical fingerprints. For numerical information, the numerical fingerprint consists of six dimensions: minimum value, first quartile, median, third quartile, maximum value, and variance.

[0054] Furthermore, the coverage of the data set of the field whose data type is the typical type of enumerated word and the dictionary value set of each corresponding candidate personal information data item is calculated respectively, and the numerical fingerprint matching degree of the field whose data type is the typical type of enumerated word and each candidate personal information data item is obtained specifically as follows:

[0055] Get a field whose data type is a typical type of an enumeration word;

[0056] Obtaining the value of the field and sorting them in ascending order according to the natural order to obtain a value set of the field;

[0057] Obtain a dictionary value set of a candidate personal information data item corresponding to the field;

[0058] Calculate the coverage of the numerical value set in the dictionary value set, and calculate the numerical fingerprint matching degree between the field and the candidate personal information data item based on the coverage and a preset fifth weight coefficient.

[0059] From the above description, we can see that for typical data of enumerated characters, the matching degree is measured by the coverage between sets.

[0060] Furthermore, the calculation of the name matching degree, entity recognition matching degree, and numerical fingerprint matching degree between each field and its corresponding candidate personal information data item, and the calculation of the final matching degree between each field and its corresponding candidate personal information data item based on the name matching degree, entity recognition matching degree, and numerical fingerprint matching degree, further includes:

[0061] Set the entity recognition matching degree of the field whose data type is not short text type and its candidate personal information data items to 0;

[0062] The numerical fingerprint matching degree of a field whose data type is not a numeric type, a date and time type, or an enumeration type and its candidate personal information data items is set to 0.

[0063] The present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned method when the program is executed by a processor.

[0064] Example 1

[0065] Please refer to Figure 2, the embodiment 1 of the present invention is: a personal information identification method, which can be applied to automatically and intelligently detect personal information involved in a database, such as Figure 2 As shown, the following steps are included:

[0066] S1: Data exploration.

[0067] First, obtain database metadata. Personal information is typically stored in a database, whether it's a traditional relational or non-relational database. The database uses metadata to describe the format of the stored data, including all table and field information, specifically table names, field names, field types (i.e., data storage types), and field descriptions (comments). Therefore, obtaining database metadata first provides support for subsequent data matching.

[0068] Then, the data type and data characteristic value of each field in the database are obtained. That is, the data of each field in the database is traversed to obtain the characteristics of the data stored in the field.

[0069] In this embodiment, data types include text type, numeric type, date and time type, short text and enumeration type, and binary type. For fields of different data types, different data characteristic values ​​are obtained, specifically, as follows:

[0070] If the data type of a field is text type, get the minimum and maximum value of the text length;

[0071] If the data type of a field is a numeric type or a date and time type, the maximum and minimum values ​​of the data are obtained. Furthermore, the average value and variance can also be obtained.

[0072] If the data type of a field is short text type and enumeration type, then get the value range list of the value;

[0073] If the data type of a field is binary, get the minimum and maximum length of the binary data.

[0074] S2: Data verification, that is, verifying each field based on the preset data type, data value range, data length and data rules of the personal information data item, as well as the data type and data characteristic value of each field, to determine the candidate personal information data item corresponding to each field.

[0075] Every data field stored in a database may contain personal information, and personal information can encompass a wide range of data items, such as name, ID number, mobile phone number, home address, registered address, religious beliefs, and so on. By using validation methods such as data type, value range, length, and rules, matching personal information data items can be identified or locked within a certain range. Data validation is a sufficient condition and must be met for a match to be considered.

[0076] In this embodiment, data verification includes data type verification, data value range verification, data length verification and data rule verification.

[0077] Regarding data type verification, which verifies the relationship between the data storage type and the personal information data item, a data item can have one or more possible data storage types. For example, the data storage type of an ID card number data item is usually text, and data types such as numbers, dates, and binary can be excluded. The data storage type of a birth date data item can be text, date and time, or even a number (milliseconds since 1970).

[0078] Most numeric data items have corresponding value ranges, such as age, height, and date of birth. Therefore, for data items with required value ranges, the data is converted and then the value range is checked. For example, for fields with numeric and date / time data types, the maximum and minimum values ​​of the data are checked to see if they are both within the value range of a certain personal information data item. If so, the personal information data item is considered to be a likely matching personal information data item for that field.

[0079] In addition, for fields with short text type and enumeration word typical type data types, since they also have corresponding value domains, these fields can also be checked for value domains.

[0080] Text and binary data items typically have length requirements. The length range can be discrete or continuous. For example, an ID card number is 15 or 18 digits long; a mobile phone number is 11, 14, or 15 digits (with international encoding); and a license plate number is 7 to 8 digits long. Therefore, length verification can be performed on text and binary data fields.

[0081] Validation rules are often set for specific personal information items. For example, the last digit of an ID number is a validation field, calculated from the first 17 digits. There are certain requirements for the first three digits of a mobile phone number, ranging from 130-139, 145-147, and so on. There are also requirements for the format of email addresses, typically consisting of a user identifier + @ + domain name. Depending on the specific validation rules, the check digit can be verified according to the rules, and regular expressions can be used to perform structural verification on the data elements.

[0082] For example, assuming that the data type of a field is text type, the minimum text length is 11, the maximum text length is 15, and the first 3 digits of each data are within the preset range, then the mobile phone number data item is considered to be a candidate personal information data item for the field.

[0083] S3: Matching degree calculation, i.e., respectively calculating the name matching degree, entity recognition matching degree and numerical fingerprint matching degree of each field and its corresponding candidate personal information data item, and calculating the final matching degree of each field and its candidate personal information based on the name matching degree, entity recognition matching degree and numerical fingerprint matching degree.

[0084] During the data verification phase, the stored data fields are typically limited to a certain range of potentially matching personal information items. Unless there are clear rules, such as for ID card numbers, data verification can generally confirm the match. Therefore, other methods are needed to select potential matching data items. This phase supports calculating matching degrees based on dimensions such as field names, entity recognition, and numerical fingerprints, intelligently matching data items based on weight thresholds and matching degree values.

[0085] Specifically, this step includes the following steps:

[0086] S301: Calculate the matching degree between the field name and field description of each field and the data item name of each corresponding candidate personal information data item, and obtain the name matching degree between each field and each candidate personal information data item.

[0087] Field names play a crucial role in field matching. However, the same data item can have multiple field names. For example, the data item "name" can be expressed as "name," "Chinese name," "former name," "other party's name," and so on. Therefore, natural language processing techniques can be used to calculate the distance between the field name and the data item name to measure the degree of match. Field names are often concise, so field descriptions (notes) in database metadata can also be used to assist in measuring match.

[0088] The natural language processing technology word2vec (wordto vector) supports the simplification of text content processing into vector operations in vector space. Using word2vec requires preparing a large amount of corpus and conducting deep learning training. The trained model results can be used to calculate word vectors. In this embodiment, the similarity between two vectors is measured by measuring the cosine of the angle between the inner product space. The formula for calculating the cosine of the angle between two vectors A and B is as follows:

[0089] ,

[0090] Therefore, the calculation method of the name matching degree in this step is as follows:

[0091] Calculate the word vector of the field name of a field to obtain a first word vector, and calculate the word vector of the field description of the field to obtain a second word vector: at the same time, calculate the word vector of the data item name of a candidate personal information data item corresponding to the field to obtain a third word vector.

[0092] For example, assuming that the field name of a field is fieldName, and the word vector converted by word2vec is vecF; the field description of the field is fieldDescription, and the word vector converted by word2vec is vecT; the data item name of a candidate personal information data item corresponding to the field is targetFieldName, and the word vector converted by word2vec is vecD, then the calculation formula for the name matching degreel between the field and the candidate personal information data item is as follows:

[0093] ,

[0094] Among them, weight1 is the first weight coefficient corresponding to the field name, and weight2 is the second weight coefficient corresponding to the field description.

[0095] S302: Using a conditional random field, calculate the matching degree between the data value of a field with a short text data type and its corresponding candidate personal information data items, and obtain the entity recognition matching degree between each field with a short text data type and its corresponding candidate personal information data items.

[0096] Named entity recognition (NER) identifies entities such as names of people, organizations, and places in text. It can further identify matching data items for textual data. Conditional random fields (CRFs) are conditional probability distribution models that output a set of random variables given a set of input variables. Their characteristic is that they assume the output random variables form a Markov random field. This machine learning technique is commonly used for entity recognition. It automatically learns from a training dataset. The trained model can then be used to identify text and match possible names, addresses, and work units.

[0097] If the data stored in the database is of the short text type and the candidate personal information data item to be matched is a recognizable named entity, the predicted score for entity recognition is used to measure the degree of match between the field and the candidate personal information data item. Otherwise, the entity recognition match value is 0. In other words, for fields whose data type is not short text, and for fields whose data type is short text but whose corresponding candidate personal information data item is not a recognizable named entity, the entity recognition match between these fields and their corresponding candidate personal information data item is directly set to 0.

[0098] In this embodiment, the calculation method of the entity recognition matching degree in this step is specifically as follows:

[0099] Get a field fieldName of short text data type, and then use the preset conditional random field to calculate the prediction score of the data value fieldValue of the field fieldName corresponding to the candidate personal information data item targetFieldName of the field. Assuming it is crfScore, the calculation formula for the entity recognition matching degree degree2 between the field fieldName and its candidate personal information data item targetFieldName is as follows:

[0100] ,

[0101] Among them, crfScore is the score of the data value fieldValue predicted by the conditional random field CRF, and weight3 is the third weight coefficient of the corresponding entity recognition.

[0102] S303: Calculate the matching degree between the numerical fingerprint of the field whose data type is a numeric type or a date and time type and the numerical fingerprint of each corresponding candidate personal information data item, and obtain the numerical fingerprint matching degree between the field whose data type is a numeric type or a date and time type and each corresponding candidate personal information data item.

[0103] The numerical distribution of some personal information, such as age and date of birth, has certain regularities. Based on the characteristics of the numerical distribution, a numerical fingerprint can be generated, and the distance between the numerical fingerprints can be used to measure the degree of match between the two.

[0104] In this embodiment, the numerical fingerprints of fields of different data types are different. For example, for numeric information, the numerical fingerprint consists of six dimensions: minimum value, first quartile, median, third quartile, maximum value, and variance. Therefore, the calculation method of the numerical fingerprint matching degree in this step is as follows:

[0105] Obtain a field whose data type is a numeric type or a date and time type; then traverse and count all the values ​​of the field to obtain the six-dimensional eigenvalues, which are the minimum value, first quartile, median, third quartile, maximum value, and variance, and then combine them to obtain the numerical fingerprint of the field. At the same time, obtain the six-dimensional eigenvalues ​​of a candidate personal information data item corresponding to the field to obtain the numerical fingerprint of the candidate personal information data item. Then sample the standardized Euclidean distance to measure the matching degree between the numerical fingerprints. The calculation formula for the numerical fingerprint matching degree3 between the field and the candidate personal information data item is as follows:

[0106] ,

[0107] Where C is the numerical fingerprint of the field fieldValue, D is the numerical fingerprint of the candidate personal information data item targetFieldValue, and n represents the dimension of the numerical fingerprint. In this embodiment, n=6, which means there are six dimensions in total. Represents the kth dimension of the numerical fingerprint. For example, C1 represents the first dimension, i.e., the minimum value; Represents the standard deviation of the data fingerprint D in the kth dimension; weight4 is the fourth weight coefficient corresponding to the digital numerical fingerprint.

[0108] S304: Calculate the coverage of the data set of the field whose data type is the typical type of enumerated character and the dictionary value set of each corresponding candidate personal information data item, and obtain the numerical fingerprint matching degree between the field whose data type is the typical type of enumerated character and each candidate personal information data item.

[0109] For typical data items of enumeration words, such as religious belief, ethnicity, education level, blood type, etc., the numerical dictionary is organized into a set in ascending order of natural order, which can represent the characteristics of numerical distribution, and the matching degree is measured by the coverage between sets.

[0110] Therefore, the calculation method of the numerical fingerprint matching degree in this step is specifically as follows:

[0111] Obtain a field whose data type is a typical enumeration type: then sort all values ​​in the field in ascending natural order to obtain a set of values ​​for the field. Simultaneously, obtain a set of dictionary values ​​for a candidate personal information data item corresponding to the field. The dictionary value set for a candidate personal information data item can refer to national, local, or industry standards. For example, nationality can refer to GB3304-91. The formula for calculating degree 3 of the numerical fingerprint match between the field and the candidate personal information data item is as follows:

[0112] ,

[0113] Among them, E is the numerical value set of the field fieldValue in the database, F is the dictionary value set of the candidate personal information data item targetFieldValue corresponding to the field, and weight5 is the fifth weight coefficient of the typical numerical fingerprint of the corresponding enumeration word.

[0114] Furthermore, for fields whose data types are not numeric types, date and time types, or typical enumeration types, the numerical fingerprint matching degree of these fields and their respective candidate personal information data items is set to 0.

[0115] S305: Calculate the final matching degree between each field and each candidate personal information data item based on the name matching degree, entity recognition matching degree and numerical fingerprint matching degree between each field and each candidate personal information data item.

[0116] That is, the matching degrees of the above multiple dimensions are accumulated to obtain the final matching degree, which is calculated as follows: degree = degree1 + degree2 + degree3.

[0117] S4: Determine the target personal information data item that matches each field.

[0118] Through the above matching degree calculation, the final matching degree between each field and its candidate personal information data items can be obtained. Then, for each field, the final matching degree of each candidate personal information data item is compared. Only when the matching degree is greater than the preset threshold G is it considered a valid match. If for a field, there are multiple candidate personal information data items with final matching degrees greater than the threshold G, the candidate personal information data item with the largest final matching degree is regarded as the best match.

[0119] That is, if the final matching degree between a field and a candidate personal information data item is the largest and greater than a preset threshold, the candidate personal information data item will be used as the target personal information data item for matching the field.

[0120] This embodiment conducts data exploration from two aspects: database metadata and traversal data distribution characteristics. It verifies the data from the data type, data value range, data length, and data rules, quickly narrows down possible matching personal information data items, and further calculates the matching degree between the field and its candidate personal information data items from three dimensions: field name, data entity identification, and data fingerprint. Finally, the target personal information data item is determined through the threshold and the final matching degree value, thereby realizing automatic intelligent detection of personal information involved in the database.

[0121] By intelligently matching data items in data resources with personal information, this approach provides scientific and efficient support for data review, serves as a reference for data governance quality management, and intelligently detects abnormal data and values. Data verification and data fingerprinting solutions can also be used to intelligently match data types during data access, reducing the workload of manually matching data items.

[0122] As regulatory agencies continue to improve laws and regulations on data security and society pays more and more attention to data security, this embodiment can be used by data security review departments to quickly survey the data resources of network operators, intelligently detect the personal information involved, and provide strong support for data asset reporting, thereby building a peaceful, secure, open, and cooperative cyberspace, and providing an efficient reference solution for data managers' self-supervision and management.

[0123] Example 2

[0124] This embodiment is a computer-readable storage medium corresponding to the above embodiment, on which a computer program is stored. When the computer program is executed by the processor, it implements the various processes in the above-mentioned personal information identification method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0125] In summary, the present invention provides a personal information identification method and computer-readable storage medium that performs data exploration through two aspects: database metadata and traversal data distribution characteristics. The data is verified through data type, data value range, data length, and data rules, thereby quickly determining possible matching personal information data items: the matching degree between the field and its candidate personal information data items is calculated through three dimensions: field name, data entity identification, and data fingerprint, and the target personal information data item is determined based on the final matching degree, thereby achieving automatic intelligent detection of personal information involved in the database and ensuring the accuracy of the detection. The present invention utilizes the metadata structure and data value range characteristics, continuously and step by step, combined with data verification and matching degree calculation, to achieve intelligent matching of data items, thereby automatically detecting stored personal information items, providing support for data security supervision and management.

[0126] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for identifying personal information, characterized in that: include: Obtain database metadata, and obtain the data type and data characteristic value of each field in the database; Based on the data type of the field, data characteristic values ​​are obtained separately, including the data length range of text type fields, statistical characteristic values ​​of numeric type or date and time type fields, the value range list of short text or enumeration type fields, and the data length range of binary fields; Based on the preset data type, data value range, data length, and data rules of the personal information data item, as well as the data type and data characteristic value of each field, each field is verified to determine the candidate personal information data item corresponding to each field; Calculate the name matching degree, entity recognition matching degree, and numerical fingerprint matching degree of each field and its corresponding candidate personal information data item respectively, and calculate the final matching degree of each field and its corresponding candidate personal information data item based on the name matching degree, entity recognition matching degree, and numerical fingerprint matching degree. Specifically, the following is the result: Calculate the degree of match between the field name and field description of each field and the name of each candidate personal information data item, to obtain the degree of match between each field and the name of each candidate personal information data item; Using the conditional random field, calculate the matching degree between the data value of the short text field and each corresponding candidate personal information data item, and obtain the entity recognition matching degree between the short text field and each candidate personal information data item; Calculate the matching degree between the numerical fingerprint of the field with a numeric data type or a date and time data type and the numerical fingerprint of each corresponding candidate personal information data item, and obtain the numerical fingerprint matching degree between the field with a numeric data type or a date and time data type and each corresponding candidate personal information data item; Calculate the coverage of the data set of the field whose data type is the typical type of enumerated word and the dictionary value set of each corresponding candidate personal information data item, and obtain the numerical fingerprint matching degree between the field whose data type is the typical type of enumerated word and each candidate personal information data item; Calculate the final matching degree between each field and each candidate personal information data item based on the name matching degree, entity identification matching degree, and numerical fingerprint matching degree between each field and each candidate personal information data item; The conditional random field is used to calculate the matching degree between the data value of the field with the short text data type and each corresponding candidate personal information data item, and the entity recognition matching degree between the field with the short text data type and each corresponding candidate personal information data item is obtained as follows: Get a field whose data type is short text type; Calculate, by using a preset conditional random field, a prediction score of a candidate personal information data item in the field corresponding to the data value of the field; Calculating an entity recognition matching degree between the field and the candidate personal information data item based on the prediction score and a preset third weight coefficient; The numerical fingerprint matching degree of the field whose data type is a numeric type or a date and time type and the corresponding numerical fingerprint of each candidate personal information data item is calculated respectively, and the numerical fingerprint matching degree of the field whose data type is a numeric type or a date and time type and each candidate personal information data item is obtained specifically as follows: Get a field whose data type is numeric or date and time type; Obtaining six-dimensional characteristic values ​​according to the value of the field to obtain a numerical fingerprint of the field, wherein the six-dimensional characteristic values ​​include a minimum value, a first quartile, a median, a third quartile, a maximum value, and a variance; Obtaining a six-dimensional feature value of a candidate personal information data item corresponding to the field, and obtaining a numerical fingerprint of the candidate personal information data item; Calculating a Euclidean distance based on the numerical fingerprint of the field and the numerical fingerprint of the candidate personal information data item, and calculating a degree of matching between the numerical fingerprint of the field and the candidate personal information data item based on the Euclidean distance and a preset fourth weight coefficient; If the final matching degree between a field and a candidate personal information data item is the largest and greater than a preset threshold, the candidate personal information data item is used as the target personal information data item for matching the field.

2. The personal information identification method according to claim 1, characterized in that: The data types include text type, number type, date and time type, short text type, enumeration type and binary type.

3. The personal information identification method according to claim 2, characterized in that: The data type and data characteristic value of each field in the database are specifically obtained as follows: If the data type of a field is text type, get the minimum and maximum value of the text length; If the data type of a field is a numeric type or a date and time type, the maximum value, minimum value, average value, and variance of the data are obtained; If the data type of a field is a short text type or an enumeration type, then the value range list of the value is obtained; If the data type of a field is binary, get the minimum and maximum length of the binary data.

4. The personal information identification method according to claim 1, characterized in that: The degree of matching between the field name and field description of each field and the name of each candidate personal information data item is calculated respectively, and the degree of matching between each field and the name of each candidate personal information data item is obtained as follows: Calculate the word vector of the field name of a field to obtain a first word vector; calculate the word vector of the field description of the field to obtain a second word vector; Calculate the word vector of the data item name of a candidate personal information data item corresponding to the field to obtain a third word vector; Calculating the cosine similarity between the field name of the field and the data item name of the candidate personal information data item based on the first word vector and the third word vector to obtain a first cosine similarity; Calculating the cosine similarity between the field description of the field and the data item name of the candidate personal information data item based on the second word vector and the third word vector to obtain a second cosine similarity; The name matching degree between the field and the candidate personal information data item is calculated based on the first cosine similarity, the second cosine similarity, and the preset first weight coefficient and second weight coefficient.

5. The personal information identification method according to claim 1, characterized in that: The coverage of the data set of the field whose data type is the typical type of enumerated word and the dictionary value set of each corresponding candidate personal information data item is calculated respectively, and the numerical fingerprint matching degree between the field whose data type is the typical type of enumerated word and each candidate personal information data item is obtained is specifically: Get a field whose data type is a typical type of an enumeration word; Obtaining the value of the field and sorting them in ascending order according to the natural order to obtain a value set of the field; Obtain a dictionary value set of a candidate personal information data item corresponding to the field; Calculate the coverage of the numerical value set in the dictionary value set, and calculate the numerical fingerprint matching degree between the field and the candidate personal information data item based on the coverage and a preset fifth weight coefficient.

6. The personal information identification method according to claim 1, characterized in that: The step of respectively calculating the name matching degree, entity recognition matching degree, and numerical fingerprint matching degree between each field and its corresponding candidate personal information data item, and calculating the final matching degree between each field and its corresponding candidate personal information data item based on the name matching degree, entity recognition matching degree, and numerical fingerprint matching degree, further includes: Set the entity recognition matching degree of the field whose data type is not short text type and its candidate personal information data items to 0; The numerical fingerprint matching degree of a field whose data type is not a numeric type, a date and time type, or an enumeration type and its candidate personal information data items is set to 0.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images