A method and system for statistical analysis of personal information disclosure cases for data tables

Abstract

The scheme discloses a personal information disclosure situation statistical analysis method and system for data tables, proposes a new data processing method, makes a data directory for data tables, and preliminarily labels and classifies personal information related data tables based on the data directory, then comprehensively identifies the field content of the screened personal information related data tables, and accurately and efficiently completes the labeling of the field identifier; on the basis of the foregoing processing, the data tables are divided according to whether there is direct identifier information record line, the data tables are split and reorganized according to whether there is direct identifier information record line, and the subsequent processing and analysis and statistical efficiency can be effectively improved; on the basis of the foregoing processing, the data tables are analyzed by using a layer-by-layer classification method, and statistical analysis is performed from multiple dimensions, and a personal information disclosure situation result report is automatically generated, so that the personal information disclosure situation of the platform is fully and completely described.

Description

Technical Field

[0001] This solution belongs to the field of personal information security technology, and proposes a method and system for statistical analysis of personal information disclosure in data tables. Background Technology

[0002] According to the "Information Security Technology: Guidelines for De-identification of Personal Information," personal information refers to various information recorded electronically or otherwise that can, alone or in combination with other information, identify a specific natural person or reflect the activities of a specific natural person. The natural person identified by personal information is called the personal information subject. Microdata refers to a structured data table where each record (row) corresponds to a personal information subject, and each field (column) of the record corresponds to an attribute. Identifiers are one or more attributes in microdata that can uniquely identify a personal information subject, and are divided into direct identifiers and quasi-identifiers. Direct identifiers are microdata attributes that can independently identify a personal information subject in a specific environment; common direct identifiers include name, ID card number, and mobile phone number. Quasi-identifiers are microdata attributes that cannot be used alone to identify an individual, but can uniquely identify a personal information subject when combined with other attributes; common quasi-identifiers include gender, occupation, and education level. Data platforms often de-identify the content of data tables before publishing them. De-identification is the process of technically processing personal information to make it impossible to identify the personal information subject without additional information. In a data table without direct identifier disclosure, rows with the same quasi-identifier value form an equivalence class. The size of the equivalence class is the number of rows where all quasi-identifier values ​​are the same, determining the risk of the individual corresponding to that row being reidentified. The equivalence class dimension is the number of quasi-identifiers forming that equivalence class, measuring the variety of personal information contained in the row; the higher the dimension, the more personal information can be disclosed. Reidentification is the process of re-associating a deidentified data table with the original personal information subject or a group of personal information subjects. For a data table to be deidentified, since rows within the same equivalence class are indistinguishable, the probability of an individual corresponding to a row within the same equivalence class being reidentified, i.e., the reidentification risk of the row, is the reciprocal of the equivalence class size.

[0003]

[0004] R b =max j∈J θ j Formula 2

[0005] In the formula: J is the equivalence class, j∈J,f j For the size of the equivalence class, θ j For the relabeling probability, |J| represents the number of equivalence classes in the data table, and R... b This represents the maximum probability of re-identification.

[0006] When analyzing the disclosure of personal information in publicly available data tables on a platform, it typically includes at least two parts: data classification and data content identification.

[0007] 1) Data classification involves grouping data with similar attributes or characteristics according to specific management or business dimensions, following certain principles and methods. Data classification tasks can categorize data tables; for example, data can be divided into personal information and non-personal information based on the individual citizen dimension. Currently, common data classification tasks mainly rely on automated tools or manual data classification and labeling. Manual classification and labeling are more accurate but significantly increase the task cycle, while fully automated labeling methods require high-quality data tables, which are often insufficient in reality.

[0008] 2) Data Content Recognition: This involves identifying the information types contained in data fields within a data table, primarily focusing on sensitive information. Traditional data content recognition methods mainly employ two approaches: manual definition and regular expressions. The manual definition method involves manually creating a sensitive word database and using keyword matching to identify information at the metadata level. However, this method requires high-quality data; if data fields are misaligned or field names do not match their content, the reliability of the recognition is limited. Regular expression methods are well-suited for structured data such as phone numbers, ID cards, and bank card numbers. However, they cannot accurately identify sensitive information in unstructured text, such as names, or accurately extract relatively simple patterns of information mixed within long, unstructured text, such as phone numbers interspersed within long text paragraphs.

[0009] In summary, the existing technology has the following drawbacks:

[0010] In real-world scenarios, data tables often exhibit complex formats, unreliable fields, and non-compliant practices such as random nesting of unstructured content within structured tables, making classification and identification difficult. Furthermore, with the continuous development of information processing and storage technologies, the problem of personal information misuse in my country is becoming increasingly serious. In specific data resource platforms, the obtained data tables often suffer from inadequate anonymization of personal information, leading to disclosures. Moreover, due to the diverse sources of these data tables, the lack of unified standards, and the fact that their data field content and field directories are often unverified, or contain non-compliant issues such as mismatched field names and content, and mixed information between fields, traditional classification and identification tasks are difficult to perform directly. Summary of the Invention

[0011] The purpose of this solution is to address the aforementioned issues by providing a method and system for statistical analysis of personal information disclosure in data tables. This allows managers of specific data resource platforms to efficiently and accurately understand the disclosure of personal privacy information related to resources published on the current platform, the potential re-identification risks of data tables, and the achievable re-identification scenarios through the system and methods of this invention.

[0012] A method for statistical analysis of personal information disclosure in a data table, comprising:

[0013] S1. Obtain the data table to be analyzed;

[0014] S2. Clean up the data tables to be analyzed and create a metadata catalog for the cleaned data tables;

[0015] Each entry in the metadata directory corresponds to a data table to be analyzed, including a set of field names of the corresponding data table and a mapping code for pointing to the corresponding data table; taking the directory table as an example, one row corresponds to one entry. When there is only one data table to be analyzed, the metadata directory table has only one row.

[0016] S3. Match the data table to be analyzed for each entry based on the mapping code; perform preliminary classification and labeling of the field values ​​of the corresponding data table with respect to the field identifier type for the set of field names in the metadata directory, and filter the entries in the metadata directory related to personal information;

[0017] S4. Based on the filtering results of step S3, extract the corresponding personal information related data table, and fully identify the field values ​​of the field names in the personal information related data table;

[0018] S5. Based on whether there are record rows containing direct identifier information, classify personal information related data tables into Type I data tables and Type II data tables;

[0019] S6. Based on whether the record rows in the data table contain identifiable information, split and reorganize the Type 1 data table at the record row level into a Type A data table consisting entirely of record rows containing identifiable information and a Type B data table consisting of record rows not containing identifiable information;

[0020] Type 2 data tables are classified as Category B data tables;

[0021] S7. Perform statistical analysis on both Category A and Category B data tables to analyze the personal information disclosure in the data tables to be analyzed.

[0022] In the above-mentioned statistical analysis method for personal information disclosure in data tables, step S2 involves cleaning the data table to be analyzed, which includes any one or more combinations of field misalignment correction, field name improvement, field name conversion, and special character processing, so that each field name in the cleaned data table exists and corresponds to the field value. The field names are mainly composed of Chinese characters, and special characters in the fields are removed. The special characters here mainly include spaces, newlines, and other special characters.

[0023] Each entry in the metadata catalog also includes any one or more combinations of the following: the data table title, webpage link, data table file name, and data table related information containing domain annotation information;

[0024] The mapping code, field name set, table title, webpage link, table file name, and related information of each table to be analyzed are mapped and integrated into the aforementioned metadata catalog.

[0025] In the above-mentioned statistical analysis method for personal information disclosure in data tables, step S3 specifically includes:

[0026] S31. Obtain the field value characteristics of each field name set in each entry, including the proportion of unique field values, the proportion distribution of various character types in the sampled samples, and the field data type;

[0027] S32. Vectorize the data table title and data table domain label of the entry containing the field name, vectorize the field value features of the field set, and merge the vector features;

[0028] S33. Input the merged vectorized field features into a trained machine learning classification model, such as a decision tree classification model, and the model outputs the identifier type label for each field;

[0029] S34. Determine whether the corresponding entry is related to personal information based on the identifier type label, so as to filter entries related to personal information in the metadata directory.

[0030] In the above-mentioned statistical analysis method for personal information disclosure in data tables, in step S33, the types of identifiers that can be annotated include direct identifiers, quasi-identifiers, and non-identifiers.

[0031] In step S34, if a field in an entry contains one of the three elements—a direct identifier, a quasi-identifier, or a de-identified identifier—it is determined that the entry is related to personal information.

[0032] In the above-mentioned statistical analysis method for personal information disclosure in data tables, step S4 involves the following comprehensive identification method:

[0033] For direct identifiers, information that strictly follows a certain compositional pattern is used for identification, such as mobile phone numbers, ID cards, bank card numbers, and license plate numbers; for descriptive texts that do not have a strict compositional pattern, such as names, the deep learning-based named entity recognition method in the LAC lexical analysis tool is used to identify and extract them.

[0034] For quasi-identifiers, such as gender, education level, and work unit, identification and matching are performed using metadata recognition technology based on keyword thesaurus, in accordance with personal information reference documents, such as Appendix B, Table B.1 of the "Network Data Classification and Grading Guide".

[0035] For de-identified identifiers, the degree of de-identification is assessed. For example, the presence of the special character "*" can be used as a criterion for determining de-identification, and the percentage of field values ​​containing "*" can be used to confirm the extent of de-identification. Before de-identification, a de-identified identifier may have been a direct identifier or a quasi-identifier.

[0036] In the aforementioned statistical analysis method for personal information disclosure in data tables, for Class A data tables, the individuals corresponding to each record row are directly disclosed. The distribution of this type of data table across different fields can be statistically analyzed, thus visually demonstrating the direct disclosure of individuals in various fields within the platform scenario. The coverage distribution of different types of personal information in specific fields can also be statistically analyzed, demonstrating the different information leakage situations of directly disclosed individuals in different fields within the platform scenario. Since the number of individuals involved in each data table varies significantly, this method also summarizes the number of records involving each relevant personal information type in each data table. For identified direct identifier information, the number of identification results for each direct identifier in each row is used as an indicator of the number of people involved. For matched quasi-identifier fields, the sum of the number of record rows is used as an indicator of the number of people involved. Furthermore, individuals with the same direct identifier information across data tables are deduplicated as a reference indicator of the number of people involved in each data table. This achieves a comprehensive characterization and quantitative display of the individuals and their personal information disclosed in a single data table within a specific platform scenario.

[0037] For Class B data tables, each record row in the table corresponds to an individual with a potential risk of being identified. First, following the re-identification risk calculation method mentioned in the background technology, the re-identification risk of each record row in each data table is calculated. The maximum re-identification risk of the data table is recorded as the maximum re-identification risk among all record rows in the data table, and the number of data tables with different maximum re-identification risks is counted. For data tables with a maximum re-identification risk greater than or equal to a set threshold, their distribution in different fields and their coverage of different types of personal information in specific fields can be statistically analyzed, demonstrating the existence of high-risk data tables on the platform in different fields and the different information disclosure situations they contain. All data rows in Class B data tables with a re-identification risk greater than or equal to the set threshold can be summarized as an indicator of the number of individuals involved in the data table, thereby achieving a comprehensive characterization and quantitative display of the personal information disclosed in data tables with a maximum re-identification risk greater than or equal to the set threshold in a specific platform scenario. The set threshold can be 1 / 2, 1 / 3, 1…, preferably 1, where, in practice, the maximum re-identification risk is 1.

[0038] In the above-mentioned statistical analysis method for personal information disclosure in data tables, step S7 also includes a correlation analysis method between data tables of type A and type B:

[0039] S71. Represent B-type data tables whose risk is greater than or equal to a set threshold as B-type data tables that can be used for association;

[0040] S72. Re-identify records in the Class B data table that have a risk greater than or equal to a set threshold and pair them with the Class A data table for association:

[0041] S721. Obtain the quasi-identifier field sets of the two data tables respectively, pair the fields that contain the same personal information type and have the same value, and obtain all the matching field pairs in the two data tables;

[0042] S722. Based on the values ​​of the two data table field pairs determined in S721, analyze the record rows of the two data tables one by one, match the record rows where all quasi-identifier field pairs have the same values ​​and the direct identifier information is also the same as the remaining information of the deidentified identifier field, and make a judgment that the record rows correspond to the same person.

[0043] S73. The number of successfully matched quasi-identifiers is used to measure the confidence that a matching record row corresponds to the same individual, for paired data table A. i and B j Data Table A i The expanded information is in data table B. j The number of quasi-identifiers minus the number of all matching field pairs in both data tables;

[0044] For paired data table A i and B j Data Table A i The confidence level of the expanded personal information is 1 / n, where n represents A. i A record row matches B j n records with a re-identification risk greater than or equal to a set threshold;

[0045] S74. Based on the association matching results, count the number of records that are re-identified under different confidence levels and different levels of certainty;

[0046] Perform statistical analysis on the association matching results similar to that on table A.

[0047] In the above-described statistical analysis method for personal information disclosure in data tables, the statistical analysis results of step S7 are visualized:

[0048] S81. Statistical analysis of Class A and Class B data tables, starting from the two perspectives of data table data and the number of people involved in the data tables, with the type of personal information involved as the X-axis, the domain label as the Y-axis, and the color indicating the number of data tables or the number of people involved, to draw a heat map to show the specific disclosure of the type of personal information involved in the data tables under each domain label;

[0049] S82. Statistical analysis of the correlation and matching between data tables A and B, and design visualizations from different perspectives:

[0050] (1) Using the type of personal information involved as the X-axis, the number of people involved as the Y-axis, and the credibility as the dimension, a multi-dimensional clustered column chart is drawn to show the specific distribution of the record rows of the data table after successful association and matching under different credibility levels for each type of personal information.

[0051] (2) Using the amount of information expanded after data pair association matching as the X-axis and the number of people involved as the Y-axis, a multi-dimensional clustered column chart is drawn with the confidence level range as the dimension to show the expansion of information after the record row is re-identified under different confidence levels.

[0052] In the above-mentioned statistical analysis method for personal information disclosure in data tables, in step S3, after the classification and labeling are completed, the user performs manual verification.

[0053] In step S4, after identifying the alignment identifier, direct identifier, and de-identified field, the user performs auxiliary verification by combining the identification results with the data table title, field value sampling, and the proportion of unique field values.

[0054] In step S5, the Type 1 data table contains record rows containing identification information that can independently identify an individual, and the corresponding individual is directly disclosed; the Type 2 data table exposes personal information in all record rows, and the corresponding individuals have the potential risk of being identified.

[0055] A system for statistical analysis of personal information disclosure, used to execute the method for statistical analysis of personal information disclosure for a data table.

[0056] The advantages of this solution are:

[0057] It has very low requirements for data quality and can perform statistical analysis on personal information disclosure in data tables with varying quality, i.e., structured data tables with random nesting of unstructured data, reducing reliance on human intervention and improving the credibility of statistical analysis.

[0058] This solution proposes a novel data processing method. First, a data catalog is created for the data tables. Based on this catalog, the personal information-related data tables are initially labeled and categorized. Then, the field content of the selected personal information-related data tables is comprehensively identified, accurately and efficiently completing the labeling of field identifiers. Building upon this process, the data tables are divided based on the presence or absence of records containing direct identifier information. Further splitting and reorganizing the data tables based on the presence or absence of such records effectively improves the efficiency of subsequent processing, analysis, and statistics. Finally, a layered classification method is used to analyze the data tables, and statistical analysis is performed from multiple dimensions to automatically generate a report on personal information disclosure, thus comprehensively and completely depicting the platform's personal information disclosure situation.

[0059] Meanwhile, this solution adopts the approach of related disclosure, analyzes the possibility of re-identifying type B data tables using type A data tables, and can realize statistical analysis of personal information disclosure of multiple related data tables, as well as statistical analysis of personal information disclosure of a set of data tables on a certain platform as a whole. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating the overall process framework of the present invention for statistical analysis of personal information disclosure in data tables.

[0061] Figure 2 This is the original data table used in this invention example;

[0062] Figure 3 for Figure 2 The example shows the first and second subplots of the original data table after data cleaning;

[0063] Figure 4 This is a schematic diagram of the directory of a certain data table;

[0064] Figure 5This is a flowchart of the personal information data table reorganization module of the present invention;

[0065] Figure 6 This is a flowchart of the personal information risk statistical analysis module of the present invention;

[0066] Figure 7 This is a flowchart of the visualization module for the analysis results of this invention;

[0067] Figure 8 This is a flowchart of the data catalog creation, data table classification, identification, and annotation modules of this invention. Detailed Implementation

[0068] The present solution will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] Figure 1 This invention provides a flowchart outlining the overall framework for analyzing and statistically analyzing personal information disclosure. It proposes a Python-based framework for statistical analysis and visualization of personal information disclosure in individual data tables and data sets. First, a data table set is obtained from a data resource platform. Then, the data table set is mapped to create a metadata directory. Next, the data table set is categorized and filtered to identify potentially personal information-related tables. Then, for each data table marked as related to personal information, the privacy information in its fields is comprehensively identified. Based on the identification and detection results and extracted information (i.e., whether direct identifiers are identified in the data table), the data table is divided, and rows with identified direct identifiers are separated from the original data table and reorganized into new data tables. Based on the different situations after the data table division and reorganization, a layer-by-layer classification and discussion method is used to analyze the data tables in the actual data table platform scenario. Statistical analysis is performed from multiple dimensions, automatically generating a personal information disclosure result report, thus comprehensively and completely depicting the platform's personal information disclosure situation. Specifically, this may include the following steps:

[0070] Step 1: Establish the data directory. Clean the data table collection obtained from a certain data resource platform. Clean the field names to make them as Chinese as possible. If they are abbreviations or English, remap the field names to Chinese according to the platform information. Remove special characters such as newlines and spaces from the field names. Ensure that each field name in the data table exists and that the field names are mainly Chinese characters. Remove special characters such as spaces from the field names.

[0071] For example, in extreme cases, such as Figure 2The image shows the original data table in XLSS format published by the data platform. It can be seen that the original data table is missing field names and has misalignments. This is due to an operational error when the data platform retrieved the data table from the original database during its release. The data platform also published the same data table in XML format, a tree structure containing complete data information. Each data element contains a tag, which represents the field name. By using BeautifulSoup to process the XML file and convert it to a standard dataframe format, the tags in the XML can be extracted as the field names needed in the XLSS format. This process cleans the original data table, transforming it step by step to obtain a more standardized dataset. Figure 3 The first subgraph and the second subgraph are shown. The first subgraph is the data table obtained by transforming the original data table through XML and XLS structures. The second subgraph is the cleaned data table obtained by using the data table metadata provided on the data platform to transform the English and abbreviation field names in the data table of the first subgraph.

[0072] Based on the actual file information of the data table set provided by a specific data resource platform, the mapping method between the data table file name and the specific data table name is first determined. This means using a mapping code to uniquely read a specific data table and obtain its correct name. Generally, mapping codes are provided by the data resource platform. These codes can be obtained synchronously when acquiring the data table set and recorded in the metadata directory. An algorithm can then be used to read the specific data table file and retrieve the data table information. If the platform does not provide such a mapping code, a unique mapping is established between the specific data table and its corresponding file name, generating a mapping code.

[0073] The system iterates through and reads the cleaned data tables, obtaining the filename, field name set, table title, and relevant information (such as domain annotations) provided by the data resource platform for each table. Based on the field name set and its mapping code in the current data table, a physical mapping is established between the table's filename, field name set, mapping code, title, and relevant information (such as domain annotations) provided by the data resource platform, integrating them into a data table directory. Each row in the data table directory corresponds to one field, and all rows represent the collection of all fields in the data table set.

[0074] Figure 4 This is a specific example of establishing a mapping for the data table directory, where title is the data table title, domain is the domain annotation information, url is the webpage URL link of the data platform corresponding to the data table, filecode is the mapping code, filenames is the specific data table file name, and columns is the cleaned set of field names.

[0075] Step two involves data table classification. Using the cleaned data table set and established data directory, the data tables are read according to the mapping. Preprocessing and feature extraction and detection of field content are performed, obtaining information such as the percentage of unique field values, the distribution of various character types in the sampled samples (e.g., whether it is Chinese, numbers, or English), and the field data type. These features are used to vectorize the fields. Text vectorization is performed using the data table title and domain annotation of the field name entry. The vector features are then merged to obtain vectorized field features. The merged vectorized features are input into the trained decision tree model for prediction to obtain classification labels for field identifier types. Preliminary classification labels are then applied to each field of the data table, classifying it as a direct identifier, a de-identified identifier, a quasi-identifier, or a non-identifier. Finally, the field labels in the data table are summarized to preliminarily confirm whether the data table is related to personal information.

[0076] Field characteristics include the title of the data table containing the field, the domain label of the data table containing the field, the field data type, the field name itself, the field data type, the proportion of character types in the sampled values ​​of the field under this field, and the proportion of unique values ​​of this field.

[0077] Possible data types for fields include int, float, object, date, and bool. Plain text fields are object; fields containing only phone numbers should be int, but there's a possibility of them being mistakenly stored as float, requiring pre-processing and conversion; fields containing both numbers and text values ​​are object; and date fields are date.

[0078] The percentage of unique values ​​for a field is calculated as follows:

[0079] Given a field j in a data table K, let all_length be the total length of field elements in field j after removing empty sets. j The total length of field j after removing null values ​​and then using unique() to remove duplicate values ​​is denoted as uni_length. j Input the lengths of the two elements into the following formula:

[0080]

[0081] The purpose of calculating the percentage of unique values ​​in a field is to use an indicator to represent the distribution characteristics of that field's values, which can then be used as a feature variable input into a decision tree model. For example, a dataset summarizing teacher information might contain a field displaying names and a field displaying institution names. Because names are direct identifiers, uniquely identifying an individual under specific circumstances, the percentage of unique values ​​in the name field is generally above 0.95 (except in cases of duplicate names and records). In contrast, institution names are quasi-identifiers; different individuals can work for the same institution. Therefore, the percentage of unique values ​​in the institution name field is generally not very high, often below 0.5. Thus, this indicator can, to some extent, distinguish between direct and quasi-identifiers and can serve as a feature variable for dataset field labeling, aiding in dataset classification.

[0082] The following three field examples illustrate the proportion of character types in the sampled values ​​of a field:

[0083] The first field is the name field from the standardized dataset, with a data type of object. Ten sample values ​​were obtained, such as Zhang San, Li Si, Wang Wu, etc. Each character in each sample is a Chinese character. Therefore, the proportion of Chinese characters in each sample is 1, while the proportions of other character types, such as English letters, numbers, separators (spaces, underscores, periods, commas), and the special character "*", are 0. Based on these ten sample values, the results can be calculated and extended to the entire field.

[0084] The second field is the ID card field from the standardized dataset. Because the last digit of the ID card's check digit contains an 'X', ID card numbers may be entirely numeric or a mixture of numbers and letters. The field data type is object. Ten sample values ​​were obtained. For numbers consisting entirely of numeric characters, the percentage of numeric characters is 1, while other character types are 0. For numbers containing an 'X', the percentage of numeric characters is 17 / 18 = 0.94, the percentage of English characters is 1 / 18 = 0.06, and the rest are 0.

[0085] The third field is the name field in the anonymized dataset. The field data type is object, and ten samples were obtained, including names like Zhang*san, Li*si, and Wang*. The field value consists of Chinese characters and the special character "*", meaning the proportion of Chinese characters in the samples is 0.66 or 0.5, and the proportion of the special character "*" is 0.33 or 0.5. The values ​​are then expanded to the entire field based on the character proportions of the samples.

[0086] For example, consider a dataset titled "Outstanding Head Teachers in a Certain City" in the "Education and Culture" category. Taking the "Teacher's Name" field as an example, after sampling ten samples, it was found that the proportion of unique values ​​was very low, while the proportion of special characters "*" was relatively high. This is because the field had been de-identified. Therefore, the classification model made a labeling decision for the field as a "de-identified identifier".

[0087] In the same data table, although the "serial number" field has a high proportion of unique values ​​and a high proportion of numeric or English characters, the field name "serial number" is usually labeled as a non-identifier in the pre-trained classification model. Therefore, the model makes the labeling decision that the field belongs to "non-identifier".

[0088] The main purpose of classification and labeling is to pre-analyze and determine whether the fields of the data table are related to personal information, so as to carry out subsequent privacy information identification and detection tasks, and analysis and statistical tasks of single data tables and multiple related data tables.

[0089] Step three involves field information identification and annotation. After obtaining the personal information-related data tables through classification and annotation, a comprehensive identification of the content of each data table in the data table set is performed. Identification algorithms are used to traverse and screen each structured and unstructured field in the data tables. The identification algorithms include regular expressions, named entity recognition methods, and keyword matching tools. Regular expression methods are used to identify strings of text with obvious regular characteristics, such as mobile phone numbers, ID card numbers, bank card numbers, and license plate numbers. The named entity recognition method uses the LAC lexical analysis tool to identify and extract names from the text.

[0090] Simultaneously, the identification of de-identified identifier fields is performed. A preliminary determination of whether a field belongs to a de-identified identifier is made by judging the proportion of field values ​​containing commonly used masked characters to the field length. For example, the presence of the special character "*" is used as a criterion for determining de-identification, and the proportion of field values ​​containing "*" is used to confirm the degree of de-identification processing.

[0091] For the remaining fields in the data table, such as gender, education level, and work unit, which are complex and diverse quasi-identifier information, a thesaurus is used, such as the personal information classification reference example in Appendix B, Table B.1 of the "Network Security Standard Practice Guide - Network Data Classification and Grading Guide" (hereinafter referred to as the "Guide"), to label the quasi-identifier fields by matching field name keywords and combining them with field value samples.

[0092] This step obtains the field completion identifier annotation, such as whether it contains direct identifier information, whether it is a directory of de-identified identifier fields and quasi-identifier fields.

[0093] Step 4: Based on the presence or absence of direct identifier information in the data table identification results, data tables involving personal information in the data resource platform can be divided into two types: (1) Data tables containing record rows that contain identifiable information that can independently identify an individual, and the corresponding individual is directly disclosed; (2) Personal information exposed in all record rows of the data table, and the corresponding individual only has the potential risk of being identified. Therefore, the disclosure of personal information by the data resource platform can be divided into two levels: direct disclosure of individuals and individuals with the risk of re-identification. Furthermore, the direct disclosure of individuals by the platform can be divided into two different methods: direct disclosure of a single data table and related disclosure of multiple data tables on the platform.

[0094] Therefore, this plan is based on... Figure 5 The process categorizes relevant data tables into Type (I) and Type (II) tables based on whether they contain direct identifier information. Then, it further breaks down the Type (I) tables by row level, separating rows containing identifiable information from those without, creating two separate data tables: one containing identifiable information (Type A) and another where no rows contain identifiable information (Type B). Identifiable information refers to information with direct identifiers; information with only quasi-identifiers or de-identified identifiers is considered unidentifiable. The five types of direct identifier information detected during the identification process are added to the Type A tables, with each added information corresponding to a row to facilitate subsequent analysis and statistics. For example, a field named "Case Description" might contain descriptive text such as ID numbers, names, and phone numbers. However, this information is contained within unstructured text and is not structured. This solution detects this information during the recognition process, extracts it, and adds it to the original data table, forming a structured field. Maintaining a correspondence between the added recognized information and the record row means that the row index of the name and ID number extracted from a segment of text in a given "Case Description" row should match the row index of that "Case Description" row, indicating that these names and ID numbers were extracted from that row of information.

[0095] For any platform scenario, the number of data tables of type A is equal to the number of data tables of type (I) above, while the number of data tables of type B will not be less than the number of data tables of type (II) above. The disclosure of personal information in all record rows of data tables of types A and B can be respectively... Figure 1 The system is divided into two levels: "direct disclosure by individuals" and "re-identification of risks by individuals".

[0096] Step 5: Based on the domain labels assigned to the data tables by the data resource platform and the classification of personal information in the "Guidelines" document, perform statistical analysis on the Class A data tables, Class B data tables, and related results using different processes, such as... Figure 6 As shown:

[0097] 1) In Category A data tables, the individuals corresponding to each record row are directly disclosed. For the field names of direct identifiers and quasi-identifiers in Category A data tables, information types are classified by establishing an information type dictionary according to Table B.1 in Appendix B of the Guidelines. The types of personal information disclosed in regular data tables mainly include: basic personal information, personal identity information, personal health and physiological information, personal education and work information, personal property information, and other information. Statistical analysis of the distribution of this type of data table across different fields can visually demonstrate the direct disclosure of individuals in each field within the platform scenario; statistical analysis of the coverage distribution of this type of data table for different types of personal information in specific fields can demonstrate the different information leakage situations of directly disclosed individuals in different fields within the platform scenario. Since the number of individuals involved in each data table varies significantly, this embodiment summarizes the number of records involving each relevant personal information type in each data table (for identified direct identifiers, the number of direct identifier identification results in each row is used as an indicator of the number of people involved; for matched quasi-identifier fields, the sum of the number of record rows is used as an indicator of the number of people involved).

[0098] In the process of classifying personal information types, a single type may include both direct identifiers and quasi-identifiers. For example, basic personal information includes name, gender, age, and ethnicity. For direct identifiers identified in the data table, the number of records for the relevant personal information type is determined by referring to the identification results of each direct identifier when counting the number of people involved. For quasi-identifier fields matched in the data table, the number of records for the relevant personal information type is determined by referring to the number of valid rows under each quasi-identifier field when counting the number of people involved. Finally, after deduplicating individuals with the same direct identifier information across data tables, the number of data tables involving the relevant personal information type and the number of people involved are calculated.

[0099] The above statistical analysis methods enable a comprehensive and quantitative portrayal of individuals and their personal information disclosed in a single data table for a specific platform scenario.

[0100] 2) For Class B data tables, each record row in the table corresponds to an individual with a potential risk of being identified. First, all quasi-identifier fields in each data table are combined to form equivalence classes. Following the re-identification risk calculation method described in the background art, the re-identification risk of each record row in each data table is calculated. Record rows unique to the equivalence class filtering criteria are counted and extracted, denoted as record rows with a re-identification risk of 1. For data tables with a maximum re-identification risk of 1, a statistical method similar to that used for Class A data tables is applied. However, since this type of data table does not contain direct identifiers, when counting the number of individuals involved in related personal information types in this type of data table, only the number of valid record rows under the matched quasi-identifier fields in the data table is referenced as an indicator of the number of individuals involved in the data table. Statistical analysis of its distribution across different fields and its coverage distribution of different types of personal information in specific fields can demonstrate the existence of high-risk data tables on the platform in different fields and the different information disclosure situations they contain. The data rows with a re-identification risk of 1 in the B-type data table are summarized as an indicator of the number of individuals involved in the data table, thereby achieving a comprehensive characterization and quantitative display of the personal information disclosed in the data table with a maximum re-identification risk of 1 in a specific platform scenario.

[0101] 3) Associate the rows in table A and table B where the re-identification risk is 1. Following the method of matching the distribution of quasi-identifier field information, first identify all matching quasi-identifier field pairs in both tables that share a common value. Then, based on the values ​​of these quasi-identifier field pairs, perform quasi-identifier matching row by row. If all quasi-identifier field pairs in both rows match, and the direct identifier information and the remaining information of the de-identified identifier field also match, then the two rows are considered a match and will be recorded in the association matching results. The association matching results will be used to statistically analyze the number of people involved, similar to the method used for table A.

[0102] After completing the record row association matching, the number of successfully matched quasi-identifiers can be used to measure the credibility of the matched record rows corresponding to the same individual. For paired data table A i and B j Data Table A i The expanded information is in data table B. j The number of quasi-identifiers minus the number of all matching field pairs in both data tables. Furthermore, given the existence of table A in class A... i A single record row, after being matched step-by-step, may correspond to record B in table B. j The n rows of records with a re-identification risk of 1 match all possible values ​​of the matching quasi-identifier field pairs. This is defined here after association matching, for paired data table A i and B j Data Table A iThe confidence level of the expanded personal information is 1 / n. Finally, based on the association matching results, the number of records that achieved re-identification under different confidence levels is counted.

[0103] This scheme employs a distributed matching approach. At the hierarchical level of the two data tables, it first identifies all matching quasi-identifier field pairs that share common values. Based on the values ​​of these quasi-identifier field pairs, it then performs quasi-identifier matching row by row. By relying on quasi-identifier matching, the data table used for matching is sliced ​​and compressed before traversing for matching, which significantly reduces computational consumption while achieving association analysis.

[0104] Step Six: For the statistical results obtained in the three parts of Step Five, according to... Figure 7 The described processes are visualized using graphs. The heatmap showing the number of data tables (Type A and Type B) and the distribution of the number of people involved is labeled with X-axis tags composed of the types of personal information involved, and Y-axis tags composed of the labels for the domains to which the data tables belong. The number of data tables or the number of people involved indicates the color value range of the specific blocks corresponding to the X and Y labels. This heatmap is used to display the specific disclosure of the types of personal information involved in the data tables under each domain label.

[0105] The credibility and certainty of associated matching records are calculated, and the number of record pairs is summarized according to relevant dimensions as a reference for the number of people involved. A multi-dimensional clustered bar chart of associated matching information disclosure uses the types of personal information involved as the X-axis label, the number of people involved as the Y-axis label, and credibility as the reference dimension of the bar chart. A multi-dimensional clustered bar chart of associated matching information expansion uses the amount of personal information expanded after association as the X-axis label, the number of people involved as the Y-axis label, and the range of certainty as the reference dimension of the bar chart. These multi-dimensional clustered bar charts are used to illustrate the expansion of information after record row re-identification at different certainty levels.

[0106] like Figure 8 As shown, in another embodiment, in step two, after initially confirming whether the data table is related to personal information, the classification labels are then manually screened for negativity only, and the data tables labeled as personal information data tables are manually verified. In step three, the identified direct identifiers and de-identified identifiers can be accompanied by manual auxiliary verification to ensure the accuracy of the labeling as much as possible. At the same time, the remaining fields other than the direct identifier field and the de-identified identifier field can also be labeled as quasi-identifier fields in conjunction with keyword matching and manual auxiliary verification. In this embodiment, a human-computer interactive mode can be used in the identification process to ensure efficient identification of direct identifier information while maintaining the accuracy of quasi-identifier information identification, minimizing misjudgments.

[0107] This solution, based on data classification and grading, and data content recognition technologies, addresses the challenge of handling routine identification and statistical analysis tasks in data resource platform scenarios where data quality varies significantly. It proposes a statistical analysis method, system, and visualization tool implemented in Python, integrating data cleaning, classification, identification, and association processes. Through the framework of this invention, a method for classifying and reorganizing the original data table set based on the identification results, a step-by-step matching process, and a visualization method defining credibility and certainty concepts, administrators can perform statistical analysis on the disclosure of personal information in individual data tables and data table sets within real-world data resource platform scenarios. The credibility of the statistical results is assessable and verifiable. This enables administrators of specific data resource platforms using this invention to efficiently and accurately understand the disclosure of personal privacy information related to resources published on the current platform, the potential re-identification risks of data tables, and the achievable re-identification scenarios.

[0108] The specific embodiments described herein are merely illustrative examples of the spirit of this solution. Those skilled in the art to which this solution pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, without departing from the spirit of this solution or exceeding the scope defined by the appended claims.

Claims

1. A method for statistical analysis of personal information disclosure cases of a data table, characterized by, The method comprises the following steps: S1. Obtain a data table to be analyzed; S2. Clean the data table to be analyzed and make a metadata directory for the cleaned data table to be analyzed; Each entry of the metadata directory corresponds to a data table to be analyzed, and comprises a field name set of the corresponding data table to be analyzed and a mapping code for pointing to the corresponding data table to be analyzed; S3. Match the data table to be analyzed of each entry based on the mapping code; Complete the preliminary classification labeling of the field identifier type with the field value information of each field of the corresponding data table as the feature vector for the field name set of each entry, and screen the entries related to personal information in the metadata directory; S4. Extract the corresponding personal information related data table based on the screening result of step S3, and comprehensively identify the field values of the field names in the personal information related data table; S5. Classify the personal information related data table into type one data table and type two data table according to whether the data table contains a record line containing direct identifier information; S6. Reclassify the type one data table into A class data table composed of record lines containing direct identifier information and B class data table not containing direct identifier information according to whether the record line in the data table contains direct identifier information; Classify the type two data table into B class data table; S7. Statistically analyze the A class data table and the B class data table to statistically analyze the personal information disclosure of the data table to be analyzed; For the A class data table, the distribution of the data table in different fields is counted, the coverage distribution of the data table to different types of personal information in a specific field is counted, the number of records related to each type of personal information of each data table is counted, for the identified direct identifier information, the number of direct identifier identification results of each row is used as an indication of the number of persons involved, and for the matched quasi identifier field, the number of record lines is used as an indication of the number of persons involved, and the same individual with the same direct identifier information in different data tables is de-duplicated as a reference indication of the number of persons involved in the data table; For the B class data table, the re-identification risk of each record line in the data table is calculated according to the calculation method of the re-identification risk, the maximum re-identification risk of the data table is recorded as the maximum re-identification risk of all record lines in the data table, and the number of data tables with different maximum re-identification risks is counted; For the data table with a maximum re-identification risk greater than or equal to a set threshold, the distribution of the data table in different fields and the coverage distribution of the data table to different types of personal information in a specific field are counted, and all data rows with a re-identification risk greater than or equal to the set threshold in the data table are counted as an indication of the number of individuals involved in the data table.

2. The method for statistical analysis of personal information disclosure situation of a data table according to claim 1, characterized in that, In step S2, the cleaning of the data table to be analyzed comprises any one or a combination of field misposition correction, field name improvement, field name conversion and special character processing, so that each field name of the cleaned data table exists and corresponds to the field value, the field name is mainly composed of Chinese characters, and the special characters in the field are removed. The metadata directory includes any one or more of a data table title, a webpage link, a data table file name, and data table related information containing field annotation information of the corresponding data table to be analyzed; and the mapping code, the field name set, the data table title, the webpage link, the data table file name, and the data table related information of each data table to be analyzed are mapped to integrate into the metadata directory.

3. The method for statistical analysis of personal information disclosure cases of data tables according to claim 1, characterized in that, Step S3 specifically includes: S31. Obtain the field value characteristics of the field name set in each entry; S32. Perform text vectorization on the data table title and the data table field annotation of the entry in which the field name is located, vectorize the field value characteristics of the field set, and combine the vector characteristics; S33. Input the combined vectorized characteristics into a trained machine learning classification model, and the model outputs the identifier type annotation of each field; S34. Determine whether the corresponding entry is related to personal information according to the identifier type annotation, so as to screen the entries related to personal information from the metadata directory. In step S33, the identifier types that can be annotated include direct identifiers, de-identified identifiers, quasi-identifiers, and non-identifiers.

4. The method for statistical analysis of personal information disclosure situation of a data table according to claim 3, characterized in that, In step S34, when one of a direct identifier, a quasi-identifier, and a de-identified identifier exists in a field in an entry, it is determined that the data table pointed to by the entry is related to personal information. In step S4, the comprehensive identification method is as follows:

5. The method for statistical analysis of personal information disclosure cases of a data table according to claim 4, characterized by, For direct identifiers, regular expressions are used to identify information that strictly follows a certain formation pattern; and named entity recognition methods are used to identify and extract information that does not have a strict formation pattern in descriptive text; For quasi-identifiers, according to personal information reference files, metadata recognition technology based on a keyword library is used for identification matching; For de-identified identifiers, the degree of de-identification is detected. In step S7, the association analysis method of the A-type data table and the B-type data table further includes:

6. The method for statistical analysis of personal information disclosure cases of data tables according to claim 1, characterized in that, S71. The B-type data table with a re-identification risk greater than or equal to a set threshold is used as a data table that can be used for association; S72. The record rows in the B-type data table that can be used for association and have a re-identification risk greater than or equal to a set threshold are paired and associated with the A-type data table: S721. Obtain the quasi-identifier field set of the two data tables respectively, pair the fields containing the same type of personal information and having the same value, and obtain all the matchable field pairs in the two data tables; S722. According to the values of the field pairs of the two data tables determined in S721, analyze the record rows of the two data tables one by one, match the record rows in which all the quasi-identifier field pairs have the same value and the direct identifier information and the remaining information of the de-identified identifier field are also the same, and make a judgment that the pair of record rows corresponds to the same person; S74. According to the association matching result, the number of records that achieve re-identification under different confidence and different certainty is counted; S73. Use the number of successfully matched quasi-identifiers to measure the confidence that the matched record rows correspond to the same individual, for pair data table A i and B j , the amount of information added by data table A i is the number of quasi-identifiers of data table B j minus the number of all matchable field pairs in both data tables; For a pair of data tables A i and B j , the confidence of the augmented personal information of A i is 1 / n, where n represents the number of record rows of A i that match n record rows of B j with a greater than or equal to a set threshold of re-identification risk; The association matching result is statistically analyzed with the A-type data table. The statistical analysis result of step S7 is also visualized.

7. The method for statistical analysis of personal information disclosure cases of a data table according to claim 6, characterized by, ​ S81. Statistical analysis of the A-type data table and the B-type data table, from the number of data tables and the number of people involved, a heat map is drawn with the type of personal information involved as the X-axis, the field label as the Y-axis, and the color indicating the number of data tables or the number of people involved, to show the specific disclosure of the data table under the field label to the type of personal information involved; S82. Statistical analysis of the association and matching of A and B data tables, and different angles of visual presentation are designed: (1) With the type of personal information involved as the X-axis and the number of people involved as the Y-axis, a multi-dimensional cluster column chart is drawn with the confidence as the dimension to show the specific distribution of the record lines after successful association and matching of the data table to the type of personal information involved under different confidence; (2) With the amount of information expanded after the association and matching of the data pair as the X-axis and the number of people involved as the Y-axis, a multi-dimensional cluster column chart is drawn with the confidence range as the dimension to show the expansion of the amount of information after the record line is re-identified under different confidence.

8. The method for statistical analysis of personal information disclosure cases of data tables according to claim 1, characterized in that, In step S3, after the classification and labeling are completed, the user performs manual verification; In step S4, after the identification of the quasi-identifier, the direct identifier and the de-identified field, the user performs auxiliary verification; In step S5, the type one data table contains record lines containing identification information that can independently identify individuals, and the corresponding individuals are directly disclosed; all the personal information exposed in the record lines in the type two data table has a potential risk of being identified.

9. A personal information disclosure situation statistical analysis system for performing the personal information disclosure situation statistical analysis method for data tables according to any one of claims 1-8.

Images