System and method for classifying and grading personal information data in combination with large language model
By combining large language models to classify and classify personal information data, the inefficiency problem of difficult to deal with complex data and null values in the prior art is solved, and a more efficient and flexible classification and classification process is achieved.
Patent Information
- Application Number
- CN202411639959.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-03-21
AI Technical Summary
When the prior art automatically classifies and safely classifies personal information, it is difficult to deal with fields that cannot summarize the regular expression matching rules of data, and when the value in the field is empty, the matching judgment cannot be made, which is inefficient and requires a lot of human resources.
The method combined with the large language model is adopted to classify and classify personal information data, obtain metadata information and classification grading label list, use the large language model to classify and judge, and classify and classify and classify and classify and rank by full traversal matching of regular expressions when classification fails.
It greatly improves the efficiency of classification and grading, can better adapt to the classification and grading of complex data, reduces the need for regular expression maintenance, and improves the flexibility and efficiency of the system.
Smart Images
Figure CN119669926A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to data processing technology, and more specifically, to a system and method for classifying and grading personal information data in combination with a large language model. Background Art
[0002] At present, the automatic classification and security classification of personal information in the field of big data is generally achieved by matching field data with regular expressions. However, the following problems exist when using this technical route:
[0003] First, regular expression matching can only classify and identify data with fixed numerical formats, such as names, ID numbers, etc. However, in standards and specifications such as the "Cybersecurity Standard Practice Guidelines - Guidelines for Classification and Classification of Network Data" and "Information Security Technology Sensitive Personal Information Processing Security Requirements", most of the standards for classifying and grading personal information, such as determining whether a field is an account number, password, company name, project name, salary amount, etc., are difficult to achieve by summarizing the fixed format of data and performing regular expression matching.
[0004] Second, if all the values in a field are empty, regular expression matching cannot be performed.
[0005] In view of the above situation, when classifying and grading the fields, it is necessary to manually classify and grade them through metadata information such as field names and field notes, but this method is inefficient and requires more human resources. Summary of the invention
[0006] The technical problem to be solved by the present invention is to address the deficiencies in the prior art and to provide a system and method for classifying and grading personal information data in combination with a large language model. Compared with the traditional method of only matching rules through regular expressions, the efficiency of classification and grading is greatly improved.
[0007] The method for classifying and grading personal information data in combination with a large language model described in the present invention comprises the following steps:
[0008] S1. Obtain metadata information of the table and field in the personal information table;
[0009] S2. extract all classification and grading labels from the personal information classification and grading label management library to form a classification and grading label list;
[0010] S3. Classify and judge the metadata information of the field by combining the large language model with the classification and grading label list, so as to classify the field into the classification and grading label corresponding to it in the classification and grading label list; if the field classification fails, classify and grade the field by full traversal matching of regular expressions;
[0011] S4. Classify and grade the metadata information of a single field according to a predetermined screening condition rule;
[0012] S5. Classify and grade the tables according to the metadata information of the tables.
[0013] As a further improvement, the metadata information is classified and judged by combining the classification and grading label list with a large language model, specifically:
[0014] The classification and grading label list is passed as knowledge to the large language model, and the large language model determines whether the field can be classified into a category in the classification and grading label list according to the metadata information of the field; if the large language model determines the classification of the field, the classification result is verified.
[0015] Furthermore, the specific method for verifying the classification result is:
[0016] The value of the field of metadata information of the field is obtained. If a non-empty value is stored under the field, the value of the field is matched with the corresponding regular expression in the classification and grading label list. If the proportion of values matching the regular expression exceeds the set threshold, the verification is passed, and the classification and grading label currently judged by the large language model is used as the classification and grading label of the field; if the proportion of values matching the regular expression does not exceed the set threshold, the value of the field is fully traversed and matched with the regular expression in the personal information classification and grading label management library to find and determine the classification of the field.
[0017] Furthermore, the regular expression full traversal matching is specifically:
[0018] The numerical value of the field is matched and compared one by one with all regular expressions in the personal information classification and grading label management library. If a corresponding regular expression is found, the classification and grading label corresponding to the current regular expression is used as the classification and grading label of the field; if no corresponding regular expression is found, it indicates that the classification and grading label of the field cannot be determined, and its classification and grading label is marked as a null value.
[0019] Furthermore, if the value of the field is a null value, it indicates that the classification and grading label of the field cannot be determined, and the classification and grading label is marked as a null value.
[0020] A system for classifying and grading personal information data in combination with a large language model, comprising an application service platform and a data middle platform; the application service platform obtains personal information from the data middle platform, and classifies and grades the personal information using the personal information data classification and grading method.
[0021] Beneficial Effects
[0022] The advantages of the present invention are:
[0023] 1. More comprehensive data understanding: Metadata (such as data type, source, creation time, update frequency, etc.) provides additional information about the data itself, which is essential for accurate classification and grading. Large models are able to comprehensively consider these metadata to more fully understand the data.
[0024] 2. Better adaptability and flexibility: For fields that cannot summarize the regular expression matching rules of the data, classification and security classification can still be performed through the large language model, which is much more flexible than classification and classification using only rules.
[0025] 3. Higher scalability and efficiency: When classifying and grading by rules, the maintenance of classification and grading is to add, delete and modify regular expressions. Maintaining a large number of regular expressions requires a lot of manpower. For large-scale classification and grading, only the new classification and grading labels need to be updated to the personal information classification and grading label library, which saves the manpower of designing and maintaining regular expressions, and has higher scalability and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 It is a schematic diagram of the method flow of the present invention;
[0027] Figure 2 It is the classification dimension information table of the present invention;
[0028] Figure 3A The personal information data classification and grading table of the present invention;
[0029] Figure 3B Based on Figure 3A Continuation of . DETAILED DESCRIPTION
[0030] The present invention will be further described below in conjunction with the embodiments, but it does not constitute any limitation to the present invention. Any limited number of modifications made by anyone within the scope of the claims of the present invention are still within the scope of the claims of the present invention.
[0031] See also Figure 1 The present invention provides a method for classifying and grading personal information data by combining a large language model. The method uses the semantic understanding ability of the large language model technology to judge the role and classification of the database table and field. The method makes up for the shortcomings of the previous data classification and grading technology route, and can be combined with the traditional rule classification method to realize intelligent personal information classification and grading. The specific introduction is as follows.
[0032] With reference to the relevant provisions on personal information classification and grading in the standards and specifications such as the "Network Security Standard Practice Guidelines - Network Data Classification and Grading Guidelines" and "Information Security Technology Sensitive Personal Information Processing Security Requirements", personal information is divided into two first-level categories: sensitive personal information and general personal information, and 16 second-level categories. It involves three classification dimensions: fields, data, and tables. Different classification dimensions require different classification rules and security grading schemes. Specific information for each classification dimension is as follows: Figure 2 shown.
[0033] based on Figure 2 The classification dimensions in the personal information classification and grading table are as follows: Figure 3A and Figure 3A Continuation of Figure 3B shown.
[0034] The following will be based on Figure 2 and Figure 3A , Figure 3B The method for classifying and grading personal information data of the present invention is described in detail. The specific implementation steps of the method are as follows:
[0035] Step 1. Obtain metadata information in the personal information table from the data warehouse, including table metadata information and field metadata information.
[0036] Step 2: Classify personal information in multiple dimensions. Specifically, it can be classified and graded from three classification dimensions: field, data, and table.
[0037] Step 2.1. Determine the field dimension.
[0038] Step 2.1.1. Extract all classification and grading labels from the personal information classification and grading label management library to form a classification and grading label list.
[0039] Step 2.1.2. Classify the field using a large language model combined with the classification and grading label list to classify the field into the classification and grading label corresponding to it in the classification and grading label list.
[0040] Specifically, the classification and grading label list is passed as knowledge to the large language model as a reference for the large speech model to perform classification and grading tasks, and it can be determined whether there are classification and grading labels corresponding to the metadata information. If the large language model determines the classification of the field, the classification result is verified. The specific verification method is implemented by the following steps.
[0041] Step 2.1.2.a1. If a non-empty value is stored in the field, the value of the field is matched with the regular expression corresponding to the classification grade label to verify whether the classification is correct.
[0042] In step 2.1.2.a2, if the proportion of values that can match the regular expression exceeds the set threshold, the verification passes, and the classification and grading label currently judged by the large language model is used as the classification and grading label of the field.
[0043] In step 2.1.2.a3, if the percentage of values that match the regular expression does not exceed the threshold, the metadata information is fully traversed and matched with the regular expression in the personal information classification and grading tag management library to find and determine the classification of the metadata information. In other words, if there is no match, jump to step 2.1.2.b1.
[0044] Step 2.1.2.a4. If the values of the field are all null values, it means that the classification and grading label of the field cannot be determined, and its classification and grading label is marked as null value.
[0045] Step 2.1.3. In step 2.1.2, if the field classification fails, the fields are classified and graded through full traversal matching of regular expressions.
[0046] In this embodiment, full traversal matching of regular expressions is achieved through the following steps.
[0047] Step 2.1.2.b1. Match the value of the field with all regular expressions in the personal information classification and grading label management library one by one.
[0048] Step 2.1.2.b2. If a corresponding regular expression is found, the classification and grading label corresponding to the current regular expression is used as the classification and grading label of the field.
[0049] Step 2.1.2.b3. If no corresponding regular expression is found, it means that the classification and grading label of the field cannot be determined, and its classification and grading label is marked as a null value.
[0050] Step 2.2. Determine the data dimension.
[0051] Each conditional data is classified and graded according to the predetermined screening condition rules. For example, after identifying the age and occupation fields in the personal information in step 2.1, it is determined whether there is personal information of minors under the age of fourteen or practitioners of specific professions (such as soldiers and police). Such personal information will be screened out, classified as sensitive personal information and saved separately.
[0052] Step 2.3. Determine the table dimension.
[0053] The entire table is classified and graded based on the table information, such as the source of the table, the metadata information of the table, and the amount of data. For example, if the amount of data in the table exceeds 1 million, it may constitute important personal information and should be classified as personal information level 5. For another example, based on the metadata information of the table, the big model identifies the table as storing the user's medical health information, financial account information, whereabouts information, criminal information, etc., and classifies the table as sensitive personal information.
[0054] Step 3. Comprehensively classify the results and output the classification and grading results of the field dimension, data dimension, and table dimension of personal information respectively, so as to facilitate the refined management of data according to different dimensions in different business scenarios.
[0055] A system for classifying and grading personal information data in combination with a large language model, including an application service platform and a data middle platform; the application service platform obtains personal information from the data middle platform, and classifies and grades the personal information using a personal information data classification and grading method.
[0056] Among them, a large language model application service is used in the application service platform. Specifically, it is a large model agent developed using the Dify large model application framework. This service is used to receive user requests, process business logic, call large model services and database query services, and finally return classification and grading results to users. The Dify framework is developed in a code-free way. Developers only need to draw workflows on the canvas to complete the development of the program. The data middle platform is developed using the SpringBoot framework. The agent can call the service to query the data warehouse and the personal information classification label management library through HTTP requests. The personal information classification label management library uses a MySQL database to store various personal information classification and grading labels, and their corresponding regular expressions and security levels.
[0057] A company has corresponding management requirements for different levels of personal information at all stages of the data life cycle. Therefore, it is necessary to use a data classification and grading system to label the database tables and then enable corresponding management strategies for the data based on the labels.
[0058] How to use the application service platform in combination with the data center:
[0059] First, add an "Automatic Classification and Grading" button to the page for viewing library and table details in the metadata management of the data center.
[0060] Second, after clicking the "Automatically Classify and Gradation" button, the connection method and metadata of the database table will be transmitted to the application service platform through the interface.
[0061] Third, the application service platform returns the classification and grading labels to the data center, where the field classification and grading labels and table classification and grading labels are directly stored in the metadata table of the data center.
[0062] Fourth, use of classification and grading labels: The data center adopts different management strategies in the four links of data storage, use, sharing, and destruction according to the classification and grading labels. For example, level 3 data needs to be encrypted when stored, and the storage medium needs to be physically destroyed when destroyed.
[0063] The above is only a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the structure of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent.
Claims
1. A method for classifying and grading personal information data in combination with a large language model, characterized in that: The method comprises the following steps: S1. Obtain metadata information of the table and field in the personal information table; S2. extract all classification and grading labels from the personal information classification and grading label management library to form a classification and grading label list; S3. Classify and judge the metadata information of the field by combining the large language model with the classification and grading label list, so as to classify the field into the classification and grading label corresponding to it in the classification and grading label list; if the field classification fails, classify and grade the field by full traversal matching of regular expressions; S4. Classify and grade the metadata information of a single field according to a predetermined screening condition rule; S5. Classify and grade the tables according to the metadata information of the tables.
2. According to claim 1, a method for classifying and grading personal information data in combination with a large language model is characterized in that: The metadata information is classified and judged by combining the classification and grading label list with the large language model, specifically: The classification and grading label list is passed as knowledge to the large language model, and the large language model determines whether the field can be classified into a category in the classification and grading label list according to the metadata information of the field; if the large language model determines the classification of the field, the classification result is verified.
3. According to claim 2, a method for classifying and grading personal information data in combination with a large language model is characterized in that: The specific method for verifying the classification result is: The value of the field of metadata information of the field is obtained. If a non-empty value is stored under the field, the value of the field is matched with the corresponding regular expression in the classification and grading label list. If the proportion of values matching the regular expression exceeds the set threshold, the verification is passed, and the classification and grading label currently judged by the large language model is used as the classification and grading label of the field; if the proportion of values matching the regular expression does not exceed the set threshold, the value of the field is fully traversed and matched with the regular expression in the personal information classification and grading label management library to find and determine the classification of the field.
4. According to claim 3, a method for classifying and grading personal information data in combination with a large language model is characterized in that: The regular expression full traversal matching is specifically: The numerical value of the field is matched and compared one by one with all regular expressions in the personal information classification and grading label management library. If a corresponding regular expression is found, the classification and grading label corresponding to the current regular expression is used as the classification and grading label of the field; if no corresponding regular expression is found, it indicates that the classification and grading label of the field cannot be determined, and its classification and grading label is marked as a null value.
5. A method for classifying and grading personal information data in combination with a large language model according to claim 3 or 4, characterized in that: If the value of the field is a null value, it indicates that the classification and grading label of the field cannot be determined, and the classification and grading label is marked as a null value.
6. A system for classifying and grading personal information data in combination with a large language model, characterized in that: It includes an application service platform and a data middle platform; the application service platform obtains personal information from the data middle platform, and classifies and grades the personal information using the method for classifying and grading personal information data as described in any one of claims 1 to 5.