A method and system for automatic classification and grading of unstructured data

Abstract

The application discloses a kind of unstructured data automatic classification grading method and system, including the following steps: scanning unstructured data file, logical convergence is formed data resource list;Set file correlation threshold and divide relevant files;Classify identification is carried out to the data in file using rule matching technique;The classified data in all relevant files are divided into security levels;The data in file is marked, and the unstructured data classification grading list is output.The above technical solution determines the influence range of data by calculating the total amount of key data in relevant files, solves the problem that the classification grading mark granularity of unstructured data in the prior art is relatively coarse, and the data security level is not accurate, classifies and grades the data in the file, outputs the unstructured data classification grading list, effectively protects the data, and effectively utilizes the value of data.

Description

Technical Field

[0001] The present invention relates to the field of data security technology, and in particular to a method and system for automatically classifying and grading unstructured data. Background Art

[0002] With the development of the digital economy, data, as a core production factor, has become a fundamental strategic resource. The value of data continues to grow, but data security risks are also increasing. Data classification and grading are fundamental to identifying important data and implementing security measures for it. Various industries have gradually introduced data classification and grading methods and guidelines, using a combination of manual and technical methods to organize, classify, and grade enterprise data resources. Enterprise data resources are divided into structured data and unstructured data based on their structural characteristics. Structured data is stored in databases and can be easily represented using two-dimensional logical database tables. Unstructured data (such as documents, images, audio, video, emails, etc.) has irregular or incomplete data structures, lacks predefined data models, and exhibits a high degree of independence.

[0003] Data indicates that common existing data classification and grading technologies can be categorized into two main types: rule-matching technologies that employ keywords, dictionaries, and regular expressions, and artificial intelligence technologies that employ machine learning and deep semantic analysis. Existing technologies employ the same or similar data classification and security grading methods for both structured and unstructured data. While the classification and grading of structured data can be fine-grained, down to the field level of a database table, the grading of unstructured data is more coarse, limited to the file level. Furthermore, existing technologies assess the data security of each file based solely on its content characteristics, ignoring two factors that impact the accuracy of data security grading. First, the varying amounts of data within different files can lead to varying data coverage, making it difficult to accurately assess the impact of a data leak on national security, the public interest, or the legitimate rights and interests of individuals or organizations based solely on file content characteristics. Second, common file attributes such as temporal, spatial, source, and format attributes can influence the correlation between files. The combined leakage of highly correlated files poses a far greater threat to national security, the public interest, or the legitimate rights and interests of individuals or organizations than a single file or files with low correlation.

[0004] Therefore, the existing technology only classifies the data security of each file based on the content characteristics of each file, which will lead to inaccurate data security level determination. If the level is too low, the data will not be effectively protected. If the level is too high, it will lead to unnecessary business expenses and it will be difficult to realize the value of the data.

[0005] Chinese patent document CN109635587A discloses a "method and device for automatic data classification and hierarchical protection." The method includes: obtaining at least one information storage system; scanning at least one information storage system to obtain a list of information storage systems; obtaining the account number of each information storage system in the list of information storage systems input by the user; scanning the information storage systems according to the account number of each information storage system to obtain a data information list for each information storage system; establishing a data information classification and hierarchical protection dictionary; and classifying and hierarchically processing the data information list of each information storage system according to the data information classification and hierarchical protection dictionary to obtain a processed data information list. The above technical solution inaccurately defines data security levels, making it difficult to maximize the value of data. Summary of the Invention

[0006] The present invention mainly solves the technical problem that the original technical solution does not accurately define the data security level and is difficult to realize the value of the data. It provides a method and system for automatic classification and grading of unstructured data. By calculating the total amount of key data in related files, the impact range of the data is determined, and the problem that the existing technology has coarse granularity in the classification and grading of unstructured data and inaccurate data security level definition is solved. The data in the file is classified and graded, and a classification and grading list of unstructured data is output to effectively protect the data and effectively realize the value of the data.

[0007] The above technical problems of the present invention are mainly solved by the following technical solutions:

[0008] A method for automatically classifying and grading unstructured data includes the following steps:

[0009] S1 scans unstructured data files and performs logical aggregation to form a data resource list;

[0010] S2 sets the file relevance threshold and divides the related files;

[0011] S3 uses rule matching technology to classify and identify data within files;

[0012] S4 classifies the classified data in all relevant documents into security levels;

[0013] S5 tags the data in the file and outputs a classification and grading list of unstructured data.

[0014] Step 1: Automatically scan all unstructured data files based on system resource paths or server information, logically aggregate the scanned unstructured data files, and form a data resource list;

[0015] Step 2: Read the common attributes of the files in the data resource list, such as time attributes, space attributes, source attributes, and format attributes, calculate the correlation between the files based on the file attributes, and then divide the related files according to the set correlation threshold;

[0016] Step 3: Use keyword, dictionary and regular expression matching technology to classify the data in the file and classify and identify different data;

[0017] Step 4: Calculate the total amount of data for each category in the relevant files, determine the impact range of the data, and then classify the security level of the classified data in all relevant files based on the importance and sensitivity of the data category;

[0018] Step 5: Index and identify the data in the file according to the classification and grading results, mark the categories and levels of the data items, and output the classification and grading list of unstructured data.

[0019] Preferably, the step S2 specifically includes reading common attributes of the files in the data resource list including time attributes, space attributes, source attributes, and format attributes, calculating file relevance based on the file attributes, and then dividing the related files according to a set relevance threshold.

[0020] Preferably, the file relevance is to vectorize common file attributes using a bag-of-words model and calculate the relevance between files using cosine distance. The file relevance calculation step includes:

[0021] Suppose the n attributes of file f constitute the string array R f =[L1, L2, ..., L n ], let L n is the nth attribute of file f, R f is the file attribute vector of f;

[0022] Let the file attribute vector of files a and b be R a and R b , where the i-th attribute is L ai and L bi , after being vectorized by the bag-of-words model, they are respectively one-dimensional integer arrays [a i1 , a i2 ,…,a ik ] and [b i1 , b i2 ,…,b ik ], the length of the array is k, and the access similarity between file a and file b at the i-th attribute is recorded as T i ,have

[0023]

[0024] The similarity of the common attributes of file a and file b forms a one-dimensional vector;

[0025] [T1(L a1 , L b1 ), T2(L a2 , L b2 ), …, T n (L an , L bn )]

[0026] The file correlation of a and b is recorded as S a,b , and

[0027]

[0028] S a,b The value range is between [-1, 1], and the correlation threshold interval of the related files is set to 0.5-1. The files with a correlation in the threshold interval are classified as related files.

[0029] As preferred, the step S3 classifies the data in the file by using the rule matching technology of keyword, dictionary and regular expression matching, specifically including:

[0030] Segment, sentence and word segmentation of the unstructured data file are performed by using NLP technology;

[0031] Rule matching of each word is performed by using the keyword, dictionary and regular expression matching technology;

[0032] The word capable of matching one or more rules is taken as a key data, and the classification corresponding to the one or more rules matched is taken as the classification of the key data.

[0033] As preferred, the step S4 calculates the total amount of data of each classification data in the related files, and determines the influence range of the data, specifically including:

[0034] The total number of each classification key data in the related files is calculated, and taken as the total amount of data of the key data;

[0035] The influence range corresponding to the data amount threshold interval where the data amount is located is determined as the influence range of the data.

[0036] As preferred, the step S4 indexes the data in the file according to the classification and grading results, and outputs the unstructured data classification and grading list, specifically including:

[0037] The category and level of each key data in the file are marked, and then the data type and security level of each file are marked;

[0038] The data type of a single file is the summary result of the categories to which each key data in the file belongs, and the security level of a single file is the highest security level among the levels of each key data in the file.

[0039] Preferably, the contents of the classification and grading list include: file name, file type, department, system, data type, security level, amount of key data, key data classification information, key data grading information, storage directory, retention period, and data processing status.

[0040] An application system for an automatic classification and grading method for unstructured data, comprising:

[0041] Unstructured data scanning module, used to scan unstructured data files and perform logical aggregation to form a data resource list;

[0042] A file correlation analysis module, used to set file correlation thresholds and classify related files;

[0043] Classification and grading standard establishment module, used to determine the data security level of each category in the classification directory tree;

[0044] Data classification management module, used to classify data based on rule matching technology;

[0045] Data classification management module, used to determine the security level of data in all relevant files;

[0046] The data classification and grading identification module is used to classify and grade the data in the file and output a classification and grading list of unstructured data.

[0047] The beneficial effects of the present invention are: by calculating the total amount of key data in relevant files, the impact range of the data is determined, and the problems of coarse granularity of classification and grading marking of unstructured data and inaccurate delineation of data security levels in the existing technology are solved. The data in the file is classified and graded, and a classification and grading list of unstructured data is output to effectively protect the data and effectively realize the value of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the present invention.

[0049] Figure 2 It is a structural diagram of an automatic classification and grading system for unstructured data of the present invention. DETAILED DESCRIPTION

[0050] The technical solution of the present invention will be further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0051] Example: This embodiment provides a method and system for automatically classifying and grading unstructured data, such as Figure 1 As shown, the following steps are included:

[0052] Step S1: Scan unstructured data files and build a data resource list;

[0053] This step reads the user-configured resource path for the system to be scanned or the account information for the server to be scanned. It then automatically scans all unstructured data files based on the system resource path or server information. The scanned unstructured data files are logically aggregated based on information such as file type, storage directory, and system to which they belong, creating a data resource list. The data resource list includes information such as file name, file type, number of files, storage directory, retention period, and data processing status.

[0054] Step S2: Calculate the correlation between the files scanned in step S1 and classify the related files;

[0055] This step reads the time attributes, space attributes, source attributes, and format attributes of the file scanned in step S1 and uses these attributes as input to calculate the cosine distance value, thereby obtaining a measure of file relevance. The file relevance calculation method is as follows:

[0056] Suppose the n attributes of file f constitute the string array R f =[L1, L2, ..., L n ], these attributes can be creation time, storage directory, system, file type, etc. Let L n is the nth attribute of file f, R f The file attribute vector of f.

[0057] Let the file attribute vector of files a and b be R a and R b , where the i-th attribute is L ai and L bi , after being vectorized by the bag-of-words model, they are respectively one-dimensional integer arrays [a i1 , a i2 ,…,a ik ] and [b i1 , b i2 ,…,b ik ], the length of the array is k, and the access similarity between file a and file b at the i-th attribute is recorded as T i ,have

[0058]

[0059] Assume that the similarity of each common attribute of file a and file b constitutes a one-dimensional vector [T1(L a1 , L b1), T2(L a2 , L b2 ),…,T n (L an , L bn )], then the file correlation between a and b is recorded as S a,b ,have

[0060] This embodiment uses the cosine distance to calculate the file correlation, which can make the file correlation range between [-1, 1]. a,b The closer the value is to 1, the closer the correlation between files a and b is. a,b The closer the value is to -1, the more opposite the correlation between the two files is. Set the correlation threshold range of related files to 0.5 to 1, and classify files with correlation within the threshold range as related files.

[0061] Step S3: using rule matching technology to classify and identify the data in the file scanned in step S1;

[0062] This step first uses NLP (Natural Language Processing) technology to segment, sentence, and word the file scanned in step S1, and then uses keyword, dictionary, and regular expression matching technology to match the rules of each word. If a word can match one or more rules, the word is regarded as a key data, and the classification corresponding to the one or more matching rules is used as the classification of this key data, thereby realizing the recognition and classification of key data in the file.

[0063] Step S4: Calculate the total amount of each classification data in the relevant files based on the relevant files divided in step S2 and the classification data identified in step S3, and perform data classification;

[0064] This step is a key step in data classification and grading. First, calculate the total number of key data of each category in the relevant files, and use this as the total amount of key data. Then, determine the impact range corresponding to the data volume threshold interval in which the data volume is located as the impact range of the data. For example, the impact range corresponding to the data volume threshold interval of 0 to 1000 is small, the impact range corresponding to the data volume threshold interval of 1000 to 10000 is medium, and the impact range corresponding to the data volume threshold interval of 10000 or above is large. After determining the impact range of the data, the security level of the classified data in all relevant files is divided into levels based on the importance and sensitivity of the classification to which the data belongs, thereby achieving the grading of the data in the file.

[0065] Step S5: Classify and grade the data in the file scanned in step S1 according to the classification data identified in step S3 and the data security level divided in step S4, and output a classification and grading list;

[0066] This step indexes and identifies the data in the files scanned in step S1 according to the classification and grading results, marking the category and level of each key data in the file, and then marking the data type and security level of each file. The data type of a single file is the summary of the categories to which each key data in the file belongs, and the security level of a single file is the highest security level among the key data levels in the file. After the data classification and grading results are marked, a classification and grading list can be output. The list content includes information such as file name, file type, department to which it belongs, system where it is located, data type, security level, data volume of key data, key data classification information, key data classification information, storage directory, retention period, data processing status, etc.

[0067] Another embodiment of the present invention is an application system of an automatic classification and grading method for unstructured data, such as Figure 2 Shown, including:

[0068] The unstructured data scanning module is used to periodically scan all unstructured data resources according to the system resource path or server information configured by the user, and then logically aggregate the scanned unstructured data files according to file type, storage directory, system to which they belong, and other information to build a data resource list.

[0069] The file correlation analysis module is used to read common attributes such as time, space, source, and format of files in the data resource list. Based on these attributes, it uses cosine distance to calculate the correlation between files. Then, it classifies related files according to a set correlation threshold. This embodiment sets a file correlation threshold interval of 0.5 to 1, and classifies files with correlations within this threshold interval as related files. Common attribute fields for files in the data resource list are shown in Table 1.

[0070] Table 1 Common attribute fields of files

[0071] Field meaning type FileName File name String(100) CreateTime Creation time String(20) LastUpdateTime Last modified time String(20) ServerInfo File server information String(100) FilePath File storage path String(255) RelatedSystem The system to which the file belongs String(100) Type File Type String(20)

[0072] The classification and grading standard establishment module is used to read and parse the data classification and grading standard files of various industries, extract the classification and grading rules in the standard files, establish a classification directory tree, and determine the data security level of each category in the classification directory tree based on the standard files.

[0073] The data classification management module is used to read the contents of the files in the data resource list, divide the file contents into segments, sentences and words, and use the rule matching technology of keyword, dictionary and regular expression matching to match the rules in the data classification and grading standards with words, and correspond the matched words to the leaf node classification of the classification directory tree.

[0074] The data classification management module is used to calculate the total amount of each classified data in the relevant files divided by the file correlation analysis module, determine the impact range of the data, and determine the security level of the data in all relevant files based on the importance and sensitivity of the data.

[0075] The data classification and grading identification module is used to classify and grade the data in the file and output a classification and grading list of unstructured data.

Claims

1. A method for automatic classification and grading of unstructured data, characterized in that: The following steps are involved: S1 scans unstructured data files and performs logical aggregation to form a data resource list; S2 sets the file relevance threshold and divides the related files; S3 uses rule matching technology to classify and identify data within files; S4 calculates the total amount of each classified data in the relevant files. Based on the relevant files divided in step S2 and the classified data identified in step S3, the total amount of each classified data in the relevant files is calculated, and the data is classified. The impact range of the data is determined, and the impact range corresponding to the data volume threshold interval in which the data volume falls is determined as the impact range of the data. The classified data in all relevant files are classified into security levels, and the security levels of the classified data in all relevant files are classified according to the importance and sensitivity of the classification to which the data belongs. S5 marks the data in the file and outputs a classification and grading list of unstructured data. After the data classification and grading results are marked, the classification and grading list can be output. The list content includes file name, file type, department, system, data type, security level, data volume of key data, key data classification information, key data grading information, storage directory, retention period, and data processing status information.

2. The method for automatic classification and grading of unstructured data according to claim 1, characterized in that: The step S2 specifically includes reading common attributes of the files in the data resource list, including time attributes, space attributes, source attributes, and format attributes, calculating file relevance based on the file attributes, and then dividing related files according to a set relevance threshold.

3. The method for automatic classification and grading of unstructured data according to claim 2, characterized in that: The file relevance is described by vectorizing common file attributes using a bag-of-words model and calculating the relevance between files using cosine distance. The file relevance calculation steps include: Suppose the n attributes of file f constitute the string array R f =[L1, L2, ..., L n ], let L n is the nth attribute of file f, R f is the file attribute vector of f; Let the file attribute vector of files a and b be R a and R b , where the i-th attribute is L ai and L bi , after being vectorized by the bag-of-words model, they are respectively one-dimensional integer arrays [a i1 , a i2 ,…,a ik ] and [b i1 , b i2 ,…,b ik ], the length of the array is k, and the access similarity between file a and file b at the i-th attribute is recorded as T i ,have Assume that the similarity of each common attribute of file a and file b constitutes a one-dimensional vector; [T1(L a1 ,L b1 ), T2(L a2 ,L b2 ),…,T n (L an ,L bn )] Then the file correlation between a and b is recorded as S a，b ,have S a，b The value range is between [-1, 1]. The correlation threshold interval of related files is set to: 0.5 to 1. Files with correlation within the threshold interval are classified as related files.

4. The method for automatic classification and grading of unstructured data according to claim 1, characterized in that: The step S3 classifies the data in the file using rule matching technology of keyword, dictionary and regular expression matching, specifically including: using NLP technology to segment, sentence and word the unstructured data file; Use keywords, dictionaries and regular expression matching technology to match the rules of each word; A word that can match one or more rules is used as a key data, and a classification corresponding to the one or more rules that match the word is used as the classification of the key data.

5. The method for automatic classification and grading of unstructured data according to claim 1, characterized in that: The step S4 calculates the total amount of data of each category in the relevant files and determines the impact range of the data, specifically including: Calculate the total number of key data of each category in the relevant documents and use this as the total amount of key data; The impact range corresponding to the data volume threshold interval in which the data volume lies is determined as the impact range of the data. After determining the impact range of the data, the classified data in all relevant files are divided into security levels based on the importance and sensitivity of the classification to which the data belongs.

6. The method for automatic classification and grading of unstructured data according to claim 1, characterized in that: The step S4 indexes and identifies the data in the file according to the classification and grading results, and outputs a classification and grading list of unstructured data, specifically including: Mark the category and level of each key data in the file, and then mark the data type and security level of each file; The data type of a single file is the summary result of the categories to which each key data in the file belongs, and the security level of a single file is the highest security level among the levels of each key data in the file.

7. The method for automatic classification and grading of unstructured data according to claim 1 or 6, characterized in that: The contents of the classification and grading list include: file name, file type, department to which it belongs, system where it is located, data type, security level, amount of key data, key data classification information, key data grading information, storage directory, retention period, and data processing status.

8. The method for automatic classification and grading of unstructured data according to claim 1, characterized in that: Said step s5 classifies and grades the data in the file scanned in step S1 according to the classification data identified in step S3 and the data security level divided in step S4, and outputs a classification and grading list.

9. An application system for an automatic classification and grading method for unstructured data, characterized in that: The system is applied to the method for automatic classification and grading of unstructured data according to claims 1 to 8, comprising: Unstructured data scanning module, used to scan unstructured data files and perform logical aggregation to form a data resource list; A file correlation analysis module, used to set file correlation thresholds and classify related files; Classification and grading standard establishment module, used to determine the data security level of each category in the classification directory tree; Data classification management module, used to classify data based on rule matching technology; Data classification management module, used to determine the security level of data in all relevant files; The data classification and grading identification module is used to classify and grade the data in the file and output a classification and grading list of unstructured data.

Images