A dynamic structured data classification method and device, system, storage medium

Abstract

The application discloses a dynamic structured data classification method and device, system and storage medium, comprising the following steps: S1, obtaining structured field attributes according to structured data; S2, forming data classification subdivision rules; S3, matching the structured field attributes and data classification policy subdivision rules to obtain matched structured data fields and unmatched structured data fields; meanwhile, the matched structured data fields are marked for data classification; S4, training a decision tree model according to the matched structured data fields; and S5, classifying and marking the unmatched structured data fields according to the trained decision tree model. The technical scheme of the application solves the problems of inaccurate structured data classification results and automatic classification.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a dynamic structured data classification method and device, system and storage medium. BACKGROUND

[0002] Data classification and grading is the basis of data management and protection, aiming to divide data into different levels according to its sensitivity and importance, so as to take corresponding protection measures. Accurate classification and grading of data provides a basis for subsequent data protection, access control, compliance management, etc.

[0003] The existing structured data classification method is for metadata management, which collects and organizes the context information about data, such as data source, ownership, creation date, etc., to assist classification or identify sensitive data and classify by predefining sensitive keywords, regular expressions, etc.

[0004] However, the structured data classification process may be affected by human factors, and different people may have different judgments on the sensitivity and importance of the same data, resulting in inconsistent classification results; or misclassification due to lack of deep understanding of business context. For example, some keywords may be incorrectly marked as sensitive information out of context. Data classification combined with business and understanding of business context can improve the accuracy of data classification. This is the reason why the accuracy of data classification has always been unable to improve. SUMMARY

[0005] The technical problem to be solved by the application is to provide a dynamic structured data classification method and device, system and storage medium, which solves the problem of inaccurate structured data classification results.

[0006] To achieve the above purpose, the application adopts the following technical solution:

[0007] A dynamic structured data classification method, comprising:

[0008] Step S1, obtaining structured field attributes according to structured data;

[0009] Step S2, forming data classification subdivision rules;

[0010] Step S3, matching the structured field attributes and data classification policy details to obtain matched structured data fields and unmatched structured data fields; and marking the matched structured data fields for data classification;

[0011] Step S4, training a decision tree model for the matched structured data fields;

[0012] Step S5, according to the trained decision tree model, the un-matched structured data field is classified and labeled.

[0013] As preferred, step S1 comprises:

[0014] The metadata information, annotations and business attribute values are extracted from the structured data as the attributes of the field.

[0015] The structured field is associated with its attributes to obtain the structured field attributes.

[0016] As preferred, step S2 comprises:

[0017] The data classification strategy is automatically labeled as a sample to form training data.

[0018] The training data is trained and predicted by DeepKE.

[0019] The sentences in the industry standard files and business knowledge files of the industry are predicted to obtain classified keywords to form data classification strategy subdivision rules.

[0020] As preferred, in step S3, the structured field attributes are traversed to find the data classification subdivision rule with the highest semantic similarity and label it to obtain the matched structured data field and the un-matched structured data field.

[0021] The application discloses a dynamic structured data classification device, comprising:

[0022] The first processing module is used for obtaining the structured field attributes according to the structured data.

[0023] The second processing module is used for forming data classification subdivision rules.

[0024] The matching module is used for matching the structured field attributes and the data classification strategy rules to obtain the matched structured data field and the un-matched structured data field, and simultaneously labeling the matched structured data field.

[0025] The training module is used for training the decision tree model of the matched structured data field.

[0026] The classification module is used for classifying and labeling the un-matched structured data field according to the trained decision tree model.

[0027] As preferred, the matching module traverses the structured field attributes to find the data classification subdivision rule with the highest semantic similarity and label it to obtain the matched structured data field and the un-matched structured data field.

[0028] The application further provides a dynamic structured data classification system, comprising a memory and a processor, wherein the memory stores a computer program which is run by the processor, and the computer program performs the dynamic structured data classification method when run by the processor.

[0029] As a preference, the application further provides a storage medium which stores a computer program, and the computer program performs the dynamic structured data classification method when run.

[0030] By using deep learning technology, the data classification strategy is refined, and the relatively rough data classification strategy content in the classification standard is associated with the industry and business to be specific. This facilitates the subsequent mapping and classification of data and classification strategies, reducing the error rate of manual classification. At the same time, the deep learning technology facilitates the addition and modification of data classification, and can flexibly adjust the classification system as the data set grows and changes, ensuring the timeliness and accuracy of classification. After the data classification strategy is subdivided by the tool, the semantic similarity between the data to be classified and the classification strategy is calculated by the natural language NLP method, and the rule with the highest semantic similarity is obtained as the classification label of the data. Semantic similarity analysis can more deeply understand the meaning behind words and sentences, thereby improving the accuracy of the text classification task. By using the decision tree method, the characteristics of structured data are judged through multiple iterations, reducing the risk of overfitting and improving the stability of classification.

[0031] Further, the application can automatically adjust the classification strategy according to the changes of different industry data and the context environment, ensuring the timeliness and adaptability of the classification result, which has a significant advantage in processing real-time data flow, social network data, and Internet of Things data. It can dynamically adjust the classification strategy according to the real-time changes of the data flow, adapt to new data features, identify and classify new data types, and ensure that the classification result is always close to the actual situation, effectively dealing with the dynamics of the data environment. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, brief introductions will be given to the drawings needed to be used in the embodiments or prior art descriptions. Obviously, the drawings in the following description are only embodiments of the application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0033] Figure 1 Flowchart of the dynamic structured data classification method of the embodiments of the application. DETAILED DESCRIPTION

[0034] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0035] In order to make the above objectives, characteristics and advantages of the present application more apparent, the present application will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0036] Embodiment 1

[0037] As shown in the accompanying drawings, the embodiments of the present application provide a dynamic structured data classification method, which comprises: Figure 1

[0038] Step S1, obtaining structured field attributes according to structured data;

[0039] Step S2, forming data classification subdivision rules;

[0040] Step S3, matching the structured field attributes and data classification policy rules to obtain matched structured data fields and unmatched structured data fields; and meanwhile, performing data classification labeling on the matched structured data fields;

[0041] Step S4, training a decision tree model for the matched structured data fields;

[0042] Step S5, classifying and labeling the unmatched structured data fields according to the trained decision tree model.

[0043] As an implementation manner of the embodiments of the present application, step S1 comprises:

[0044] extracting metadata information, annotations and business attribute values from the structured data as the attributes of the field;

[0045] establishing an association between the structured field and its attributes to obtain the structured field attributes.

[0046] As an implementation manner of the embodiments of the present application, step S2 comprises:

[0047] automatically labeling the data classification policy as a sample to form training data;

[0048] training and predicting the training data through DeepKE;

[0049] ​The sentences in the industry standard documents and business knowledge documents of the industry are predicted to obtain classified keywords, and data classification strategy subdivision rules are formed.

[0050] Further, the data classification strategy subdivision rule construction process is automatically labeled by a simple and easy-to-use entity recognition automatic labeling tool based on dictionary matching, and the data classification strategy is automatically labeled using an entity dictionary + jieba part-of-speech tagging, and the automatically labeled data is divided into a training set, a validation set and a test set in a ratio of 8:1:1, wherein the entity dictionary is shown in Table 1, and the automatic text labeling result is shown in Table 2.

[0051] Table 1

[0052] Transmission voltage class SDPJ Scheduling mechanism SDPJ Transformer PWYW Transformer winding PWYW Shunt reactive compensator PWYW Transmission dispatch plan DDYX Scheduling device safety analysis DDYX Load history analysis FHYC Load operation summer vacation FHYC Scheduling demand DDJH Scheduling plan DDJH Scheduling execution DDJH

[0053] Table 2

[0054]

[0055] After the data is prepared, a BiLSTM-CRF model is selected for training to predict the sentences in the industry standard documents, business knowledge documents and other materials, obtain classified keywords, supplement the classified keywords to the data classification strategy, form data classification strategy subdivision rules, and the manifestation is a subdivision rule table.

[0056] As an embodiment of the present application, in step S3, the structured field attribute is traversed to find the data classification subdivision rule with the highest semantic similarity and label it, and the matched structured data field and the unmatched structured data field are obtained.

[0057] Further, in step S3, there are two inputs, which are the structured field attribute and the subdivision rule of the classification strategy, both of which are structured data.

[0058] Step S3 is specifically: extracting the information to be matched from the structured field attribute table and the subdivision rule of the classification strategy, and converting it into an array form; traversing the data of the structured field attribute, calculating the similarity of each classification strategy subdivision rule in the data; finding the classification strategy subdivision rule with the highest similarity to the data to be classified, and taking the subdivision rule as the label of the data to be classified.

[0059] The similarity calculation step includes: converting words into vectors using a word vector model; calculating the weighted average of the word vectors of each text to obtain the vector representation of the text; and calculating the similarity between two text vectors using cosine similarity.

[0060] Word vector model maps words to vector representations in a high-dimensional space that can capture semantic information of words. The Word2Vec class in gensim library of Python language is used to train the model, and the word vector can be loaded from the trained model.

[0061] In natural language processing (NLP), skip-gram model is one of the two main architectures of Word2Vec model. In skip-gram model, each word is represented by two d-dimensional vectors to calculate conditional probability. Assuming that the word is indexed as i in the dictionary, the vector representation is vi∈R d when it is the center word, and the vector representation is u i ∈R d when it is the context word. Let the center word w c be indexed as c in the dictionary, and the context word w o be indexed as o in the dictionary, the conditional probability of generating the context word given the center word can be obtained by doing softmax operation on the inner product of vectors:

[0062]

[0063] where the set of dictionary indices V = 0, 1,..., |V|-1. Assuming that a text sequence of length T is given, let the word at time step t be w (t) . Assuming that the generation of context words is independent given the center word, the likelihood function of skip-gram model, i.e. the probability of generating all context words given any center word, is

[0064]

[0065] The TF-IDF weight of each word is calculated, and for each text, the TF-IDF weight of each word is multiplied by its vector in the Word2Vec model, and the result is accumulated to obtain the vector representation of the text. The TF-IDF importance of word x in sentence y can be represented as

[0066]

[0067] The cosine value of the angle between two vectors in vector space is used as a measure of the difference between two individuals, and the value is closer to 1, which means the angle is closer to 0°, that is, the two vectors are more similar.

[0068] As an embodiment of the present application, in step S4, the matched structured data field is standardized or normalized; the matched structured data field is divided into a training set and a test set according to a ratio of 8:2 as a data set. Before the division, the whole data set needs to be randomly shuffled to ensure that the training set and the test set can represent the distribution of the whole data. According to the determined ratio, the training set and the test set are divided from the shuffled data set. It is checked whether the divided training set and test set maintain similar distribution on key features as the original data set. If the distribution is biased, the division may need to be re-divided; the decision tree model is constructed using the data set. There are potential features in the corresponding attributes in the data, and the Gini coefficient is used to judge the influence of these attributes on the classification task, and the formula is as follows:

[0069]

[0070] Wherein, 1 minus the square of all category probability, the value of Gini coefficient is between 0 and 1, the lower the value, the smaller the uncertainty of the model, and the better the classification effect. The attribute with the smallest Gini coefficient value and the greatest influence on the classification task is selected as the best feature for constructing the decision tree. Starting from the root node, the best feature is selected to divide the data. For each division, a child node is created, and the best feature is selected for the subset and the same division process is applied, until all data samples belong to the same category, or there are no more features to be used for division; the performance of the model is evaluated using the test set. The accuracy and recall of the model are evaluated, and the model parameters such as the depth of the tree and the splitting standard are adjusted according to the model evaluation results to optimize the performance of the model. The model training is completed.

[0071] The embodiment of the present application can release personnel from tedious and tiring manual classification, realize automatic classification of structured data, and greatly improve the speed and efficiency of structured data classification; at the same time, the structured data to be classified is classified multiple times, which can greatly improve the accuracy of structured data classification.

[0072] Embodiment 2:

[0073] The embodiment of the present application discloses a dynamic structured data classification device, comprising:

[0074] The first processing module is configured to obtain the structured field attribute according to the structured data;

[0075] The second processing module is configured to form a data classification subdivision rule;

[0076] The matching module is configured to match the structured field attribute and the data classification policy rule to obtain a matched structured data field and an unmatched structured data field; and to perform data classification tagging on the matched structured data field;

[0077] a training module configured to train a decision tree model with the structured data fields that are successfully matched;

[0078] a classification module configured to classify the structured data fields that are not successfully matched according to the trained decision tree model.

[0079] As an embodiment of the present application, the matching module traverses the structured field attributes to find the data classification subdivision rule with the highest semantic similarity and labels it, obtaining the structured data fields that are successfully matched and the structured data fields that are not successfully matched.

[0080] Embodiment 3:

[0081] The present application also provides a dynamic structured data classification system, comprising a memory and a processor, wherein the memory stores a computer program that is run by the processor, and the computer program performs the dynamic structured data classification method when run by the processor.

[0082] Embodiment 4:

[0083] The present application also provides a storage medium that stores a computer program, and the computer program performs the dynamic structured data classification method when run.

[0084] The above embodiments only describe the preferred modes of the present application, and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope defined by the claims of the present application.

Claims

1. A method of dynamically structuring data classification, characterized by, The method comprises the following steps: Step S1, obtaining structured field attributes according to structured data; Step S2, forming data classification subdivision rules; Step S3, matching the structured field attributes and the data classification policy rules to obtain matched structured data fields and unmatched structured data fields; and meanwhile, performing data classification tagging on the matched structured data fields; Step S4, training a decision tree model by using the matched structured data fields; Step S5, classifying and tagging the unmatched structured data fields according to the trained decision tree model; Step S1 comprises the following steps: extracting metadata information, annotations and business attribute values from the structured data as the attributes of the field; establishing an association between the structured field and the attributes to obtain the structured field attributes; Step S2 comprises the following steps: automatically annotating the data classification policy as a sample to form training data; training and predicting the training data by using DeepKE; predicting the sentences in the industry standard files and the business knowledge files of the industry to obtain classified keywords, supplementing the classified keywords into the data classification policy, and forming data classification policy subdivision rules; In step S3, the information to be matched is extracted from the structured field attribute table and the subdivision rules of the classification policy, and is converted into an array form; the structured field attributes are traversed to find the data classification subdivision rule with the highest semantic similarity, the subdivision rule is taken as the label of the data to be classified and is tagged, and the matched structured data fields and the unmatched structured data fields are obtained.

2. A dynamically structured data classification apparatus, characterized by, The method comprises the following steps: a first processing module configured to obtain structured field attributes according to structured data; a second processing module configured to form data classification subdivision rules; a matching module configured to match the structured field attributes and the data classification policy rules to obtain matched structured data fields and unmatched structured data fields; and meanwhile, perform data classification tagging on the matched structured data fields; a training module configured to train a decision tree model by using the matched structured data fields; a classification module configured to classify and tag the unmatched structured data fields according to the trained decision tree model; The first processing module extracts metadata information, annotations and business attribute values from the structured data as the attributes of the field; and establishes an association between the structured field and the attributes to obtain the structured field attributes. The second processing module automatically annotates the data classification policy as a sample to form training data; trains and predicts the training data by using DeepKE; predicts the sentences in the industry standard files and the business knowledge files of the industry to obtain classified keywords, supplements the classified keywords into the data classification policy, and forms data classification policy subdivision rules. The matching module extracts information to be matched from the structured field attribute table and the classification strategy subdivision rule, and converts it into an array form; the structured field attribute is traversed to find the data classification subdivision rule with the highest semantic similarity, the subdivision rule is taken as the label of the data to be classified and is labeled, and the matched structured data field and the unmatched structured data field are obtained.

3. A dynamically structured data classification system, characterized by Comprise: A memory and a processor, the memory has a computer program run by the processor stored thereon, the computer program executes the dynamic structured data classification method as claimed in claim 1 when run by the processor.

4. A storage medium, characterized by The storage medium has a computer program stored thereon, the computer program executes the dynamic structured data classification method as claimed in claim 1 when run.

Images