Dynamic structured data classification method, device and system and storage medium

Abstract

The invention discloses a dynamic structured data classification method, device and system and a storage medium, and the method comprises the steps: S1, obtaining structured field attributes according to structured data; s2, forming a data classification subdivision rule; s3, the structured field attributes and the data classification strategy detailed rules are matched, and structured data fields which are successfully matched and structured data fields which are not successfully matched are obtained; meanwhile, data classification marking is conducted on the successfully matched structured data fields; s4, training a decision tree model by using the successfully matched structured data fields; and S5, according to the trained decision tree model, carrying out classification marking on unsuccessfully matched structured data fields. By adopting the technical scheme of the invention, the problems of inaccurate structured data classification result and incapability of automatic classification are solved.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a dynamic structured data classification method and device, system, and storage medium. Background Art

[0002] Data classification and grading is the basis of data management and protection. It aims to classify data into different levels according to its sensitivity and importance so that appropriate protection measures can be taken. Accurately classifying and grading data provides a basis for subsequent data protection, access control, compliance management, etc.

[0003] Existing structured data classification methods are aimed at metadata management, collecting and organizing contextual information about the data, such as data source, ownership, creation date, etc., to assist in classification or identify and classify sensitive data through preset sensitive keywords, regular expressions, etc.

[0004] The structured data classification process may be affected by human factors. Different people may have different judgments on the sensitivity and importance of the same data, resulting in inconsistent classification results; or misclassification due to lack of in-depth understanding of the business context. For example, certain keywords may be mistakenly marked as sensitive information out of context. Data classification can only be improved in accuracy if it is combined with business and business context understanding. This is also the reason why the accuracy of data classification has not been improved so far. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a dynamic structured data classification method and device, system, and storage medium to solve the problem of inaccurate structured data classification results.

[0006] To achieve the above object, the present invention adopts the following technical solution:

[0007] A dynamic structured data classification method, comprising:

[0008] Step S1, obtaining structured field attributes according to structured data;

[0009] Step S2, forming data classification and subdivision rules;

[0010] Step S3, matching the structured field attributes with the data classification strategy details to obtain successfully matched structured data fields and unmatched structured data fields; and at the same time, data classification and labeling are performed on the successfully matched structured data fields;

[0011] Step S4: training a decision tree model with the successfully matched structured data fields;

[0012] Step S5: Classify and label the unmatched structured data fields according to the trained decision tree model.

[0013] Preferably, step S1 comprises:

[0014] Extract metadata information, annotations and business attribute values ​​from structured data as attributes of the field;

[0015] An association relationship is established between the structured field and its attributes to obtain the structured field attributes.

[0016] Preferably, step S2 comprises:

[0017] Automatically label the data classification strategy as a sample to form training data;

[0018] Use DeepKE to train and predict the training data;

[0019] Predict the sentences in the industry's industry standard documents and business knowledge documents to obtain classified keywords and form data classification strategy segmentation rules.

[0020] Preferably, in step S3, the structured field attributes are traversed to find the data classification and subdivision rules with the highest semantic similarity and are labeled to obtain successfully matched structured data fields and unmatched structured data fields.

[0021] The present invention discloses a dynamic structured data classification device, comprising:

[0022] A first processing module is used to obtain structured field attributes according to the structured data;

[0023] The second processing module is used to form data classification and subdivision rules;

[0024] The matching module is used to match the structured field attributes with the data classification strategy details to obtain the successfully matched structured data fields and the unmatched structured data fields; at the same time, the successfully matched structured data fields are subjected to data classification and labeling;

[0025] A training module is used to train a decision tree model with successfully matched structured data fields;

[0026] The classification module is used to classify and label the unmatched structured data fields based on the trained decision tree model.

[0027] Preferably, the matching module traverses the structured field attributes to find the data classification and subdivision rules with the highest semantic similarity and labels them, thereby obtaining successfully matched structured data fields and unmatched structured data fields.

[0028] The present invention also provides a dynamic structured data classification system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes the dynamic structured data classification method when executed by the processor.

[0029] Preferably, the present invention further provides a storage medium, on which a computer program is stored, and the computer program executes the dynamic structured data classification method when running.

[0030] Using deep learning technology, the data classification strategy is refined, and the rough data classification strategy content in the classification standard is related to the industry and business. It is convenient to map and classify the data with the classification strategy later, reducing the error rate of manual classification. At the same time, deep learning technology is convenient for adding and modifying data classification, and can flexibly adjust the classification system as the data set grows and changes, ensuring the timeliness and accuracy of classification. After the data classification strategy is subdivided by the tool, the semantic similarity between the data to be classified and the classification strategy is calculated by the natural language NLP method, and the rule with the highest semantic similarity is the classification label of the data. Semantic similarity analysis can more deeply understand the meaning behind words and sentences, thereby improving the accuracy of text classification tasks. The decision tree method is used to judge the characteristics of structured data through repeated optimization, reducing the risk of overfitting and improving the stability of classification.

[0031] Furthermore, the present invention can automatically adjust the classification strategy according to the changes in data and contextual environments of different industries to ensure the timeliness and adaptability of the classification results, which has significant advantages for processing real-time data streams, social network data, Internet of Things data, etc. It can dynamically adjust the classification strategy according to the real-time changes in the data stream, adapt to new data features, identify and classify newly emerging data types, ensure that the classification results are always close to the actual situation, and effectively cope with the dynamic nature of the data environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0033] Figure 1 The figure is a flow chart of a method for dynamically classifying structured data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Embodiment 1:

[0037] like Figure 1 As shown, an embodiment of the present invention provides a dynamic structured data classification method, including:

[0038] Step S1, obtaining structured field attributes according to structured data;

[0039] Step S2, forming data classification and subdivision rules;

[0040] Step S3, matching the structured field attributes with the data classification strategy details to obtain successfully matched structured data fields and unmatched structured data fields; and at the same time, data classification and labeling are performed on the successfully matched structured data fields;

[0041] Step S4: training a decision tree model with the successfully matched structured data fields;

[0042] Step S5: Classify and label the unmatched structured data fields according to the trained decision tree model.

[0043] As an implementation of the embodiment of the present invention, step S1 includes:

[0044] Extract metadata information, annotations and business attribute values ​​from structured data as attributes of the field;

[0045] An association relationship is established between the structured field and its attributes to obtain the structured field attributes.

[0046] As an implementation of the embodiment of the present invention, step S2 includes:

[0047] Automatically label the data classification strategy as a sample to form training data;

[0048] Use DeepKE to train and predict the training data;

[0049] Predict the sentences in the industry's industry standard documents and business knowledge documents to obtain classified keywords and form data classification strategy segmentation rules.

[0050] Furthermore, in the process of constructing data classification strategy segmentation rules, automatic labeling is achieved through a simple and easy-to-use entity recognition automatic labeling tool based on dictionary matching. The entity dictionary + Jieba part-of-speech tagging is used to automatically label the data classification strategy, and the automatically labeled data is divided into training set, validation set and test set in a ratio of 8:1:1. The entity dictionary example is shown in Table 1, and the automatic text labeling result example is shown in Table 2.

[0051] Table 1

[0052] Transmission voltage level SDP Dispatching Agency SDP transformer PW Transformer windings PW Shunt Reactance Compensator PW Power transmission dispatch plan DDYX Dispatching equipment safety analysis DDYX Load history analysis FHYC Load operation summer vacation FHYC Scheduling requirements DDJ Scheduling plan DDJ Scheduling Execution DDJ

[0053] Table 2

[0054]

[0055] After the data is prepared, the BiLSTM-CRF model is selected for training to predict the sentences in the industry standard documents, business knowledge documents and other materials to obtain classified keywords. The classified keywords are added to the data classification strategy to form data classification strategy segmentation rules, which are expressed in the form of a segmentation rule table.

[0056] As an implementation method of an embodiment of the present invention, in step S3, the structured field attributes are traversed to find the data classification and subdivision rules with the highest semantic similarity and are labeled to obtain successfully matched structured data fields and unmatched structured data fields.

[0057] Furthermore, in step S3, there are two inputs: structured field attributes and segmentation rules of classification strategy, both of which are structured data.

[0058] Step S3 is specifically as follows: extracting the information to be matched from the structured field attribute table and the classification strategy segmentation rules, and converting it into an array form; traversing the data of the structured field attributes, calculating its similarity with each classification strategy segmentation rule in the data; finding the classification strategy segmentation rule with the highest similarity to the data to be classified, and using the segmentation rule as the label of the data to be classified.

[0059] The similarity calculation steps include: using the word vector model to convert words into vectors; calculating the weighted average of the word vectors for each text to obtain the vector representation of the text; and using cosine similarity to calculate the similarity between two text vectors.

[0060] The word vector model maps words to vector representations in a high-dimensional space. These vectors can capture the semantic information of words. Use the Word2Vec class in the gensim library of the Python language to train the model, and then load the word vectors from the trained model.

[0061] In natural language processing (NLP), the skip-gram model is one of the two main architectures of the Word2Vec model. In the skip-gram model, each word is represented as two d-dimensional vectors to calculate the conditional probability. Assume that the word is indexed as i in the dictionary, and when it is the center word, the vector is represented as vi∈R d , and when it is a background word, the vector is represented as u i ∈R d Let the central word w c The index in the dictionary is c, the background word w o The conditional probability of generating a background word given a central word at index o in the dictionary can be obtained by performing a softmax operation on the vector inner product:

[0062]

[0063] Wherein, the dictionary index set V = 0, 1, ..., |V|-1. Assume that a text sequence of length T is given, and the word at time step t is w (t) Assuming that the generation of background words is independent of each other given the central word, when the background window size is m, the likelihood function of the skip-word model is the probability of generating all background words given any central word

[0064]

[0065] Calculate the TF-IDF weight of each word, traverse all the words in each text, multiply the TF-IDF weight of each word by its vector in the Word2Vec model, and accumulate the results to get the vector representation of the text. The TF-IDF importance of word x in sentence y can be expressed as

[0066]

[0067] The cosine value of the angle between two vectors in the vector space is used as a measure of the difference between two individuals. The closer the value is to 1, the closer the angle is to 0°, that is, the more similar the two vectors are.

[0068] As an implementation method of an embodiment of the present invention, in step S4, the successfully matched structured data fields have been standardized or normalized; the successfully matched structured data fields are used as data sets and divided into training sets and test sets in a ratio of 8:2. Before division, the entire data set needs to be randomly shuffled to ensure that both the training set and the test set can represent the distribution of the entire data. According to the determined ratio, the training set and the test set are divided from the shuffled data set. Check whether the divided training set and test set maintain a distribution similar to that of the original data set in terms of key features. If the distribution is biased, it may need to be re-divided; use the data set to build a decision tree model. There are potential features corresponding to the attributes in the data, and the Gini coefficient is used to judge the impact of these attributes on the classification task. The formula is as follows:

[0069]

[0070] Among them, 1 minus the square of all category probabilities, the value of the Gini coefficient ranges from 0 to 1, and the lower the value, the smaller the uncertainty of the model and the better the classification effect. Select the attribute with the smallest Gini coefficient value that has the greatest impact on the classification task as the best feature for building a decision tree. Starting from the root node, select the best feature to split the data. For each split, create child nodes, select the best features for the subsets and apply the same splitting process until all data samples belong to the same category, or there are no more features to be used for splitting; use the test set to evaluate the performance of the model. Evaluate the accuracy and recall of the model, and adjust the model parameters, such as the depth of the tree, splitting criteria, etc. according to the model evaluation results to optimize the model performance. Model training is completed.

[0071] The embodiments of the present invention can free personnel from the complicated and tiring manual classification, realize the automatic classification of structured data, and greatly improve the speed and efficiency of structured data classification; at the same time, the structured data to be classified is classified multiple times, which can greatly improve the accuracy of structured data classification.

[0072] Embodiment 2:

[0073] The embodiment of the present invention discloses a dynamic structured data classification device, comprising:

[0074] A first processing module is used to obtain structured field attributes according to the structured data;

[0075] The second processing module is used to form data classification and subdivision rules;

[0076] The matching module is used to match the structured field attributes with the data classification strategy details to obtain the successfully matched structured data fields and the unmatched structured data fields; at the same time, the successfully matched structured data fields are subjected to data classification and labeling;

[0077] A training module is used to train a decision tree model with successfully matched structured data fields;

[0078] The classification module is used to classify and label the unmatched structured data fields based on the trained decision tree model.

[0079] As an implementation method of an embodiment of the present invention, the matching module traverses the structured field attributes to find the data classification and subdivision rules with the highest semantic similarity and marks them, thereby obtaining successfully matched structured data fields and unmatched structured data fields.

[0080] Embodiment 3:

[0081] An embodiment of the present invention further provides a dynamic structured data classification system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a dynamic structured data classification method when executed by the processor.

[0082] Embodiment 4:

[0083] An embodiment of the present invention further provides a storage medium, on which a computer program is stored, and the computer program executes the dynamic structured data classification method when running.

[0084] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A dynamic structured data classification method, characterized in that: include: Step S1, obtaining structured field attributes according to structured data; Step S2, forming data classification and subdivision rules; Step S3, matching the structured field attributes with the data classification strategy details to obtain successfully matched structured data fields and unmatched structured data fields; and at the same time, performing data classification marking on the successfully matched structured data fields; Step S4: training a decision tree model with the successfully matched structured data fields; Step S5: Classify and label the unmatched structured data fields according to the trained decision tree model.

2. The dynamic structured data classification method according to claim 1, characterized in that: Step S1 includes: Extract metadata information, annotations and business attribute values ​​from structured data as attributes of the field; An association relationship is established between the structured field and its attributes to obtain the structured field attributes.

3. The dynamic structured data classification method according to claim 2, characterized in that: Step S2 includes: Automatically label the data classification strategy as a sample to form training data; Use DeepKE to train and predict the training data; Predict the sentences in the industry's industry standard documents and business knowledge documents to obtain classified keywords and form data classification strategy segmentation rules.

4. The dynamic structured data classification method according to claim 3, characterized in that: In step S3, the structured field attributes are traversed to find the data classification and subdivision rule with the highest semantic similarity and label it, thereby obtaining the successfully matched structured data fields and the unmatched structured data fields.

5. A dynamic structured data classification device, characterized in that: include: A first processing module is used to obtain structured field attributes according to the structured data; The second processing module is used to form data classification and subdivision rules; The matching module is used to match the structured field attributes with the data classification strategy details to obtain the successfully matched structured data fields and the unmatched structured data fields; at the same time, the successfully matched structured data fields are subjected to data classification and labeling; A training module is used to train a decision tree model with successfully matched structured data fields; The classification module is used to classify and label the unmatched structured data fields based on the trained decision tree model.

6. The dynamic structured data classification device according to claim 5, characterized in that: The matching module traverses the structured field attributes to find the data classification and segmentation rules with the highest semantic similarity and labels them, obtaining the successfully matched structured data fields and the unmatched structured data fields.

7. A dynamic structured data classification system, characterized in that: include: A memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the dynamic structured data classification method according to any one of claims 1 to 4 is executed.

8. A storage medium, characterized in that: The storage medium stores a computer program, which executes the dynamic structured data classification method according to any one of claims 1 to 4 when running.

Images