Data classification method and device based on machine learning, equipment and storage medium

Abstract

The invention discloses a data classification method and device based on machine learning, equipment and a storage medium, and relates to the field of machine learning. Constructing and training a classification model by taking the extracted bag-of-words vector as model input and the classification identifier as target output, and constructing a structured knowledge graph by taking the keywords extracted from the model, the classification and the weight coefficient as knowledge entries; to-be-classified field information is obtained, keyword matching is carried out based on the knowledge graph, graph keywords contained in the to-be-classified field information are extracted, and a graph keyword set is constructed; and extracting graph keywords, matching candidate classifications by matching knowledge entries, and screening target classifications based on weight coefficient scores. According to the scheme, explicit expression and dynamic fusion of machine learning are achieved by constructing the knowledge graph, the contradiction that classification decision cannot be explained and knowledge is difficult to update is effectively solved, the defects that a rule method is rigid and a machine learning method is black in a traditional technology are overcome, and the adaptability and reliability of a system in a dynamic service environment are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine learning, and in particular to a data classification method and device based on machine learning, equipment and storage medium. BACKGROUND

[0002] In the current data security governance practice, the automatic identification and classification of database sensitive fields mainly rely on two technical paths. However, both of these two paths have inherent defects, and it is difficult to achieve a good balance between efficiency, accuracy and interpretability. The traditional technical solutions are as follows: 1. Method based on static rules and keyword matching This method is a traditional solution widely used at present. Its core principle is: a set of sensitive keyword rules library (for example, containing words such as "name", "ID card", "account number", etc.) is defined by domain experts in advance, and then through string matching or regular expression, the field name and annotation of the database are scanned and compared with the rule library to determine whether the field is sensitive and its classification.

[0003] This technical solution has the problems of rigidity and lag, poor generalization ability, and difficulty in sharing of knowledge.

[0004] 2. Method based on pure machine learning (especially deep learning) In order to overcome the limitations of rule-based methods, in recent years, data classification solutions based on machine learning have appeared. This method regards field classification as a text classification problem, usually using Bag-of-Words, Word2Vec or deep neural networks (such as BERT) to extract features of field names and annotations, and then using classification models (such as SVM, random forest or neural network) to make classification decisions.

[0005] This solution has the problems of "black box" decision-making, poor interpretability, cold start problem, data dependence, difficulty in knowledge updating and drift.

[0006] The fundamental defect of the above two technical paths is the separation of human expert knowledge and automated systems: rule-based methods have interpretability but lose adaptability, and machine learning methods have generalization ability but sacrifice transparency. Both of them fail to build a mechanism for dynamic fusion and closed-loop evolution of knowledge, resulting in the system being unable to optimize itself in the evolution of business. Especially in the context of sensitive data classification, this separation makes the technical solution neither meet the requirements of decision traceability for compliance audit, nor cope with the rapidly changing business context, seriously restricting the effectiveness of data security governance.

[0007] In view of the above problems, the related technology needs to be improved. SUMMARY

[0008] The application provides a data classification method, device and equipment based on machine learning and a storage medium, realizes the explainability of classification decision, supports the dynamic updating and fusion of knowledge, overcomes the defects of rigid rules and black box of machine learning, and improves the adaptability and reliability of the system in a dynamic business environment.

[0009] In one aspect, the application provides a data classification method based on machine learning, which comprises: A classification model is constructed and trained by taking the extracted bag-of-words vector as the model input and the classification identifier as the target output, and a structured knowledge graph is constructed by taking the keywords, the belonging classification and the weight coefficient extracted from the model as knowledge items; The field information to be classified is obtained, the keywords in the knowledge graph are matched based on the knowledge graph, the graph keywords contained therein are extracted, and a graph keyword set is constructed; The graph keywords are extracted, the candidate classification is filtered by matching the knowledge items, and the target classification is filtered based on the weight coefficient score.

[0010] Specifically, the step of obtaining the bag-of-words vector comprises: A training sample set of artificial classification identifiers is obtained, the field name and the field annotation are processed by a word segmenter to be segmented into independent words, and the words are split; A bag-of-words model is constructed based on all the words, and the unstructured text data is converted into the structured bag-of-words vector , which is represented as:

[0011] wherein n is the size of the dictionary, represents the frequency or TF-IDF value of the i-th word in the field text.

[0012] Specifically, the process of generating knowledge items and constructing a structured knowledge graph comprises: The classification model is constructed based on a random forest classifier, the bag-of-words vector is taken as the input, and the artificial classification identifier is taken as the output target, and the model is trained; The importance score of each input word is obtained by calculating the feature importance score of the input word through the random forest algorithm; the importance score is the Gini impurity reduction or average accuracy drop value, which is used to describe the contribution degree to the model; The top key words with the highest contribution degree under each classification result are extracted , and the weight coefficient of each keyword under each classification is calculated.The keyword-subject classification-weight coefficient mapping relationship set is determined as a knowledge item and stored in a structured knowledge base to form an initial knowledge graph.

[0013] Specifically, the constructing the graph keyword set comprises: receiving field information to be classified , generating a corresponding regularization expression based on the knowledge item, and performing rapid scanning and matching on the field text to extract the graph keywords that are successfully matched; all the graph keywords form the graph keyword set ; wherein the field that does not match the keyword is determined as non-sensitive data and is directly filtered out.

[0014] Specifically, the filtering out the candidate classification by matching the knowledge item and screening the target classification based on the weight coefficient score comprises: For each of the candidate classifications , polling the graph keywords in the graph keyword set, calculating the total weight score of the graph keyword set under the candidate classification ; and the total weight score of the graph keyword set under the candidate classification ; is represented as follows:

[0015] wherein, is the graph keyword under the candidate classification , the weight coefficient value, represents the number of set keywords; comparing the total weight scores under all candidate classifications, determining the candidate classification corresponding to the highest classification score as the target classification of the field to be classified.

[0016] Specifically, the system confidence is set during initialization. After calculating the target classification, the method further comprises: calculating the target confidence score of the target classification result ; the target confidence score is the highest classification score , or the difference between the highest classification score and the second highest score . comparing the target confidence score with the system confidence, when , outputting the final classification result; when , pushing to artificial adjudication; the adjudication and correction results are re-integrated into the training sample set; when the number of artificial adjudication reaches a set threshold, triggering re-tokenization and updating the bag-of-words vector.

[0017] Specifically, the knowledge graph is provided with an intervention interface, when updating or adding domain knowledge, new keywords are supplemented in the knowledge graph, synonyms, near synonyms or business jargons are added to the existing keywords, and the default value is set for the newly added words initial weight is given ; the initial weight Based on the semantic similarity calculation of the existing keywords, or set the default value.

[0018] In another aspect, the present application provides a data classification device based on machine learning, the device comprises: The graph construction module is used for constructing and training a classification model with the extracted bag-of-words vector as the model input and the classification identification as the target output, and constructing a structured knowledge graph with the keywords, the belonging classification and the weight coefficient extracted from the model as the knowledge items. The set construction module is used for obtaining field information to be classified, matching keywords based on the knowledge graph, extracting the graph keywords contained therein, and constructing a graph keyword set. The classification output module is used for extracting the graph keywords, filtering out candidate classifications by matching the knowledge items, and filtering out target classifications based on the weight coefficient score.

[0019] In another aspect, the present application provides a computer device, which comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by the processor to realize the machine learning-based data classification method of the above-mentioned aspect.

[0020] In another aspect, the present application provides a computer readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by the processor to realize the machine learning-based data classification method of the above-mentioned aspect.

[0021] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects: by constructing a structured knowledge graph, explicit expression and dynamic fusion of machine learning knowledge are realized, effectively solving the technical contradiction of uninterpretable classification decision and difficult knowledge update, by converting the knowledge extracted by the machine learning model into a structured knowledge graph, the interpretability of the classification decision is realized, while supporting dynamic update and fusion of knowledge, effectively overcoming the defects of rigid rules and black box of machine learning in traditional technology, and improving the adaptability and reliability of the system in dynamic business environment. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 This is a flowchart of a machine learning-based data classification method provided in an embodiment of this application; Figure 2 A schematic diagram of a two-stage process provided in an embodiment of this application is shown; Figure 3 This is an algorithm flowchart of a machine learning-based data classification method provided in an exemplary embodiment; Figure 4 A structural block diagram of a machine learning-based data classification device provided in this application embodiment; Figure 5 A structural block diagram of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0024] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0025] 1. A method based on static rules and keyword matching Rigidity and lag: The rule base is static and cannot adapt to new field names, industry jargon or abbreviations that emerge as the business develops. It requires continuous manual maintenance and updates, which is costly.

[0026] Poor generalization ability: It relies heavily on the standardization of field names and is basically ineffective for deliberately disguised, misspelled, or meaningless field names (such as col_01, temp).

[0027] Knowledge becomes rigid and difficult to share: Rule bases are essentially the solidification of expert experience, but the experience of different experts is difficult to integrate and iterate systematically, which makes it impossible for the knowledge base to evolve and the quality of rule bases varies between different systems.

[0028] 2. Methods based on pure machine learning (especially deep learning) "Black box" decision-making and poor explainability: In particular, deep learning models have opaque decision-making processes, making it difficult to explain to business personnel, auditors, or regulatory agencies "why a field is deemed sensitive," which is a fatal flaw in financial scenarios with high compliance requirements.

[0029] Cold start problem and data dependency: Model performance heavily relies on a large amount of high-quality labeled data. In the early stages of a project or when dealing with a specific domain lacking labeled data, the model performs poorly and cannot be quickly deployed for practical use.

[0030] Knowledge updates are difficult: Once a model is trained, its knowledge becomes fixed. When new business concepts or regulatory requirements emerge, it is necessary to collect samples again and retrain the model, which is a cumbersome process with slow response, making it difficult to achieve real-time knowledge injection and continuous learning.

[0031] Inadequate handling of concept drift: For new and sensitive field types that emerge rapidly in the business, pure machine learning models cannot perform logical reasoning and learn quickly like human experts, which may lead to continuous misjudgments.

[0032] Traditional methods based on static rules and keyword matching, due to the static nature of the rule base, cannot dynamically adapt to new field names, industry terminology abbreviations, or non-standard naming conventions emerging in business scenarios, thus limiting the system's ability to parse field semantics. Furthermore, these methods heavily rely on the standardization of field names, failing to effectively identify fields without clear semantic identifiers (such as automatically generated column names). While pure machine learning methods possess some generalization ability, their decision-making process lacks transparency, making it difficult to trace the classification criteria. Moreover, model knowledge is fixed during the training phase, unable to be injected with new knowledge in real time when business requirements change, resulting in insufficient performance during the cold start phase and a lack of ability to handle concept drift. These problems collectively lead to the continuous accumulation of misjudgment risks in classification systems within dynamic business environments, impacting the reliability of data governance frameworks and the feasibility of compliance audits.

[0033] If the above issues are not addressed, the classification system will be unable to effectively cope with the rapid iteration of business terminology, leading to a continuous deterioration in the accuracy of sensitive field identification over time. In a strictly regulated financial environment, the unexplainable nature of classification decisions will hinder the audit verification process and may cause compliance verification failures. At the same time, the lack of a knowledge update mechanism will reduce the system's adaptability to new business concepts, causing a disconnect between data governance strategies and actual business needs, ultimately affecting the effectiveness of the overall data security protection system.

[0034] To address this, this application provides a data classification method based on machine learning, such as... Figure 1 As shown, the method includes the following steps: S1. Using the extracted bag-of-words vectors as model input and the classification label as target output, construct and train a classification model, and use the keywords, classifications, and weight coefficients extracted from the model as knowledge entries to construct a structured knowledge graph; This technical solution mainly consists of two major phases. Figure 2The diagram illustrates a two-stage process provided in an embodiment of this application: the knowledge graph construction and training stage, and the automatic classification and evolution stage. This step is for constructing the knowledge graph.

[0035] Bag-of-words vectors (BODs) are a representation of unstructured text data converted into structured numerical vectors. In practical applications, BODs can be extracted using statistical methods based on word frequency, such as calculating the absolute number of times each word appears in the text field, or using binarized indicators to show whether a word appears. The main purpose is to quantify text features into input data that the model can process.

[0036] Furthermore, the construction and training of the classification model can employ machine learning algorithms such as support vector machines or Naive Bayes, using the extracted bag-of-words vectors as input and manually labeled classification labels as the target output, to establish the mapping relationship between text features and classification categories.

[0037] Weight coefficients can be determined based on the importance assessment of features during model training, such as by calculating information gain or chi-square statistics, which are mainly used to quantify the contribution of keywords to the classification results.

[0038] A knowledge entry is a structured data unit that stores keywords, its category, and weight coefficients. This entry acts as a data index and is stored in a knowledge graph. Therefore, a structured knowledge graph can be stored using a graph database or a relational database, where nodes represent keywords or categories, and edges represent the relationships between them, used to organize and manage knowledge entries.

[0039] S2. Obtain the information of the fields to be classified, perform keyword matching based on the knowledge graph, extract the graph keywords contained therein, and construct a graph keyword set; This step is primarily the second stage, namely the automatic classification and evolution based on the input fields during the actual runtime. After obtaining the information of the fields to be classified, keyword matching based on the knowledge graph can be performed using string exact matching algorithms or edit distance algorithms. The matched keywords are extracted to form a graph keyword set, mainly to identify key elements in the fields that are relevant to the knowledge graph. All matched keywords (words) are collectively formed into a set, which is then classified and confirmed one by one in subsequent steps.

[0040] S3. Extract keywords from the graph, filter candidate categories by matching knowledge entries, and filter target categories based on weight coefficient scores.

[0041] This stage polls each keyword in the set, filters candidate categories by matching knowledge entries, and selects the target category based on the weight coefficient score. The weight coefficient score is obtained by summing the weight values ​​of all matching keywords under the candidate category. Its main purpose is to determine the most likely classification result based on the contribution of the keywords (for example, sorting the weight scores in descending order and selecting the category option with the highest score as the output).

[0042] In implementing this data classification method, the extracted bag-of-words vectors are first used as model input, and the classification label is used as the target output to construct and train the classification model. Keywords, their corresponding categories, and weight coefficients extracted from the model are used as knowledge entries to construct a structured knowledge graph. After obtaining the information of the field to be classified, keyword matching is performed based on the knowledge graph to extract the graph keywords contained therein, and a graph keyword set is constructed. Subsequently, candidate categories are selected by matching knowledge entries, and the target category is selected based on the weight coefficient scores. The knowledge graph construction process utilizes the efficient computational characteristics of structured text features, avoiding the complexity of deep learning models, while making the decision-making basis transparent and facilitating the tracing of classification reasons. The keyword matching stage combines the real-time efficiency of rule matching with the evolutionary capabilities of the knowledge graph, providing accurate input for subsequent classification.

[0043] Specifically, in one implementation, consider the classification scenario of the database field "Customer Name". The knowledge graph stores the keyword "Name", belonging to the category "Personal Information" with a weight coefficient of 0.85. When the system processes this field, it performs keyword matching based on the knowledge graph, extracts the graph keyword "Name", and constructs a graph keyword set containing this keyword. Further, it filters candidate categories "Personal Information" by matching knowledge entries and determines the target category based on a weight coefficient score of 0.85. As a preferred implementation, this process does not rely on the standardization of field names; even if field names are disguised or abbreviated, the dynamic knowledge of the knowledge graph can adapt to new business terminology.

[0044] In summary, this technical solution effectively addresses the balance between efficiency, accuracy, and interpretability in existing data classification methods. By integrating machine learning with structured knowledge representation, the decision-making process becomes transparent, allowing experts to directly trace the logical connections between keywords and weights. Simultaneously, the structured storage of the knowledge graph provides a platform for dynamic updates, enabling the system to adapt to emerging business jargon and industry terminology, avoiding the rigidity and lag inherent in rule-based methods. Furthermore, the selection mechanism based on weight coefficient scores quantifies the actual contribution of keywords to classification, effectively distinguishing contextual relevance and overcoming the poor generalization ability of rule-based methods and the slow cold-start response of machine learning methods. This achieves dynamic integration of expert knowledge and continuous system evolution.

[0045] Figure 3 This is an exemplary embodiment of the algorithm flowchart for a machine learning-based data classification method. The following describes the knowledge graph construction and training phases: The method for obtaining bag-of-words vectors based on the training set or information to be classified includes the following steps: Obtain the training sample set of manually classified labels, and use a word segmenter to segment the field names and field annotations into independent words; A bag-of-words model is built based on all word segmentation to transform unstructured text data into structured bag-of-words vectors. , represented as:

[0046] Where n is the size of the dictionary. This represents the frequency or TF-IDF value of the i-th word in the field text.

[0047] The training sample set for manual classification and labeling refers to database field sample data pre-annotated by domain experts. This can be achieved using a manual review platform or a semi-automatic annotation tool to ensure the accuracy and representativeness of the samples, thereby providing a high-quality data foundation for feature extraction. For example, metadata containing field names and field annotations can be collected from the target database system, and then manually classified and labeled by data security experts according to a predefined classification system (such as "personal sensitive information," "corporate trade secrets," "non-sensitive information," etc.) to form a high-quality initial training sample set.

[0048] A word segmenter can be understood as a text segmentation component based on rule matching or statistical models. It can be implemented using the maximum matching algorithm, conditional random field model, or pre-trained language model. It is used to intelligently identify semantic units in field names and annotations, adapting to the diversity of abbreviations, slang, or dialects in business scenarios.

[0049] The bag-of-words model can construct a feature space framework through global word segmentation statistics, such as by using a dynamic dictionary based on hash tables or a fixed dictionary structure. The goal is to transform unstructured text into a numerical vector of a uniform dimension.

[0050] Bag-of-words vectors In It is a quantitative indicator representing the importance of words, with each dimension corresponding to a word in the dictionary. It can be implemented using word frequency statistics, TF-IDF weighting, or information gain calculation, with the aim of highlighting the distinguishability of key feature words and suppressing the interference of common words.

[0051] As one specific implementation method, the solution of this application is implemented as follows: In financial data classification scenarios, a training sample set containing labeled fields is obtained, such as "user name" labeled as the "personal identity information" category. A conditional random field-based word segmenter is used to segment the "user name" field name and its annotation "store customer full name information," resulting in independent word segments such as "user," "name," "store," "customer," "full name," and "information." A bag-of-words model is constructed based on all the segmented words to form a global dictionary containing several terms. Finally, the "user name" field is converted into a bag-of-words vector. The word "name" The TF-IDF value is calculated by multiplying the word frequency in the current field text with the inverse document frequency in all training samples, thereby strengthening its weight contribution in classification decision.

[0052] Through the above scheme, this application realizes a standardized process for bag-of-words vector acquisition, improves the accuracy and adaptability of feature extraction, effectively alleviates the problem of model performance degradation caused by insufficient data in the cold start stage, and enhances the system's ability to generalize to newly emerging fields in business scenarios.

[0053] In some embodiments, the process of generating knowledge entries and constructing a structured knowledge graph includes: A. A classification model is built based on a random forest classifier, and the model is trained with bag-of-words vectors as input and manually classified labels as output targets. B. The importance score of each input word is calculated by using the random forest algorithm. The importance score is the reduction in Gini impurity or the decrease in average precision, which describes the degree of contribution to the model. Random Forest (RF) is a supervised learning model based on decision tree ensemble. It can be implemented by training multiple decision trees in parallel and outputting classification results through a voting mechanism. Its purpose is to improve the model's generalization ability and anti-overfitting properties through ensemble learning.

[0054] Feature importance score can be understood as an indicator that quantifies the degree of influence of input features on classification decisions. It can be calculated based on the reduction in Gini impurity or the decrease in average precision. Its purpose is to objectively reflect the actual contribution of word segmentation in distinguishing different categories and avoid the subjective bias of manual rules.

[0055] C. Extract the top contributors for each classification result. Key words And calculate each keyword Weighting coefficients under each category ; Extract the top contributors The goal is to select the most influential subset of keywords. This can be achieved by using a ranking algorithm to sort the feature importance scores in descending order and then selecting the top M items. The aim is to focus on core semantic features and reduce noise interference. Each keyword... Weighting coefficients under each category Specifically, it can be obtained by normalizing the feature importance score or calculating the relative proportion, with the aim of providing an operable numerical basis for the subsequent matching process.

[0056] D. The set of relationships that establish a mapping between keywords, their categories, and weight coefficients is determined as knowledge entries and stored in a structured knowledge base to form an initial knowledge graph.

[0057] In a potential financial data governance scenario, when processing training samples containing the field "customer ID number," for the "personal identification information" category, the system, based on the analysis results of a random forest model and combined with category-specific evaluation methods such as the chi-square test, extracts the five keywords with the highest contribution (e.g., "ID card," "number," "card," "document," "ID"), and calculates their discriminative weight coefficients for this category as 0.85, 0.72, 0.68, 0.65, and 0.60, respectively. These <keyword, category, weight> triples are stored as prior knowledge in the knowledge base. When the new field "cust_ID" is input, the system matches it with the knowledge base through word segmentation and identifies the keyword "ID." The high weight of this keyword (0.60) serves as key evidence, and the decision engine (possibly combined with other features) quickly infers that this field belongs to the "personal identification information" category.

[0058] Through the above scheme, this application realizes the objective quantification of keyword contribution and the automatic generation of structured knowledge entries, so that the construction of knowledge graph has a reliable mathematical basis; the structured knowledge entries support the system to dynamically update the knowledge base when business terms evolve, effectively improving the adaptability to new business scenarios; at the same time, the weight coefficient mechanism provides a traceable quantitative basis in classification decision-making, solving the technical problem that it is difficult to balance classification accuracy and interpretability in traditional methods.

[0059] In some embodiments, although a graph keyword set is proposed to extract keywords from the fields to be classified, keyword matching relies on a simple string matching mechanism in this process, which cannot efficiently handle complex patterns, business jargon, or synonym variations, resulting in slow matching speed and high missed matching rate. In particular, it has insufficient generalization ability when facing non-standard field names (such as abbreviations, dialects, or disguised fields), which affects the real-time response capability and overall accuracy of the classification system.

[0060] Traditional knowledge graphs are statically constructed and cannot be dynamically updated to adapt to new domain knowledge, business jargon, or changes in regulatory requirements. This results in the need to retrain the model or manually maintain the rule base when facing business development, leading to slow response, high maintenance costs, and difficulty in achieving continuous knowledge evolution.

[0061] To provide a mechanism for updating and expanding the knowledge graph, this application also provides a manual intervention interface for the knowledge graph, supporting data security experts to leverage their domain knowledge. When updating or adding domain knowledge, new keywords are added to the knowledge graph, synonyms, near-synonyms, or business jargon are added to existing keywords, and new words are added accordingly. Assign initial weights .

[0062] Initial weights The similarity can be calculated based on its semantic similarity with existing core keywords or directly specified by experts. For example, it can be calculated based on cosine similarity in word vector space or by using preset baseline weight values ​​to ensure that new knowledge has a reasonable ability to participate in the classification process. Semantic similarity calculation refers to the technical means of quantifying the degree of semantic association between new words and existing keywords. It can use pre-trained word embedding models (such as Word2Vec) or path analysis methods based on knowledge graph topology to ensure that the weight of new words is logically consistent with related concepts.

[0063] The above operations transform the knowledge graph from a static database into a dynamic, expandable, and reproducible system that integrates data patterns and expert wisdom. Furthermore, the real-time dynamic updating capability of the knowledge graph effectively avoids classification blind spots caused by the evolution of business terminology; the system can immediately apply new knowledge without retraining the model, significantly reducing maintenance response time and manual intervention costs; simultaneously, the semantic similarity-based weight initialization mechanism ensures a reasonable contribution of new knowledge to classification decisions, maintaining the system's classification accuracy and stability during knowledge updates.

[0064] The content regarding automatic classification and evolutionary stages is described as follows: This application provides a method based on field information to be classified. Methods for constructing a graph keyword set may include: Received field information to be categorized At that time, the corresponding regular expression is generated based on the knowledge entries, and the field text is quickly scanned and matched to extract the successfully matched graph keywords; Combine all the graph keywords into a graph keyword set. Fields that did not match the keyword were identified as non-sensitive data and were filtered out directly.

[0065] Regular expressions dynamically construct matching rules using keywords and their associated variants stored in a knowledge graph. This can be achieved using a regular expression engine combined with a thesaurus, aiming to expand the coverage of keyword recognition to accommodate business jargon and language variations. Fast scanning of field text involves batch text processing operations implemented through pre-compiled regular expressions. Specifically, a DFA (Deterministic Finite Automaton)-based scanning algorithm can be used to improve processing efficiency, aiming to reduce time complexity to meet real-time classification requirements.

[0066] For example, when processing field information F containing the field annotation "Customer Nickname (English ID)", the system generates a regular expression "(Name|name|Formal Name)" based on the keyword "Name|Name|Formal Name" in the "Name" category of the knowledge entry. The regular expression engine quickly scans the field text and successfully matches "name" in "English ID". The matching result "name" is then used as a graph keyword. The data is stored in a collection; however, for fields annotated as "temporary cache", the collection is empty after scanning, and the system directly filters them out and determines them as non-sensitive data.

[0067] Through the above scheme, this application effectively improves the generalization ability of keyword matching, can accurately identify non-standard expressions such as abbreviations and dialects in business scenarios, and significantly shortens the matching time through the efficient processing mechanism of regularization expressions, thus meeting the real-time processing requirements while ensuring classification accuracy.

[0068] In some embodiments, candidate categories are filtered by matching knowledge entries, and target categories are filtered based on weighted coefficient scores, including: For each candidate category Poll the graph keywords in the graph keyword set and calculate their role in the candidate classification. The total weight score of the keyword set in the graph below ; indicates the following:

[0069] in, Keyword in the graph In candidate classification The weighting coefficient values ​​below, Indicates the number of set keys; Compare the total weight scores under all candidate categories, and select the candidate category corresponding to the highest category score. The target category for the field to be categorized is determined.

[0070] As a specific implementation method, the solution of this application is implemented as follows: When the field to be classified is "user identifier", the system extracts the graph keyword set containing "identifier" and "user"; for the candidate category "identity information", the weight coefficient of "identifier" is 0.75, the weight coefficient of "user" is 0.65, and the total weight score is 1.40; for the candidate category "transaction information", the weight coefficient of "identifier" is 0.25, the weight coefficient of "user" is 0.35, and the total weight score is 0.60; after comparing the total weight scores of all candidate categories, the system determines "identity information" corresponding to the highest score as the target category, and provides the user with the basis for score calculation to enhance interpretability.

[0071] Through the above technical solutions, the classification decision-making process becomes interpretable, the system can objectively evaluate the reliability of candidate classifications, effectively handle scenarios with ambiguous or conflicting field information, reduce misjudgments, and provide clear classification criteria. In some embodiments, the system may be unable to assess the reliability of the classification results. When the highest classification score is close to the second highest score, the output classification result may be inaccurate. Furthermore, the lack of a mechanism to handle low confidence situations increases the risk of misclassification. At the same time, the system cannot automatically trigger optimization to cope with the continuous uncertainty.

[0072] To complete the closed-loop processing of the system and improve the reliability of the classification results, in another embodiment, the system confidence level can also be set during initialization. After calculating the target classification, perform the following steps: A. Calculate the target confidence score of the target classification results. The target confidence score is the highest classification score. Or the difference between the highest category score and the second highest score. ; B. Compare the target confidence score with the system confidence score, when When, output the final classification result; when When necessary, the results are pushed to manual adjudication; the adjudication and correction results are then re-integrated into the training sample set (or the knowledge graph is directly updated). C. When the number of pushes for manual adjudication reaches the set threshold, trigger word resegmentation and update the bag-of-words vector.

[0073] Through the above solution, a continuous learning and evolution closed-loop of "model training -> knowledge graph construction -> automatic classification -> manual verification -> feedback optimization" is formed, realizing the dynamic evaluation and adaptive optimization of the reliability of classification results, effectively reducing the risk of misclassification caused by fuzzy decision boundaries. At the same time, through the closed-loop mechanism of manual feedback and feature update, it ensures that the system can continuously absorb domain knowledge and adapt to business changes, significantly improving the robustness and maintainability of data classification in complex business scenarios.

[0074] In addition, the present application can further construct multi-language independent sub-knowledge graphs, and use existing cross-language dictionaries and machine translation technologies to establish link relationships between sub-knowledge graphs; for example, concepts such as "name", "name", and "formal name" (dialect) are linked as equivalent nodes. In the bag-of-words vector extraction stage, the MUSE (Multilingual Unsupervised or Supervised word Embeddings) algorithm is used to map word vectors in different languages to the same vector space, enabling the model to capture semantically similar words in different languages and improving cross-language understanding ability. When performing classification, a lightweight language detection algorithm (such as an n-gram-based statistical method) is used to quickly determine the main language type of each field to be classified; this information will be used in the subsequent classification process to select appropriate sub-knowledge graphs and feature extraction strategies.

[0075] Furthermore, a language-related query optimization mechanism is introduced into the knowledge graph. When processing database field information, the system first uses an n-gram-based lightweight language detection algorithm to identify the field language type (such as "customer name" being standard Chinese and "customer_name" being English). The system maintains a multi-language knowledge graph, in which synonymous concepts in different languages are associated through cross-language links, such as establishing an equivalent relationship between the Chinese "name" node and the English "name" node.

[0076] The solution of the present application is specifically implemented as follows: when processing database field information, the system first uses an n-gram-based lightweight language detection algorithm to identify that the field "customer name" is standard Chinese and "customer_name" is English; for Chinese fields, the Chinese sub-knowledge graph is called, and the "name" node is linked to the "name" node of the English sub-graph through a cross-language dictionary; in the bag-of-words vector extraction stage, the MUSE algorithm maps the word vectors of "name" and "name" to similar vector positions.

[0077] During classification, the system preferentially matches in the sub-graphs of the corresponding languages based on the detected language type. For fields in mixed languages or rare languages, concept expansion queries are performed through cross-language links. For example, the "Customer Name" is directly matched to the "Personal Information" classification in the Chinese sub-graph; although the "cust_name" field is not directly included, through the semantic association from "cust" to "customer" and the cross-language link from "name" to "姓名", it can still be accurately inferred that it belongs to the "Personal Information" classification.

[0078] When the system receives field information containing dialect words, such as Cantonese expressions like "靓仔" in the field annotation, after the language detection module identifies it as Cantonese, the dynamic weight adjustment mechanism automatically increases the weight value of the "Name" node in the Cantonese sub-knowledge graph, so that in the subsequent matching process, this field is more likely to be accurately classified as the "Name" category, while avoiding the risk of misclassification caused by dialect expression differences.

[0079] Through the above solution, this application effectively bridges the semantic gap in a multilingual environment, avoids matching omissions or errors caused by language differences, and improves the accuracy of sensitive field classification and the applicability of the system in multilingual business scenarios.

[0080] To further enhance the self-learning ability of the system, in some embodiments, an expert feedback and cross-language knowledge transfer mechanism can also be introduced. When an expert corrects the classification result of a field in a certain language, the system not only updates the node weights in the sub-knowledge graph of that language, but also synchronizes the semantic relationship (e.g., the strong association between a certain word and a specific classification) contained in this correction to the sub-graphs of other languages through pre-set cross-language links. This mechanism greatly accelerates the knowledge accumulation and adaptation process of the system in the entire multilingual environment, forming a continuously optimized closed loop.

[0081] For example, when an expert corrects the Chinese field "身份证号" from the "Basic Information" category to the "Identity Authentication" category, the system not only updates the Chinese sub-knowledge graph, but also automatically identifies the corresponding "IDnumber" node in the English sub-graph and the equivalent node of "身份证号码" in the dialect sub-graph through cross-language links, and synchronously migrates the corrected classification information and weight coefficients to these associated nodes, so that the English and dialect sub-graphs automatically obtain the same classification logic as the Chinese sub-graph.

[0082] Combined with the foregoing embodiments, it can effectively solve the technical problem of low knowledge update efficiency in a multilingual environment, ensure the consistency of knowledge between different language sub-graphs, significantly improve the adaptive ability of the system in multilingual scenarios, and enable expert feedback to be quickly transformed into classification optimization ability in a multilingual environment.

[0083] Figure 4The structural block diagram of the machine learning-based data classification device provided in this application embodiment includes: The knowledge graph construction module 410 uses the extracted bag-of-words vectors as model input and the classification label as target output to build and train a classification model. The keywords, categories, and weight coefficients extracted from the model are used as knowledge entries to build a structured knowledge graph. The collection construction module 420 obtains the field information to be classified, performs keyword matching based on the knowledge graph, extracts the graph keywords contained therein, and constructs a graph keyword set; The classification output module 430 extracts the graph keywords, filters candidate categories by matching the knowledge entries, and filters the target category based on the weight coefficient score.

[0084] It should be noted that the machine learning-based data classification device provided in this embodiment is only an example of the above-described division of functional modules / units. In practical applications, the above functions can be assigned to different functional modules / units as needed, that is, the internal structure of the machine learning-based data classification device can be divided into different functional modules / units to complete all or part of the functions described above. Furthermore, the implementation methods of the machine learning-based data classification method provided in the above-described method embodiments and the implementation methods of the machine learning-based data classification device provided in this embodiment belong to the same concept. The specific implementation process of the machine learning-based data classification device provided in this embodiment is detailed in the above-described method embodiments and will not be repeated here.

[0085] Figure 5 This application provides a structural block diagram of a computer device according to an exemplary embodiment. The computer device can be a desktop computer, laptop computer, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor and memory. The processor and memory can be connected via a bus or other means. The processor can be a central processing unit (CPU) or other general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a graphics processing unit (GPU), an embedded neural network processor (NPU) or other dedicated deep learning coprocessor, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0086] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the above embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include a program storage area and a data storage area, wherein the program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0087] Those skilled in the art will understand that the structure shown in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0088] This application also discloses a computer-readable storage medium. Specifically, the computer-readable storage medium is used to store a computer program, which, when executed by a processor, implements the methods described in the above-described method embodiments. Those skilled in the art will understand that implementing all or part of the processes in the methods described above can be accomplished by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0089] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A data classification method based on machine learning, characterized in that, The method includes: The extracted bag-of-words vectors are used as model inputs and classification labels are used as target outputs to build and train a classification model. The keywords, categories, and weight coefficients extracted from the model are used as knowledge entries to build a structured knowledge graph. Obtain the field information to be classified, perform keyword matching based on the knowledge graph, extract the graph keywords contained therein, and construct a graph keyword set; Extract the graph keywords, filter candidate categories by matching the knowledge entries, and filter the target category based on the weight coefficient score.

2. The method according to claim 1, characterized in that, The steps for obtaining the bag-of-words vectors include: Obtain the training sample set of manually classified labels, and use a word segmenter to segment the field names and field annotations into independent words; Based on all word segmentation, a bag-of-words model is constructed to transform unstructured text data into structured bag-of-words vectors. , is represented as: Where n is the size of the dictionary. This represents the frequency or TF-IDF value of the i-th word in the field text.

3. The method according to claim 2, characterized in that, The process of generating knowledge entries and constructing a structured knowledge graph includes: The classification model is constructed based on a random forest classifier, and the bag-of-words vector is used as input and the manually classified label is used as the output target for model training. The importance score of each input word is obtained by calculating the feature importance score of the input word segmentation using the random forest algorithm; the importance score is the reduction in Gini impurity or the decrease in average precision, which is used to describe the degree of contribution to the model. Extract the top contributors for each classification result. Key words And calculate each keyword Weighting coefficients under each category ; The set of relationships that establish a mapping between keywords, their categories, and weight coefficients is determined as knowledge entries and stored in a structured knowledge base to form an initial knowledge graph.

4. The method according to claim 1, characterized in that, The constructed graph keyword set includes: Received field information to be categorized At that time, a corresponding regular expression is generated based on the knowledge entry, and the field text is quickly scanned and matched to extract the successfully matched graph keywords; All the aforementioned graph keywords are combined into the graph keyword set. Fields that did not match the keyword were identified as non-sensitive data and were filtered out directly.

5. The method according to any one of claims 1-4, characterized in that, The step of filtering candidate categories by matching the knowledge entries and filtering target categories based on weighted coefficient scores includes: For each of the candidate classifications The graph keywords in the graph keyword set are polled to calculate the candidate classification. The total weight score of the following graph keyword set ; indicates the following: in, Keyword in the graph In the candidate classification The weighting coefficient values ​​below, Indicates the number of set keys; Compare the total weight scores under all candidate categories, and select the candidate category corresponding to the highest category score. The target category for the field to be categorized is determined.

6. The method according to claim 5, characterized in that, Set system confidence during initialization ; After calculating the target classification, the method further includes: Calculate the target confidence score of the target classification result. The target confidence score is the highest classification score. Or the difference between the highest category score and the second highest score. ; The target confidence score is compared with the system confidence score. When, output the final classification result; when When the time comes, the data is pushed to a human adjudicator; the adjudication and correction results are then re-integrated into the training sample set. When the number of pushes for manual adjudication reaches a set threshold, word resegmentation and updating of the bag-of-words vector are triggered.

7. The method according to claim 1, characterized in that, The knowledge graph is equipped with an intervention interface. When updating or adding domain knowledge, it can add new keywords to the knowledge graph, add synonyms, near-synonyms, or business jargon to existing keywords, and add new words. Assign initial weights The initial weights The result is calculated based on the semantic similarity of existing keywords, or a default value can be set.

8. A data classification device based on machine learning, characterized in that, The device includes: The knowledge graph construction module is used to build and train a classification model by taking the extracted bag-of-words vectors as model input and the classification label as target output. The keywords, categories, and weight coefficients extracted from the model are used as knowledge entries to build a structured knowledge graph. The collection construction module is used to obtain the field information to be classified, perform keyword matching based on the knowledge graph, extract the graph keywords contained therein, and construct a graph keyword set; The classification output module is used to extract the graph keywords, filter candidate categories by matching the knowledge entries, and filter the target category based on the weight coefficient score.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the machine learning-based data classification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the machine learning-based data classification method as described in any one of claims 1 to 7.

Images