Data resource classification method and device based on combination of rules and semantic approximation and AI
By combining rule matching, semantic approximation, and AI models to classify data resources, this method addresses the shortcomings in efficiency, accuracy, and convenience in existing technologies, achieving efficient, accurate, and convenient data classification applicable to structured or unstructured data such as text, images, and audio.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHONGDIAN DATA IND CO LTD
- Filing Date
- 2025-09-05
- Publication Date
- 2026-07-14
Smart Images
Figure CN121456535B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a data resource classification method and apparatus based on a combination of rules and semantic approximation with AI. Background Technology
[0002] With the development of big data and artificial intelligence technologies, automated data classification has become a core requirement in areas such as enterprise data governance, intelligent customer service, and content moderation. Traditional data classification methods mainly rely on manual rules or pure AI models. However, manual rule methods are only suitable for structured or simple text and struggle to cover the diversity of natural language; for example, they are difficult to cover synonyms and complex expressions. Pure AI model methods, on the other hand, rely on large-scale labeled data, suffer from severe cold-start problems, and have high model training and iteration costs. Therefore, how to achieve a classification scheme that balances efficiency, accuracy, and ease of implementation has become an urgent problem to be solved. Summary of the Invention
[0003] This invention provides a data resource classification method and apparatus based on rules and semantic approximation combined with AI, in order to solve the problem that existing data classification methods cannot simultaneously achieve efficiency, accuracy and ease of implementation.
[0004] In a first aspect, the present invention provides a data resource classification method based on rules and semantic approximation combined with AI, the method comprising:
[0005] Set up a rule matching module, a semantic approximation matching module, and an AI model;
[0006] Upon receiving the data resources to be classified, the rule matching module, the semantic approximation matching module, and the AI model are sequentially activated to classify the data resources to be classified until the classification of the data resources to be classified is completed.
[0007] The rule matching module classifies the data resources to be classified based on preset hard rules. The semantic approximation matching module calculates the semantic similarity of the data resources to be classified that are not matched by the rule matching module, and classifies the data resources based on the calculated semantic similarity. When the semantic approximation matching module does not match the classification of the data resources to be classified, the AI model calculates the semantic similarity between the data resources to be classified and the standard classification labels based on the pre-trained embedding model.
[0008] Optionally, the hard rules include keyword matching, regular expression matching, and logical condition matching based on keywords, regular expressions, and logical conditions in the rule base.
[0009] Optionally, the method further includes: when a new category is added, dynamically updating the keywords, the regular expression, and the logical conditions by expanding the rule base.
[0010] Optionally, calculating the semantic similarity of the data resources to be classified includes: calculating the similarity of the data resources to be classified through word vectors and sentence vectors, and matching data that are semantically similar but have different expressions.
[0011] Optionally, the step of calculating the similarity of the data resources to be classified through word vectors and sentence vectors, and matching data that are semantically similar but have different expressions, includes: storing word vector semantics and sentence vector semantics in a preset semantic library, and matching data that are semantically similar but have different expressions based on the word vector semantics and sentence vector semantics in the semantic library and the word vectors and sentence vectors of the data resources to be classified.
[0012] Optionally, the method further includes: when a new category is added, dynamically updating the word vector semantics and sentence vector semantics by expanding the semantic library.
[0013] Optionally, the method further includes: training the embedding model through deep learning and machine learning, and calculating the semantic similarity between the data resource to be classified and the standard classification label through the embedding model.
[0014] Optionally, the embedding model is a BERT model. The step of calculating the semantic similarity between the data resource to be classified and the standard classification label through the embedding model includes: pre-generating vectors for a large number of sentences and storing them in a vector database; when using the model, inputting the input sentence into the same BERT model to obtain a sentence vector representation; then comparing the vector representation with the vectors in the vector database to calculate the similarity between the vector and the vectors in the database; and based on a preset similarity threshold, identifying sentences with similarity higher than the similarity threshold as belonging to the corresponding classification.
[0015] Secondly, the present invention provides a data resource classification device based on rules and semantic approximation combined with AI, the device comprising:
[0016] The configuration unit is used to configure the rule matching module, the semantic approximation matching module, and the AI model.
[0017] The processing unit is configured to, upon receiving data resources to be classified, sequentially activate the rule matching module, the semantic approximation matching module, and the AI model to classify the data resources until the classification of the data resources to be classified is completed. The rule matching module classifies the data resources to be classified based on preset hard rules. The semantic approximation matching module calculates the semantic similarity of the data resources to be classified for those not matched by the rule matching module, and classifies the data resources based on the calculated semantic similarity. When the semantic approximation matching module fails to match the classification of the data resources to be classified, the AI model calculates the semantic similarity between the data resources to be classified and the standard classification labels based on a pre-trained embedding model.
[0018] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the methods described above.
[0019] The beneficial effects of this invention are as follows:
[0020] This invention sets up a rule matching module, a semantic approximation matching module, and an AI model. By sequentially activating the rule matching module, the semantic approximation matching module, and the AI model to classify the data resources to be classified until the classification of the data resources to be classified is completed, the invention achieves accurate and convenient data classification while taking into account classification efficiency, thereby greatly improving the user experience.
[0021] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description
[0022] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0023] Figure 1 This is a flowchart illustrating a data resource classification method based on rules and semantic approximation combined with AI, provided by an embodiment of the present invention.
[0024] Figure 2 This is a schematic diagram of the structure of a data resource classification device based on rules and semantic approximation combined with AI, provided in an embodiment of the present invention. Detailed Implementation
[0025] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of the invention.
[0026] Existing methods for classifying data resources using manual rules struggle to capture the diversity of natural language, are costly to maintain, and require extensive manual rule writing for new categories. Pure AI model methods, on the other hand, rely on large-scale labeled data, suffer from severe cold-start problems, high training and iteration costs, and their black-box nature makes decision-making difficult to interpret. Furthermore, cold-start difficulties arise because new business scenarios lack initial labeled data, preventing direct application of AI models. Pure rule-based methods also exhibit poor scalability, while pure AI methods have long implementation cycles. Additionally, there is insufficient coverage of long-tail data, meaning that rare or diverse data cannot be accurately classified using a single method.
[0027] To address the aforementioned problems, embodiments of the present invention provide a data resource classification method based on rules and semantic approximation combined with AI, see [link to relevant documentation]. Figure 1 The method includes:
[0028] S101. Set up the rule matching module, semantic approximation matching module, and AI model;
[0029] S102. Upon receiving the data resources to be classified, the rule matching module, the semantic approximation matching module, and the AI model are sequentially activated to classify the data resources to be classified until the classification of the data resources to be classified is completed.
[0030] In other words, the embodiments of the present invention set up a rule matching module, a semantic approximation matching module, and an AI model, and sequentially start the rule matching module, the semantic approximation matching module, and the AI model to classify the data resources to be classified until the classification of the data resources to be classified is completed. Thus, while taking into account classification efficiency, the present invention achieves accurate and convenient data classification, thereby effectively solving the above-mentioned problems.
[0031] In specific implementation, the rule matching module of this embodiment of the invention matches and classifies the data resources to be classified based on preset hard rules. The semantic approximation matching module calculates the semantic similarity of the data resources to be classified that are not matched by the rule matching module, and classifies the data resources based on the calculated semantic similarity. When the semantic approximation matching module does not match the classification of the data resources to be classified, the AI model is used to calculate the semantic similarity between the data resources to be classified and the standard classification labels based on the pre-trained embedding model.
[0032] In short, the method described in this invention addresses the problem that existing data classification methods primarily rely on pure AI models (such as deep learning and machine learning), requiring extensive manual data annotation for model training. This leads to several issues: firstly, high annotation costs, as new domains or classifications necessitate the re-annotation of massive amounts of data, resulting in a large workload; secondly, a cold start problem, where model performance is poor when initial labeled data is lacking; and thirdly, low flexibility, requiring model retraining for changes in classification logic, leading to long cycles. While existing rule matching or semantic approximation methods do not require training data, their accuracy is insufficient, making them unsuitable for handling complex scenarios. Therefore, this invention combines rule matching, semantic approximation calculation, and AI model-based data resource classification methods to effectively automate the classification of structured or unstructured data such as text, images, and audio.
[0033] Furthermore, in this embodiment of the invention, the rule matching module (which can also be simply called the rule layer) predefines rules such as keywords and regular expressions, and prioritizes matching explicit categories; the semantic approximation matching module (which can be simply called the semantic approximation layer) calculates similarity through word vectors and sentence vectors, and matches data that are similar but have different expressions; the AI model layer, namely the AI model mentioned above, calls the pre-trained model to classify complex cases that are not covered by the first two layers.
[0034] Practice has proven that the method described in the embodiments of the present invention can achieve at least the following beneficial effects:
[0035] When adding a new category, the method described in this embodiment of the invention only requires expanding the rule base or semantic base, without retraining the model. The method described in this invention can reduce annotation requirements by more than 70%, especially for cold start scenarios, which can greatly reduce reliance on manual intervention. Furthermore, this invention can filter simple cases through rules and semantic layers, while the AI layer focuses on complex decisions, thereby improving overall accuracy by 15%-30%. Moreover, this invention can adapt to new classification requirements simply by adding or deleting rules, shortening the implementation cycle by 50%, thus achieving flexible rule expansion.
[0036] In specific implementation, the hard rules in this embodiment of the invention include keyword matching, regular expression matching, and logical condition matching based on keywords, regular expressions, and logical conditions in the rule base. When a new category is added, the keywords, regular expressions, and logical conditions are dynamically updated by expanding the rule base, thereby avoiding the complexity of retraining the model.
[0037] In other words, the rule-based layer of this invention can quickly match data through predefined hard rules (such as keywords, regular expressions, and logical conditions), and is suitable for clear and structured classification scenarios.
[0038] In practical implementation, this invention involves constructing a rule base that includes category keywords, regular expressions, and logical judgments (such as "IF-THEN" rules). Then, the input data is matched against these rules; if a match is successful, it is directly categorized.
[0039] For example, text classification includes:
[0040] Rule 1: If the text contains "fault" or "unusable" → classify it as "problem feedback".
[0041] Rule 2: If the text matches the regular expression price[high / low], it is classified as "price-related".
[0042] The rule-based matching method for classifying data resources has zero training cost and can process some data without the need for an AI model. Furthermore, it is highly interpretable, as the classification results are entirely determined by manually defined rules.
[0043] Furthermore, the calculation of the semantic similarity of the data resources to be classified, as described in this embodiment of the invention, includes:
[0044] The similarity of the data resources to be classified is calculated using word vectors and sentence vectors, matching data that are semantically similar but have different expressions. Specifically, a pre-defined semantic database stores word vector semantics and sentence vector semantics. Based on the word vector semantics and sentence vector semantics in this database, as well as the word vectors and sentence vectors of the data resources to be classified, data that are semantically similar but have different expressions is matched. When a new category is added, the word vector semantics and sentence vector semantics are dynamically updated by expanding the semantic database. This approach also avoids the complexity of retraining the model while achieving accurate classification of data resources.
[0045] Specifically, the semantic similarity layer in this embodiment of the invention classifies data that is not matched in the rule layer by using semantic similarity calculation (such as word vector and sentence vector matching), which is suitable for data with diverse expressions but similar meanings.
[0046] In specific implementation, the method described in this embodiment of the invention uses a pre-trained word embedding model (such as Word2Vec, GloVe) or sentence embedding model (such as SBERT, Universal Sentence Encoder) to calculate the semantic similarity between the input data and the standard classification label.
[0047] And by setting a similarity threshold (such as 0.8), if the similarity exceeds the threshold, it will be classified.
[0048] For example, input: "Delivery is too slow", calculate its similarity to the standard label "Logistics delay" (e.g., cosine similarity 0.85), and classify it if it exceeds the threshold.
[0049] This classification method reduces reliance on strict rules and adapts to the diversity of natural language. Compared to pure AI models, it requires less computation and is suitable for real-time classification.
[0050] In addition, embodiments of the present invention train the embedding model through deep learning and machine learning, and calculate the semantic similarity between the data resources to be classified and the standard classification labels through the embedding model.
[0051] In this embodiment of the invention, the embedding model is the BERT model, and the step of calculating the semantic similarity between the data resource to be classified and the standard classification label through the embedding model includes:
[0052] A large number of sentences are pre-generated into vectors and stored in a vector database. When needed, the input sentence is fed into the same BERT model to obtain a sentence vector representation. Then, this vector representation is compared with the vectors in the vector database to calculate the similarity between the vector and the vectors in the database. Based on a preset similarity threshold, sentences with similarity higher than the threshold are identified as matching the corresponding category.
[0053] Specifically, the AI model (AI Model Layer) in this embodiment of the invention is used to perform the final classification of data that is not matched in the first two layers, using an AI model (such as deep learning or machine learning), which is suitable for complex, fuzzy, or long-tailed data.
[0054] In specific implementation, the embodiments of the present invention use pre-trained classification models (such as BERT, CNN, XGBoost, etc.) to predict only the data not covered by the first two layers, so as to reduce computational overhead.
[0055] For example: Input: "This phone takes good photos, but the battery life is average" → The model judges it as "mixed evaluation".
[0056] This invention can use AI models to handle complex situations that cannot be covered by rules and semantic layers, and the model performance can be iteratively optimized as data accumulates.
[0057] In general, the embodiments of the present invention set up a rule matching module, a semantic approximation matching module, and an AI model, and sequentially start the rule matching module, the semantic approximation matching module, and the AI model to classify the data resources to be classified until the classification of the data resources to be classified is completed. In this way, while taking into account classification efficiency, the present invention achieves accurate and convenient data classification, thereby greatly improving the user experience.
[0058] In addition, embodiments of the present invention also provide a data resource classification device based on rules and semantic approximation combined with AI, see [link to related document]. Figure 2 The device includes:
[0059] The configuration unit is used to configure the rule matching module, the semantic approximation matching module, and the AI model.
[0060] The processing unit is configured to, upon receiving data resources to be classified, sequentially activate the rule matching module, the semantic approximation matching module, and the AI model to classify the data resources until the classification of the data resources to be classified is completed. The rule matching module classifies the data resources to be classified based on preset hard rules. The semantic approximation matching module calculates the semantic similarity of the data resources to be classified for those not matched by the rule matching module, and classifies the data resources based on the calculated semantic similarity. When the semantic approximation matching module fails to match the classification of the data resources to be classified, the AI model calculates the semantic similarity between the data resources to be classified and the standard classification labels based on a pre-trained embedding model.
[0061] In other words, the embodiments of the present invention, through the setting unit and the processing unit, can achieve accurate and convenient data classification while taking into account classification efficiency, thereby greatly improving the user experience.
[0062] Meanwhile, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described in any of the embodiments of the present invention.
[0063] The relevant content of the device embodiment and storage medium embodiment of the present invention can be understood by referring to the method embodiment of the present invention, and will not be discussed in detail here.
[0064] For a detailed understanding, please refer to the embodiments of the method of this invention; they will not be discussed in detail here.
[0065] Although preferred embodiments of the invention have been disclosed for illustrative purposes, those skilled in the art will recognize that various modifications, additions, and substitutions are possible, and therefore the scope of the invention should not be limited to the embodiments described above.
Claims
1. A data resource classification method based on rule-based and semantic approximation combined with AI, characterized in that, The method includes: Set up a rule matching module, a semantic approximation matching module, and an AI model; Upon receiving the data resources to be classified, the rule matching module, the semantic approximation matching module, and the AI model are sequentially activated to classify the data resources to be classified until the classification of the data resources to be classified is completed. The rule matching module matches and classifies the data resources to be classified based on preset hard rules. The semantic approximation matching module calculates the semantic similarity of the data resources to be classified that are not matched by the rule matching module, and classifies the data resources based on the calculated semantic similarity. When the semantic approximation matching module does not match the classification of the data resources to be classified, the AI model calculates the semantic similarity between the data resources to be classified and the standard classification labels based on the pre-trained embedding model. The hard rules include keyword matching, regular expression matching, and logical condition matching based on keywords, regular expressions, and logical conditions in the rule base; The step of calculating the semantic similarity of the data resources to be classified includes: calculating the similarity of the data resources to be classified through word vectors and sentence vectors, and matching data that are semantically similar but have different expressions; The method further includes: training the embedding model through deep learning and machine learning, and calculating the semantic similarity between the data resources to be classified and the standard classification labels through the embedding model.
2. The method according to claim 1, characterized in that, The method further includes: When a new category is added, the keywords, regular expressions, and logical conditions are dynamically updated by expanding the rule base.
3. The method according to claim 1, characterized in that, The process of calculating the similarity of the data resources to be classified using word vectors and sentence vectors, and matching data that are semantically similar but have different expressions, includes: The semantic library stores word vector semantics and sentence vector semantics. Based on the word vector semantics and sentence vector semantics in the semantic library and the word vectors and sentence vectors of the data resources to be classified, data with similar semantics but different expressions are matched.
4. The method according to claim 3, characterized in that, The method further includes: When a new category is added, the semantic base is expanded to dynamically update the word vector semantics and sentence vector semantics.
5. The method according to claim 1, characterized in that, The embedding model is the BERT model. The process of calculating the semantic similarity between the data resource to be classified and the standard classification label using the embedding model includes: A large number of sentences are pre-generated into vectors and stored in a vector database. When needed, the input sentence is fed into the same BERT model to obtain a sentence vector representation. Then, this vector representation is compared with the vectors in the vector database to calculate the similarity between the vector and the vectors in the database. Based on a preset similarity threshold, sentences with similarity higher than the threshold are identified as matching the corresponding category.
6. A data resource classification apparatus based on rule-based and semantic approximation combined with AI for implementing the method of any one of claims 1-5, characterized in that, The device includes: The configuration unit is used to configure the rule matching module, the semantic approximation matching module, and the AI model. The processing unit is configured to, upon receiving data resources to be classified, sequentially activate the rule matching module, the semantic approximation matching module, and the AI model to classify the data resources until the classification of the data resources to be classified is completed. Specifically, the rule matching module classifies the data resources to be classified based on preset hard rules; the semantic approximation matching module calculates the semantic similarity of the data resources to be classified for those not matched by the rule matching module, and classifies the data resources based on the calculated semantic similarity; when the semantic approximation matching module fails to match the classification of the data resources to be classified, the AI model calculates the semantic similarity between the data resources to be classified and the standard classification labels based on a pre-trained embedding model.
7. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the data resource classification method based on rule-based and semantic approximation combined with AI as described in any one of claims 1-5.