A data classification method based on semantic understanding
Semantic analysis and classification model construction through deep learning technology solve the problem that traditional data classification methods are difficult to capture deep semantic information, and achieve more accurate and flexible data classification.
Patent Information
- Application Number
- CN202411820484.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-11
- Publication Date
- 2025-04-04
AI Technical Summary
Traditional data classification methods based on keyword matching are difficult to capture the deep semantic information and contextual relationships of text, resulting in poor classification results and difficulty in coping with data in different fields.
The semantic analysis technology based on deep learning is used to preprocess text data and model deep semantic information, and a classification model is built based on semantic analysis results, and the model is dynamically adjusted through real-time data and user feedback.
It improves the accuracy of classification results, can adapt to data changes and user needs in real time, and ensures that the model has high classification performance.
Smart Images
Figure CN119760516A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data classification, and specifically is a data classification method based on semantic understanding. Background Art
[0002] Data classification is a key link in information retrieval and data mining analysis. Traditional data classification methods based on keyword matching often find it difficult to accurately capture the semantic information and contextual relationships of the data when faced with complex and ambiguous text data, resulting in unsatisfactory classification results.
[0003] At present, data classification methods are mainly implemented through keyword matching methods. Keyword matching methods extract keywords from texts and match them to preset classification labels. This type of method is simple and easy to implement, but it is easily affected by keyword sparsity and ambiguity, resulting in poor classification results.
[0004] According to the current existing technology, the keyword matching-based method is difficult to capture the deep semantic information and contextual relationships of the text, resulting in poor classification effect and difficulty in dealing with data from different fields, which leads to the classification results not meeting the actual application needs. Summary of the invention
[0005] In view of the above problems, the present invention is proposed to provide a data classification method based on semantic understanding that overcomes the above problems or at least partially solves the above problems.
[0006] To achieve the above object, the present invention adopts the following technical solutions:
[0007] A data classification method based on semantic understanding, the method comprising:
[0008] Preprocess the input text data;
[0009] Use deep learning technology to perform semantic analysis, model the deep semantic information and contextual relationships of the text, and extract feature representations of the text;
[0010] Combine the semantic analysis results to build a classification model;
[0011] Input the text to be classified into the classification model and output the classification result;
[0012] Adjust classification models based on real-time data and user feedback.
[0013] Optionally, the input text data is preprocessed including word segmentation, stop word removal and part-of-speech tagging.
[0014] Optionally, the deep learning technology includes neural networks, transfer models and large model technologies.
[0015] Optionally, the method for constructing the classification model includes:
[0016] Supervised learning method: input the semantically analyzed text features into the SVM model, and train the SVM model using labeled training data to obtain an SVM model that can classify text;
[0017] The unsupervised learning method uses K-means cluster analysis to classify the text. The text features after semantic analysis are input into the K-means algorithm to classify the text.
[0018] Optionally, methods for outputting classification results include:
[0019] Sort the categories by the predicted probability of the classification model, and give priority to displaying categories with higher predicted probabilities;
[0020] Through text summarization and keyword extraction, the classification results are displayed in the form of charts.
[0021] Optionally, adjust the classification model based on real-time data and user feedback including:
[0022] Real-time data collection: regularly collect new text data to update the training set, and collect new text data by writing data crawling scripts in Python;
[0023] User feedback: set a feedback button on the classification result display page so that users can evaluate the classification results. Based on user feedback, adjust the parameters of the classification model to guide the optimization of the classification model.
[0024] Classification model adjustment: add the collected new data to the training set and retrain the classification model to adapt to the changes in the data.
[0025] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0026] 1. The present invention performs semantic analysis through deep learning to extract deep semantic information and contextual relationships of text data, which can greatly improve the accuracy of classification results; it can also collect data and user feedback in real time, dynamically adjust the classification model algorithm, and ensure that the model always has high classification performance.
[0027] 2. The present invention uses Python general programming language to write data crawling scripts, which reduces the difficulty of implementation and makes the method easy to apply to data sets of different scenarios and different sizes; the processing quality of text data is improved through preprocessing techniques such as word segmentation, stop word removal and part-of-speech tagging, which can lay the foundation for subsequent deep learning analysis.
[0028] 3. In the whole classification process of the present invention, from data preprocessing to model training and classification, automation can be achieved, greatly reducing manual intervention and improving data processing efficiency. Description of the Drawings
[0029] Figure 1 It is a schematic flowchart of a data classification method based on semantic understanding provided by an embodiment of the present application. Detailed Embodiments
[0030] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.
[0031] Please refer to Figure 1 , this embodiment provides a data classification method based on semantic understanding, and the method includes:
[0032] S1. Preprocess the input text data.
[0033] Preprocessing the input text data includes operations such as word segmentation, stop word removal, and part-of-speech tagging for subsequent semantic analysis.
[0034] Text preprocessing is the first step in data classification, and its purpose is to convert the original text data into a format suitable for semantic analysis.
[0035] Word segmentation: Split continuous text into meaningful lexical units to eliminate redundant information between words and improve the accuracy of subsequent analysis. The word segmentation algorithm can adopt a dictionary-based method combined with word embedding technology, considering not only the boundaries of words but also the meanings of words in the context.
[0036] For example, for the text "Big data analysis helps enterprise development", this method can recognize "big data analysis" as a whole concept instead of simply splitting it.
[0037] Stop word removal: Based on the stop word removal strategy of word frequency statistics and context importance, only remove those words that contribute the least to the meaning of the text, thereby retaining more useful information.
[0038] For example, "of", "and", "is", etc. Removing stop words can reduce the amount of calculation and improve the classification efficiency. For the text "Artificial intelligence is changing our lives", we get: "Artificial intelligence / is / changing / our / lives / ."
[0039] Part-of-speech tagging: Assign a part-of-speech tag to each word in the text, such as noun, verb, adjective, etc. Part-of-speech tagging helps identify the grammatical role of words in a sentence and assists semantic analysis. For example: "artificial intelligence / noun / is / adverb / change / verb / we / pronoun / life / noun / ".
[0040] S2. Use deep learning technology to perform semantic analysis, model the deep semantic information and contextual relationships of the text, and extract feature representations of the text.
[0041] The deep learning technology includes neural network, transfer model and large model technology.
[0042] Semantic analysis: Semantic analysis is an important part of this embodiment, and its purpose is to capture the deep semantic information and contextual relationship of the text.
[0043] This embodiment adopts the following semantic analysis method:
[0044] Deep learning technology: Use neural networks, transfer models, large models and other technologies to model text and automatically learn the characteristics of text. Deep learning technology can effectively capture the deep semantic information and contextual relationships of text and improve classification results.
[0045] Attention mechanism: Introducing the attention mechanism into the deep learning model enables the model to focus on the key parts of the text, automatically identify the core elements of the event, and improve the accuracy of semantic analysis.
[0046] For example: When processing the sentence "Artificial intelligence is changing our lives", the model will automatically give higher weights to "artificial intelligence" and "change".
[0047] S3. Combine the semantic analysis results to build a classification model. The classification model can be implemented through decision trees, vector machines, cluster analysis, etc.
[0048] The methods for building classification models include:
[0049] The supervised learning method inputs the text features after semantic analysis into the SVM model, and trains the SVM model using labeled training data to obtain an SVM model that can classify text.
[0050] The unsupervised learning method uses K-means cluster analysis to classify the text. The text features after semantic analysis are input into the K-means algorithm to classify the text.
[0051] S4. Input the text to be classified into the classification model and output the classification result.
[0052] Methods for outputting classification results include:
[0053] Sorting: sort the categories by the predicted probability of the classification model, and give priority to displaying categories with higher predicted probabilities.
[0054] For example: “Science and technology (90%) > society (80%) > education (70%)”.
[0055] Tag display: Display classification results in the form of tag clouds, charts, etc., so that users can quickly understand the classification situation.
[0056] For example: display keywords such as "artificial intelligence" and "technological development" in different sizes and colors on the page.
[0057] Visualization, through text summarization and keyword extraction, the classification results are displayed in the form of charts.
[0058] For example, a traditional bar chart can be used to show the distribution of the number of texts in each category. With an interactive timeline chart, users can view the changes in classification results at different time points by sliding the timeline, thereby obtaining richer information.
[0059] S5. Adjust the classification model based on real-time data and user feedback.
[0060] Adjusting classification models based on real-time data and user feedback includes:
[0061] Real-time data collection: regularly collect new text data to update the training set, and write data crawling scripts in Python to collect new text data.
[0062] For example, by defining crawling functions and storage functions in Python and setting timed tasks, asynchronous IO (such as aiohttp) or distributed crawling frameworks (such as Scrapy) can be used to improve crawling efficiency. This will implement a basic real-time data collection function that can automatically crawl text data from a specified data source according to a preset time interval and save it to a local file for use in subsequent data classification models.
[0063] User feedback: A feedback button is set on the classification result display page for users to evaluate the classification results. Based on user feedback, the parameters of the classification model are adjusted to guide the optimization of the classification model.
[0064] Classification model adjustment: add the collected new data to the training set and retrain the classification model to adapt to the changes in the data.
[0065] This embodiment performs semantic analysis through deep learning to extract deep semantic information and contextual relationships of text data, which can greatly improve the accuracy of classification results. It can also collect data and user feedback in real time and dynamically adjust the classification model algorithm to ensure that the model always has high classification performance.
[0066] This embodiment uses Python general programming language to write data crawling scripts, which reduces the difficulty of implementation and makes the method easy to apply to different scenarios and data sets of different sizes; the processing quality of text data is improved through preprocessing techniques such as word segmentation, stop word removal and part-of-speech tagging, which can lay the foundation for subsequent deep learning analysis.
[0067] This embodiment can automate the entire classification process, from data preprocessing to model training and classification, greatly reducing manual intervention and improving data processing efficiency.
[0068] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.
Claims
1. A data classification method based on semantic understanding, characterized in that: The method comprises: Preprocess the input text data; Use deep learning technology to perform semantic analysis, model the deep semantic information and contextual relationships of the text, and extract feature representations of the text; Combine the semantic analysis results to build a classification model; Input the text to be classified into the classification model and output the classification result; Adjust classification models based on real-time data and user feedback.
2. A data classification method based on semantic understanding as claimed in claim 1, characterized in that: Preprocessing of input text data includes word segmentation, stop word removal and part-of-speech tagging.
3. A data classification method based on semantic understanding as claimed in claim 1, characterized in that: The deep learning technology includes neural network, transfer model and large model technology.
4. A data classification method based on semantic understanding as claimed in claim 1, characterized in that: The method for constructing the classification model includes: Supervised learning method: input the semantically analyzed text features into the SVM model, and train the SVM model using labeled training data to obtain an SVM model that can classify text; The unsupervised learning method uses K-means cluster analysis to classify the text. The text features after semantic analysis are input into the K-means algorithm to classify the text.
5. A data classification method based on semantic understanding as claimed in claim 1, characterized in that: Methods for outputting classification results include: Sort the categories by the predicted probability of the classification model, and give priority to displaying categories with higher predicted probabilities; Through text summarization and keyword extraction, the classification results are displayed in the form of charts.
6. A data classification method based on semantic understanding as claimed in claim 1, characterized in that: Adjusting classification models based on real-time data and user feedback includes: Real-time data collection: regularly collect new text data to update the training set, and collect new text data by writing data crawling scripts in Python; User feedback: set a feedback button on the classification result display page so that users can evaluate the classification results. Based on user feedback, adjust the parameters of the classification model to guide the optimization of the classification model. Classification model adjustment: add the collected new data to the training set and retrain the classification model to adapt to the changes in the data.