Window enhancement feature fusion text classification method based on word frequency
By performing frequency analysis of named entities and combining with improved multi-channel convolutional neural networks, the problem of neglected semantic information and entity roles in traditional text classification methods is solved, and the accuracy and robustness of text classification are improved.
Patent Information
- Application Number
- CN202510597032.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-09
- Publication Date
- 2025-08-26
AI Technical Summary
When facing complex text information, traditional text classification methods ignore the key roles of semantic information and specific entities, resulting in insufficient classification accuracy.
By extracting the frequency of the named entity, dividing it into high-frequency words and low-frequency words, different processing strategies are used to generate feature vectors, and text classification is combined with improved multi-channel convolutional neural networks, and a focus loss function is introduced to enhance the sensitivity of the model.
Improve the accuracy and robustness of text classification, especially when dealing with complex text, key features can be better extracted and overcome the neglect of low-frequency information in traditional methods.
Smart Images
Figure CN120541224A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a word frequency-based window-enhanced feature fusion text classification method, belonging to the technical field of Chinese text classification. Background Art
[0002] With the rapid development of information technology, text classification technology is increasingly being applied in various fields, such as news classification, sentiment analysis, social media analysis, advertising recommendations, and automated question answering. However, traditional text classification methods typically rely on feature extraction based on word frequency, such as TF-IDF (Term Frequency-Inverse Document Frequency) and the bag-of-words model. While effective in many scenarios, these methods have limitations. Especially when faced with complex text, these methods often overlook the semantic information and the critical role of specific entities in the text, resulting in insufficient classification accuracy.
[0003] Named entity recognition (NER), a key task in natural language processing, aims to identify meaningful entities (such as names of people, places, organizations, times, dates, etc.) from text. These entities often carry a wealth of critical information within the text. Existing text classification methods, particularly those based on deep learning, typically use the original text's vocabulary as input, ignoring the deeper semantic role of entity information and failing to fully explore the impact of entities on classification results.
[0004] To improve the performance of text classification, some studies in recent years have proposed methods that combine named entity information with the original text content. These methods extract named entities from the text and merge them with the original content to enhance the richness and semantic expression of the text representation. This method can better capture the key information in the text, especially in certain application scenarios, where the presence of named entities plays a decisive role in text classification. However, entity processing in existing technologies is often relatively simple, mainly extracting all entities and directly classifying them. However, these methods fail to fully consider the frequency distribution of entities in the text and ignore the potential value of low-frequency words in specific scenarios.
[0005] Furthermore, current text classification methods mostly rely on traditional deep learning models, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). These methods can automatically extract features from text and perform classification. However, when faced with long or complex texts, how to effectively combine named entities with text content and fully tap the potential of entity information remains an urgent problem. Summary of the Invention
[0006] The technical problem to be solved by the present invention is that traditional text classification methods usually rely on feature extraction based on word frequency. Especially when faced with complex text information, they often ignore the key role of semantic information and specific entities in the text, resulting in insufficient classification accuracy.
[0007] The technical solution of the present invention is: a window-enhanced feature fusion text classification method based on word frequency, the method comprising:
[0008] Step 1: Extract original text data from various text files and perform preprocessing;
[0009] Step 2: Perform named entity recognition on the preprocessed text data to extract named entities in the text. The named entities are entity information with clear references, count the frequency of each entity, set the entity frequency threshold according to the classification rules, and classify the entities into high-frequency words and low-frequency words according to the frequency threshold;
[0010] Step 3: For high-frequency words, use the pre-trained word vector model and TF-IDF weighting to generate word vectors for high-frequency words. For low-frequency words, use a sliding window mechanism to extract contextual vocabulary and vectorize it to generate enhanced features for low-frequency words. The word vectors for high-frequency words and the enhanced features for low-frequency words are fused into a new feature vector through weighted averaging.
[0011] Step 4: Based on the fused new feature vector, an improved multi-channel convolutional neural network is constructed. The improved multi-channel convolutional neural network integrates the attention mechanism and introduces a focal loss function to achieve text classification.
[0012] The step 1 is specifically as follows:
[0013] Extract tags, titles, and content, remove invalid data, and convert them into a standard JSON file; load a stop word list;
[0014] Define a regular expression that matches all non-alphanumeric characters, spaces, and Chinese characters.
[0015] The specific classification rules are:
[0016] Using the percentile method, a specific percentile in the data set is selected as the threshold, where the formula for calculating the percentile position is:
[0017]
[0018] Where n is the total number of entities, P is the set percentile, and K is the index of the position;
[0019] Use f(e k ) as percentiles:
[0020] T P =f(e k )
[0021] Among them, T P represents the percentile threshold of frequency;
[0022] According to the percentile threshold T P , divide named entities into two categories: high-frequency words and low-frequency words. If an entity e i The frequency f(e i ) is greater than or equal to T P , then the entity is a high-frequency word; if an entity e i The frequency f(e i ) is less than T P , then the entity is a high-frequency word.
[0023] The step three is specifically as follows:
[0024] Different strategies are used for feature modeling and fusion for high-frequency words and low-frequency words respectively:
[0025] For high-frequency words, a pre-trained word vector model is used to directly obtain the word vector of the high-frequency words, and weighted processing is performed in combination with the TF-IDF weight so that the classification results are not affected by common words.
[0026] For low-frequency words, a sliding window mechanism is used to extract the contextual vocabulary at the location of the low-frequency word. The contextual vocabulary within the window is vectorized, and the contextual word vectors of the low-frequency words are aggregated to generate an enhanced feature vector for the low-frequency word.
[0027] The low-frequency word features enhanced by the window mechanism and the high-frequency word features directly represented by the word vector are fused, and the word vectors of high-frequency words and the enhanced features of low-frequency words are fused into a new feature vector through weighted averaging.
[0028] The step 4 is specifically as follows:
[0029] Read the fused feature vector and corresponding label data;
[0030] Encode the label data to facilitate multi-classification training of the model;
[0031] Constructing an improved multi-channel convolutional neural network, wherein the improved multi-channel convolutional neural network includes an input layer, an embedding vector input map, multiple convolutional layers with different convolution kernel sizes, a global maximum pooling layer, a dropout regularization layer, an attention mechanism module, a fully connected layer, and an output layer;
[0032] The focal loss function is introduced to improve the convolutional neural network to enhance the sensitivity to difficult-to-classify samples and ensure the balance of labels;
[0033] Perform model iteration and finally output multi-classification results through the Softmax activation function;
[0034] This paper combines the window enhancement mechanism to effectively improve the robustness and generalization ability of the model in text classification tasks.
[0035] The beneficial effects of the present invention are as follows: by combining named entity information and text content, the present invention comprehensively considers the overall information and important entities of the text, thereby improving the accuracy and robustness of text classification. In particular, the present method treats high-frequency and low-frequency words differently through frequency analysis of named entities, and processes them in different ways, thus avoiding the important problem of traditional methods ignoring low-frequency words. In addition, the present invention further improves the effect of feature extraction and classification through an improved multi-channel convolutional neural network, so that the method exhibits better classification performance when processing complex texts. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 Schematic diagram of the implementation process of the present invention;
[0037] Figure 2 This is a schematic diagram of the feature fusion process implemented in the present invention;
[0038] Figure 3 This is a schematic diagram of the classification network model implemented in the present invention. DETAILED DESCRIPTION
[0039] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0040] Example 1: Figure 1 As shown in Figure 1, a window-enhanced feature fusion text classification method based on word frequency is shown in Figure 1. The specific steps are as follows:
[0041] Step 1: Extract original text data from various text files and perform preprocessing.
[0042] Specifically, we extract raw text data from various text files (such as news, microblogs, comments, etc.). For each text record, we extract its title, text content, and corresponding tags. Then, we preprocess the text data, including:
[0043] Remove invalid data (such as blank lines, incorrectly formatted text, etc.).
[0044] Use regular expressions to match and remove non-alphanumeric characters, spaces, and Chinese characters, and clean up Chinese and English punctuation marks.
[0045] Convert text into a standard JSON file. Each record contains title, content, and tag fields, providing a unified data interface for subsequent operations.
[0046] Preload a stop word list and ignore meaningless words (such as "of", "already", etc.) to improve text processing efficiency. The deletion of stop words can be completed through the text processing library in Python.
[0047] Step 2: Perform named entity recognition on the preprocessed text data, extract the named entities in the text. The named entities are entity information with clear references. Count the occurrence frequency of each entity, set an entity frequency threshold according to the grading rules, and classify the entities into high-frequency words and low-frequency words according to the frequency threshold. [[ID=~]]
[0048] Specifically, for the preprocessed text data, use the LAC model to perform word segmentation and named entity recognition on the text. The LAC model can recognize entities such as person names, place names, organization names, times, dates, currencies, etc. in the text, and extract all named entity information. Its expression results are as follows:
[0049] In the text "Li Hua will go to Beijing on May 1, 2025", the LAC model will recognize "Li Hua" as a person name, "May 1, 2025" as a date, and "Beijing" as a place name.
[0050] All named entities extracted from the text will be saved in an entity list for subsequent analysis.
[0051] Specifically, through the extracted named entity list, for each text record, count the named entities it contains, and record the occurrence frequencies of these entities in different text categories. For example, in the news classification task, the frequency of "Beijing" as a place name may be counted, or the number of occurrences of "Li Hua" as a person name.
[0052] According to the label of each text (such as "sports", "entertainment"), count the frequency of each entity in each category. For example, in the "sports" category, the frequency of "XX" as a player may be higher, while in the "entertainment" category, the occurrence frequency of "XX" is lower.
[0053] Specifically, clarify high-frequency words and low-frequency words through the grading rules. The grading rule is to use the percentile method and select a specific percentile in the dataset as the threshold. The formula for calculating the percentile position is:
[0054]
[0055] where n is the total number of entities, P is the set percentile, and K is the index of this position;
[0056] Use f(e k ) as the percentile:
[0057] T P =f(e k )
[0058] Among them, T P represents the percentile threshold of frequency;
[0059] According to the percentile threshold T P , divide named entities into two categories: high-frequency words and low-frequency words. If an entity e i The frequency f(e i ) is greater than or equal to T P , then the entity is a high-frequency word; if an entity e i The frequency f(e i ) is less than T P , then the entity is a high-frequency word.
[0060] Furthermore, high-frequency words are usually more representative of the topic of the text and therefore have stronger classification information. Although low-frequency words appear less frequently in the training set, they may still contain valuable information in certain specific categories or contexts.
[0061] Step 3: For high-frequency words, use the pre-trained word vector model and TF-IDF weighted processing to generate word vectors for high-frequency words; for low-frequency words, use the sliding window mechanism to extract contextual vocabulary and vectorize them to generate enhanced features for low-frequency words; the word vectors of high-frequency words and the enhanced features of low-frequency words are fused into a new feature vector through weighted averaging.
[0062] Specifically, feature fusion is the core step of the present invention, which improves the accuracy of the text classification model by combining high-frequency and low-frequency named entity information. The feature fusion process is as follows: Figure 2 shown.
[0063] Furthermore, the text data processed in this embodiment is news articles, which contain different types of named entities, such as names of people, places, organizations, and time periods. In step 2, these entities were divided into high-frequency and low-frequency words based on their frequency information. Next, we will describe the processing of high-frequency and low-frequency words, respectively, and explain how to perform feature fusion for them.
[0064] For high-frequency words, pre-trained word vector models (such as Word2Vec or BERT) are directly used to obtain word vector representations of these words. Pre-trained word vectors can accurately capture the semantic information of high-frequency words and ensure that their expression in text features is not diluted. In order to reduce the interference of common high-frequency words on the classification task, high-frequency words are processed in combination with TF-IDF weighting. The specific approach is: calculate the TF-IDF value of each high-frequency word and adjust its weight according to the relative importance of the word frequency. In this way, frequently appearing high-frequency words will be weighted according to their relative importance in the document, avoiding too much attention to common but less informative words. The expression results are as follows:
[0065] For example, "XX Company" is a high-frequency word that appears in multiple news articles. Using the BERT or Word2Vec model, we can obtain a word vector representation for "XX Company," such as [V_apple]. Using TF-IDF to weight "XX Company," we dynamically adjust its importance across different articles based on its relative frequency within the article, resulting in a word vector like [V_apple_weighted]. If "XX Company" appears more frequently in a particular news article, its weight will be lowered.
[0066] For low-frequency words, although they appear less frequently in the dataset, they may have potential classification value in specific categories or contexts. To enhance the semantic representation of low-frequency words, a sliding window mechanism is used to extract contextual information of low-frequency words. Specifically, several words before and after the low-frequency word (for example, three words before and after each) are taken as context, and this contextual information is used to enrich the semantic expression of the low-frequency word. These context vectors can be aggregated with the word vector of the low-frequency word itself to generate an enhanced feature vector for the low-frequency word. The expression results are as follows:
[0067] If the context of the low-frequency word "quantum computing" is "breakthrough" and "technology," we can merge the word vectors of "breakthrough" and "technology" with the word vector of "quantum computing" to generate a new feature vector [V_quantum_enhanced]. This way, the low-frequency word not only contains its own semantic information, but also combines the semantics of the context, enhancing its representational power.
[0068] Next, we combine the enhanced features of low-frequency words with the features of high-frequency words using a weighted average method to generate a new feature vector. When performing weighted averaging, the weights of high-frequency and low-frequency words are adjusted based on their relative importance. The concatenation method directly concatenates the feature vectors of the two words to form a single long vector. The resulting representation is as follows:
[0069] The feature vectors of high-frequency words [V_apple_weighted] and the enhanced feature vectors of low-frequency words [V_quantum_enhanced] have been processed.
[0070] For high-frequency words and low-frequency words, weights are assigned to them according to their importance. In this example, the weight of high-frequency words is 0.7, and the weight of low-frequency words is 0.3:
[0071] V_combined=0.7×V_apple_weighted+0.3×V_quantum_enhanced
[0072] V_combined is the final fused feature vector, which combines the feature information of the high-frequency word "XX Company" and the low-frequency word "quantum computing". This vector will be used as the input feature for model training.
[0073] Step 4: Based on the fused new feature vector, an improved multi-channel convolutional neural network is constructed. The improved multi-channel convolutional neural network integrates the attention mechanism and introduces a focal loss function to achieve text classification.
[0074] Specifically, in step 3, the feature vectors of high-frequency and low-frequency words in the text are extracted and integrated. The labels are encoded, converting the text labels into digital format. This embodiment uses digital encoding, so that each label corresponds to a unique number (e.g., "politics" corresponds to 0, "entertainment" corresponds to 1, and "sports" corresponds to 2).
[0075] The processed text data and labels are divided into training and test sets. Using the train_test_split method, the data is allocated in a ratio of 80% training set and 20% test set to ensure that the model can be fully learned and verified during the training process.
[0076] Construct an improved multi-channel convolutional neural network to extract text features and perform classification. The process is as follows Figure 3 shown.
[0077] Input layer: Receives the fused feature vector extracted in step 3. Assume that the feature vector of each text is (seq_len, feature_dim), where seq_len is the maximum length of the text and feature_dim is the feature dimension of each word.
[0078] Multi-channel convolution layer: Multiple convolution kernel sizes (such as 2, 3, and 4) are used to extract n-gram features of different granularities in parallel. The multi-channel convolution structure can effectively capture information of different scales in the text.
[0079] Pooling layer: Each convolutional layer is followed by a pooling layer, which reduces the dimension through max pooling and extracts the most important features.
[0080] Dropout layer: To prevent overfitting, the Dropout layer is used. By randomly discarding some neuron connections, the generalization ability of the model can be improved.
[0081] Attention mechanism layer: Introducing the attention mechanism to enhance the model's focus on key information, automatically learning and weighting important parts. The attention mechanism helps the model find important words in the text and assign them greater weight.
[0082] Fully connected layer: After convolution and pooling operations, a fully connected layer is used to process the extracted features to generate a higher-level abstraction. The output dimension of the fully connected layer can be adjusted through experimentation.
[0083] Output layer: Based on the number of categories in the classification task, the Softmax activation function is used for multi-classification prediction and the probability distribution of each category is output. The calculation method is as follows:
[0084] For multi-class classification problems, assume that the output of the model is Z = (z1, z2, ..., z C ), that is, the model's predicted value is a vector of length C, where C is the number of categories. The Softmax function converts this vector into the probability of each category. Its calculation formula is as follows:
[0085]
[0086] in, is the predicted probability of category c; Z c is the original output value of the model for category c (the output without activation function, usually logits); e is the base of the natural logarithm.
[0087] After building the network structure, use the optimizer and loss function to compile the model:
[0088] This embodiment selects the Adam optimizer (Adam) for training. The optimizer has the characteristic of adaptive learning rate, which helps to accelerate the training process and improve the training effect.
[0089] The loss function introduces the focus loss function, which is suitable for multi-classification tasks with integer labels. It is very effective in solving the problem of class imbalance and calculates the error between the model prediction result and the true label. The prediction probability of the model for category c obtained by the Softmax function
[0090]
[0091] Among them, L is the loss value, γ is the adjustment factor, which controls the penalty for difficult-to-classify samples. In this embodiment, it is set to 2, and y c Is the one-hot encoding value of the actual label, indicating the label of the true category c. If the sample belongs to category c, then y c =1, otherwise y c =0.
[0092] Furthermore, the model continuously adjusts parameters on the training set to achieve high performance on the test set. During training, accuracy and loss are primarily used as metrics based on business needs. Evaluation also includes recall and F1 scores for each category.
[0093] To verify the effectiveness of the present invention in text classification, especially its performance in processing semi-structured text, the present invention conducted a series of experiments to compare the performance of traditional classification methods and the model after entity information fusion. The model of the present invention is named the classification model combining frequency feature fusion (Frequency-based Window-enhanced Convolutional Neural Network, F-WCNN) to facilitate subsequent comparison with other models. The experimental design is as follows:
[0094] Dataset: The dataset covers 10 categories, including sports, finance, real estate, education, politics, games, home furnishings, technology, entertainment, and fashion. The original dataset contains 5,000 samples in each category (4,000 in the training set, 500 in the validation set, and 500 in the test set), for a total of 50,000 samples.
[0095] Experimental method: Multiple models were used for comparison, including traditional text classification models (based on TF-IDF and SVM, CNN, and Text CNN). The best model was selected from these models for comparison with the F-WCNN model proposed in this paper.
[0096] Evaluation Metrics: We first conducted a macro comparison of multiple traditional large models, using metrics such as accuracy, minimum loss, and F1 score to assess their classification performance. Finally, we selected the best performing model and compared it with the proposed F-WCNN model, evaluating it using accuracy, precision, recall, and F1 score.
[0097] The experimental results are shown in Tables 1 and 2 below:
[0098] Table 1 Experimental results of four classification models
[0099] Model Accuracy(%) Min-Loss <![CDATA[F1-score(%) <!-- 6 -->]]> TF-IDF 89.26 - 85 Support Vector Machine 92.39 - 91 CNN 96.26 0.15 96 Text CNN 93.49 0.49 93
[0100] According to the comparative experimental results in Table 1, it can be found that the performance of CNN is better than that of several other models. Next, various classification results and indicators of CNN are evaluated and compared in detail with the F-WCNN proposed in this paper.
[0101] Table 2 Classification comparison between CNN and F-WCNN
[0102]
[0103]
[0104] The classification comparison results in Table 2 demonstrate that the proposed F-WCNN outperforms the CNN model. Therefore, the present invention effectively leverages the combination of raw text data and named entity information to improve the accuracy and robustness of text classification. By distinguishing and fusing high-frequency and low-frequency named entities, the model enhances its ability to recognize texts of different categories. This is particularly true when processing complex text data, enabling better extraction of key features and overcoming the neglect of low-frequency information in traditional methods.
[0105] In summary, the embodiments of the present invention have been described above. The above embodiments are merely illustrative and do not limit the scope of protection of the present invention. A person skilled in the art can, based on the spirit of the present invention and without departing from the technical solution of the present invention, make appropriate changes, modifications, and optimizations to the embodiments.
Claims
1. A window-enhanced feature fusion text classification method based on word frequency, characterized in that: The following steps are involved: Step 1: Extract original text data from various text files and perform preprocessing; Step 2: Perform named entity recognition on the preprocessed text data to extract named entities in the text. The named entities are entity information with clear references, count the frequency of each entity, set the entity frequency threshold according to the classification rules, and classify the entities into high-frequency words and low-frequency words according to the frequency threshold; Step 3: For high-frequency words, use the pre-trained word vector model and TF-IDF weighting to generate word vectors for high-frequency words. For low-frequency words, use a sliding window mechanism to extract contextual vocabulary and vectorize it to generate enhanced features for low-frequency words. The word vectors for high-frequency words and the enhanced features for low-frequency words are fused into a new feature vector through weighted averaging. Step 4: Based on the fused new feature vector, an improved multi-channel convolutional neural network is constructed. The improved multi-channel convolutional neural network integrates the attention mechanism and introduces a focal loss function to achieve text classification.
2. A text classification method based on word frequency and window-enhanced feature fusion according to claim 1, characterized in that: The step 1 is specifically as follows: Extract tags, titles, and content, remove invalid data, and convert them into a standard JSON file; Load the stop word list; Define a regular expression that matches all non-alphanumeric characters, spaces, and Chinese characters.
3. The text classification method based on word frequency and window-enhanced feature fusion according to claim 1 is characterized in that: The specific classification rules are: Using the percentile method, a specific percentile in the data set is selected as the threshold, where the formula for calculating the percentile position is: Where n is the total number of entities, P is the set percentile, and K is the index of the position; Use f(e k ) as percentiles: T P =f(e k ) Among them, T P represents the percentile threshold of frequency; According to the percentile threshold T P , divide named entities into two categories: high-frequency words and low-frequency words. If an entity e i The frequency f(e i ) is greater than or equal to T P , then the entity is a high-frequency word; if an entity e i The frequency f(e i ) is less than T P , then the entity is a high-frequency word.
4. The text classification method based on word frequency and window-enhanced feature fusion according to claim 1 is characterized in that: The step three is specifically as follows: Different strategies are used for feature modeling and fusion for high-frequency words and low-frequency words respectively: For high-frequency words, a pre-trained word vector model is used to directly obtain the word vector of the high-frequency words, and weighted processing is performed in combination with the TF-IDF weight so that the classification results are not affected by common words. For low-frequency words, a sliding window mechanism is used to extract the contextual vocabulary at the location of the low-frequency word. The contextual vocabulary within the window is vectorized, and the contextual word vectors of the low-frequency words are aggregated to generate an enhanced feature vector for the low-frequency word. The low-frequency word features enhanced by the window mechanism and the high-frequency word features directly represented by the word vector are fused, and the word vectors of high-frequency words and the enhanced features of low-frequency words are fused into a new feature vector through weighted averaging.
5. The text classification method based on word frequency and window-enhanced feature fusion according to claim 1 is characterized in that: The step 4 is specifically as follows: Read the fused feature vector and corresponding label data; Encode the label data to facilitate multi-classification training of the model; Constructing an improved multi-channel convolutional neural network, wherein the improved multi-channel convolutional neural network includes an input layer, an embedding vector input map, multiple convolutional layers with different convolution kernel sizes, a global maximum pooling layer, a dropout regularization layer, an attention mechanism module, a fully connected layer, and an output layer; The focal loss function is introduced to improve the convolutional neural network to enhance the sensitivity to difficult-to-classify samples and ensure the balance of labels; Perform model iteration and finally output multi-classification results through the Softmax activation function.