Text classification optimization method based on social appeal data based on incremental pre-training

Abstract

The present invention discloses a text classification optimization method based on incremental pre-trained social appeal data. The method collects social appeal data from various social appeal platforms, presenting the social appeal data in text format. The collected social appeal data is pre-processed. After pre-processing, new words are mined using an improved tree-based word mining algorithm, segmented, and then keywords are extracted using a term frequency inverse document frequency (TF-IDF) algorithm. The keyword dataset is expanded using GPT-based data augmentation technology. A pre-trained BERT model is input and an incremental pre-training strategy is selected to train the pre-trained BERT model. The text information classification of the social appeal data is implemented using a text classification algorithm and a named entity recognition algorithm based on the trained BERT model. The present invention aims to address social appeals and optimize text classification functions in the field of social appeals, and to effectively extract deep information from social appeal data.

Description

Technical Field

[0001] The present invention relates to text classification technology, and in particular to a text classification optimization method based on incremental pre-training of social appeal data. Background Art

[0002] With the advancement of natural language processing (NLP) technology, pre-trained models such as BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformers), and T5 (Text-to-Text Transfer Transformers) have achieved impressive results on various text processing tasks. These models are pre-trained on large amounts of unlabeled text data and then fine-tuned for specific tasks to achieve reasonable performance. However, when focusing on specific domains, such as social appeals, traditional pre-trained models may not fully capture the details and characteristics of that domain. Summary of the Invention

[0003] In order to address the deficiencies in the prior art, the present invention aims to provide a text classification optimization method based on incremental pre-training of social appeal data.

[0004] To achieve the purpose of the present invention, the technical solution adopted by the present invention is:

[0005] A text classification optimization method based on incremental pre-training of social appeal data, comprising the following steps:

[0006] (1) Collect social demand data from various social demand platforms and present the social demand data in text format;

[0007] (2) Preprocessing the collected social appeal data;

[0008] (3) After preprocessing, the improved tree-based word mining algorithm is used to mine new words, segment them, and then the term frequency-inverse document frequency TF-IDF algorithm is used to extract keywords;

[0009] (4) Expand keyword datasets using GPT-based data enhancement technology;

[0010] (5) Input the pre-trained BERT model and select the incremental pre-training strategy to train the pre-trained BERT model;

[0011] (6) Use the text classification algorithm and named entity recognition algorithm based on the trained BERT model to realize the text information classification of social appeal data.

[0012] Furthermore, in step (2), the social appeal data preprocessing includes: removing irrelevant symbols, special characters and noise information, and converting all texts into a unified format. Then, a preset stop word library is used to delete words that have no practical significance for text analysis.

[0013] Furthermore, in step (3), the improved tree-based word mining algorithm identifies high-frequency words that show strong contextual relevance in text data, relying on two key statistical indicators: mutual information (MI) and entropy.

[0014] Furthermore, in step (3), the enhanced dictionary tree structure takes Chinese characters as the main nodes of the tree, and each tree root starts from the first Chinese character of the Chinese vocabulary to form a new tree root, and the subtrees under each root correspond to all potential words starting with the specific Chinese character.

[0015] Furthermore, in step (3), the term frequency inverse document frequency algorithm is used to measure the importance of each word in the text, where the term frequency TF represents the frequency of occurrence of a word in a specific document, and the inverse document frequency IDF measures the rarity of the word in the entire document collection;

[0016] The higher the TF-IDF value, the more important the word is in distinguishing documents. Based on the TF-IDF value of each word, the most representative terms in each document are extracted to generate a keyword list for the document.

[0017] Furthermore, in step (4), the Tongyi Qianwen model is used for data enhancement to generate diverse texts related to the category based on the input. The generation parameters are adjusted to ensure that the generated text matches the original category features. The generated text will be merged with the original dataset after strict screening to form a balanced dataset.

[0018] Furthermore, in step (5), the pre-trained BERT model includes two tasks: mask language modeling (MLM) and next sentence prediction (NSP).

[0019] Masked Language Modeling (MLM) randomly masks some tokens in the input sentence, predicts the masked tokens through a pre-trained model, and trains the model to recognize incomplete text.

[0020] Next Sentence Prediction (NSP) provides two sentences to the pre-trained model and determines whether the second sentence is a direct successor of the first.

[0021] Furthermore, in step (6), the text classification algorithm based on the BERT model has the following specific steps: using the word embedding table of the BERT model to map each character of the text to a vector representation, and generating a word-level vector representation after combining the context. BERT introduces a [CLS] tag at the beginning of each text segment, and uses the output vector corresponding to the tag as the global semantic representation of the text.

[0022] Furthermore, in step (6), the named entity recognition algorithm based on the BERT model has the following specific steps: using BERT or BERT-bi-LSTM as the underlying text feature encoder and using CRF for entity label prediction.

[0023] The beneficial effect of the present invention is that, compared with the existing technology, the present invention uses specific domain data for model retraining to improve the domain adaptability of the model compared with the traditional pre-training model; introduces a new word mining algorithm to optimize the classification and recognition of data text, uses GPT-based data enhancement technology for data enhancement, and finally adopts text classification algorithm and named entity recognition algorithm to fully extract deep information in social demand data.

[0024] This paper adopts incremental pre-training technology, combined with an improved tree-based word mining algorithm and a text classification optimization strategy based on GPT data enhancement technology, aiming to solve social demands in urban governance and optimize the text classification function in the field of social demands, and realize the effective extraction of deep information of social demand data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flowchart of the text classification optimization method based on incremental pre-training of social appeal data;

[0026] Figure 2 This is a flowchart of the word mining algorithm based on the improved tree;

[0027] Figure 3 It is a schematic diagram of new word discovery;

[0028] Figure 4 This is a flowchart of the text classification algorithm based on the BERT model;

[0029] Figure 5 This is a flowchart of the named entity recognition algorithm based on the BERT model;

[0030] Figure 6 It is a flowchart for introducing named entity recognition algorithm to identify missing addresses. DETAILED DESCRIPTION

[0031] The following embodiments are only used to more clearly illustrate the technical solution of the present invention and are not intended to limit the scope of protection of this application.

[0032] like Figure 1 As shown, the text classification optimization method based on incremental pre-training of social appeal data described in the present invention is used to solve social appeals in urban governance and optimize the text classification function of social appeal data. Specifically, it includes the following steps:

[0033] (1) Collect social demand data from various social demand platforms, including government hotlines, mayor mailboxes, and other social demand platforms. Social demand data are presented in text format, including titles and content. The titles summarize the social demands, while the content provides detailed information on these social demands.

[0034] (2) Preprocessing the collected social appeal data;

[0035] After collecting and loading the data, the text is cleaned, including removing irrelevant symbols, special characters, and noise. This process also involves converting all text to a uniform format, such as lowercase, to ensure consistency in subsequent processing. Next, a pre-defined stop word library is used to remove words that are not meaningful for text analysis, thereby reducing data noise and improving analysis efficiency.

[0036] (3) After preprocessing, the improved tree-based word mining algorithm is used to mine new words, segment them, and then the term frequency-inverse document frequency TF-IDF algorithm is used to extract keywords;

[0037] like Figure 2 As shown, the present invention improves the accuracy of feature extraction based on social appeal data by improving the new word discovery algorithm. First, social appeal data is extracted from the business system database and preprocessed as raw data. The preprocessing stage includes the introduction of an improved new word discovery algorithm. The core of this algorithm is to construct a dynamically updated new social appeal vocabulary by identifying and recording new, high-frequency words that appear in social media or business systems. The construction of this new vocabulary helps improve the accuracy of word segmentation and avoids the omission of new words in the traditional word segmentation process.

[0038] After the text is processed, it needs to be segmented, that is, the text is divided into vocabulary units to prepare for the subsequent word frequency calculation.

[0039] On this basis, the TF-IDF algorithm is applied to the preprocessed social appeal text data for keyword extraction. By leveraging the optimized segmentation results from the new vocabulary, the TF-IDF algorithm can more accurately extract key terms from the text, thereby obtaining more representative features of social appeal text. These features can then be fed into the incremental pre-training model for learning, further enhancing the model's performance when processing social appeal text in specific domains. Ultimately, the incremental pre-trained model can more accurately identify the features of social appeals and provide stronger support for subsequent tasks such as text classification and trend analysis.

[0040] The Term Frequency-Inverse Document Frequency (TF-IDF) algorithm is used to measure the importance of each word in a document. Term Frequency (TF) indicates how often a word appears in a specific document, while Inverse Document Frequency (IDF) measures the rarity of that word across the entire document collection. A higher TF-IDF value indicates a greater role for the word in distinguishing documents. Finally, based on the TF-IDF value of each word, the most representative terms in each document are extracted to generate a keyword list for that document for further analysis or application.

[0041] The social appeal data preprocessing operation is to address the input limitations of the BERT model. The BERT model only accepts input of up to 512 characters. The input text data is crucial for the training of subsequent models.

[0042] This paper introduces an improved tree-based new word mining algorithm. The new word discovery algorithm identifies high-frequency words that show strong contextual relevance in text data. This algorithm relies on two key statistical indicators: mutual information (MI) and entropy.

[0043] Mutual information evaluates the degree of association between words, and the expression is as follows:

[0044] (1)

[0045] in, is the probability of words appearing at the same time; and Represents the probability of a single word appearing alone.

[0046] Entropy quantifies the diversity of a word's appearance in different contexts and is calculated as follows:

[0047] (2)

[0048] Among them, given a word w, Indicates the probability of a specific situation c occurring. Through the above, new words can be effectively mined from large amounts of text data.

[0049] Aiming at the unique properties of Chinese new word discovery, this paper proposes an enhanced dictionary tree structure. Chinese characters are used as the main nodes of the tree. Each tree root starts from the first Chinese character of a Chinese vocabulary, forming a new tree root. The subtrees under each root correspond to all potential words starting with this specific Chinese character. This effectively reduces the average depth of nodes in the dictionary tree and improves retrieval efficiency. Figure 3 As shown, the improved schematic diagram of new word discovery.

[0050] (4) Expand keyword datasets using GPT-based data enhancement technology;

[0051] The present invention uses the Tongyi Qianwen large model for data enhancement, aiming to solve the problem of data imbalance. First, by analyzing the data set, the categories with fewer samples are determined, and representative samples are selected from these categories as generated input prompts. The Tongyi Qianwen model can generate diverse text related to the category based on the input, thereby improving data coverage. By adjusting the generation parameters (such as temperature, Top-k and Top-p sampling), it can be ensured that the generated text not only matches the original category features, but also has sufficient diversity to avoid duplication. The generated text is strictly screened to exclude poor quality or irrelevant content. The screened enhanced data will be merged with the original data set to form a more balanced data set. Finally, this enhanced balanced data set is used to retrain the model, which significantly improves its recognition ability on a few categories, effectively alleviates the long-tail effect, and enhances the model's generalization ability and classification accuracy, especially in unbalanced classification tasks.

[0052] (5) Input the pre-trained model and select the incremental pre-training strategy to train the existing pre-trained model and integrate the characteristics of social demand data;

[0053] Incremental pre-training methods build on existing language models by further training them with domain-specific data, allowing the model to gradually integrate and adapt to the characteristics of a specific domain. When processing social demand data, incremental pre-training can enhance the model's understanding of domain terminology and specific expressions, and capture complex contextual relationships. By continuously introducing new data, the model can continuously update its understanding of social demands and adapt to the dynamically changing social environment, thereby improving its performance in text classification.

[0054] The BERT model is selected as the pre-training basic model. The social appeal incremental pre-training model includes two tasks: MLM (masked language modeling) and NSP (next sentence prediction). MLM is used to prevent text missing, and NSP is used to understand the relationship between the previous and next contexts.

[0055] MLM (Masked Language Modeling) randomly masks some tokens in the input sentence, predicts the masked tokens through a pre-trained model, and trains the model to recognize incomplete text, achieving true bidirectional contextual dependency.

[0056] NSP (next sentence prediction) provides two sentences to the pre-trained model and determines whether the second sentence is a direct successor of the first sentence, helping to understand the semantic connection and coherence between sentences.

[0057] (6) Use the text classification algorithm and named entity recognition algorithm based on the trained BERT model to realize the text information classification of social appeal data.

[0058] like Figure 4As shown in the figure, the text classification algorithm based on the BERT model has the following specific steps: using the word embedding table of the BERT model to map each character of the text to a vector representation, and then generating a word-level vector representation after combining the context. BERT introduces a [CLS] tag at the beginning of each text segment, and uses the output vector corresponding to this tag as the global semantic representation of the text, which facilitates text classification and improves text classification accuracy.

[0059] like Figure 5 The following figure shows a named entity recognition algorithm based on the BERT model. The specific steps are: using BERT or BERT-Bit-LSTM as the underlying text feature encoder and using CRF for entity label prediction. BL, IL, BO, IO, and O are labels. The BL and IL labels represent locations, indicating the beginning and end of a location. The BO and IO labels represent organizations, indicating the beginning and end of an organization.

[0060] It's important to note that CRF uses label sequences as prediction targets and can impose constraints on the final predicted label sequence based on logits to ensure that the predicted entity label sequence is valid. These constraints are automatically learned by the CRF layer from the training dataset during training. Because many label paths are incorrectly predicted, the correlation between labels and the characteristics of the labels are crucial in entity label prediction. CRF learns label transformation relationships and constraints from the data, helping the model select the correct and reasonable entity label sequence, reducing the prediction of invalid entity label sequences and improving the accuracy of named entity recognition.

[0061] like Figure 6 As shown, the named entity recognition algorithm introduced in this invention is mainly used to solve the problem of users not filling in their addresses when submitting social appeals on Weibo, WeChat and APP platforms. The missing address will significantly affect the classification accuracy of social appeals, and thus affect the response efficiency of relevant departments. The specific process is as follows:

[0062] Data source: Social appeal text data comes from multiple platforms, including Weibo, WeChat, and apps. Users can submit appeals through these channels.

[0063] Address missing check: The system first analyzes the social appeal text to determine whether there is address information. If the address is missing, the system will trigger the address completion process.

[0064] Named Entity Recognition Algorithm: When a missing address is detected, the system will call the Named Entity Recognition (NER) algorithm to process the social appeal text and automatically identify the geographical location-related named entities (such as place names, street names, building names, etc.) that may be contained therein, thereby extracting potential address information.

[0065] Geographic Information System (GIS) Matching: Address entities extracted through NER are matched against a Geographic Information System (GIS) database to further confirm the accuracy and completeness of the address. The system uses GIS geographic information to complete and verify the identified address, ensuring the accurate location of the request.

[0066] Optimized text classification model: After confirming the address information, the system inputs the completed social appeal text into an optimized text classification model for classification. This model, trained for different social appeal scenarios, accurately assigns appeals to relevant departments based on their content, address information, and other characteristics.

[0067] Classification and allocation: Based on the classification results, social demands are assigned to corresponding departments (such as Department 1, Department 2, ..., Department n) to ensure that the demands can be handled quickly and accurately.

[0068] This process significantly improves the classification accuracy in cases of missing addresses through the introduction of NER algorithms and GIS systems, ensuring that social demands can be accurately classified and processed, while also improving the response efficiency of government departments.

[0069] The beneficial effect of the present invention is that, compared with the existing technology, the application of the method of the present invention provides good data processing and text output for optimizing and improving the social appeal scheduling process, demonstrates the potential of the BERT model in text processing and its application value in processing social appeals, promotes the advancement of text classification technology through targeted training and model optimization of social appeal data, and provides new perspectives and tools for research and practice in the field of social appeals.

[0070] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation plans of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, and is not a limitation on the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. A text classification optimization method based on incremental pre-training of social appeal data, characterized in that: Including steps: (1) Collect social demand data from various social demand platforms and present the social demand data in text format; (2) Preprocessing the collected social appeal data; (3) After preprocessing, the improved tree-based word mining algorithm is used to mine new words, segment them, and then the term frequency-inverse document frequency TF-IDF algorithm is used to extract keywords; The improved tree-based word mining algorithm identifies high-frequency words that exhibit strong contextual relevance in text data, relying on two key statistical indicators: mutual information (MI) and entropy. The enhanced dictionary tree structure uses Chinese characters as the main nodes of the tree. Each tree root starts from the first Chinese character in the Chinese vocabulary, forming a new tree root. The subtrees under each root correspond to all potential words that begin with that specific Chinese character. The term frequency-inverse document frequency algorithm is used to measure the importance of each word in the text. The term frequency (TF) indicates how often a word appears in a specific document, while the inverse document frequency (IDF) measures the rarity of the word in the entire document collection. The higher the TF-IDF value, the greater the role of the word in distinguishing documents. Based on the TF-IDF value of each word, the most representative terms in each document are extracted to generate a keyword list for the document. (4) Expand keyword datasets using GPT-based data enhancement technology; We use the Tongyi Qianwen model for data augmentation, generating diverse, category-related text based on the input. We adjust generation parameters to ensure that the generated text matches the original category features. After rigorous screening, the generated text is merged with the original dataset to form a balanced dataset. (5) Input the pre-trained BERT model and select the incremental pre-training strategy to train the pre-trained BERT model; The pre-trained BERT model includes two tasks: mask language modeling (MLM) and next sentence prediction (NSP). Masked Language Modeling (MLM) randomly masks some tokens in the input sentence, predicts the masked tokens through a pre-trained model, and trains the model to recognize incomplete text. Next sentence prediction (NSP) involves providing two sentences to the pre-trained model and determining whether the second sentence is a direct successor of the first. (6) Use the text classification algorithm and named entity recognition algorithm based on the trained BERT model to realize the text information classification of social appeal data.

2. The text classification optimization method based on incremental pre-training of social appeal data according to claim 1 is characterized in that In step (2), the social appeal data preprocessing includes: removing irrelevant symbols, special characters and noise information, and converting all texts into a unified format. Then, using the preset stop word library to delete words that have no practical significance for text analysis.

3. The text classification optimization method based on incremental pre-training of social appeal data according to claim 1 is characterized in that In step (6), the text classification algorithm based on the BERT model has the following specific steps: using the word embedding table of the BERT model to map each character of the text to a vector representation, and generating a word-level vector representation after combining the context. BERT introduces a [CLS] tag at the beginning of each text segment, and uses the output vector corresponding to the tag as the global semantic representation of the text.

4. The text classification optimization method based on incremental pre-training of social appeal data according to claim 1, characterized in that: In step (6), the named entity recognition algorithm based on the BERT model has the following specific steps: using BERT or BERT-bi-LSTM as the underlying text feature encoder and using CRF for entity label prediction.

Images