Text classification optimization method for social appeal data based on incremental pre-training

Abstract

The invention discloses a text classification optimization method for social appeal data based on incremental pre-training, and the method comprises the steps: collecting the social appeal data of each social appeal platform, and presenting the social appeal data in a text format; preprocessing the collected social appeal data; after preprocessing, mining new words based on a word mining algorithm of an improved tree, carrying out word segmentation processing on the new words, and then extracting keywords by utilizing a term frequency inverse document frequency TF-IDF algorithm; expanding the keyword data set based on a GPT data enhancement technology; inputting a pre-training BERT model, and selecting an increment pre-training strategy to train the pre-training BERT model; and adopting a text classification algorithm and a named entity recognition algorithm based on the trained BERT model to realize text information classification of the social appeal data. The method aims at solving the problem of social appeals and optimizing the text classification function in the social appeal field, and effective extraction of deep information of social appeal data is achieved.

Description

Technical Field

[0001] The present invention relates to text classification technology, and in particular to a text classification optimization method based on incremental pre-training of social appeal data. Background Art

[0002] With the development of natural language processing (NLP) technology, pre-trained models such as BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformers), and T5 (Text-to-Text Transfer Transformers) have achieved impressive results on various text processing tasks. These models are pre-trained on a large amount of unlabeled text data and then fine-tuned on specific tasks to achieve reasonable performance. However, when focusing on a specific field, such as social appeals, traditional pre-trained models may not be able to fully capture the details and characteristics of the field. Summary of the invention

[0003] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a text classification optimization method based on incremental pre-training of social appeal data.

[0004] To achieve the purpose of the present invention, the technical solution adopted by the present invention is: A text classification optimization method based on incremental pre-training of social appeal data, comprising the steps of: (1) Collect social demand data from various social demand platforms, and present the social demand data in text format; (2) Preprocessing the collected social appeal data; (3) After preprocessing, the improved tree-based word mining algorithm is used to mine new words, segment them, and then the term frequency-inverse document frequency TF-IDF algorithm is used to extract keywords; (4) Expand keyword datasets using GPT-based data enhancement technology; (5) Input the pre-trained BERT model and select the incremental pre-training strategy to train the pre-trained BERT model; (6) Use the text classification algorithm and named entity recognition algorithm based on the trained BERT model to realize the text information classification of social appeal data.

[0005] Furthermore, in step (2), the social appeal data preprocessing includes: removing irrelevant symbols, special characters and noise information, and converting all texts into a unified format, and then using a preset stop word library to delete words that have no practical significance for text analysis.

[0006] Furthermore, in step (3), the improved tree-based word mining algorithm identifies high-frequency words that show strong contextual relevance in the text data, relying on two key statistical indicators: mutual information MI and entropy.

[0007] Furthermore, in step (3), the enhanced dictionary tree structure takes Chinese characters as the main nodes of the tree, and each tree root starts from the first Chinese character in the Chinese vocabulary to form a new tree root, and the subtrees under each root correspond to all potential words starting with the specific Chinese character.

[0008] Furthermore, in step (3), the term frequency inverse document frequency algorithm is used to measure the importance of each word in the text, where the term frequency TF represents the frequency of occurrence of a word in a specific document, and the inverse document frequency IDF measures the rarity of the word in the entire document collection; The higher the TF-IDF value, the greater the role of the word in distinguishing documents. According to the TF-IDF value of each word, the most representative terms in each document are extracted to generate a keyword list for the document.

[0009] Furthermore, in step (4), the Tongyi Qianwen model is used for data augmentation to generate diverse texts related to the category based on the input. The generation parameters are adjusted to ensure that the generated text matches the original category features. The generated text is strictly screened and then merged with the original dataset to form a balanced dataset.

[0010] Furthermore, in step (5), the pre-trained BERT model includes two tasks: mask language modeling (MLM) and next sentence prediction (NSP); Masked Language Modeling (MLM) randomly masks some tokens in the input sentence, predicts the masked tokens through a pre-trained model, and trains the model to recognize incomplete text; Next Sentence Prediction (NSP) provides two sentences to the pre-trained model and determines whether the second sentence is a direct successor of the first one.

[0011] Furthermore, in step (6), the text classification algorithm based on the BERT model has the following specific steps: using the word embedding table of the BERT model to map each character of the text to a vector representation, and generating a word-level vector representation after combining the context. BERT introduces a [CLS] tag at the beginning of each text segment, and uses the output vector corresponding to the tag as the global semantic representation of the text.

[0012] Furthermore, in step (6), the named entity recognition algorithm based on the BERT model specifically comprises the following steps: using BERT or BERT-BI-LSTM as the underlying text feature encoder and using CRF for entity label prediction.

[0013] The beneficial effect of the present invention lies in that, compared with the prior art, the present invention, compared with the traditional pre-trained model, utilizes specific domain data for model retraining to improve the domain adaptability of the model; introduces a new word mining algorithm to optimize the classification and recognition of data text, uses GPT-based data enhancement technology for data enhancement, and finally adopts text classification algorithm and named entity recognition algorithm to fully extract deep information in social demand data.

[0014] This paper adopts incremental pre-training technology, combined with an improved tree word mining algorithm and a text classification optimization strategy based on GPT data enhancement technology, aiming to solve social demands in urban governance and optimize the text classification function in the field of social demands, and realize the effective extraction of deep information of social demand data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the text classification optimization method based on incremental pre-training social appeal data; Figure 2 This is a flowchart of the word mining algorithm based on the improved tree; Figure 3 It is a schematic diagram of new word discovery; Figure 4 This is a flowchart of the text classification algorithm based on the BERT model; Figure 5 This is a flowchart of the named entity recognition algorithm based on the BERT model; Figure 6 It is a flowchart for introducing named entity recognition algorithm to identify missing addresses. DETAILED DESCRIPTION

[0016] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of this application.

[0017] like Figure 1 As shown, the text classification optimization method of social appeal data based on incremental pre-training described in the present invention is used to solve social appeals in urban governance and optimize the text classification function of social appeal data. Specifically, it includes the following steps: (1) Collect social demand data from various social demand platforms, including government hotlines, mayor mailboxes, and other social demand platforms. Social demand data are presented in text format, including titles and content. The titles are a summary of social demands, while the content provides detailed information on these social demands. (2) Preprocessing the collected social appeal data; Collect and load the data, and then clean the text, including removing irrelevant symbols, special characters, and noise information. This process also involves converting all text to a unified format, such as lowercase, to ensure consistency in subsequent processing. Then, use the preset stop word library to delete words that have no practical meaning for text analysis, thereby reducing data noise and improving analysis efficiency.

[0018] (3) After preprocessing, the improved tree-based word mining algorithm is used to mine new words, segment them, and then the term frequency-inverse document frequency TF-IDF algorithm is used to extract keywords; like Figure 2 As shown, the present invention improves the accuracy of feature extraction based on social appeal data by improving the new word discovery algorithm. First, the social appeal data is extracted from the business system database and preprocessed as raw data. The preprocessing stage includes the introduction of an improved new word discovery algorithm. The core of the algorithm is to build a dynamically updated social appeal new word library by identifying and recording new and high-frequency words that appear in social media or business systems. The construction of this new word library helps to improve the accuracy of word segmentation and avoid the phenomenon of missing new words in the traditional word segmentation process.

[0019] After the text is processed, it needs to be segmented, that is, the text is segmented according to vocabulary units to prepare for the subsequent word frequency calculation.

[0020] On this basis, the TF-IDF algorithm is applied to extract keywords from the preprocessed social appeal text data. By using the word segmentation results optimized by the new vocabulary, the TF-IDF algorithm can more accurately extract key terms in the text, thereby obtaining more representative social appeal text features. These features can then be input into the incremental pre-training model for learning to further enhance the model's performance when processing social appeal text in specific fields. Ultimately, the incremental pre-trained model can more accurately identify the characteristics of social appeals and provide stronger support for subsequent tasks such as text classification and trend analysis.

[0021] The Term Frequency-Inverse Document Frequency (TF-IDF) algorithm is used to measure the importance of each word in the text, where the term frequency (TF) indicates the frequency of a word in a specific document, while the inverse document frequency (IDF) measures the rarity of the word in the entire document collection. The higher the TF-IDF value, the greater the role of the word in distinguishing documents. Finally, based on the TF-IDF value of each word, the most representative terms in each document are extracted to generate a keyword list for the document for further analysis or application.

[0022] The social appeal data preprocessing operation is to address the input limitations of the BERT model. The BERT model only accepts input of up to 512 characters. The input text data is crucial to the training of subsequent models.

[0023] The present invention introduces a new word mining algorithm based on an improved tree, and the new word discovery algorithm identifies high-frequency words that show strong contextual relevance in text data. The algorithm relies on two key statistical indicators: mutual information MI and entropy.

[0024] Mutual information evaluates the degree of association between words, and the expression is as follows: (1) in, is the probability of words appearing at the same time; and Represents the probability of a single word appearing alone.

[0026] Entropy quantifies the diversity of a word's occurrence in different contexts and is calculated as follows: (2) Among them, given a word w, Indicates the probability of a specific situation c occurring. Through the above, new words can be effectively mined from a large amount of text data.

[0028] Aiming at the unique properties of Chinese new word discovery, the present invention proposes an enhanced dictionary tree structure. Chinese characters are used as the main nodes of the tree. Each tree root starts from the first Chinese character in the Chinese vocabulary, forming a new tree root. The subtrees under each root correspond to all potential words starting with the specific Chinese character. This effectively reduces the average depth of the nodes in the dictionary tree and improves the retrieval efficiency. Figure 3 As shown, schematic diagram of improved new word discovery.

[0029] (4) Expand keyword datasets using GPT-based data enhancement technology; The present invention uses the Tongyi Qianwen large model for data enhancement, aiming to solve the problem of data imbalance. First, by analyzing the data set, the categories with fewer samples are determined, and representative samples are selected from these categories as generated input prompts. The Tongyi Qianwen model can generate diversified text related to the category based on the input, thereby improving data coverage. By adjusting the generation parameters (such as temperature, Top-k and Top-p sampling), it can be ensured that the generated text not only matches the original category features, but also has sufficient diversity to avoid duplication. The generated text is strictly screened to exclude poor quality or irrelevant content. The screened enhanced data will be merged with the original data set to form a more balanced data set. Finally, the model is retrained using this enhanced balanced data set, so that its recognition ability on a few categories is significantly improved, the long tail effect is effectively alleviated, and the generalization ability and classification accuracy of the model are enhanced, especially in unbalanced classification tasks.

[0030] (5) Input the pre-trained model and select the incremental pre-training strategy to train the existing pre-trained model and integrate the characteristics of social demand data; The incremental pre-training method further trains domain-specific data on the basis of the existing language model, so that the model gradually integrates and adapts to the characteristics of the specific field. When processing social demand data, incremental pre-training can enhance the model's understanding of domain terms and specific expressions, and capture complex contextual associations. By continuously introducing new data, the model can continuously update its understanding of social demands and adapt to the dynamically changing social environment, thereby improving its performance in text classification.

[0031] The BERT model is selected as the pre-training basic model. The social appeal incremental pre-training model includes two tasks, MLM (masked language modeling) and NSP (next sentence prediction). MLM is used to prevent text missing, and NSP is used to understand the relationship between the previous and next contexts.

[0032] MLM (Masked Language Modeling) randomly masks some tokens in the input sentence, predicts the masked tokens through a pre-trained model, trains the model to recognize incomplete text, and realizes true bidirectional context dependency.

[0033] NSP (Next Sentence Prediction) provides two sentences to the pre-trained model and determines whether the second sentence is a direct successor of the first sentence, helping to understand the semantic association and coherence between sentences.

[0034] (6) Use the text classification algorithm and named entity recognition algorithm based on the trained BERT model to realize the text information classification of social appeal data.

[0035] like Figure 4As shown in the figure, the text classification algorithm based on the BERT model has the following specific steps: use the word embedding table of the BERT model to map each character of the text to a vector representation, and generate a word-level vector representation after combining the context. BERT introduces a [CLS] label at the beginning of each text segment, and uses the output vector corresponding to the label as the global semantic representation of the text to facilitate text classification and improve text classification accuracy.

[0036] like Figure 5 As shown in the figure, the named entity recognition algorithm based on the BERT model has the following specific steps: use BERT or BERT-BI-LSTM as the underlying text feature encoder and use CRF to predict entity labels. BL, IL, BO, IO, and O are labels. Labels BL and IL represent locations, indicating the beginning and end of the location. Labels BO and IO represent organizations, indicating the beginning and end of the organization.

[0037] It should be noted that CRF uses the label sequence as the prediction target, and can impose constraints on the final predicted label sequence based on logits to ensure that the predicted entity label sequence is valid. These constraints can be automatically learned from the training data set by the CRF layer during the training process. Since many label paths are predicted incorrectly, the correlation between labels and the characteristics of labels are crucial in entity label prediction. CRF learns label conversion relationships and constraints from data to help the model select correct and reasonable entity label sequences, reduce the prediction of invalid entity label sequences, and improve the accuracy of named entity recognition.

[0038] like Figure 6 As shown, the named entity recognition algorithm introduced in the present invention is mainly used to solve the problem that users do not fill in the address when submitting social demands on Weibo, WeChat and APP platforms. The missing address will significantly affect the classification accuracy of social demands, and then affect the response efficiency of relevant departments. The specific process is as follows: Data source: The social appeal text data comes from multiple platforms, including Weibo, WeChat, and APP. Users can submit appeals through these channels.

[0039] Address missing check: The system first analyzes the social appeal text to determine whether there is address information. If the address is missing, the system will trigger the address completion process.

[0040] Named Entity Recognition Algorithm: When a missing address is detected, the system will call the Named Entity Recognition (NER) algorithm to process the social appeal text and automatically identify the geographical location-related named entities (such as place names, street names, building names, etc.) that may be contained therein, thereby extracting potential address information.

[0041] Geographic Information System (GIS) Matching: The address entities extracted through NER are matched with the Geographic Information System (GIS) database to further confirm the accuracy and completeness of the address. The system uses GIS geographic information to complete and verify the identified address to ensure the accurate positioning of the request.

[0042] Optimized text classification model: After confirming the address information, the system inputs the completed social appeal text into the optimized text classification model for classification. The model is trained for different social appeal scenarios and can accurately assign appeals to relevant departments based on the content, address information and other features of the appeal.

[0043] Classification and allocation: Based on the classification results, social demands are allocated to corresponding departments (such as Department 1, Department 2, ..., Department N) to ensure that the demands can be handled quickly and accurately.

[0044] This process significantly improves the classification accuracy in cases of missing addresses through the introduction of NER algorithms and GIS systems, ensuring that social demands can be accurately classified and processed, while also improving the response efficiency of government departments.

[0045] The beneficial effect of the present invention is that, compared with the prior art, the application of the method of the present invention provides good data processing and text output for optimizing and improving the social appeal scheduling process, demonstrates the potential of the BERT model in text processing and its application value in processing social appeals, and promotes the advancement of text classification technology through targeted training and model optimization of social appeal data, and also provides new perspectives and tools for research and practice in the field of social appeals.

[0046] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. A text classification optimization method based on incremental pre-training of social appeal data, characterized in that: Includes steps: (1) Collect social demand data from various social demand platforms, and present the social demand data in text format; (2) Preprocessing the collected social appeal data; (3) After preprocessing, the improved tree-based word mining algorithm is used to mine new words, segment them, and then the term frequency-inverse document frequency TF-IDF algorithm is used to extract keywords; (4) Expand keyword datasets using GPT-based data enhancement technology; (5) Input the pre-trained BERT model and select the incremental pre-training strategy to train the pre-trained BERT model; (6) Use the text classification algorithm and named entity recognition algorithm based on the trained BERT model to realize the text information classification of social appeal data.

2. The text classification optimization method based on incremental pre-training social appeal data according to claim 1 is characterized in that: In step (2), the social appeal data preprocessing includes: removing irrelevant symbols, special characters and noise information, and converting all texts into a unified format. Then, using a preset stop word library to delete words that have no practical significance for text analysis.

3. The text classification optimization method based on incremental pre-training social appeal data according to claim 1 is characterized in that: In step (3), the improved tree-based word mining algorithm identifies high-frequency words that show strong contextual relevance in text data, relying on two key statistical indicators: mutual information MI and entropy.

4. The text classification optimization method based on incremental pre-training social appeal data according to claim 1 is characterized in that: In step (3), the enhanced dictionary tree structure takes Chinese characters as the main nodes of the tree. Each tree root starts from the first Chinese character in the Chinese vocabulary to form a new tree root. The subtrees under each root correspond to all potential words starting with the specific Chinese character.

5. The text classification optimization method based on incremental pre-training social appeal data according to claim 1 is characterized in that: In step (3), the term frequency inverse document frequency algorithm is used to measure the importance of each word in the text, where the term frequency TF represents the frequency of occurrence of a word in a specific document, and the inverse document frequency IDF measures the rarity of the word in the entire document collection; The higher the TF-IDF value, the greater the role of the word in distinguishing documents. According to the TF-IDF value of each word, the most representative terms in each document are extracted to generate a keyword list for the document.

6. The text classification optimization method based on incremental pre-training social appeal data according to claim 1 is characterized in that: In step (4), the Tongyi Qianwen model is used for data augmentation to generate diverse texts related to the category based on the input. The generation parameters are adjusted to ensure that the generated text matches the original category features. The generated text is strictly screened and then merged with the original dataset to form a balanced dataset.

7. The text classification optimization method based on incremental pre-training social appeal data according to claim 1 is characterized in that: In step (5), the pre-trained BERT model includes two tasks: mask language modeling (MLM) and next sentence prediction (NSP). Masked Language Modeling (MLM) randomly masks some tokens in the input sentence, predicts the masked tokens through a pre-trained model, and trains the model to recognize incomplete text; Next Sentence Prediction (NSP) provides two sentences to the pre-trained model and determines whether the second sentence is a direct successor of the first one.

8. The text classification optimization method based on incremental pre-training social appeal data according to claim 1 is characterized in that: In step (6), the text classification algorithm based on the BERT model has the following specific steps: using the word embedding table of the BERT model to map each character of the text to a vector representation, and generating a word-level vector representation after combining the context. BERT introduces a [CLS] tag at the beginning of each text segment, and uses the output vector corresponding to the tag as the global semantic representation of the text.

9. The text classification optimization method based on incremental pre-training social appeal data according to claim 1, characterized in that: In step (6), the named entity recognition algorithm based on the BERT model has the following specific steps: using BERT or BERT-bi-LSTM as the underlying text feature encoder and using CRF for entity label prediction.

Images