Text classification method by integrating optimized pre-training model and graph convolutional network model

Abstract

This paper discloses a text classification method that integrates an optimized pre-trained model with a graph convolutional network model, relating to the field of natural language processing. This method extracts text features and performs classification predictions using an optimized pre-trained model that performs feature interpolation and a graph convolutional network model that employs a scale fusion mechanism. The two prediction results are then weighted and fused to achieve efficient text classification. The more comprehensive text classification technology provided by this paper can effectively improve text classification accuracy and is widely applicable in various fields.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a high-performance text classification method that integrates an optimized pre-training model with a graph convolutional network model. Background Art

[0002] With the rapid development of the internet, the amount of information is increasing exponentially, providing easy access to a wealth of knowledge across a wide range of fields. Through digital platforms like search engines and social media, users can quickly find the information they need, promoting the widespread dissemination of innovative thinking and knowledge. This allows people to more easily exchange and absorb new perspectives, ideas, and technologies, thereby accelerating the development of science, technology, and culture. Furthermore, the rapid flow of information has significantly improved communication efficiency, making business collaboration, government operations, and personal interactions more efficient.

[0003] However, this surge in information volume has also led to information overload for users, making it difficult to sift through the vast ocean of data to identify truly valuable information. This not only reduces work efficiency but can also impact the quality of decision-making. Furthermore, as data volumes continue to grow, so does the complexity of information, requiring more powerful algorithms and techniques for effective information retrieval and analysis. Against this backdrop, text classification technology has emerged. It utilizes computer algorithms and models to automatically assign text documents to predefined categories or labels. Automatic text classification is achieved through text preprocessing, feature extraction, and model training using supervised learning algorithms. This significantly improves the ability to process large amounts of text data, making the information more clearly organized and easier to retrieve and understand.

[0004] Currently, both pre-trained models and graph convolutional networks (GCNNs) are widely used in text classification tasks. GCNNs can extract and combine features from multi-scale local spatial data, possessing strong representational capabilities. They perform a weighted average of information about each node's neighbors and its own data to produce a result vector that can be fed into a neural network. Pre-trained models typically use unsupervised learning methods, training on unlabeled data to learn a universal data representation, enabling them to achieve greater performance in text classification tasks.

[0005] However, when using graph convolutional networks for text classification, there are problems such as a lack of valid node information and slow model convergence, resulting in poor classification results. While pre-trained models perform well in constructing contextual information, the extracted text features lack local and global structural information. Summary of the Invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a text classification method that integrates an optimized pre-trained model with a graph convolutional network model. In response to problems such as the lack of effective information in graph neural network nodes and the inability of pre-trained models to capture text structure information, this patent proposes to integrate the optimized pre-trained model into the graph convolutional network model. The optimized pre-trained model is first used to extract text feature representation from the text, and then the text feature representation is input into the graph convolutional network model to extract the text feature representation again, and this is used as the basis for determining the category to which the text belongs, thereby achieving high-performance text classification.

[0007] The technical solutions of the present invention are as follows:

[0008] A text classification method that integrates an optimized pre-trained model with a graph convolutional network model. The optimized pre-trained model and the graph convolutional network are used to extract text features, including semantic features and structural features of the text, to achieve efficient text classification. The method includes the following steps:

[0009] By using an optimized pre-trained model and graph convolutional networks to extract text features, including semantic features and structural features, efficient text classification is achieved. The steps include:

[0010] 1) Construct a heterogeneous graph corresponding to the text dataset based on the relevance of documents and words in the text dataset; the nodes of the heterogeneous graph represent the feature representation of documents or words, and the edge weights of the heterogeneous graph represent the relevance between documents and words or between words; the information of all edges of the heterogeneous graph is stored in an adjacency matrix;

[0011] 2) Extracting feature representations of text data and obtaining prediction results by optimizing the pre-trained model; including:

[0012] Perform data enhancement on the original text data sample to generate two data enhancement samples;

[0013] The original sample and two augmented samples are input into the word segmenter of the pre-trained model to segment the text data into vocabulary units and obtain text sequences. The word segmenter is used to add a marker to indicate sequence classification information at the corresponding position of each text sequence.

[0014] Convert the processed text data into tensor format and generate attention mask and unified text sequence length;

[0015] The original and augmented samples are loaded in batches and input into the pre-trained model for feature extraction and propagation. During the feature extraction and propagation process, two layers are randomly selected from all layers of the pre-trained model for feature interpolation. In the first layer, the hidden states obtained by propagating the original sample and the first augmented sample are weighted mixed. In the second layer, the hidden states obtained by propagating the original sample and the second augmented sample are weighted mixed. The results of the two interpolation operations are used as the hidden states of the original samples and continue to propagate in each layer of the model, allowing the pre-trained model to learn different combinations of features. The hidden state of the last layer of the pre-trained model is then used as the final extracted feature representation and output.

[0016] Assign the node features of the heterogeneous graph to the output features of the optimized pre-trained model;

[0017] The output features of the optimized pre-trained model are processed by the Softmax classification layer, and the output is the probability of each text category, that is, the prediction result;

[0018] 3) Feature extraction and propagation based on graph convolutional network model:

[0019] The node features and adjacency matrix of the heterogeneous graph are fed into the improved deep graph convolutional network model in batches. The node features are propagated and transformed through a series of convolutional layers of the model. Each layer uses residual connections and identity mapping to update the node features.

[0020] A new feature scale fusion mechanism is designed. Multiple convolution operations are performed in each convolutional layer. Each convolution operation uses an adjacency matrix of different powers to extract features of different scales. The power of the adjacency matrix represents the set of nodes that can be reached in k steps from the current node. The features of different scales are then weighted and summed.

[0021] The feature representation of the last output layer is used as the final extracted feature input into the Softmax classification layer for processing, and the probability of each category is output, that is, the prediction result;

[0022] 4) Perform weighted fusion on the two prediction results to obtain the final text classification result.

[0023] In specific implementation, the present invention also provides a text classification system that utilizes the above-mentioned text classification method of integrating the optimized pre-trained model and the graph convolutional network model, including: a heterogeneous graph construction module, a feature extraction module based on the optimized pre-trained model, a feature extraction module based on the graph convolutional network, and a text classification module that integrates the prediction results of the two models.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] Using an optimized pre-trained model and a graph convolutional network (GCNN) to extract text features, they not only mine the semantic features of the text but also capture its structural information. This allows for more precise representation of text information, effectively improving classification accuracy. The optimization of the pre-trained model enables the model to better understand the contextual meaning and complex linguistic structure of language, improving its performance and robustness. The introduction of GCNNs models text as a graph structure, with words or phrases as nodes and their relationships as edges. This structure enables the model to capture the complex relationships between various components of the text. By performing computations on the graph structure, GCNNs can effectively extract structural features of the text, further deepening the model's understanding of the text. Combining these two approaches not only captures the overall semantics of the text but also understands its inherent structure. This combination of depth and breadth provides a more comprehensive and accurate approach to text classification, with broad application across multiple fields. For example, in news analysis, this approach can accurately identify the topic of an article; in social media analysis, it helps identify user sentiment and intent; and in biomedical research, it can efficiently classify and retrieve large volumes of research literature. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a structural block diagram of a text classification system provided in a specific embodiment of the present invention.

[0027] Figure 2 This is a flowchart of the text classification method that integrates an optimized pre-trained model and a graph convolutional network provided by the present invention.

[0028] Figure 3 This is a flowchart of constructing a heterogeneous graph module in a specific embodiment of the present invention.

[0029] Figure 4 This is a flowchart of the workflow of the feature extraction module based on the optimized pre-training model in a specific embodiment of the present invention.

[0030] Figure 5 This is a flowchart of the workflow of the feature extraction module based on the graph convolutional network in a specific embodiment of the present invention.

[0031] Figure 6 This is a flowchart of the workflow of a text classification module that integrates the prediction results of two models in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described below by way of examples in conjunction with the accompanying drawings, but the scope of the present invention is not limited in any way. The present invention provides a text classification method that integrates an optimized pre-training model and a graph convolutional network model. In one embodiment, Figure 1 As shown, the method consists of four main steps:

[0033] Step 1: Construct a heterogeneous graph. First, preprocess the text dataset, including unifying the data format and performing sentence segmentation. Then, treat each document in the text dataset as a document node, and extract all words that appear in the documents as word nodes. Then, set the features of the word nodes and document nodes to zero vectors. Finally, calculate the correlation between nodes as the edge weight.

[0034] Step 2: Use an optimized pre-trained model to extract text features. First, perform data augmentation on the original text sample to obtain an enhanced sample. Then, use the pre-trained model to extract features from both the original and enhanced samples. The features of the two samples are then fused, and the document node features in the heterogeneous graph are assigned to the fused feature representation. This ensures that the document node features contain rich semantic information, which facilitates final classification. Finally, the fused feature representation is processed and output through a Softmax layer. The Softmax function exponentially amplifies each element of the input vector and then normalizes it (i.e., the sum of all elements is 1), forming a probability distribution.

[0035] Step 3: Use graph convolutional network to extract features. First, use graph convolutional network to read the information of heterogeneous graph, including node information and edge information. Then use graph convolutional network to extract features, propagate between each layer, and then process the extracted features through the Softmax layer and output them.

[0036] Step 4: Fusion optimizes the pre-trained model and the output of the graph convolutional network, uses a weighted approach to fuse the two results, obtains the final classification result and outputs it.

[0037] Specifically, the present invention proposes a text classification method that integrates an optimized pre-trained model and a graph convolutional network model. First, the text data is processed using the optimized pre-trained model to extract high-quality feature representations. These feature representations can fully capture the semantic information in the text. Next, these initially extracted feature representations are input into the graph convolutional network for further processing. Through the calculation of the graph convolutional network, new and more expressive feature representations can be obtained. Finally, the prediction results of the optimized pre-trained model and the prediction results obtained by the graph convolutional network are fused to form a comprehensive prediction result. This fusion method can make full use of the advantages of the two models, thereby significantly improving the accuracy of text classification. Through this method, text data can be classified more accurately, improving the overall classification performance.

[0038] The method of the present invention includes: constructing a heterogeneous graph corresponding to a text dataset; using an optimized pre-trained model to extract text feature representations to initialize node features in the heterogeneous graph; processing the heterogeneous graph using a graph convolutional network to obtain new feature representations; and fusing the predictions of the two models to obtain a final result, which serves as the primary basis for determining the category to which the text data belongs. This method can effectively improve the accuracy of text classification, and its performance is not significantly affected by insufficient training sample resources.

[0039] In specific implementation, the present invention also provides a text classification system that utilizes the above-mentioned text classification method of integrating the optimized pre-trained model and the graph convolutional network model, including: a heterogeneous graph construction module, a feature extraction module based on the optimized pre-trained model, a feature extraction module based on the graph convolutional network, and a text classification module that integrates the prediction results of the two models.

[0040] The heterogeneous graph construction module constructs a heterogeneous graph based on the relevance of documents and words in the dataset. The nodes of the graph store the feature representation of documents or words, and the weight of the edge represents the relevance between documents and words or between words.

[0041] A feature extraction module based on an optimized pre-trained model is used to extract semantic features from text data. The feature representation output by the model is used to initialize document nodes in the heterogeneous graph.

[0042] The feature extraction module based on graph convolutional networks is used to extract and propagate features on heterogeneous graphs, process the data of each node and its neighbors, and obtain feature representations containing semantic and structural information of the text.

[0043] The text classification module that fuses the prediction results of the two models is used to fuse the prediction results output by the optimized pre-trained model and the prediction results output by the graph convolutional network. The fused result of the two is used as the output of the model and then sent to the classifier to obtain the final classification result.

[0044] In the above text classification system, further:

[0045] 1. The heterogeneous graph construction module includes:

[0046] 1.1 Data Preprocessing

[0047] (1) Text standardization

[0048] Process the text data, converting all text to a uniform format, such as standardizing uppercase and lowercase letters and removing unnecessary whitespace and blank lines. Then, detect the language of the text and filter out text in non-target languages ​​to ensure data consistency.

[0049] (2) Sentence segmentation

[0050] Use punctuation marks to segment text into independent sentences, and handle special symbols and other situations that may affect the segmentation effect to ensure the accuracy of sentence segmentation.

[0051] (3) Special mark processing

[0052] Remove non-text information and delete special tags such as URLs and emails to ensure the purity of the text.

[0053] 1.2 Constructing nodes of heterogeneous graphs

[0054] This method constructs a large heterogeneous text graph consisting of word nodes and document nodes, which can explicitly model global word co-occurrence and easily adapt to graph convolution. This method treats each document in the dataset as a document node, then extracts the words that appear in all documents and treats each word as a word node.

[0055] 1.3 Constructing edges of heterogeneous graphs

[0056] (1) Calculate the edge between the word node and the document node. The edge between these two nodes uses TF-IDF as the weight. The larger the TF-IDF value, the more important the word is to the document.

[0057] (2) Calculate the edges between word nodes. The PMI is used as the weight for the edges between these two types of nodes. A positive PMI indicates a high semantic correlation between the words, while a negative PMI indicates a low or nonexistent semantic connection between the two words. Therefore, only edges with positive PMIs are retained. At the same time, a self-loop is added to each node with a weight of 1.

[0058] (3) Finally, the information of all edges is stored in the heterogeneous graph adjacency matrix.

[0059] 2. The feature extraction module based on the optimized pre-trained model includes:

[0060] 2.1 Data enhancement of original data samples

[0061] (1) Prepare a perturbation operation set that includes simple token-level data augmentation techniques such as back-translation, synonym replacement, and Gaussian noise perturbation.

[0062] (2) Adjust different hyperparameters and randomly sample the operation set.

[0063] (3) Use the selected operation to generate two new samples, namely the data augmented samples.

[0064] 2.2 Word Segmentation and Tokenization

[0065] (1) The original sample and two augmented samples are fed into the tokenizer of the pre-trained model XLNet (Generalized Autoregressive Pretraining for Language Understanding, a natural language processing model for generalized autoregressive pretraining for language understanding). The tokenizer segments the text data into vocabulary units and generates text sequences. The XLNet tokenizer processes the words in the vocabulary to ensure that the vocabulary size is within the model's processing range.

[0066] (2) Adding special tags. The XLNet tokenizer adds special tags [CLS] and [SEP] to the corresponding positions of each text sequence. The [CLS] tag is used to indicate the classification information of the entire sequence, and the [SEP] represents the sentence separator.

[0067] (3) The processed text sequence data is converted into a tensor format and an attention mask is generated to distinguish actual tokens from padded tokens (the actual word position is represented by 1 and the padded position is represented by 0). Then the length of the text sequence is unified, short texts are padded to the maximum length, and texts exceeding the maximum length are truncated.

[0068] 2.3 Feature extraction and propagation

[0069] (1) Create a data loader for batch loading and processing original samples and enhanced samples.

[0070] (2) Set the pre-trained model Adam optimizer and cross entropy loss function, and then input all samples into the XLNet pre-trained model for feature extraction and propagation.

[0071] (3) During the text feature propagation process, two layers are randomly selected from all layers of the XLNet model for feature interpolation (including the first and second layer operations). In the first layer, the hidden states obtained by propagating the original sample and the first enhanced sample are weighted and mixed, and the weights are obtained from the Dirichlet distribution. The Dirichlet distribution is a continuous multivariate probability distribution that is often used to model the parameters of multinomial distributions in statistics and machine learning. The numpy library of Python is used to extract samples from the Dirichlet distribution to obtain a set of weight values, and then the original sample and the enhanced sample are mixed. The resulting mixed sample contains the features of the original sample and the enhanced sample, which helps the model learn a wider feature space. In the second layer, the hidden states obtained by propagating the original sample and the second enhanced sample are mixed in the same way. The results of the two interpolation operations in the first and second layers are both used as the hidden states of the original sample to continue to propagate. Through the above feature interpolation method, the pre-trained model can better learn the combination of different features, thereby improving the generalization ability of the model and the accuracy of text classification. Finally, the hidden state of the last layer of XLNet is used as the final extracted feature representation and output.

[0072] Unlike the existing technology, the method of the present invention specifically inputs both the original samples and the enhanced samples into the model. During the feature extraction and propagation process, the hidden state obtained by extracting the original samples and the hidden state obtained by extracting the enhanced samples are innovatively mixed. The obtained mixed hidden state is used to continue to propagate in each layer of the model. The hidden state of the last layer is the final extracted feature.

[0073] 2.4 Initializing the Heterogeneous Graph

[0074] The document node features of the heterogeneous graph are assigned to the output features of the optimized pre-trained model, so that the nodes of the heterogeneous graph contain rich semantic information, which is conducive to the final classification.

[0075] 2.5 Output Results

[0076] The output features of the optimized pre-trained model are input into the Softmax classification layer, and the output is processed by the Softmax layer. The output result is the probability of each category, that is, the prediction result.

[0077] 3. Feature extraction module based on graph convolutional network, including:

[0078] 3.1 Feature extraction and propagation based on graph convolutional network model

[0079] (1) The node features and adjacency matrix of the heterogeneous graph are fed into the improved deep graph convolutional network model (GCNII model) in batches. The GCNII model (Simple and Deep Graph Convolutional Networks) is an improved graph convolutional network that can solve the common over-smoothing phenomenon in deep convolutional networks.

[0080] (2) In the model, node features are propagated and transformed through a series of GCNII convolutional layers. Each layer uses residual connections and identity mapping to update node features. During the feature propagation process, the features of each layer retain a portion of the initial features to ensure that the text information is retained in the deep network. At the same time, the present invention designs a new feature scale fusion mechanism, performing multiple convolution operations in each layer, and each convolution operation uses an adjacency matrix of different powers to extract features of different scales. Specifically, the features of different scales in each layer can be obtained by multiplying the features of the previous layer with adjacency matrices of different powers and the weight matrix of the convolution operation. The power of the adjacency matrix represents the set of nodes that can be reached in k steps from the current node. For example, A 1 represents the first-order neighbor, A 2 Denotes the second-order neighbor, and so on, finally performing a weighted summation of features at different scales. This feature extraction method not only focuses on the neighborhood information of the current layer but also introduces neighborhood features from a wider range, effectively utilizing the multi-scale text information in the graph structure and improving the model's performance in text classification tasks.

[0081] (3) An activation function and a random dropout layer are set after each layer of GCNII convolution to prevent overfitting.

[0082] 3.2 Output Results

[0083] The feature representation output by the last GCNII layer is input into the Softmax classification layer as the final extracted feature. After processing by the Softmax layer, the output result is the probability of each category, that is, the prediction result.

[0084] 4. The text classification module that integrates the prediction results of the two models includes:

[0085] 4.1 The optimized pre-trained model prediction results and the graph convolutional network prediction results are weighted by the parameter λ. Specifically, the weight of the optimized pre-trained model prediction results is λ, and the weight of the graph convolutional network prediction results is 1-λ.

[0086] 4.2 The final prediction result is calculated according to the above weighted summation method. This result combines the advantages of the optimized pre-training model and the graph convolutional network method, thereby improving the accuracy and robustness of the classification.

[0087] 4.3 Finally, the final prediction result after integration is output. This result is the final classification label for the text data, reflecting the effect of the combined effect of the optimized pre-training model and the graph convolutional network.

[0088] Through the above steps, text classification is achieved by integrating the optimized pre-trained model and the graph convolutional network model.

Claims

1. A text classification method that integrates an optimized pre-trained model with a graph convolutional network model, characterized in that: The method extracts text features and performs predictions by optimizing a pre-trained model through feature interpolation and designing a graph convolutional network model with a scale fusion mechanism, and then performs weighted fusion on the two prediction results to achieve efficient text classification. The method comprises the following steps: 1) Construct a heterogeneous graph corresponding to the text dataset based on the relevance of documents and words in the text dataset; the nodes of the heterogeneous graph represent the feature representation of documents or words, and the edge weights of the heterogeneous graph represent the relevance between documents and words or between words; the information of all edges of the heterogeneous graph is stored in the adjacency matrix; 2) Extracting feature representations of text data and obtaining prediction results by optimizing the pre-trained model; including: Perform data enhancement on the original text data sample to generate two data enhancement samples; Input the original sample and two enhanced samples into the pre-trained model to obtain a text sequence; and add a text sequence classification tag to the corresponding position of each text sequence; The original and augmented samples are loaded in batches and input into the pre-trained model for feature extraction and propagation. During the feature extraction and propagation process, two layers are randomly selected from all layers of the pre-trained model for feature interpolation. In the first layer, the hidden states obtained by propagating the original sample and the first augmented sample are weighted mixed. In the second layer, the hidden states obtained by propagating the original sample and the second augmented sample are weighted mixed. The results of the two interpolation operations are used as the hidden states of the original samples and continue to propagate in each layer of the model, allowing the pre-trained model to learn different combinations of features. The hidden state of the last layer of the pre-trained model is then used as the final extracted feature representation and output. Assign the node features of the heterogeneous graph to the output features of the optimized pre-trained model; The output features of the optimized pre-trained model are processed by the Softmax classification layer, and the output is the probability of each text category, that is, the prediction result; 3) Feature extraction and propagation based on graph convolutional network model: The node features and adjacency matrix of the heterogeneous graph are fed into the improved deep graph convolutional network model in batches. The node features are propagated and transformed through a series of convolutional layers of the model, and each layer uses residual connections and identity mapping to update the node features. Design a new feature scale fusion mechanism to extract features during feature propagation and transformation. Perform multiple convolution operations in each convolution layer, each using an adjacency matrix of different powers to extract features of different scales. Then, perform a weighted summation of the features of different scales. The feature representation of the last output layer is used as the final extracted feature input into the Softmax classification layer for processing, and the probability of each category is output, that is, the prediction result; 4) Perform weighted fusion on the two prediction results to obtain the final text classification result.

2. The text classification method of integrating the optimized pre-trained model and the graph convolutional network model as claimed in claim 1, characterized in that: Text features include the semantic features of the text and the structural features of the text.

3. The text classification method of integrating the optimized pre-trained model and the graph convolutional network model as claimed in claim 1, characterized in that: In step 2), the original sample and the two enhanced samples are input into the word segmenter of the pre-trained model to segment the text data into vocabulary units and then obtain a text sequence; the word segmenter is used to add tags to represent sequence classification information; and an attention mask and a unified text sequence length are generated for the text data converted into tensor format.

4. The text classification method of integrating the optimized pre-trained model and the graph convolutional network model as claimed in claim 1, characterized in that: Step 2) weighted mixing is performed during feature extraction and propagation, where the weights are specifically obtained by Dirichlet distribution.

5. The text classification method of integrating the optimized pre-trained model and the graph convolutional network model as claimed in claim 1, characterized in that: In step 3), the power of the adjacency matrix represents the set of nodes that can be reached in k steps from the current node.

6. A system for text classification using the method of integrating the optimized pre-trained model and the graph convolutional network model according to claim 1, comprising: Construct a heterogeneous graph module, a feature extraction module based on an optimized pre-trained model, a feature extraction module based on a graph convolutional network, and a text classification module; wherein: The heterogeneous graph construction module is used to construct a heterogeneous graph based on the relevance of documents and words in the dataset; A feature extraction module based on an optimized pre-trained model is used to extract semantic features from text data. The feature representation output by the model is used to initialize document nodes in the heterogeneous graph. The feature extraction module based on graph convolutional networks is used to propagate and extract data from heterogeneous graphs, processing the data of each node and its neighbors in the heterogeneous graph to obtain feature representations containing semantic and structural information of the text; The text classification module is used to fuse the feature representation output by the optimized pre-trained model and the feature representation output by the graph convolutional network. The fusion result of the two is used as the output and sent to the classifier for prediction to obtain the final classification result.

Images