A hierarchical classification method for government hotline text based on pre-trained model and graph convolutional network

Abstract

A hierarchical classification method for government hotline text based on a pre-trained model and a graph convolutional network relates to the technical field of intelligent text classification. Currently, deep learning methods have shown higher classification accuracy in hierarchical multi-label text classification. However, most of them simply treat labels as supervisory information and do not fully utilize the correlation between labels and text. In order to hierarchically classify government hotline texts more quickly and efficiently so that problems encountered by citizens can be promptly handed over to relevant departments for processing, the model proposed in the present invention combines a large-scale pre-trained model with a graph neural network, effectively utilizing the hierarchical structure of labels and text semantic information, and significantly improving classification accuracy and efficiency. The model constructs a heterogeneous graph containing labels, documents, and words on the dataset, and fully utilizes the advantages of the two models, providing an effective solution for the automated classification of government hotline texts.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent text classification, and in particular to a government hotline text hierarchical classification method based on a pre-training model and a graph convolutional network. Background Art

[0002] With the development of modern society, government hotlines play a vital role as a bridge of communication between the government and the public. With the increasing prevalence of information technology and the internet, the text messages received by government hotlines are becoming increasingly large and diverse. By hierarchically classifying the content of citizen calls, government hotlines can better understand the content and needs of calls and assign them to relevant departments for processing, providing faster and more accurate services. Given the large number of calls received daily by government departments, traditional manual classification is not only inefficient but also wastes significant manpower and resources. Therefore, how to automatically, efficiently, and accurately classify and categorize these text messages has become an unresolved issue.

[0003] Text classification is a fundamental task in natural language processing, which aims to assign one or more categories to a given document based on its content. Unlike traditional classification tasks, in hierarchical text classification tasks, a document may be labeled into multiple categories, and these categories have a hierarchical parent-child relationship. How to exploit the relationships between label hierarchies for hierarchical classification is an important research topic. In hierarchical text classification, the label hierarchy is usually modeled as a tree or a directed acyclic graph, where nodes represent labels. See Figure 1 , an input text can be classified into different levels of label nodes, and there is a parent-child relationship between labels. For example, "urban and rural construction" has two subcategories "urban and rural planning" and "project quality".

[0004] Hierarchical multi-label text classification, which involves classifying text into hierarchically structured label categories based on its content, plays a crucial role in natural language processing. This problem has garnered significant attention in both industry and academia and has become a research hotspot. Compared to simple large-scale text classification, hierarchical multi-label text classification is more challenging, primarily due to its complex hierarchical structure. To address this issue, incorporating the parent-child relationship of hierarchical labels as prior knowledge into the model, enabling it to recognize and utilize the hierarchical structure, is a natural and effective approach. The key issue in hierarchical text classification research is how to fully and effectively utilize this information to improve classification performance.

[0005] Recent research on hierarchical multi-label text classification mainly adopts a dual encoder framework, which consists of a text encoder responsible for text representation and a structural encoder that integrates label information into the text. The text encoder can be a traditional model for text classification, such as TextRCNN and BERT. The structural encoder is a graph neural network that regards the label hierarchy as a directed acyclic graph and propagates information between labels. As a subtask of multi-label classification, the key challenge facing hierarchical text classification is how to model large-scale, unbalanced and structured label hierarchies. Depending on whether the hierarchical information of the labels is utilized and how it is utilized, hierarchical multi-label classification algorithms are mainly divided into two categories: non-hierarchical methods and hierarchical methods. Non-hierarchical methods are also called planar methods. They ignore the hierarchical dependencies between labels and convert the problem into a planar multi-label classification problem for processing. Hierarchical methods make full use of the hierarchical structure of labels and are further subdivided into local methods, global methods and hybrid methods. As Figure 2 shown.

[0006] A graph neural network (GNN) is a model for information processing based on graph structures. It represents data through nodes and edges in a graph and transmits information through connections between nodes. GNNs can model nodes, edges, or entire graphs, enabling tasks such as node classification, link prediction, and graph classification. Compared to traditional convolutional neural networks (CNNs) and recurrent neural networks (RNNs), GNNs are better able to process data with non-Euclidean structures, such as graphs. In traditional neural networks, the input data format is typically fixed, such as an image pixel matrix or a text sequence. However, in real-world applications, many data have complex structures, such as social networks and knowledge graphs, which cannot be represented using simple matrices or sequences. By representing data as a graph, GNNs can better capture dependencies and relationships between data. For example, in a social network, GNNs can represent relationships between users through connections between nodes and user attributes through node features. GNNs can use this information to predict user behavior, such as whether a user will click on a link or be interested in a product. Another advantage of GNNs is their ability to process graphs of varying sizes. Traditional neural networks require input data of a fixed format, such as the size of an image or the length of a text file. However, in real-world applications, many graph-structured data have varying sizes. GNNs can also have multiple layers. The more layers, the larger the "receptive field" of the graph neural network, allowing each node to capture more information. This helps the GNN understand graph-structured data more comprehensively, thereby improving model performance. In summary, GNNs are powerful models that can effectively process graph-structured data. They offer many advantages, such as better capturing dependencies and relationships between data, handling graphs of varying sizes, and supporting multiple layers.

[0007] Graph convolutional networks (GCNs) belong to the category of semi-supervised learning and can be trained even if not all nodes have labels. When calculating the loss, only the labeled nodes need to be considered. In order to reduce the loss of labeled nodes, their neighboring nodes will also be adjusted accordingly, which is a unique feature of the graph structure. Therefore, in graph neural networks and graph convolutional networks, training can be performed without all nodes having labels. The core idea of ​​the graph convolutional network is to first calculate the features of each node, then fuse the features of each node, and finally input the fused features into the fully connected network for classification. The graph convolutional network is a model based on node relationships. The embedding expression of all nodes in the first layer is H(1). The output of the l+1 layer of the graph convolutional network is:

[0008]

[0009] in is the adjacency matrix, which reflects the connection information of all nodes, and The diagonal matrix consisting of the degree of each node, The adjacency matrix is ​​normalized, and σ is a nonlinear activation function. The graph convolutional network is iterated repeatedly using the above formula until the representation of the last layer is output.

[0010] With the continuous advancement of natural language processing technology, the Transformer model has been widely used in numerous natural language processing (NLP) tasks due to its outstanding performance. ROBERTA, a variant of the Transformer model, was proposed by Facebook AI Research in 2019. Its goal is to improve the performance of the BERT model by improving pre-training strategies. ROBERTA improves on several of BERT's pre-training strategies. Results indicate that the original BERT model may be undertrained and fails to fully learn the linguistic knowledge in the training data. OBERTA's key improvements include: 1. When preparing training data, BERT masks some tokens and forces the model to predict these tokens during training. This masking method, which ensures that the training data remains unchanged and is used throughout training, is known as static masking. ROBERTA uses dynamic masking, continuously changing the mask position during training and masking multiple copies of the original training data, allowing the same data to be trained with different mask positions. 2. ROBERTA has found that increasing the batch size during training improves downstream task metrics. 3. ROBERTA uses a byte-based BPE (Byte Pair Encoding) algorithm to represent words and generate a vocabulary, while BERT uses a character-based BPE algorithm. ROBERTA's vocabulary contains approximately 50,000 words, broken down at the byte level, effectively avoiding the problem of unknown words. 4. ROBERTA uses more training data than BERT.

[0011] In summary, hierarchical multi-label text classification is one of the key tasks in the field of natural language processing, and is particularly complex and challenging due to the hierarchical structure of its labels. Hierarchical text classification tasks can be applied in many scenarios such as news content classification and academic paper classification, and are of great significance for information organization and retrieval. The current strategies for handling such tasks can be roughly divided into traditional methods and deep learning-based methods. From the perspective of performance evaluation, deep learning methods have shown higher classification accuracy in hierarchical multi-label text classification and have become the mainstream trend. However, most of these methods simply treat labels as supervisory information and do not fully utilize the correlation between labels and text. In order to hierarchically classify government hotline texts more quickly and efficiently, so that the problems encountered by citizens can be promptly handed over to relevant departments for processing, the present invention proposes a government hotline text hierarchical classification model based on a pre-trained model (ROBERTA) and a graph convolutional network (GCN). Summary of the Invention

[0012] This paper proposes a hierarchical classification method for government hotline text based on a pretrained model and a graph convolutional network. Combining a large-scale pretrained model with a graph neural network effectively leverages the hierarchical structure of labels and text semantics, significantly improving classification accuracy and efficiency. The model constructs a heterogeneous graph consisting of labels, documents, and words on the dataset. By jointly training the ROBERTA and GCN modules, it leverages the strengths of both models, providing an effective solution for the automated classification of government hotline text.

[0013] In order to achieve the above object, the technical solution adopted by the present invention is:

[0014] A hierarchical classification method for government hotline text based on a pre-trained model and graph convolutional network. The steps are as follows:

[0015] Step 1: Get the dataset and process it

[0016] Obtain a dataset of text descriptions of government hotline numbers and clean it. Remove extra spaces, special characters, and irrelevant symbols from the text to ensure neatness and consistency. Handle missing values ​​and outliers by marking or deleting them. Use a Chinese word segmentation tool to segment the cleaned text into words or phrases. Divide the processed data into a training set, a validation set, and a test set. The training set is used for model training and optimization, and the test set is used to evaluate model performance.

[0017] Step 2: Build a heterogeneous graph

[0018] Define three types of nodes: document nodes, word nodes, and tag nodes; build edges and calculate weights;

[0019] Step 3: Model initialization

[0020] Before model training begins, the ROBERTA model is fine-tuned using the dataset to obtain the weight parameters of the ROBERTA model to speed up model training. Node representation initialization: The pre-trained ROBERTA model is used to obtain document embeddings and word embeddings as the initial values ​​of node representations.

[0021] Step 4: Model training

[0022] Joint training: GCN and ROBERTA models are trained simultaneously, leveraging GCN's ability to handle complex structures combined with ROBERTA's powerful semantic representation capabilities to improve classification accuracy and efficiency. During training, a memory bank M is used to store the input features of all document nodes. That is, at the beginning of each training round, all document embeddings are first calculated using the ROBERTA model and stored in M. A batch of data of size 16 is sampled from M in each epoch of training. The data is then fed into the ROBERTA module to obtain a new embedding representation, which is then updated to the corresponding document node in M. The GCN module is then used to train the entire graph. Loss function optimization: Using a binary crossover loss function, the gradient descent optimization algorithm is used to minimize the loss function and update the model parameters.

[0023] Step 5: Prediction and Evaluation

[0024] Prediction phase: Predictions are made on the text data in the test set, with the output of GCN regarded as the final representation of the document, and the softmax function is used to calculate the probability of each label. During the training process, the document embedding obtained by the ROBERTA module is also used as input to the sotmax function to calculate the probability of each label. The final predicted probability of each label is the linear interpolation of the predictions of GCN and ROBERTA. Evaluation indicators: Macro-F1 and Micro-F1 are used as evaluation indicators to evaluate the performance of the model to reflect the classification accuracy and generalization ability of the model.

[0025] As a preferred technical solution of the present invention, the three types of nodes defined in step 2 are

[0026] Document node: represents each piece of text data received by the government hotline;

[0027] Word node: represents a word or phrase in the text;

[0028] Label node: represents the hierarchical classification label to which the text belongs;

[0029] The constructed edges are:

[0030] Word-document edge: constructs an edge between a word node and a document node based on the occurrence of the word in the document;

[0031] Document-label edge: constructs an edge between a document node and a label node based on the label to which the document belongs;

[0032] Label-label edge: constructs edges between label nodes based on the hierarchical parent-child relationship between labels;

[0033] The weight is calculated as:

[0034] Word-word edge weight: Use point-wise mutual information (PMI) to calculate the degree of association between words and construct edges;

[0035] Document-label edge weight: The weight between the document and the label is calculated using the average of the weighted co-occurrence probabilities of the words;

[0036] Label-label edge weight: The weight between labels is calculated using the logarithm of the co-occurrence frequency in the training data.

[0037] As a preferred technical solution of the present invention, in the constructed heterogeneous graph in step 2, the weight of the word-word edge is calculated using point-wise mutual information (PMI); point-wise mutual information is a common method for measuring the degree of association between two words i and j. The point-wise mutual information value of words i and j is calculated as:

[0038]

[0039] Where p(i,j) is the probability of two words i and j appearing together, p(i) is the probability of word i appearing, p(j) is the probability of word j appearing, #w represents the number of sliding windows, #w(i) is the number of sliding windows in which a word appears, and #w(i,j) indicates that there is little or no semantic correlation between word i and the sliding windows in which it appears together. For the edge between a label and a document, the average of the weighted co-occurrence probabilities of the words is used for calculation:

[0040]

[0041] Among them, (LW) ij represents the weight between document i and label j, n i represents the total number of words in the document associated with label j, p(j|i) represents the frequency of word j in the document associated with label i, and p(j) is the frequency of word j in the corpus; the higher its value, the higher the relevance of the word to the label;

[0042] For the edges between labels, the logarithm of the co-occurrence frequency in the training data is used:

[0043]

[0044] Where #L(i,j) represents the number of documents whose categories belong to tags i and j, and #L(i) represents the number of documents whose category is i. For parent tags and child tags in the tag hierarchy, the proportion of documents containing child tags in all parent tag categories is calculated. The higher the proportion, the stronger the correlation between the child tag and the parent tag, and the greater the weight.

[0045] The weight of the edge between them is calculated using the term frequency inverse document frequency (TF-IDF). The calculation formula is as follows:

[0046]

[0047] Among them, TF i,j (Term Frequency) represents the frequency of word i in document j, n ij Indicates the number of times word i appears in document j, n k,j Represents the total number of occurrences of all words in document j:

[0048]

[0049] Among them, IDF i (Inverse Document Frequency) represents the inverse document frequency of word i, |D| represents the number of all documents in the corpus, |j:t i ∈d j | represents the number of documents containing word i;

[0050] TF-IDF=TF*IDF

[0051] The TF-IDF value comprehensively considers the frequency of occurrence of a word in a document and the sparsity of the word itself, thus effectively measuring the relevance between a word and a document, thereby better reflecting the dependency relationship between document nodes and word nodes when constructing a heterogeneous graph;

[0052] In LDWHGCN, the ROBERTA model is used to obtain document embeddings and they are regarded as input representations of document nodes. Document node embeddings are expressed as Represented by , where d is the embedding dimension. In general, the initial node feature matrix is:

[0053]

[0054] where n doc is the number of document nodes, n word is the number of word nodes, n labelis the number of labels; X is input into the model, which trains and propagates information; where the output feature matrix L of the i-th layer (i) Calculated as:

[0055]

[0056] Where η is the activation function, is the normalized adjacency matrix, is the weight matrix of the layer, L (0) =X is the input feature matrix for model initialization.

[0057] As a preferred technical solution of the present invention, the model training step in step 4 is optimized using a memory library, specifically:

[0058] During training, a memory bank M is used to store the input features of all document nodes. At the beginning of each batch, all document embeddings are first calculated using the current ROBERTA module and stored in M. In each iteration, a mini-batch index set K = {k0, k1, ..., kn} is sampled from M, where n is the batch size, and then their document embeddings M are also calculated using the current ROBERTA module. k , and update the corresponding memory in M; Next, use the updated M as input to derive the GCN output and calculate the loss of the current mini-batch; For backpropagation, M is considered to be constant except for the records in K;

[0059] Using the memory bank, the network containing the ROBERTA module can be effectively trained. Before training begins, the dataset can be used to fine-tune the ROBERTA model and initialize the ROBERTA parameters in the model to speed up training.

[0060] As a preferred technical solution of the present invention, in step 5, interpolation ROBERTA and GCN prediction are performed, and the steps are:

[0061] In the prediction phase, the hierarchy of multi-label classification is flattened; the output of the GCN is considered as the final representation of the document, which is then input into a linear layer and the probability is calculated using the softmax function:

[0062] S GCN =softmax(G(X,A))

[0063] Where G represents the GCN model; during training, the ROBERTA classifier is used to optimize the model to achieve faster convergence and better performance, so an auxiliary classifier is constructed by directly inputting the document embedding X into a dense layer with a softmax activation function:

[0064] S ROBERTA=softmax(WX)

[0065] The final training goal is the linear interpolation of the predictions of GCN and ROBERTA, and the probability of document i appearing on label j is:

[0066] S ij =θS GCN +(1-θ)S ROBERTA

[0067] The interpolation parameter θ is used to control the weights of the GCN and ROBERTA models in the prediction process; θ = 1 means that the GCN model is used entirely for prediction, and θ = 0 means that only the ROBERTA model is used for prediction; when θ ∈ (0, 1) balances the prediction results of the two models, the entire model can be better optimized;

[0068] During model training, the binary cross loss function is used to jointly optimize the parameters of ROBERTA and GCN. The loss value of document i on label j is:

[0069]

[0070] Among them, y ij is the true value of the label, is the total loss value.

[0071] The present invention mainly studies the problem of hierarchical multi-label text classification, which is challenging in the field of natural language processing and has a wide range of applications in the real world. For text classification tasks, the embedded representation of documents or words is the key to determining the accuracy of the task. In the hierarchical multi-label text classification task, in addition to the representation of documents or words, the use of label hierarchical structure also plays a very important role. In order to solve the problem of hierarchical multi-label text classification, the present invention proposes the LDWHGCN model, which constructs the corpus into a heterogeneous graph containing label, document and word nodes, and uses the ROBERTA model to initialize the node representation, and then uses GCN for hierarchical classification. GCN iteratively aggregates the information of neighboring nodes and gradually integrates the global graph structure information into the feature representation of the node, thereby improving the representation ability of the node feature. The LDWHGCN model can make full use of label hierarchical information, document semantic information and word semantic information. It makes full use of the advantages of large-scale pre-training models and GCN learning to perform hierarchical text classification, improving classification accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is an example of hierarchical text classification.

[0073] Figure 2 It is a hierarchical text classification method.

[0074] Figure 3 It is a structural diagram of the model constructed by the present invention. DETAILED DESCRIPTION

[0075] The present invention proposes a hierarchical classification model for government hotline text based on a pre-trained model (ROBERTA) and a graph convolutional network (GCN), which includes the following steps:

[0076] Step 1: Get the dataset and process it

[0077] Obtain a dataset of text descriptions of government hotline numbers, including citizens' complaints, suggestions, and consultations. Clean the obtained dataset, remove extra spaces, special characters, and irrelevant symbols in the text to ensure the neatness and consistency of the text content, handle missing values ​​and outliers, and mark or delete outliers. Correct misclassified samples to ensure data accuracy. Use Chinese word segmentation tools to segment the cleaned text and split the text into words or phrases. Word segmentation is the basis for subsequent feature extraction and representation, and helps to more accurately capture the semantic information in the text. Divide the processed data into training set, validation set, and test set. The training set is used for model training and optimization, and the test set is used to evaluate the performance of the model.

[0078] Step 2: Build a heterogeneous graph

[0079] In the heterogeneous graph, three types of nodes are defined: document nodes, word nodes, and label nodes to fully utilize the hierarchical information.

[0080] Document node: represents each piece of text data received by the government hotline.

[0081] Word nodes: Represent words or phrases in the text.

[0082] Label node: represents the hierarchical classification label to which the text belongs.

[0083] Build edges:

[0084] Word-document edge: Build an edge between a word node and a document node based on the occurrence of the word in the document.

[0085] Document-label edge: Build an edge between a document node and a label node based on the label to which the document belongs.

[0086] Label-label edge: Build edges between label nodes based on the hierarchical parent-child relationship between labels.

[0087] Weight calculation:

[0088] Word-word edge weight: Use pointwise mutual information (PMI) to calculate the degree of association between words and construct edges.

[0089] Document-label edge weight: The weight between a document and a label is calculated using the average of the weighted co-occurrence probabilities of the words.

[0090] Label-label edge weight: The weight between labels is calculated using the logarithm of the co-occurrence frequency in the training data.

[0091] Step 3: Model initialization

[0092] Before model training begins, fine-tune the ROBERTA model using the dataset to obtain the model's weight parameters to accelerate model training. Initialize node representations: Use the pre-trained ROBERTA model to obtain document and word embeddings as initial values ​​for node representations. Set model hyperparameters such as the optimizer and learning rate.

[0093] Step 4: Model training

[0094] Joint training: The GCN model and the ROBERTA model are trained simultaneously, leveraging GCN's ability to handle complex structures combined with ROBERTA's powerful semantic representation capabilities to improve classification accuracy and efficiency. During training, a memory bank M is used to store the input features of all document nodes. That is, at the beginning of each training round, all document embeddings are first calculated using the ROBERTA model and stored in M. A batch of data with a batch size of 16 is sampled from M in each epoch of training. The data is then fed into the ROBERTA module to obtain a new embedding representation, which is updated to the corresponding document node in M, and then the GCN module is used to train the entire graph. Loss function optimization: Using the binary cross loss function, the gradient descent optimization algorithm is used to minimize the loss function and update the model parameters.

[0095] Step 5: Prediction and Evaluation

[0096] Prediction phase: Predictions are made on the text data in the test set, using the GCN output as the final representation of the document and using the softmax function to calculate the probability of each label. During training, the document embeddings obtained from the ROBERTA module are also used as input to the softmax function to calculate the probability of each label. The final predicted probability for each label is a linear interpolation of the predictions from the GCN and ROBERTA modules. Evaluation metrics: Model performance is evaluated using metrics such as Macro-F1 and Micro-F1 to reflect the model's classification accuracy and generalization ability.

[0097] The present invention is further explained in detail below with reference to Examples 1 and 2:

[0098] Example 1

[0099] Text classification task is a basic task in NLP. The text classification task needs to make full use of the features and semantic information of the text in order to accurately classify the text into the correct category. In the present invention, a Label, Document, and Word-based Heterogeneous Graph Convolutional Networks (LDWHGCN) is proposed. The network makes full use of the advantages of large-scale pre-training and graph neural networks for hierarchical text classification. The present invention constructs the corpus into a graph convolutional network module containing words, documents, labels, and successfully combines large-scale pre-training with graph convolutional networks. Since graph convolutional networks have a powerful ability to process complex structures, they have been widely used in text classification tasks in recent years. In the hierarchical text classification task of the present invention, labels, documents, and words are constructed as nodes to promote their respective learning. This graph convolutional network structure design of labels, documents, and words takes into account the close connection between the three. The overall structure of the model designed by the present invention is as follows Figure 3 shown.

[0100] 1. Construction of heterogeneous graph

[0101] The present invention constructs a heterogeneous graph containing label, document, and word nodes to fully utilize hierarchical information. Note that the present invention does not connect words and labels, as connecting words and labels would result in an excessive number of edges connecting each node. To avoid this, the GCN network continuously integrates information from neighboring nodes at each layer, causing node embeddings to become similar as the number of layers increases, thus affecting classification performance. Through this design, the present invention not only preserves the semantic relationship between nodes, but also avoids the problem of oversmoothing, thereby improving model performance. Unlike traditional hierarchical classification models, the present invention innovatively incorporates label nodes into the heterogeneous graph structure, thereby fully utilizing the rich semantic information inherent in the categories in hierarchical text classification. This design not only enhances the interaction between category information and document nodes, but also promotes semantic learning of category, document, and word nodes. Ultimately, the document representation output by the GCN is fed into a softmax classifier for prediction. By combining the advantages of pre-trained models and graph models, the present invention achieves more accurate hierarchical text classification. In the constructed heterogeneous graph, the weights of word-word edges are calculated using point-wise mutual information (PMI). Pointwise mutual information is a common method to measure the degree of association between two words i and j. The pointwise mutual information value of words i and j is calculated as:

[0102]

[0103] Where p(i,j) is the probability of two words i and j appearing together, p(i) is the probability of word i appearing, p(j) is the probability of word j appearing, #w represents the number of sliding windows, #w(i) is the number of sliding windows in which the word appears, and #w(i,j) indicates that there is little or no semantic correlation between word i and the sliding windows in which it appears together. Therefore, the present invention only constructs edges when the point-by-point mutual information value of the word pair is positive. For edges between tags and documents, the average value of the co-occurrence probability after word weighting is used for calculation:

[0104]

[0105] Among them, (LW) ij represents the weight between document i and label j, n i represents the total number of words in documents associated with label j, p(j|i) represents the frequency of word j in documents associated with label i, and p(j) is the frequency of word j in the corpus. The higher its value, the higher the relevance of the word to the label. This calculation method ensures that the more frequently a word appears in articles of a certain category, the greater the relevance of the word to the category label, and the greater the weight of the corresponding document with the category label. For the edges between labels, the logarithm of the co-occurrence frequency in the training data is used for calculation:

[0106]

[0107] Where #L(i,j) represents the number of documents belonging to categories i and j, and #L(i) represents the number of documents belonging to category i. For parent and child tags in the tag hierarchy, the proportion of documents containing the child tag among all parent tag categories is calculated. A higher proportion indicates a stronger association between the child tag and the parent tag, and a greater weight.

[0108] In order to better reflect the association between document nodes and word nodes when constructing a heterogeneous graph, the present invention uses term frequency-inverse document frequency (TF-IDF) to calculate the weight of the edge between them. Term frequency-inverse document frequency is a statistical indicator commonly used in information retrieval, which is used to measure the importance of a word in a specific document. It is widely used in text classification tasks and can effectively process the dependency between document nodes and word nodes. Specifically, term frequency-inverse document frequency combines the frequency of a word in a document and the distribution of the entire expected library to more accurately reflect the relevance between the word and the document. The calculation formula is as follows:

[0109]

[0110] Among them, TF i,j (Term Frequency) represents the frequency of word i in document j, n ijIndicates the number of times word i appears in document j, n k,j Represents the total number of occurrences of all words in document j:

[0111]

[0112] Among them, IDF i (Inverse Document Frequency) represents the inverse document frequency of word i, |D| represents the number of all documents in the corpus, |j:t i ∈d j | represents the number of documents containing word i.

[0113] TF-IDF=TF*IDF

[0114] The TF-IDF value comprehensively considers the frequency of occurrence of a word in a document and the sparsity of the word itself, so it can more effectively measure the relevance between a word and a document, thereby better reflecting the dependency relationship between document nodes and word nodes when constructing a heterogeneous graph.

[0115] In LDWHGCN, the ROBERTA model is used to obtain document embeddings and they are regarded as input representations of document nodes. Document node embeddings are expressed as Represented by , where d is the embedding dimension. In general, the initial node feature matrix is:

[0116]

[0117] where n doc is the number of document nodes, n word is the number of word nodes, n label is the number of labels. X is input to the model, which trains and propagates information. The output feature matrix L of the i-th layer (i) Calculated as:

[0118]

[0119] Where η is the activation function, is the normalized adjacency matrix, is the weight matrix of the layer, L (0) =X is the input feature matrix for model initialization.

[0120] 2. Interpolating ROBERTA and GCN predictions

[0121] In the prediction phase, the hierarchy of multi-label classification is flattened. The output of the GCN is considered as the final representation of the document, which is then input into the linear layer and the probability is calculated using the softmax function:

[0122] SGCN =softmax(G(X,A))

[0123] Where G represents the GCN model. During training, the ROBERTA classifier is used to optimize the model to achieve faster convergence and better performance. Therefore, an auxiliary classifier is constructed by directly inputting the document embedding X into a dense layer with a softmax activation function:

[0124] S ROBERTA =softmax(WX)

[0125] The final training goal is the linear interpolation of the predictions of GCN and ROBERTA, and the probability of document i appearing on label j is:

[0126] S ij =θS GCN +(1-θ)S ROBERTA

[0127] The interpolation parameter θ is used to control the weights of the GCN and ROBERTA models during the prediction process. θ = 1 means that the GCN model is used entirely for prediction, while θ = 0 means that only the ROBERTA model is used for prediction. When θ∈(0,1), the prediction results of the two models are balanced, which can better optimize the entire model. The advantages of the interpolation mechanism are:

[0128] ROBERTA operates directly on the GCN input, ensuring that the GCN input is adjusted and optimized towards the target. This helps the multi-layer GCN model overcome inherent defects such as gradient vanishing or over-smoothing, thereby achieving better performance.

[0129] During model training, the binary cross loss function is used to jointly optimize the parameters of ROBERTA and GCN. The loss value of document i on label j is:

[0130]

[0131] Among them, y ij is the true value of the label, is the total loss value.

[0132] 3. Use memory library for optimization

[0133] The original GCN model uses full-batch gradient descent for training, but this method requires storing all node information. Roberta also requires a large amount of memory during training, which makes full-batch methods unsuitable for Roberta. To address this issue, a memory bank M is introduced to store all document embeddings, decoupling the training batch size from the total number of nodes in the graph.

[0134] Specifically, during training, a memory bank M is used to store the input features of all document nodes. At the beginning of each batch, all document embeddings are first calculated using the current ROBERTA module and stored in M. In each iteration, a mini-batch index set K = {k0, k1, ..., kn} is sampled from M, where n is the batch size, and then the present invention also calculates their document embeddings M using the current ROBERTA module. k , and update the corresponding memory in M. Next, use the updated M as input to derive the GCN output and calculate the loss of the current mini-batch. For backpropagation, M is considered to be constant except for the records in K.

[0135] Using the memory bank, networks containing the ROBERTA module can be efficiently trained. However, during training, using the ROBERTA module to calculate the embeddings in the memory bank at different steps within a batch can lead to inconsistent embeddings. To overcome this problem, a smaller learning rate is set for ROBERTA to improve the consistency of stored embeddings. Due to the low learning rate, training takes longer. To speed up training, the present invention uses a dataset to fine-tune the ROBERTA model and initialize the ROBERTA parameters in the model before training begins, thereby speeding up training.

[0136] Example 2 Experiment

[0137] 1. Dataset introduction and preprocessing

[0138] For the dataset, this paper uses government hotline data from a district in Chengdu in January 2023, totaling 40,457 data points. The first-level categories include eight categories, such as transportation and environmental supervision, and the second-level categories include 30 categories, such as market supervision and labor relations. Table 1 shows the detailed information of the dataset. The data in the table shows that the sample label distribution is very unbalanced. For example, the second-level category "market supervision" has 22,105 data points, while "solid waste pollution (environmental supervision)" has only 50 data points, a significant difference.

[0139] Because the original dataset contains a significant amount of noise—for example, a small number of data points have the same primary and secondary labels, and some data points have null secondary labels—we reclassify these samples. We remove extra spaces and special characters from the text content of the samples, and then perform word segmentation on the text. The dataset is divided into training, validation, and test sets, totaling 25,892, 6,473, and 8,092 entries, respectively.

[0140] Table 1: Dataset statistics

[0141]

[0142]

[0143] 2. Evaluation Metrics

[0144] For hierarchical multi-label text classification problems, the choice of evaluation indicators is also crucial. In order to more accurately reflect the performance of the model, this paper uses Macro-F1 and Micro-F1 to evaluate the experimental results. These two indicators are used to evaluate multi-classification tasks and can be calculated based on precision and recall. The Precision and Recall of the i-th class can be expressed as:

[0145]

[0146] Among them TP i is the number of predicted positives that are actually positive, FP i is the number of predicted positive results that are actually negative, FN i It is the number of predicted negative numbers that are actually positive. Micro-F1 calculation method:

[0147]

[0148] Since Micro-F1 calculation takes into account the number of categories, it is more suitable for cases where the data distribution is unbalanced. In this case, the larger the number of categories, the greater the impact of Micro-F1. Macro-F1 calculation method:

[0149]

[0150] Since Macro-F1 averages the Precision and Recall of each category, it does not take the data quantity into account. In this case, categories with higher Precision and Recall will have a greater impact on Macro-F1.

[0151] 3. Experimental Setup

[0152] Since the government hotline dataset is in Chinese, the pre-training model uses Chinese-RoBerta, and the feature embedding dimension dh size is 768. Adam is used as the optimizer. The batch size is set to 16 in the training set and the batch size is set to 32 in the test set. The sliding window w size is set to 20. The deep learning framework PyTorch1.9.0 is used to implement the model of the present invention and the model is trained end-to-end. The present invention trains the model with the training set, and after each batch, the model is evaluated with the validation set, and the training is stopped if Macro-F1 does not increase within 6 batches. The early stopping setting is to stop training if Macro-Fl does not increase within 6 batches. The weight parameter θ is set to 0.7. For ROBERTA, the present invention uses the output features of the [CLS] token as document embedding and uses two layers of GCN to implement the model. For the learning rate, in the GCN module, the initial learning rate is 1e -3 For the fine-tuned ROBERTA module, the initial learning rate of this invention is 1e -5 .

[0153] 4. Experimental results

[0154] The present invention is compared with several powerful hierarchical text classification baselines. HiAGM proposes a hierarchical attention mechanism to obtain hierarchical representation of text. HTCInfoMax adopts text label mutual information maximization and label prior matching technology. HiMatch groups text representations with their hierarchical label representations, converting the problem into a matching problem. HGCLR directly embeds the hierarchical structure into the text encoder for hierarchical text classification and performs contrastive learning on the generated positive samples. HiTIN uses structural entropy as a guide to convert the label hierarchy into an unweighted tree structure. The structural encoder is designed to integrate hierarchical perceptual information in the encoding tree into the text representation.

[0155] In order to verify the superiority of the present invention over other models in the hierarchical text classification task, the present invention added a comparative experiment with the baseline model to study the performance of the mainstream hierarchical text classification algorithm in recent years. In order to fully demonstrate the limitations of the present invention, the GCN in the model was replaced with GAT for experimentation. GAT (Graph Attention Network) is a model based on graph neural network. GAT introduces an attention mechanism between nodes to enable the model to better capture the relationship and feature information between nodes. The experimental results are shown in Table 2. The experimental results show that the present invention is very competitive.

[0156] As can be seen in the table, LDWHGCN and LDWHGAT perform best in the dataset. It should be noted that the baseline model, HGCLR, directly injects the label hierarchy into the text encoder and performs contrastive learning on the generated positive samples. This can introduce noise and ignore label correlations. Compared to HGCLR, our proposed method incorporates labels into the heterogeneous graph and links them to text via GCN, significantly improving classification performance.

[0157] Compared to the state-of-the-art hierarchical text classification model, HiTIN, our method improves Macro-F1 by 0.84% ​​and Macro-F1 by 1.08%, respectively, demonstrating its effectiveness. Because the label distribution in this dataset is highly imbalanced, a common situation in real-world datasets, our model incorporates labels into a heterogeneous graph, fully leveraging the connections between labels and between labels and text. This improves the performance of hierarchical text classification, demonstrating the practical application of our method.

[0158] Table 2: Performance comparison of text classification models at different levels (LDWHGAT replaces GCN in the model with GAT, and bold indicates the best)

[0159] Model Micro-F1 Macro-F1 HiAGM(2020) 94.67 82.27 HTC InfoMax (2021) 94.52 83.41 MiMatch(2021) 94.72 83.65 BERT 94.80 81.25 HGCLR(2022) 95.00 82.74 HiTIN(2023) 95.14 83.95 LDWHGAT 95.92 85.01 LDWHGCN 95.98 85.03

[0160] 5 Ablation Experiment

[0161] In order to verify the performance of the invented model, the following ablation experiments were conducted to evaluate the effectiveness of the improvement strategy. As can be seen from Table 3, when the labels are removed from the heterogeneous graph, only document nodes and word nodes remain in the graph. Regardless of whether GCN or GAT is used for graph propagation learning, the performance of the model drops significantly. This shows that incorporating labels into heterogeneous graphs can effectively improve model performance. In hierarchical text classification, the input text corresponds to multiple labels in a label structure. By constructing the labels as nodes in a heterogeneous graph, the model can learn a more discriminative embedding representation. Removing the label nodes reduces the correlation between labels and between documents and labels, resulting in a decline in model performance. This shows that the introduction of label information effectively guides the model to learn embedded representations and improves classification accuracy.

[0162] Table 3: Ablation experiment (GCN w / o label means removing the label nodes in the heterogeneous graph and using GCN to learn the graph, GAT w / olabel means removing the label nodes in the heterogeneous graph and using GAT to learn the graph)

[0163] model Micro-F1 Macro-F1 Baseline GCN 95.98 85.03 Baseline GAT 95.92 85.01 GCN w / o label 95.25 84.86 GAT w / o label 94.83 84.72

[0164] The above content is merely an example and explanation of the concept of the present invention. Those skilled in the art may make various modifications or additions to the described specific embodiments or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the claims, they should all fall within the scope of protection of the present invention.

Claims

1. A hierarchical classification method for government hotline text based on pre-trained models and graph convolutional networks, characterized by: Here are the steps: Step 1: Get the dataset and process it Obtain a dataset of text descriptions of government hotline numbers and clean it. Remove extra spaces, special characters, and irrelevant symbols from the text to ensure neatness and consistency. Handle missing values ​​and outliers by marking or deleting them. Use a Chinese word segmentation tool to segment the cleaned text into words or phrases. Divide the processed data into a training set, a validation set, and a test set. The training set is used for model training and optimization, and the test set is used to evaluate model performance. Step 2: Build a heterogeneous graph Define three types of nodes: document nodes, word nodes, and tag nodes; build edges and calculate weights; Step 3: Model initialization Before model training begins, the ROBERTA model is fine-tuned using the dataset to obtain the weight parameters of the ROBERTA model to speed up model training. Node representation initialization: The pre-trained ROBERTA model is used to obtain document embeddings and word embeddings as the initial values ​​of node representations. Step 4: Model training Joint training: GCN and ROBERTA models are trained simultaneously, leveraging GCN's ability to handle complex structures and ROBERTA's powerful semantic representation capabilities to improve classification accuracy and efficiency. During training, a memory bank M is used to store input features of all document nodes. That is, at the beginning of each training round, the ROBERTA model is first used to calculate all document embeddings and store them in M. In each training iteration, a batch of data with a batch size of 16 is sampled from M. The data is then fed into the ROBERTA module to obtain a new embedding representation, which is updated to the corresponding document node in M. The GCN module is then used to train the entire graph. Loss function optimization: Using the binary cross loss function, the gradient descent optimization algorithm is used to minimize the loss function and update the model parameters. Step 5: Prediction and Evaluation Prediction phase: Predictions are made on the text data in the test set, with the GCN output considered as the final representation of the document, and the softmax function is used to calculate the probability of each label. During the training process, the document embedding obtained by the ROBERTA module is also used as input to the softmax function to calculate the probability of each label. The final predicted probability of each label is a linear interpolation of the predictions of GCN and ROBERTA. Evaluation metrics: Macro-F1 and Micro-F1 are used as evaluation metrics to evaluate the performance of the model to reflect the classification accuracy and generalization ability of the model.

2. The hierarchical classification method for government hotline texts according to claim 1, characterized in that: The three types of nodes defined in step 2 are: Document node: represents each piece of text data received by the government hotline; Word nodes: represent words or phrases in the text; Label node: represents the hierarchical classification label to which the text belongs; The constructed edges are: Word-document edge: constructs an edge between a word node and a document node based on the occurrence of the word in the document; Document-label edge: constructs an edge between a document node and a label node based on the label to which the document belongs; Label-label edge: constructs edges between label nodes based on the hierarchical parent-child relationship between labels; The weight is calculated as: Word-word edge weight: Use point-wise mutual information (PMI) to calculate the degree of association between words and construct edges; Document-label edge weight: The weight between the document and the label is calculated using the average of the weighted co-occurrence probabilities of the words; Label-label edge weight: The weight between labels is calculated using the logarithm of the co-occurrence frequency in the training data.

3. The hierarchical classification method for government hotline texts according to claim 2, characterized in that: In the constructed heterogeneous graph in step 2, the weight of the word-word edge is calculated using point-wise mutual information (PMI). Point-wise mutual information is a common method for measuring the degree of association between two words i and j. The point-wise mutual information value of words i and j is calculated as: Where p(i,j) is the probability of two words i and j appearing together, p(i) is the probability of word i appearing, p(j) is the probability of word j appearing, #w represents the number of sliding windows, #w(i) is the number of sliding windows in which a word appears, and #w(i,j) indicates that there is little or no semantic correlation between word i and the sliding windows in which it appears together. For the edge between a label and a document, the average of the weighted co-occurrence probabilities of the words is used for calculation: Among them, (LW) ij represents the weight between document i and label j, n i represents the total number of words in the document associated with label j, p(j|i) represents the frequency of word j in the document associated with label i, and p(j) is the frequency of word j in the corpus; the higher its value, the higher the relevance of the word to the label; For the edges between labels, the logarithm of the co-occurrence frequency in the training data is used: Where #L(i,j) represents the number of documents whose categories belong to tags i and j, and #L(i) represents the number of documents whose category is i. For parent tags and child tags in the tag hierarchy, the proportion of documents containing child tags in all parent tag categories is calculated. The higher the proportion, the stronger the correlation between the child tag and the parent tag, and the greater the weight. The weight of the edge between them is calculated using the term frequency inverse document frequency (TF-IDF). The calculation formula is as follows: Among them, TF i,j (Term Frequency) represents the frequency of word i in document j, n ij Indicates the number of times word i appears in document j, n k,j Represents the total number of occurrences of all words in document j: Among them, IDF i (Inverse Document Frequency) represents the inverse document frequency of word i, |D| represents the number of all documents in the corpus, |j:t i ∈d j | represents the number of documents containing word i; TF-IDF=TF*IDF The TF-IDF value comprehensively considers the frequency of occurrence of a word in a document and the sparsity of the word itself, thus effectively measuring the relevance between a word and a document, thereby better reflecting the dependency relationship between document nodes and word nodes when constructing a heterogeneous graph; In LDWHGCN, the ROBERTA model is used to obtain document embeddings and they are regarded as input representations of document nodes. Document node embeddings are expressed as Represented by , where d is the embedding dimension. In general, the initial node feature matrix is: where n doc is the number of document nodes, n word is the number of word nodes, n label is the number of labels; X is input into the model, which trains and propagates information; where the output feature matrix L of the i-th layer (o) Calculated as: Where η is the activation function, is the normalized adjacency matrix, is the weight matrix of the layer, L (0) =X is the input feature matrix for model initialization.

4. The hierarchical classification method for government hotline texts according to claim 3, characterized in that: The model training step in step 4 is optimized using the memory library, specifically: During training, a memory bank M is used to store the input features of all document nodes. At the beginning of each batch, all document embeddings are first calculated using the current ROBERTA module and stored in M. In each iteration, a mini-batch index set K = {k0, k1, ..., kn} is sampled from M, where n is the batch size, and then their document embeddings M are also calculated using the current ROBERTA module. k , and update the corresponding memory in M; Next, use the updated M as input to derive the GCN output and calculate the loss of the current mini-batch; For backpropagation, M is considered to be constant except for the records in K; Using the memory bank, the network containing the ROBERTA module can be effectively trained. Before training begins, the dataset can be used to fine-tune the ROBERTA model and initialize the ROBERTA parameters in the model to speed up training.

5. The hierarchical classification method for government hotline texts according to claim 4, characterized in that: In step 5, interpolation ROBERTA and GCN predictions are performed as follows: In the prediction phase, the hierarchy of multi-label classification is flattened; the output of the GCN is considered as the final representation of the document, which is then input into a linear layer and the probability is calculated using the softmax function: S GCN =softmax(G(X,A)) Where G represents the GCN model; during training, the ROBERTA classifier is used to optimize the model to achieve faster convergence and better performance, so an auxiliary classifier is constructed by directly inputting the document embedding X into a dense layer with a softmax activation function: S ROBERTA =softmax(WX) The final training goal is the linear interpolation of the predictions of GCN and ROBERTA, and the probability of document i appearing on label j is: S ij =θS GCN +(1-θ)S ROBERTA The interpolation parameter θ is used to control the weights of the GCN and ROBERTA models in the prediction process; θ = 1 means that the GCN model is used entirely for prediction, and θ = 0 means that only the ROBERTA model is used for prediction; when θ ∈ (0, 1) balances the prediction results of the two models, the entire model can be better optimized; During model training, the binary cross loss function is used to jointly optimize the parameters of ROBERTA and GCN. The loss value of document i on label j is: Among them, y ij is the true value of the label, is the total loss value.

Images