A multi-label text classification method based on graph convolutional network

Abstract

The present invention discloses a multi-label text classification method based on a graph neural network, and relates to the field of natural language processing. The method comprises the following steps: extracting text semantic vectors of text to be classified through a text encoder; establishing a graph G with category labels as nodes; inputting the graph G into a graph convolutional network to obtain a node representation vector of each category label; combining the node representation vector of the category label with an additionally defined category expression vector to obtain a classification layer fusion weight; inputting the text semantic vector into the classification layer, and performing category prediction on the text according to the classification layer fusion weight. The method can improve the performance of the model in multi-label text classification of various text genres, and can be combined with powerful pre-trained text feature extractors such as BERT and Elmo, and can be used in actual multi-label text classification systems to improve classification effects.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a multi-label text classification method based on a graph convolutional network. Background Art

[0002] Text classification is one of the basic problems in natural language processing, and one of its sub-problems is multi-label text classification. The goal of multi-label text classification is to label text with one or more category labels. Multi-label text classification technology has been applied in many real-world application scenarios such as label recommendation, text sentiment analysis, and text tagging in social media, and has broad application prospects.

[0003] An important feature of the multi-label text classification problem is that the category labels have multiple relationships. One example is the co-occurrence relationship, that is, different category labels can correspond to the same article at the same time. For example, in paper classification, since computational linguistics papers in recent years usually use machine learning methods, the two category labels "computational linguistics" and "machine learning" often correspond to the same article, that is, there is a certain degree of co-occurrence relationship between the two category labels. In addition, there are hierarchical relationships, semantic relevance of label texts, and so on. If the co-occurrence relationship can be effectively used to assist classification, it will help improve the classification accuracy of the multi-label classification system.

[0004] The traditional method of multi-label text classification is to establish a category expression vector for each category label in the classification layer, and predict the probability that the text to be classified belongs to a certain category based on the inner product of the category expression vector and the text semantic vector of the text to be classified. Its advantage is that the category expression vector fully models the information of each category label, but its disadvantage is that it independently predicts the probability of text belonging to different categories without considering the relationship between category labels.

[0005] The Chinese invention patent application with application number 202010868076.1, "A Chinese text classification method based on multi-feature fusion of graph neural networks", establishes a graph with the text to be classified and the named entities in the text as nodes, and models this graph through a graph neural network to obtain the node representation vector of each node. The shortcoming is that when predicting the probability of the text belonging to each category, it is still an independent prediction, and the relationship between category labels is not considered.

[0006] Zhou et al. published a paper titled "Hierarchy-Aware Global Model for Hierarchical Text Classification" at the Annual Meeting of the Association for Computational Linguistics. (58th, 2020). The paper uses graph convolutional networks and Tree-LSTM models to calculate node vectors in the label graph, and then fuses the node vectors with the text semantic vectors of the text to be classified to obtain text-specific node vectors for predicting the probability that the text belongs to a certain category. The disadvantage is that the category expression vector of the category is ignored, which affects the model's ability to represent each category.

[0007] The shortcoming of DocBERT: BERT for Document Classification, published by Adhikari et al. on the arxiv preprint platform (2019 No. 1904.08398), is that it ignores the relationship between categories.

[0008] An analysis of relevant domestic and foreign patents and related research leads to the following conclusions: Currently, in multi-label text classification scenarios, there is no method that takes into account both the correlation between category labels and the effectiveness of category label representation.

[0009] Therefore, technicians in this field are committed to developing a multi-label text classification method based on graph convolutional networks to take into account the correlation between category labels and the effectiveness of category label representation. Summary of the invention

[0010] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to balance the correlation between category labels and the effectiveness of category label representation.

[0011] To achieve the above object, the present invention provides a multi-label text classification method based on graph convolutional network, comprising the following steps:

[0012] Step 1: Extract the text semantic vector of the text to be classified through the text encoder;

[0013] Step 2: Create a graph G with category labels as nodes;

[0014] Step 3: Input the graph G into the graph convolutional network to obtain a node representation vector for each category label;

[0015] Step 4: Combine the node representation vector of the category label with the additionally defined category expression vector to obtain the classification layer fusion weight;

[0016] Step 5: Input the text semantic vector into the classification layer, and perform category prediction on the text according to the classification layer fusion weight obtained in step 4.

[0017] Furthermore, the step 1 includes: inputting the segmented text into the text encoder to obtain the text semantic vector h.

[0018] Furthermore, in step 3, each layer of graph convolution in the graph convolution network updates the node representation vector of each node based on the graph G.

[0019] Furthermore, the representation vector of the 0th layer of the graph convolutional network is the language description of the category label by the text encoder, and the given language representation vector h l .

[0020] Furthermore, each of the category labels has its own category expression vector The class expression vector is part of the model parameters.

[0021] Furthermore, in step 4, for a certain category label Li, its classification layer fusion weight l i The node is represented by a vector of class labels The vector expressing the category Obtained through fusion mechanism.

[0022] Furthermore, the fusion mechanism includes

[0023] Step 4.1: Obtain a scalar α between [0, 1] through a linear layer F with a sigmoid activation function, which represents the ratio of the node representation vector after fusion.

[0024]

[0025] Step 4.2: Use the following formula to calculate the classification layer fusion weight l i

[0026]

[0027] Furthermore, the step 5 comprises: fusion weight l of the classification layer i The inner product of the text semantic vector h predicts that the text to be classified belongs to category L i The probability p i =σ(l i h).

[0028] Furthermore, the objective function used in the method for training the model is

[0029]

[0030] Among them, y i Indicates whether the text belongs to category L i , if yes, it is 1; CE() represents the cross entropy loss function, Representative based on The text to be classified predicted by the inner product of the text semantic vector h belongs to category L i The probability p i =σ(l i h), λ is an artificially defined parameter.

[0031] Furthermore, the It is only used for auxiliary training. The predicted probability output by the model is p i .

[0032] Compared with the prior art solutions, the beneficial technical effects of the present invention include:

[0033] (1) The present invention takes into account both the relationship between category labels and the category expression vectors of category labels, and adopts a gate mechanism to flexibly determine the proportion of node representation vectors and category expression vectors in the classification layer fusion weight according to the difference between the text to be classified and the category label, thereby enhancing the adequacy of the category labels in modeling.

[0034] (2) Practice has proved that compared with the algorithm that only considers the category expression vector or node representation vector of the category label, the present invention has a better classification effect.

[0035] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a model structure diagram of a preferred embodiment of the present invention.

[0037] Figure 2 It is a training flow chart of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0039] Next, we will introduce how to use the present invention in a paper abstract classification scenario. In this scenario, given an English paper abstract, the model needs to predict the probability that the paper belongs to a certain discipline. A paper may belong to multiple disciplines. The following is an example of an abstract and its discipline:

[0040] summary:

[0041] Various hand-crafted features and metric learning methods prevail in the field of personre-identification. Compared to these methods, this paper proposes a more general way that can learn a similarity metric from imagepixels directly. By using a "siamese" deep neural network, the proposed method can jointly learn the color feature, texture feature and metric in a unified framework. The network has a symmetry structure with two sub-networks which are connected by Cosine function. To deal with the big variations of personimages, binomial deviance is used to evaluate the cost between similarities and labels, which is proved to be robust to outliers.

[0042] Compared to existing researches, a more practical setting is studied in the experiments that is training and test on different datasets (crossdataset person re-identification). Both in “intra dataset” and “cross dataset” settings, the superiorities of the proposed method are illustrated on VIPeRand PRID.

[0043] Discipline:

[0044] Computer Vision and Pattern Recognition

[0045] Machine Learning

[0046] Neural and Evolutionary Computing

[0047] In order to apply the present invention to this scenario, a text classification model must first be trained on the training set. The model architecture of the model is as follows: Figure 1 After training, the model will predict the probability that a text to be classified belongs to each category.

[0048] Next, combine Figure 2 Describe the training process.

[0049] Step 1: Text preprocessing. For the given English text to be classified, perform preprocessing to remove uppercase and lowercase letters, and then segment the text into words. The word segmentation method is determined by the text encoder in the text classification model. For example, when the text encoder is BERT, the corresponding word segmentation method is BPE (Byte Pair Encoder).

[0050] Step 2: Construct a label graph. This embodiment uses the co-occurrence relationship of category labels to construct a graph. The co-occurrence relationship means that multiple category labels correspond to the same text. The specific method is to i and L j , calculate L i When L appears j The conditional probability P(L j|L i ), if the conditional probability is greater than the manually specified threshold, a line consisting of L is added to the graph i To L j Edge

[0051] Step 3: Text encoder processing. The input of the text encoder is the text after word segmentation, and the output is the text representation vector of the entire text. This embodiment uses BERT as the text encoder, and the vector at the "[CLS]" position output by the last layer of BERT is used as the text representation vector.

[0052] Step 4: Graph encoder processing. In this embodiment, a graph convolutional network is used as a graph encoder. The input of the graph convolutional network is the label graph and the 0th layer node representation vector of each node. The label graph is represented by the adjacency matrix A (if nodes i and j are connected, then the element A in the i-th row and j-th column of the adjacency matrix ij is 1, otherwise it is 0; before entering the graph convolutional network, the category expression vector needs to be standardized. The standardization process is as follows

[0053]

[0054] The 0th-layer node representation vector is a text representation vector described by the label, and the acquisition process is as described in steps 1 to 3.

[0055] The output of the graph encoder is a node representation for each label.

[0056] Step 5: Gate fusion mechanism processing.

[0057] For label i, let its node representation vector be Category Expression Vector The text representation vector is h, then the linear layer F with sigmoid activation function is first used to obtain a scalar α between [0, 1], which represents the proportion of the node representation vector in the fusion classifier weight.

[0058]

[0059] Then the node representation vector l is obtained by the following formula: i

[0060]

[0061] Step 6: Class probability prediction. By calculating l i The inner product p of i =σ(l i h), and obtain the probability that the text belongs to category i, where σ represents the sigmoid function.

[0062] Step 7: Calculate the loss function and use the back propagation algorithm to update the model parameters. The loss function is obtained by comparing the probability predicted by the model with the actual category of the text to be classified. The loss function used in training is based on the cross entropy loss function. The cross entropy loss function can be intuitively understood as a distance measure between the probability predicted by the model and the actual category. The specific loss function is as follows:

[0063]

[0064] Among them, y i Indicates whether the text belongs to category L i , if yes, it is 1, if not, it is 0; CE() represents the cross entropy loss function, Representative based on The inner product of and h predicts that the text to be classified belongs to category L i The probability p i =σ(l i h). It should be noted that It is only used for auxiliary training. The predicted probability of the model output is p i .

[0065] During the training process, the gradient back-propagation algorithm is used to update the model parameters. In this embodiment, the parameters that need to be updated include the graph convolutional network, the parameters in the fusion mechanism, and the label representation vector; the optimization algorithm used is the Adam algorithm. At regular intervals, the loss function of the model on the validation set is evaluated, and the above process is repeated until the loss function on the validation set no longer decreases. The overall model consisting of the text encoder, graph convolutional network, and classifier is saved, such as Figure 1 shown.

[0066] Step 8: Category prediction. After the model training is completed, the text given in the embodiment is input into the saved model to obtain the probability that the text belongs to each category. The top three categories predicted by the model are:

[0067] Computer Vision and Pattern Recognition (0.869);

[0068] Machine Learning (0.572);

[0069] Neural and Evolutionary Computation (0.366);

[0070] In summary, the core idea of ​​the present invention is to introduce the correlation between category labels into the traditional classification method. There are multiple independent sub-classifiers in the traditional classification method, and each sub-classifier is responsible for predicting the probability that the text belongs to a certain category. This does not take into account the correlation between each category. The present invention builds the correlation between category labels into a graph and uses the information in the graph to enhance the traditional classification method. Specifically, the category expression vector of each category label in the traditional classifier is fused with the node representation vector of the category label to generate the classifier fusion weight. Taking into account that different articles and category labels have different requirements for node representation vectors, the present invention dynamically determines the proportion of node representation vectors in the classifier fusion weight by adopting a gate mechanism. In this way, the advantages of the traditional method of fully modeling category label information are retained, and the correlation between categories is considered during classification, taking into account the correlation between category labels and the effectiveness of category label representation.

[0071] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A multi-label text classification method based on graph convolutional network, characterized in that: The following steps are involved: Step 1: extract the text semantic vector h of the text to be classified through a text encoder; Step 2: Create a graph G with category labels as nodes; Step 3: Input the graph G into the graph convolutional network to obtain a node representation vector for each category label; Step 4: Combine the node representation vector of the category label with the additionally defined category expression vector to obtain the classification layer fusion weight; Step 5: input the text semantic vector into the classification layer, and perform category prediction on the text according to the classification layer fusion weight obtained in step 4; In step 4, for a certain category label L i , its classification layer fusion weight l i The node is represented by a vector of class labels The vector expressing the category Obtained through fusion mechanism; The fusion mechanism includes Step 4.1: Get the scalar α between [0,1] through the linear layer F with sigmoid activation function, which represents the ratio of the node representation vector after fusion. Step 4.2: Use the following formula to calculate the classification layer fusion weight l i 2. The multi-label text classification method based on graph convolutional network according to claim 1, characterized in that: The step 1 comprises: inputting the segmented text into the text encoder to obtain the text semantic vector h.

3. The multi-label text classification method based on graph convolutional network as claimed in claim 2, characterized in that: In step 3, each layer of graph convolution in the graph convolution network updates the node representation vector of each node based on the graph G.

4. The multi-label text classification method based on graph convolutional network as claimed in claim 3, characterized in that: The 0th layer representation vector of the graph convolutional network is the language description of the category label by the text encoder, and the given language representation vector h l .

5. The multi-label text classification method based on graph convolutional network as claimed in claim 4, characterized in that: Each of the category labels has its own category expression vector The class expression vector is part of the model parameters.

6. The multi-label text classification method based on graph convolutional network as claimed in claim 5, characterized in that: The step 5 comprises: fusing the classification layer weights l i The inner product of the text semantic vector h predicts that the text to be classified belongs to category L i The probability p i =σ(l i h).

7. The multi-label text classification method based on graph convolutional network as claimed in claim 6, characterized in that: The objective function used in the method to train the model is: Among them, y i Indicates whether the text belongs to category L i , if yes, it is 1; CE() represents the cross entropy loss function, Representative based on The text to be classified predicted by the inner product of the text semantic vector h belongs to category L i The probability p i =σ(l i h), λ is an artificially defined parameter.

8. The multi-label text classification method based on graph convolutional network as claimed in claim 7, characterized in that: Said It is only used for auxiliary training. The predicted probability output by the model is p i .

Images