A hierarchical text classification method based on pre-trained generative model

Abstract

The present invention discloses a hierarchical text classification method based on a pre-trained generative model, the purpose of which is to solve the problems of label inconsistency, the need to manually set thresholds, and the inability to effectively learn the co-occurrence relationship between target labels in existing hierarchical text classification methods. The present invention transforms the hierarchical text classification problem into a hierarchical label sequence generation problem, and by constructing a label sequence that incorporates hierarchical information, the model can learn the hierarchical information and co-occurrence relationship between labels, thereby solving the label inconsistency problem. Moreover, the label generation process of this method adopts an autoregressive method, so there is no need to manually set a threshold in the process of predicting labels. Experiments show that the classification performance of the present invention is better than that of other hierarchical text classification methods.

Description

Technical Field

[0001] The invention relates to a hierarchical text classification method based on a pre-training generation model, and belongs to the field of natural language processing in the computer field. Background Art

[0002] Hierarchical text classification (HTC) is an important subtask under multi-label text classification. Different from multi-label text classification, the labels in hierarchical text classification have a hierarchical structure, which is usually represented by a tree or a directed acyclic graph. At present, the mainstream hierarchical text classification algorithm takes the hierarchical structure of text and labels as input during the training process, uses CNN and RNN series models to obtain text representation, and uses graph convolutional neural network GCN to encode the hierarchical information between labels to obtain label representation that incorporates hierarchical structure information, and then projects the text representation and label representation into the same space, and finally uses a multi-label classifier to calculate the probability of each category. This type of method has obvious defects:

[0003] 1) The model finally obtains the probability of each category through a multi-label classifier without considering the consistency of the labels.

[0004] 2) After obtaining the probability of each category, it is often necessary to manually set a threshold to filter out the final predicted label. Different thresholds often have different effects. In order to obtain the best results, it often takes a lot of time to adjust.

[0005] 3) The hierarchical structure of labels is static and has nothing to do with text samples. Therefore, for different text samples, the co-occurrence relationship between target labels cannot be effectively learned. Summary of the invention

[0006] Purpose of the invention: In view of the above-mentioned prior art, a hierarchical text classification method based on a pre-trained generative model is proposed

[0007] Technical solution: A hierarchical text classification method based on a pre-trained generative model, comprising the following steps:

[0008] 1) Construction of label vocabulary;

[0009] 2) Construction of hierarchical label sequences: The labels corresponding to the samples are converted into hierarchical label sequences through the hierarchical structure of labels and predefined sequence construction rules;

[0010] 3) Preprocessing of training data: segment the text using a segmentation algorithm, count the length of all segmented texts, set the maximum text length based on the distribution of lengths, and truncate the text that exceeds the maximum text length;

[0011] 4) After training the model with the training set, the model is obtained, and the effect of the model is tested with the test set.

[0012] Preferably, in step 1): using the pre-trained model BART, in the hierarchical label sequence generation task of the pre-trained model BART text, the input of the decoder part of the pre-trained model BART is a label sequence, the label sequence includes multiple labels, one word represents one label, and a label vocabulary is constructed.

[0013] Preferably, the implementation process of step 2) is:

[0014] 2.1) The sequence set consisting of the label vocabulary is regarded as a label language, and the grammatical rules of three label languages ​​are constructed: top-down hierarchical traversal rule, bottom-up hierarchical traversal rule and path rule; wherein the top-down hierarchical traversal rule adopts a hierarchical traversal of a multi-branch tree, and in each layer, each label is traversed from left to right. If it is a label corresponding to the text, the label will be added to the hierarchical label sequence to obtain the final hierarchical label sequence; the hierarchical label sequence obtained by the bottom-up hierarchical traversal mode is in the opposite order to the hierarchical label sequence obtained by the top-down hierarchical traversal rule; the path rule traverses each label path from top to bottom in a left-to-right order, and each label path is connected with a special symbol;

[0015] 2.2) According to the hierarchical structure and grammatical rules of the labels, the labels corresponding to the samples are converted into hierarchical label sequences.

[0016] Preferably, the implementation process of step 3) is:

[0017] 3.1) Segment the text using a word segmentation algorithm and count the length of all segmented texts;

[0018] 3.2) Set the maximum text length according to the distribution of text length, and truncate the text that exceeds the maximum text length;

[0019] 3.3) Calculate the maximum text length based on the length of the statistical text.

[0020] Preferably, in step 3.2), it is ensured that the maximum text length does not exceed the maximum length 1024 that the BART model can learn, and at the same time, the number of truncated texts does not exceed 5% of the total number of texts.

[0021] Preferably, the implementation process of step 4 is:

[0022] 4.1) Use the training set to train the model, and use the validation set to evaluate the effect of each training step during the training process. Select the model with the best evaluation index Micro-F1 or Macro-F1 in the validation set as the final model;

[0023] 4.2) After training, test the model; use the evaluation indicators Micro-F1 and Macro-F1 to evaluate the classification performance of the model.

[0024] Beneficial effects: A method for converting hierarchical text classification into hierarchical label sequence generation is provided, which converts labels organized in a hierarchical structure into a label sequence that incorporates hierarchical rules. The labels in such a label sequence have a progressive relationship, which can better represent the hierarchical information between labels and also solve the problem of label inconsistency. In addition, since the prediction of labels adopts a generative paradigm, there is no need to manually set the threshold for label prediction. Experiments show that the classification performance of the present invention is better than other methods of hierarchical text classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the overall flow of the proposed algorithm framework. DETAILED DESCRIPTION

[0026] The present invention will be further explained below in conjunction with the accompanying drawings.

[0027] The overall framework of the attack algorithm proposed in this invention is as follows: Figure 1 As shown in the figure, it is divided into three parts: 1. Construction of label vocabulary. 2. Construction of hierarchical label sequence. 3. Preprocessing of training data. 4. Training and testing of the model.

[0028] Please refer to Figure 1 :A hierarchical text classification method based on a pre-trained generative model, comprising the following steps:

[0029] 5) Construction of label vocabulary;

[0030] 6) Construction of hierarchical label sequences: The labels corresponding to the samples are converted into hierarchical label sequences through the hierarchical structure of labels and predefined sequence construction rules;

[0031] 7) Preprocessing of training data: segment the text using a segmentation algorithm, count the length of all segmented texts, set the maximum text length based on the distribution of lengths, and truncate the text that exceeds the maximum text length;

[0032] 8) After training the model with the training set, the model is obtained and the effect of the model is tested with the test set.

[0033] In step 1): using the pre-trained model BART, in the hierarchical label sequence generation task of the pre-trained model BART text, the input of the decoder part of the pre-trained model BART is a label sequence, wherein the label sequence includes multiple labels, one word is used to represent one label, and a label vocabulary is constructed.

[0034] The implementation process of step 2) is:

[0035] 2.1) The sequence set consisting of the label vocabulary is regarded as a label language, and the grammatical rules of three label languages ​​are constructed: top-down hierarchical traversal rule, bottom-up hierarchical traversal rule and path rule; wherein the top-down hierarchical traversal rule adopts a hierarchical traversal of a multi-branch tree, and in each layer, each label is traversed from left to right. If it is a label corresponding to the text, the label will be added to the hierarchical label sequence to obtain the final hierarchical label sequence; the hierarchical label sequence obtained by the bottom-up hierarchical traversal mode is in the opposite order to the hierarchical label sequence obtained by the top-down hierarchical traversal rule; the path rule traverses each label path from top to bottom in a left-to-right order, and each label path is connected with a special symbol;

[0036] 2.2) According to the hierarchical structure and grammatical rules of the labels, the labels corresponding to the samples are converted into hierarchical label sequences.

[0037] The implementation process of step 3) is:

[0038] 3.1) Segment the text using a word segmentation algorithm and count the length of all segmented texts;

[0039] 3.2) Set the maximum text length according to the distribution of text length, and truncate the text that exceeds the maximum text length;

[0040] 3.3) Calculate the maximum text length based on the length of the statistical text.

[0041] In step 3.2), ensure that the maximum text length does not exceed the maximum length 1024 that the BART model can learn, and ensure that the number of truncated texts does not exceed 5% of the total number of texts.

[0042] The implementation process of step 4 is:

[0043] 4.1) Use the training set to train the model, and use the validation set to evaluate the effect of each training step during the training process. Select the model with the best evaluation index Micro-F1 or Macro-F1 in the validation set as the final model;

[0044] 4.2) After training, test the model; use the evaluation indicators Micro-F1 and Macro-F1 to evaluate the classification performance of the model.

[0045] Embodiment 1:

[0046] (1) Construction of label vocabulary

[0047] In this method, the pre-trained model BART is used as the backbone network of the model.

[0048] The original BART model uses the same vocabulary at the input of the encoder and decoder. The vocabulary is obtained through the BPE algorithm, and the vocabulary size is 50265. The word segmentation method of the model proposed in the present invention follows the setting of the BART model in the encoder part. However, in the task of generating hierarchical label sequences of text, the input of the decoder part is a label sequence, and each word represents a category. For different data sets, the total number of predefined categories is also different. Therefore, the vocabulary constructed by the BPE algorithm is not suitable for the task of this article. In order to solve the above problems, HTC-BART regards each category predefined in the data set as a word separately, and constructs another special vocabulary-the label vocabulary separately.

[0049] (2) Construction of hierarchical label sequences

[0050] According to the characteristics of text generation, the first generated label will be added to the prediction process of the subsequent label, that is, the first generated label can guide the generation of the subsequent label. On the one hand, for the labels on a certain path in the hierarchy, the information of these labels has an intersection. If a label on the path has been generated, it will guide the generation of other labels on the path. On the other hand, intuitively, regular label sequences must be easier to learn than irregular label sequences. In language, this rule is also called grammar. This paper also regards the sequence set composed of labels as a vocabulary as a special language - label language, and constructs three different label grammar rules for this label language.

[0051] -Top-down hierarchical traversal mode. This mode uses a multi-branch tree hierarchical traversal. In each layer, each category is traversed from left to right. If it is a label corresponding to the text, the label will be added to the hierarchical label sequence. The characteristic of this mode is that the high-level coarse-grained labels are in front and the low-level fine-grained labels are in the back. Generally speaking, the number of samples corresponding to the low-level fine-grained labels on the same path is much less than the number of samples corresponding to the high-level coarse-grained labels. Therefore, it is more difficult to predict the bottom-level fine-grained labels than the high-level coarse-grained labels. Setting the high-level coarse-grained labels in front can guide the generation of fine-grained labels.

[0052] - Bottom-up hierarchical traversal mode. The hierarchical label sequence generated by this mode is in the opposite order to that generated by the top-down hierarchical traversal mode.

[0053] -Path mode. This mode traverses each label path from top to bottom in order from left to right, and connects the paths with the '&' symbol. This setting ensures that when predicting the label of a moment, the label most relevant to the moment is the label of moment -1.

[0054] (3) Preprocessing of training data

[0055] In this method, the text is first segmented by a word segmentation algorithm, and the length of all segmented texts is counted. The maximum text length is set according to the distribution of length, and the text that exceeds the maximum text length is truncated. The maximum length should not be too large or too small. It is necessary to ensure that the maximum length does not exceed the maximum length of 1024 that the BART model can learn, and the truncated text is as small as possible. In this method, when the maximum length does not exceed 1024, the number of truncated texts does not exceed 5% of the total number of texts, and the maximum text length is calculated based on the length of the statistical text.

[0056] (4) Model training and testing

[0057] In order to train the model proposed in the present invention, the AdamW algorithm was selected as the optimization algorithm, the learning rate was 0.00002, and the training was performed for 100 cycles. The model was trained with a training set, wherein the data set content includes all samples, each sample includes text and corresponding multiple labels, and the data set is divided according to training set + validation set: test set = 8:2, training set: validation set = 8:2. During the training process, the validation set is used to evaluate the effect of each training, and the model with the best performance of Micro-F1 or Macro-F1 in the validation set is selected as the final model. After the training is completed, the model is tested. Micro-F1 and Macro-F1 are used to evaluate the classification performance of the model.

[0058] This method converts labels organized in a hierarchical structure into a label sequence that incorporates hierarchical rules. In such a label sequence, the labels have a progressive relationship, which can better represent the hierarchical information between labels and solve the problem of label inconsistency. In addition, since the label prediction adopts a generative paradigm, there is no need to manually set the threshold for label prediction. Experiments show that the classification performance of this invention is better than other methods of hierarchical text classification.

[0059] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A hierarchical text classification method based on a pre-trained generative model, characterized in that: The following steps are involved: 1) Construction of label vocabulary; 2) Construction of hierarchical label sequences: The labels corresponding to the samples are converted into hierarchical label sequences through the hierarchical structure of labels and predefined sequence construction rules; 3) Preprocessing of training data: segment the text using a segmentation algorithm, count the length of all segmented texts, set the maximum text length based on the length distribution, and truncate the text that exceeds the maximum text length; 4) After training the model with the training set, the model is obtained, and the effect of the model is tested with the test set; In step 1), the pre-trained model BART is used. In the hierarchical label sequence generation task of the pre-trained model BART text, the input of the decoder part of the pre-trained model BART is a label sequence, wherein the label sequence includes multiple labels, and one word is used to represent one label, and a label vocabulary is constructed; The implementation process of step 2) is: 2.1) The sequence set consisting of the label vocabulary is regarded as a label language, and the grammatical rules of three label languages ​​are constructed: top-down hierarchical traversal rule, bottom-up hierarchical traversal rule and path rule; wherein the top-down hierarchical traversal rule adopts a hierarchical traversal of a multi-branch tree, and in each layer, each label is traversed from left to right. If it is a label corresponding to the text, the label will be added to the hierarchical label sequence to obtain the final hierarchical label sequence; the hierarchical label sequence obtained by the bottom-up hierarchical traversal rule is in the opposite order to the hierarchical label sequence obtained by the top-down hierarchical traversal rule; the path rule traverses each label path from top to bottom in a left-to-right order, and each label path is connected with a special symbol; 2.2) According to the hierarchical structure and grammatical rules of the labels, the labels corresponding to the samples are converted into hierarchical label sequences.

2. A hierarchical text classification method based on a pre-trained generative model as claimed in claim 1, characterized in that: The implementation process of step 3) is: 3.1) Segment the text using a word segmentation algorithm and count the length of all segmented texts; 3.2) Set the maximum text length according to the distribution of text length, and truncate the text that exceeds the maximum text length; 3.3) Calculate the maximum text length based on the length of the statistical text.

3. A hierarchical text classification method based on a pre-trained generative model as claimed in claim 2, characterized in that: In step 3.2), ensure that the maximum text length does not exceed the maximum length of 1024 that the BART model can learn, and ensure that the number of truncated texts does not exceed 5% of the total number of texts.

4. A hierarchical text classification method based on a pre-trained generative model as claimed in claim 3, characterized in that: The implementation process of step 4) is: 4.1) Use the training set to train the model, and use the validation set to evaluate the effect of each training during the training process. Select the model with the best evaluation index Micro-F1 or Macro-F1 in the validation set as the final model; 4.2) After training, test the model; use the evaluation indicators Micro-F1 and Macro-F1 to evaluate the classification performance of the model.

Images