Multi-label text classification method and device

Abstract

The application discloses a multi-label text classification method and device, the method comprises the following steps: obtaining a plurality of texts of known label categories, constructing a training set and a test set; inputting each text in the training set and a plurality of label categories into a BERT model to output a first embedded sequence; converting the established relationship matrix into a second embedded sequence; inputting the first embedded sequence and the second embedded sequence into a relative attention network (RAT) to output semantic correlation information and a plurality of internal correlation information, and converting the semantic correlation information and the plurality of internal correlation information into a corresponding one-dimensional vector through a bidirectional LSTM network; mapping the one-dimensional vector to the plurality of label categories in the training set through a linear network to obtain a label category prediction result of each text; calculating a loss value of the network model, updating network model parameters according to the loss value to obtain a trained network model; and classifying to-be-classified texts by using the tested network model, so that the accuracy of multi-label text classification can be improved under low resource conditions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a multi-label text classification method and device. BACKGROUND

[0002] This section is intended to provide background or context to the embodiments of the application recited in the claims. The description herein does not constitute admission that the prior publication, square, or subject matter described herein and / or the material contained therein is or was prior art to the claims at issue.

[0003] Single-label text classification task refers to determining which predefined category a given text belongs to, such as news category, sentiment analysis, etc. Multi-label text classification task refers to determining which predefined categories a given text belongs to, such as product label, topic label, etc. Each text can have multiple label categories, such as a product review that can contain price, quality, service, etc. different labels.

[0004] The prior art has the following defects in the process of multi-label text classification by training a classification model: 1. The association between the text to be classified and each label category is not considered; 2. The co-occurrence between each label category is not considered. Multi-label classification task is different from single-label, there is co-occurrence between labels, some label combinations appear frequently, while some combinations never appear.

[0005] The above can be summarized as the lack of prior knowledge. Without these prior knowledge, the model can only rely on a large amount of training data to obtain a classification model through training. Under the condition of insufficient training data resources, the accuracy of the classification model is affected. SUMMARY

[0006] The embodiments of the present application provide a multi-label text classification method to inject prior knowledge into the model, effectively improve the utilization rate of training data, and improve the accuracy of multi-label text classification under low resource conditions. The method comprises:

[0007] A plurality of texts of known label categories are obtained, and a training set and a test set are constructed, the training set and the test set comprising a plurality of texts and a plurality of label categories, wherein the known label categories of each text are multiple;

[0008] Each text and multiple label categories in the training set are input into a BERT model to output a first embedded sequence; a relationship matrix is established according to each text and multiple label categories in the training set, and the relationship matrix is converted into a second embedded sequence;

[0009] The first embedded sequence is input into a relative attention network RAT to output semantic association information between each text and multiple label categories; the second embedded sequence is input into the relative attention network RAT to output multiple internal association information within the range of each text and multiple label categories;

[0010] The bidirectional LSTM network is used to reduce dimensions of semantic association information and a plurality of internal association information output by the relative attention network RAT and convert the dimensions into corresponding one-dimensional vectors; the one-dimensional vectors are mapped to a plurality of label categories in the training set through a linear network; and a label category prediction result of each text is obtained according to a mapping result of the one-dimensional vectors;

[0011] According to the label category prediction result of each text and the known label category of each text, a loss value of the network model is calculated based on a loss function; when the loss value is not less than a preset threshold, the network model parameters are updated according to the loss value, and the label category prediction result of each text is continuously obtained by using the updated network model until the loss value of the network model is less than the preset threshold, and a trained network model is obtained;

[0012] The trained network model is tested according to a test set, and the network model that passes the test is used to classify the text to be classified.

[0013] The embodiment of the present application also provides a multi-label text classification device for injecting prior knowledge into a model, effectively improving the utilization rate of training data, and improving the accuracy of multi-label text classification under low resource conditions. The device comprises:

[0014] A training set construction module is configured to obtain a plurality of texts with known label categories, and construct a training set, wherein the training set comprises a plurality of texts and a plurality of label categories, and the known label categories of each text are multiple;

[0015] An embedded sequence conversion module is configured to input each text and the plurality of label categories in the training set into a BERT model to output a first embedded sequence; and establish a relationship matrix according to each text and the plurality of label categories in the training set, and convert the relationship matrix into a second embedded sequence;

[0016] An association information output module is configured to input the first embedded sequence into a relative attention network RAT to output semantic association information between each text and the plurality of label categories; and input the second embedded sequence into the relative attention network RAT to output a plurality of internal association information within a range of each text and the plurality of label categories;

[0017] A label category prediction module is configured to use a bidirectional LSTM network to reduce dimensions of semantic association information and a plurality of internal association information output by the relative attention network RAT and convert the dimensions into corresponding one-dimensional vectors; map the one-dimensional vectors to a plurality of label categories in the training set through a linear network; and obtain a label category prediction result of each text according to a mapping result of the one-dimensional vectors;

[0018] The network model updating module is configured to calculate a loss value of the network model based on a loss function according to a label category prediction result of each text and a known label category of each text, update a network model parameter according to the loss value when the loss value is not less than a preset threshold, continue to obtain the label category prediction result of each text by using the updated network model, and obtain a trained network model until the loss value of the network model is less than the preset threshold.

[0019] The text classification module is configured to test the trained network model according to a test set and classify a text to be classified by using the network model that passes the test.

[0020] The embodiment of the present application further provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and capable of being executed on the processor, and the processor implements the multi-label text classification method when executing the computer program.

[0021] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed on a processor to implement the multi-label text classification method.

[0022] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed on a processor to implement the multi-label text classification method.

[0023] In the embodiment of the present application, a plurality of texts of known label categories are obtained, and a training set and a test set are constructed; each text in the training set and a plurality of label categories are input into a BERT model, and a first embedded sequence is output; a relationship matrix is established according to each text in the training set and the plurality of label categories, and the relationship matrix is converted into a second embedded sequence; the first embedded sequence is input into a relative attention network RAT, and semantic association information between each text and the plurality of label categories is output; the second embedded sequence is input into the relative attention network RAT, and a plurality of internal association information within the range of each text and the plurality of label categories is output; the semantic association information and the plurality of internal association information output by the relative attention network RAT are reduced in dimension by using a bidirectional LSTM network, and are converted into corresponding one-dimensional vectors; the one-dimensional vectors are mapped to the plurality of label categories in the training set through a linear network; a label category prediction result of each text is obtained according to a mapping result of the one-dimensional vectors; a loss value of the network model is calculated according to the label category prediction result of each text and the known label category of each text based on a loss function, when the loss value is not less than a preset threshold, the network model parameters are updated according to the loss value, the label category prediction result of each text is continuously obtained by using the updated network model, until the loss value of the network model is less than the preset threshold, and a trained network model is obtained; the trained network model is tested according to the test set, and the text to be classified is classified by using the network model that passes the test.

[0024] The present application can effectively improve the utilization rate of training data by mining the semantic association, implicit association and internal association between the text to be classified and each category, and injecting prior knowledge into the network model, thereby greatly improving the accuracy of multi-label text classification under low resource conditions. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. In the drawings:

[0026] Figure 1 The flow chart of the multi-label text classification method in the embodiment of the present application;

[0027] Figure 2 The specific example diagram of the relationship matrix in the embodiment of the present application;

[0028] Figure 3 The method flow chart for establishing the relationship matrix between each element in the text in the embodiment of the present application;

[0029] Figure 4 A method flowchart for establishing a relationship matrix between each text and multiple label categories in an embodiment of the present application;

[0030] Figure 5 A specific example schematic diagram of a relationship matrix between a text to be classified and a category description in an embodiment of the present application;

[0031] Figure 6 A method flowchart for establishing a relationship matrix between multiple label categories in an embodiment of the present application;

[0032] Figure 7 A method flowchart for predicting a category of a text to be classified in an embodiment of the present application;

[0033] Figure 8 A specific example structural schematic diagram of a multi-label text classification device in an embodiment of the present application;

[0034] Figure 9 A structural schematic diagram of a computer device of an embodiment of the present application. DETAILED DESCRIPTION

[0035] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer and more apparent, the embodiments of the present application are further described in detail below with reference to the drawings. Herein, the illustrative embodiments of the present application and the descriptions thereof are used to explain the present application, but are not as a limitation on the present application.

[0036] First, technical terms in the embodiments of the present application are introduced:

[0037] BERT model (Bidirectional Encoder Representations from Transformers, BERT): The BERT model is a pre-trained language model (PLM), and the BERT is an extractor, which inputs a sentence (word sequence) and outputs an extracted embedding sequence. In a simple understanding, it is an encoder. We can use the transformer to pre-train the language model. Because the transformer is an encoder-decoder structure, the language model only needs the encoder part, and the BERT uses the encoder of the transformer to pre-train.

[0038] Relation Aware Self-Attention Network (RAT): RAT is a deep learning model based on Transformer. Transformer was proposed by Google team in 2017, which uses self-attention mechanism in the model training process, making the model can effectively capture global information. RAT is improved on the basis of Transformer, which is a model for embedding representation of semi-structured input sequence. RAT embeds the pre-existing relationship between elements in the sequence and integrates the pre-existing relationship between elements into the self-attention mechanism.

[0039] Bi-directional Long Short-Term Memory (BiLSTM): is a bidirectional long short-term memory network, which is an improvement based on LSTM (Long Short-Term Memory Network). LSTM is a recurrent neural network that can effectively handle long-term dependencies in sequence data. However, LSTM can only consider the context information before the current time, and cannot capture subsequent context information. Therefore, BiLSTM introduces a reverse network that can consider both forward and backward context information, thus better handling sequence data. The forward and reverse networks in BiLSTM are independent, and their outputs are concatenated as the input to the next layer. During training, the parameters of the forward and reverse networks are shared. BiLSTM has more parameters than unidirectional LSTM, but can achieve better performance in some tasks. BiLSTM is widely used in natural language processing, such as sentiment analysis, named entity recognition, natural language generation, etc. Through the extraction of bidirectional information, BiLSTM can better understand the semantic and grammatical structure of natural language.

[0040] In view of the technical defects of lacking prior knowledge, consuming a large amount of training data, and affecting the accuracy of the classification model in the process of using a classification model for multi-label text classification in the prior art, the applicant of the present application proposes a multi-label text classification method, which injects prior knowledge into the network model by mining the semantic association, implicit association, and internal association between the text to be classified and each category, which can effectively improve the utilization rate of training data and greatly improve the accuracy of multi-label text classification under low resource conditions.

[0041] The specific technical solutions of the present application are introduced below. The present application proposes a multi-label text classification method, Figure 1 The flowchart of the multi-label text classification method in the embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the multi-label text classification method in the embodiment of the present application can include: Figure 1

[0042] ​Step 101, obtaining a plurality of texts of known label categories, constructing a training set and a test set, the training set and the test set containing a plurality of texts and a plurality of label categories, wherein the known label category of each text is a plurality of label categories;

[0043] Step 102, inputting each text and a plurality of label categories in the training set into a BERT model to output a first embedded sequence; establishing a relationship matrix according to each text and a plurality of label categories in the training set, and converting the relationship matrix into a second embedded sequence;

[0044] Step 103, inputting the first embedded sequence into a relative attention network RAT to output semantic association information between each text and a plurality of label categories; inputting the second embedded sequence into the relative attention network RAT to output a plurality of internal association information within the range of each text and a plurality of label categories;

[0045] Step 104, using a bidirectional LSTM network to reduce the dimension of the semantic association information and the plurality of internal association information output by the relative attention network RAT, and converting them into corresponding one-dimensional vectors; mapping the one-dimensional vectors to a plurality of label categories in the training set through a linear network; obtaining a label category prediction result of each text according to the mapping result of the one-dimensional vector.

[0046] Step 105, based on a loss function, calculating the loss value of the network model according to the label category prediction result of each text and the known label category of each text, when the loss value is not less than a preset threshold, updating the network model parameters according to the loss value, and using the updated network model to continue to obtain the label category prediction result of each text, until the loss value of the network model is less than the preset threshold, obtaining a trained network model;

[0047] Step 106, testing the trained network model according to the test set, and classifying the text to be classified using the network model that passes the test.

[0048] In the implementation, first, a training set and a test set need to be constructed. It should be noted that in step 101, the plurality of label categories refers to all label categories that can be defined within the plurality of texts obtained, and the more comprehensive the label categories in the training set, the more accurately the trained network model can mine the plurality of association relationships between the text to be classified and each category.

[0049] After constructing the training set and the test set according to the plurality of texts of known label categories, the training set is used to train and update a plurality of network models, and the specific training process is introduced as follows:

[0050] First, the text of each known label category and the combination of multiple label categories (all label categories that can be defined within the multiple texts in the training set) are taken as a training sample, that is, the number of training samples can be the same as the number of texts of known label categories.

[0051] In step 102, for each training sample, the existing association between the elements in the training sample can be embedded before capturing the multiple association information contained in the training sample using the relative attention network RAT.

[0052] To obtain the semantic association information between each text and multiple label categories, the text sequence formed by each text and multiple label categories in the training set can be input into the BERT model to output the first embedded sequence. In specific implementation, the text sequence can include two parts: the text and the multiple label categories, wherein the text and the multiple label categories are separated by the [SEP] symbol; the categories are separated by the [SEP] symbol.

[0053] To obtain the multiple intrinsic association information within the range of each text and multiple label categories, a relationship matrix can be established according to each text and multiple label categories in the training set, and the relationship matrix is converted into a second embedded sequence.

[0054] In specific implementation, the rows and columns of the relationship matrix can be composed of the names of the text and the label category. Figure 2 A specific example of the relationship matrix in the embodiment of the present application is shown in the following figure. Figure 2 As shown, SW1 represents the first word of the text to be classified, SWn represents the nth word in the text to be classified; C1W1 represents the first word in the name of category 1, and CnWm represents the mth word in the name of category n. The relationship matrix records the multiple relationships within the range of the text to be classified and the multiple label categories.

[0055] In one embodiment, the relationship matrix can include the following relationship types: the relationship between the elements in each text, the relationship between each text and multiple label categories, and the relationship between multiple label categories.

[0056] The relationship matrix establishment method corresponding to each relationship type is introduced as follows:

[0057] Figure 3 The flow chart of the method for establishing the relationship matrix between the elements in each text in the embodiment of the present application is shown in the following figure. Figure 3 As shown, in one embodiment, the relationship matrix can be established according to each text and multiple label categories in the training set, which can include:

[0058] Step 301, for the relationship between the elements in each text, the distance between the elements in each text is calculated.

[0059] Step 302, the distance between each element in each text is taken as the relationship between each element in each text to establish a relationship matrix.

[0060] In an embodiment, the distance between each element in each text can be calculated according to the following formula:

[0061] r ij = |i-j|

[0062] Wherein, r ij represents the element in the i-th row and the j-th column of the relationship matrix.

[0063] Since r ij is a positive integer and less than the maximum length of the text, it does not need to be discretized before being converted into a second embedded sequence.

[0064] Figure 4 is a flow chart of the method for establishing a relationship matrix between each text and multiple label categories in an embodiment of the present application. As shown in Figure 4 each text and multiple label categories in the training set, the relationship matrix can include:

[0065] Step 401, for the relationship between each text and multiple label categories, a word segmentation tool is used to divide the description information of each text and multiple label categories to obtain a word segmentation result, the word segmentation result including entity words, stop words and other words;

[0066] Step 402, according to the word segmentation result, the relationship between each text and multiple label categories is determined to establish a relationship matrix; the relationship between each text and multiple label categories includes multiple relationships between entity words, stop words and other words.

[0067] In an embodiment, according to the word segmentation result, the relationship between each text and multiple label categories can include:

[0068] The relationship between entity words and entity words is queried from a pre-constructed knowledge graph, and the knowledge graph is constructed according to the association relationship between different entities;

[0069] The relationship between entity words and other words or other words and other words is determined by calculating the similarity coefficient between entity words and other words or other words and other words, the numerical range of the similarity coefficient, and the relationship corresponding to the numerical range of different similarity coefficients preset;

[0070] The relationship between stop words and entity words or stop words and other words is determined according to a manually established stop word list.

[0071] Stop Words refers to some words or characters that are automatically filtered out before or after processing natural language data (or text) in information retrieval in order to save storage space and improve search efficiency. These words or characters are called Stop Words. These stop words are manually inputted and not automatically generated. The generated stop words form a stop word list. However, there is no clear stop word list that can be applied to all tools. Even some tools explicitly avoid using stop words to support phrase search.

[0072] In specific implementation, a segmentation tool such as jieba can be used to segment each text and the description information of multiple label categories; the segmented results can be divided into three categories: Entity, StopWords and Others.

[0073] Entity represents an entity word, for example, "loan" and "bank". It can be identified by a named entity recognition tool;

[0074] StopWords represents meaningless stop words, for example, "of" and "of". It can be detected by a tool such as jieba;

[0075] Others represents words other than Entity and Stopwords.

[0076] Figure 5 A specific example of a relationship matrix between the text to be classified and the category description in the embodiment of the present application is shown in the following table: Figure 5 As shown in the table, the relationship between the elements of the text to be classified and the category description can be divided into three categories:

[0077] E-E: represents the relationship between Entity and Entity, that is, the relationship between two entities;

[0078] O-EO: represents the relationship between Entity and Others or Others and Others;

[0079] SW: represents the relationship between StopWords and Entity or Others.

[0080] The relationship between E-E is obtained by querying the knowledge graph. If the two entities have a direct association in the knowledge graph, the relationship between them is defined as follows:

[0081] E-E-isA

[0082] Indicates that one entity is a "kind" of another entity. For example, an apple is a kind of fruit, so fruit is a kind of apple.

[0083] E-E-subclassof

[0084] Indicates that one entity is a "subclass" of another entity. For example, an apple is a kind of fruit, so apple is a subclass of fruit.

[0085] E-E-partof

[0086] Indicates that one entity is a "part" of another entity. For example, a wheel is a part of a car, so wheel is a part of car.

[0087] E-E-hasA

[0088] Indicates that one entity "has" another entity. For example, a car has wheels, so car "has" wheels.

[0089] E-E-instanceof

[0090] Indicates that one entity is a specific instance of a certain "class". For example, iPhone 14 is a kind of smartphone produced by Apple, so iPhone 14 is a specific instance of the class of smartphones.

[0091] E-E-typeof

[0092] Indicates that one entity belongs to a certain "type". For example, iPhone 14 belongs to the type of smartphone.

[0093] If the two entities do not have a direct connection in the knowledge graph, the shortest path between them can be used to define it, for example:

[0094] E-E-SP-Short: The shortest path between the two entities is less than or equal to 3;

[0095] E-E-SP-Medium: The shortest path between the two entities is greater than 3 and less than or equal to 10;

[0096] E-E-SP-Long: The shortest path between the two entities is greater than 10.

[0097] The relationship between O-E is defined by the similarity of the two words, and the similarity coefficient can be calculated by tools such as sentence-transformer. The similarity coefficient is a number between 0 and 1, and according to the similarity, the relationship between O-E can be divided as follows:

[0098] O-E-Sim-High: Similarity coefficient >= 0.7;

[0099] O-E-Sim-Medium: 0.4 <= similarity coefficient < 0.7;

[0100] O-E-Sim-Low: similarity coefficient > 0.4.

[0101] Figure 6 A flow chart of the method for establishing a relationship matrix between multiple label categories in an embodiment of the present application. As shown in the figure, in one embodiment, the relationship matrix is established according to each text in the training set and the multiple label categories, which can include: Figure 6

[0102] Step 601, for the relationship between the multiple label categories, calculate the relationship coefficient between each label category, which represents the correlation density between each label category.

[0103] Step 602, according to the relationship coefficient between each label category, and the relationship corresponding to the numerical range of different relationship coefficients set in advance, determine the relationship between the multiple label categories, and establish the relationship matrix.

[0104] In one embodiment, the relationship coefficient between each label category is calculated according to the following formula:

[0105]

[0106]

[0107] Where k is the number of training data, i and j represent the i-th and j-th label in the category label.

[0108] In specific implementation, after the relationship matrix is established according to each text in the training set and the multiple label categories, since the relationship matrix needs to be converted into embedded representation when it is transmitted into the RAT network, each relationship coefficient in the relationship matrix needs to be discretely represented.

[0109] The relationship coefficient calculated according to the above formula is a value between 0 and 1. Find the quartile separation point of these relationship coefficients, and map each quartile to a label, as shown in Table 1:

[0110] Table 1

[0111] Decile Label One decile L_L_Zero Two deciles L_L_Low Three deciles L_L_Middle Four deciles L_L_High

[0112] ​After outputting the first embedded sequence and the second embedded sequence, step 103 is performed, the first embedded sequence can be input into a relative attention network RAT to output semantic association information between each text and multiple label categories; and the second embedded sequence can be input into the relative attention network RAT to output multiple internal association information within a range of each text and multiple label categories.

[0113] In a specific implementation, the first embedded sequence and the second embedded sequence can be input into the relative attention network RAT respectively, or the first embedded sequence and the second embedded sequence can be input into the relative attention network RAT simultaneously.

[0114] The RAT is improved on the basis of the Transformer, and is a model for embedding representation of a semi-structured input sequence. The RAT embeds the pre-existing relationship between elements in the sequence and integrates the pre-existing relationship between the elements into the self-attention mechanism. The key to training the RAT model lies in how to define the relationship between the elements, that is, how to define the relationship matrix between each text and category.

[0115]

[0116]

[0117]

[0118] wherein x represents embedding of an element in the input sequence, W is a learnable matrix, d is a constant, H is the number of layers of the transformer in the RAT network, and h represents the hth layer of the transformer in the RAT network, wherein r represents a coefficient of the i th row and the j th column in the relationship matrix, and softmax is an activation function that can map the output of a neuron to a probability value in the interval of 0 to 1, so as to perform classification. As shown in the above formula, when calculating the attention of element i and element j, the pre-existing relationship r between the two elements is added. ij .

[0119] Next, since the semantic association information and the multiple internal association information output by the relative attention network RAT are multi-dimensional vectors, step 104 is performed, and a bidirectional LSTM network can be used to reduce the dimension of the semantic association information and the multiple internal association information output by the relative attention network RAT; then the one-dimensional vector can be mapped to multiple label categories in the training set through a linear network, and the multiple label categories that can be mapped by the one-dimensional vector are the multi-label text classification prediction result of the corresponding text.

[0120] From the above steps, it can be known that the embodiment of the application predicts multiple label categories of the text under the combined action of multiple network models. In order to improve the prediction accuracy, in step 105, the loss value of the network model can be calculated according to the label category prediction result of each text and the known label category of each text, and the parameters of each network model mentioned above can be updated according to the loss value, which can specifically include the BERT model, the relative attention network RAT, the bidirectional LSTM network and the linear network. And the updated each network model is used to continue to obtain the label category prediction result of each text, until the loss value of the network model is less than the preset threshold, and the trained network model is obtained; finally, in step 106, the trained network model can be tested according to the test set, and the network model passed through the test is used for classifying the text to be classified.

[0121] Figure 7 A method flowchart for predicting the category of the text to be classified in the embodiment of the application is shown in FIG. 1. Figure 7 As shown in the figure, the flow from bottom to top includes:

[0122] (1) converting the text to be classified and N categories into a first embedded sequence by the BERT model, and converting the relationship matrix established according to the text to be classified and the N categories into a second embedded sequence;

[0123] (2) then inputting the first embedded sequence and the second embedded sequence into the relative attention network RAT, and outputting the semantic association information between the text to be classified and the N categories, and the multiple internal association information in the range of the text to be classified and the N categories;

[0124] (3) then converting the semantic association information and the multiple internal association information into a corresponding one-dimensional vector through the bidirectional LSTM network;

[0125] (4) mapping the one-dimensional vector to the N categories through the linear network, and the multiple label categories that the one-dimensional vector can be mapped to are the multi-label text classification prediction result of the text to be classified.

[0126] The embodiment of the application also provides a multi-label text classification device, as described in the following embodiment. Since the principle of solving the problem of the device is similar to that of the multi-label text classification method, the implementation of the device can be referred to the implementation of the multi-label text classification method, and the repeated parts will not be described here.

[0127] Figure 8 A specific example structure diagram of the multi-label text classification device in the embodiment of the application is shown in FIG. 2. Figure 8 As shown in the figure, the multi-label text classification device in the embodiment of the application can include:

[0128] The training set construction module 801 is configured to obtain a plurality of texts with known label categories, and construct a training set containing the plurality of texts and the plurality of label categories, wherein the known label category of each text is multiple;

[0129] The embedded sequence conversion module 802 is configured to input each text and the plurality of label categories in the training set into a BERT model to output a first embedded sequence, and establish a relationship matrix according to each text and the plurality of label categories in the training set, and convert the relationship matrix into a second embedded sequence;

[0130] The association information output module 803 is configured to input the first embedded sequence into a relative attention network RAT to output semantic association information between each text and the plurality of label categories, and input the second embedded sequence into the relative attention network RAT to output a plurality of internal association information within a range of each text and the plurality of label categories;

[0131] The label category prediction module 804 is configured to use a bidirectional LSTM network to reduce dimensions of the semantic association information and the plurality of internal association information output by the relative attention network RAT, and convert the semantic association information and the plurality of internal association information into corresponding one-dimensional vectors, map the one-dimensional vectors to the plurality of label categories in the training set through a linear network, and obtain a label category prediction result of each text according to a mapping result of the one-dimensional vectors;

[0132] The network model update module 805 is configured to calculate a loss value of a network model based on a loss function according to the label category prediction result of each text and the known label category of each text, update network model parameters according to the loss value when the loss value is not less than a preset threshold, continue to obtain the label category prediction result of each text by using the updated network model, and obtain a trained network model when the loss value of the network model is less than the preset threshold.

[0133] The text classification module 806 is configured to test the trained network model according to a test set, and classify a text to be classified by using the network model that passes the test.

[0134] In an embodiment, the relationship matrix includes the following relationship types:

[0135] Relationships between elements in each text, relationships between each text and the plurality of label categories, and relationships between the plurality of label categories.

[0136] In an embodiment, the embedded sequence conversion module 802 includes:

[0137] The distance calculation unit is configured to calculate distances between elements in each text for relationships between elements in each text.

[0138] The first relationship matrix establishing unit is configured to establish a relationship matrix by taking the distance between each element in each text as the relationship between each element in each text.

[0139] In one embodiment, the distance calculating unit is specifically configured to calculate the distance between each element in each text according to the following formula:

[0140] r ij = |i-j|

[0141] wherein, r ij represents an element in the i-th row and the j-th column of the relationship matrix.

[0142] In one embodiment, the embedded sequence conversion module 802 comprises:

[0143] The word segmentation dividing unit is configured to divide the description information of each text and the plurality of label categories by using a word segmentation tool to obtain a word segmentation dividing result, the word segmentation dividing result comprising entity words, stop words and other words.

[0144] The second relationship matrix establishing unit is configured to determine the relationship between each text and the plurality of label categories according to the word segmentation dividing result to establish a relationship matrix, the relationship between each text and the plurality of label categories comprising a plurality of relationships between entity words, stop words and other words.

[0145] In one embodiment, the second relationship matrix establishing unit is specifically configured to:

[0146] The relationship between entity words is queried from a pre-constructed knowledge graph, the knowledge graph being constructed according to the association relationship between different entities.

[0147] The relationship between entity words and other words or between other words and other words is determined by calculating the similarity coefficient between entity words and other words or between other words and other words, the value range of the similarity coefficient and the pre-set relationship corresponding to the value range of different similarity coefficients.

[0148] The relationship between stop words and entity words or between stop words and other words is determined by calculating the similarity coefficient between stop words and entity words or between stop words and other words, the value range of the similarity coefficient and the pre-set relationship corresponding to the value range of different similarity coefficients.

[0149] In one embodiment, the embedded sequence conversion module 802 comprises:

[0150] The relationship coefficient calculating unit is configured to calculate the relationship coefficient between each label category for the relationship between the plurality of label categories, the relationship coefficient representing the association closeness between each label category.

[0151] The third relationship matrix establishing unit is configured to determine the relationship among the plurality of label categories according to the relationship coefficients among the label categories and the relationship corresponding to the numerical range of different relationship coefficients preset in advance, and establish a relationship matrix.

[0152] In one embodiment, the relationship coefficient calculating unit is specifically configured to calculate the relationship coefficient among the label categories according to the following formula:

[0153]

[0154]

[0155] wherein, r ij represents an element in the i-th row and the j-th column of the relationship matrix.

[0156] Based on the foregoing inventive concept, as shown in Figure 9 The application further provides a computer device 900, which comprises a memory 910, a processor 920, and a computer program 930 stored in the memory 910 and capable of running on the processor 920, wherein the processor 920 implements the foregoing multi-label text classification method when executing the computer program 930.

[0157] The application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the foregoing multi-label text classification method.

[0158] The application further provides a computer program product, which comprises a computer program, wherein the computer program is executed by a processor to implement the foregoing multi-label text classification method.

[0159] In summary, in the embodiment of the present application, a plurality of texts of known label categories are acquired to construct a training set and a test set; each text in the training set and a plurality of label categories are input into a BERT model to output a first embedded sequence; a relationship matrix is established according to each text in the training set and the plurality of label categories, and the relationship matrix is converted into a second embedded sequence; the first embedded sequence is input into a relative attention network RAT to output semantic association information between each text and the plurality of label categories; the second embedded sequence is input into the relative attention network RAT to output a plurality of internal association information within the range of each text and the plurality of label categories; a bidirectional LSTM network is used to reduce dimensions of the semantic association information and the plurality of internal association information output by the relative attention network RAT and convert them into corresponding one-dimensional vectors; the one-dimensional vectors are mapped to the plurality of label categories in the training set through a linear network; a label category prediction result of each text is obtained according to a mapping result of the one-dimensional vectors; a loss value of a network model is calculated according to the label category prediction result of each text and the known label category of each text based on a loss function, and when the loss value is not less than a preset threshold, the network model parameters are updated according to the loss value, the label category prediction result of each text is continuously obtained by using the updated network model, until the loss value of the network model is less than the preset threshold, and a trained network model is obtained; the trained network model is tested according to the test set, and the network model that passes the test is used to classify the text to be classified.

[0160] The present application can effectively improve the utilization rate of training data by mining the semantic association, implicit association, internal association and the like between the text to be classified and each category, and injecting prior knowledge into the network model, and can greatly improve the accuracy of multi-label text classification under low resource conditions.

[0161] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0162] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0163] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0164] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0165] The above-described specific embodiments, the purpose, technical solutions and advantages of the present application are further described in detail, it should be understood that the above-described is only the specific embodiments of the present application, and is not used to limit the protection scope of the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A multi-label text classification method, characterized in that, include: Obtain multiple texts with known label categories, and construct a training set and a test set, wherein the training set and the test set contain multiple texts and multiple label categories, and each text has multiple known label categories; Input each text and multiple label categories in the training set into the BERT model, and output the first embedded sequence; build a relation matrix based on each text and multiple label categories in the training set, and convert the relation matrix into the second embedded sequence; The first embedded sequence is input into the Relative Attention Network (RAT), which outputs semantic association information between each text and multiple label categories; the second embedded sequence is input into the RAT, which outputs various intrinsic association information between each text and multiple label categories within their respective ranges. A bidirectional LSTM network is used to reduce the dimensionality of semantic association information and various intrinsic association information output by the relative attention network (RAT) into corresponding one-dimensional vectors. The one-dimensional vectors are then mapped to multiple label categories in the training set through a linear network. Based on the mapping results of the one-dimensional vectors, the label category prediction results for each text are obtained. Based on the loss function, the loss value of the network model is calculated according to the predicted label category of each text and the known label category of each text. When the loss value is not less than the preset threshold, the network model parameters are updated according to the loss value. The updated network model is then used to continue to obtain the predicted label category of each text until the loss value of the network model is less than the preset threshold, thus obtaining the trained network model. The trained network model is tested using the test set, and the tested network model is then used to classify the text to be classified.

2. The method as described in claim 1, characterized in that, The relation matrix includes the following relation types: The relationships between elements within each text, the relationships between each text and multiple tag categories, and the relationships between multiple tag categories.

3. The method as described in claim 1, characterized in that, Based on each text and multiple label categories in the training set, a relationship matrix is ​​constructed, including: For the relationships between elements in each text, calculate the distance between elements in each text; The distances between elements in each text are used as the relationships between elements in each text to establish a relationship matrix.

4. The method as described in claim 3, characterized in that, include: Calculate the distance between elements in each text using the following formula: r ij =|i-j| Where, r ij This represents the element in the i-th row and j-th column of the relation matrix.

5. The method as described in claim 1, characterized in that, Based on each text and multiple label categories in the training set, a relationship matrix is ​​constructed, including: For the relationship between each text and multiple tag categories, a word segmentation tool is used to segment the descriptive information of each text and multiple tag categories to obtain the word segmentation result, which includes entity words, stop words and other words; Based on the word segmentation results, the relationship between each text and multiple tag categories is determined, and a relationship matrix is ​​established; the relationship between each text and multiple tag categories includes various relationships between entity words, stop words, and other words.

6. The method as described in claim 5, characterized in that, Based on the word segmentation results, the relationship between each text and multiple tag categories is determined, including: The relationships between entity words are retrieved from a pre-built knowledge graph, which is constructed based on the associations between different entities. The relationship between entity words and other words or other words is determined by calculating the similarity coefficient between entity words and other words or other words and other words. The similarity coefficient is determined by the numerical range of the similarity coefficient and the relationship corresponding to the numerical range of different pre-set similarity coefficients. The relationship between stop words and entity words, or between stop words and other words, is determined based on a manually created stop word list.

7. The method as described in claim 1, characterized in that, Based on each text and multiple label categories in the training set, a relationship matrix is ​​constructed, including: For the relationship between multiple tag categories, calculate the relationship coefficient between each tag category, and the relationship coefficient represents the degree of association between each tag category; Based on the relationship coefficients between various label categories and the relationships corresponding to the pre-set numerical ranges of different relationship coefficients, the relationships between multiple label categories are determined, and a relationship matrix is ​​established.

8. The method as described in claim 7, characterized in that, include: Calculate the relationship coefficients between the various label categories using the following formula: Where k is the number of training data, and i and j represent the i-th and j-th labels in the category labels.

9. A multi-label text classification device, characterized in that, include: The training set construction module is used to obtain multiple texts with known label categories and construct a training set, wherein the training set contains multiple texts and multiple label categories, and each text has multiple known label categories; The embedded sequence transformation module is used to input each text and multiple label categories in the training set into the BERT model and output the first embedded sequence; based on each text and multiple label categories in the training set, a relation matrix is ​​built and the relation matrix is ​​transformed into the second embedded sequence. The association information output module is used to input the first embedded sequence into the relative attention network (RAT) and output the semantic association information between each text and multiple label categories; and to input the second embedded sequence into the relative attention network (RAT) and output various intrinsic association information between each text and multiple label categories within their respective ranges. The label category prediction module uses a bidirectional LSTM network to reduce the dimensionality of semantic association information and various intrinsic association information output by the Relative Attention Network (RAT) into corresponding one-dimensional vectors; the one-dimensional vectors are then mapped to multiple label categories in the training set via a linear network; and the label category prediction result for each text is obtained based on the mapping result of the one-dimensional vectors. The network model update module is used to calculate the loss value of the network model based on the loss function, the predicted label category of each text and the known label category of each text. When the loss value is not less than a preset threshold, the network model parameters are updated according to the loss value. The updated network model is then used to continue to obtain the predicted label category of each text until the calculated loss value of the network model is less than the preset threshold, thus obtaining the trained network model. The text classification module is used to test the trained network model against the test set, and then use the network model that passes the test to classify the text to be classified.

10. The apparatus as claimed in claim 9, characterized in that, The relation matrix includes the following relation types: The relationships between elements within each text, the relationships between each text and multiple tag categories, and the relationships between multiple tag categories.

11. The apparatus as claimed in claim 9, characterized in that, The embedded sequence conversion module includes: The distance calculation unit is used to calculate the distance between elements in each text, considering the relationship between elements in each text. The first relation matrix establishment unit is used to establish a relation matrix by taking the distance between each element in each text as the relationship between each element in each text.

12. The apparatus as claimed in claim 11, characterized in that, The distance calculation unit is specifically used to calculate the distance between each element in each text according to the following formula: r ij =|i-j| Where, r ij This represents the element in the i-th row and j-th column of the relation matrix.

13. The apparatus as claimed in claim 9, characterized in that, The embedded sequence conversion module includes: The word segmentation unit is used to segment the descriptive information of each text and multiple tag categories using a word segmentation tool to obtain the word segmentation result, which includes entity words, stop words and other words; The second relation matrix establishment unit is used to determine the relationship between each text and multiple tag categories based on the word segmentation results, and to establish a relation matrix; the relationship between each text and multiple tag categories includes various relationships between entity words, stop words and other words.

14. The apparatus as claimed in claim 13, characterized in that, The second relation matrix establishment unit is specifically used for: The relationships between entity words are retrieved from a pre-built knowledge graph, which is constructed based on the associations between different entities. The relationship between entity words and other words or other words is determined by calculating the similarity coefficient between entity words and other words or other words and other words. The similarity coefficient is determined by the numerical range of the similarity coefficient and the relationship corresponding to the numerical range of different pre-set similarity coefficients. The relationship between stop words and entity words, or between stop words and other words, is determined based on a manually created stop word list.

15. The apparatus as claimed in claim 9, characterized in that, The embedded sequence conversion module includes: The relationship coefficient calculation unit is used to calculate the relationship coefficient between multiple tag categories, wherein the relationship coefficient represents the degree of association between the tag categories; The third relation matrix establishment unit is used to determine the relationship between multiple label categories and establish a relation matrix based on the relation coefficients between each label category and the relationships corresponding to the numerical ranges of different preset relation coefficients.

16. The apparatus as claimed in claim 15, characterized in that, The relationship coefficient calculation unit is specifically used to calculate the relationship coefficient between each tag category according to the following formula: Where k is the number of training data, and i and j represent the i-th and j-th labels in the category labels.

17. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

19. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images