An optimization method for multi-label text classification based on WTLBert

Abstract

The invention provides an optimization method for multi-label text classification based on WTLBert. The method comprises the following steps: performing word segmentation on a text training set, controlling the word number of each training text within 510 words, and initializing an embedding vector for each text; constructing a label forest for all the labels marked in the text training set according to the semantic relationship between the labels, dividing labels with similar semantics into the same tree in the forest, and then pruning the trees with the same branches and leaves in the forest; initializing an embedding vector for each label according to the constructed label forest; generating a label-related weight matrix according to the constructed label forest; calculating a final embedding vector for each label according to F-Attention; inputting the label embedding vector and the text initialization embedding vector into a WTLBert model, and obtaining an aggregated embedding vector by iteratively training the WTLBert model through 12 Transformer encoders and 12 WTL encoders, wherein the vector not only considers the semantic relationship of the text context, but also takes into account the semantic relationship between the text content and the label; and finally inputting the aggregated vector into a classification network to obtain a multi-label classification result.

Description

Technical Field

[0001] The present invention relates to a multi-label text classification method, in particular to an optimization method for multi-label text classification based on WTLBert, and belongs to the technical field of natural language processing. Background Art

[0002] Multi-label text classification is a challenging task in the field of natural language processing. The multi-label classification methods proposed at this stage still have some problems. Problem 1: Due to the great success in the field of single-label text classification, most multi-label text classification problems are directly converted to single-label classification for processing. However, in the process of single-label classification, the label is only used as a classification symbol, and each label is independent. The semantic correlation between labels is not considered. The implicit association information between labels is an important part of improving classification accuracy. Problem 2: In the problem of multi-label text classification, there is actually a great deal of semantic correlation between fine-grained text content and labels. However, the existing multi-label multi-text classification methods cannot deeply explore the semantic relationship between fine-grained text content and labels. Summary of the invention

[0003] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide an optimization method for multi-label text classification based on WTLBert. Compared with the existing deep learning method, it can calculate the label embedding vector with semantic relationship at low cost and high efficiency, and the proposed WTL model integrates the contextual relationship of text content and the semantic relationship between text content and labels more closely.

[0004] The present invention provides an optimization method for multi-label text classification based on WTLBert, and the specific steps are as follows:

[0005] Step 1: Preprocess the text and initialize the text word embedding vector for the preprocessed text;

[0006] Step 2: Construct a label forest according to the degree of semantic association and the proposed construction rules, generate a semantic weight matrix, and obtain the semantic weights between labels. Then, based on the constructed label forest, initialize an embedding vector for each label. This vector initially reflects the semantic relevance between labels. Then, combined with the F-Attention mechanism, the semantic weights between the two labels in the semantic weight matrix are combined with the attention score to calculate the embedding vector corresponding to each label. These vectors more accurately reflect the semantic relationship between labels. The complete label embedding module is as follows: Figure 2 As shown;

[0007] Step 3: The text word embedding vector initialized in step 1 and the label embedding vector obtained in step 2 are used as the input of the WTLBert model. The CLS vector output by the Transformer encoder is input into the WTL encoder in the model. The WTL encoder analyzes the semantics of each word and label through the WTL-Attention mechanism, and finally reflects the semantic relevance between the two in the output vector. The WTLBert model is iteratively trained through multiple Transformer encoders and multiple WTL encoders to continuously explore the semantic relevance between fine-grained text content and labels, and finally calculates the aggregated embedding vector. The complete WTLBert model and the WTL encoder model are shown in Figure 2. Figure 3 shown.

[0008] Step 4: Input the aggregated embedding vector into the classification network and obtain the final classification network model through multiple iterative training.

[0009] The method of the present invention deeply mines the semantic relationship between fine-grained text content and labels, as well as the semantic relevance between labels. In response to the aforementioned problem one, the present invention proposes a label embedding training method based on a label forest and an F-Attention mechanism to generate label embeddings with semantic relationships. A label forest is constructed according to the semantic relationship of all labels in the data set, and labels with semantic relationships are divided into a tree, and the label nodes on each tree are arranged according to the degree of semantic association. According to the constructed label forest, a label semantic weight matrix is ​​generated and used as the input of the F-Attention mechanism, and finally the F-Attention mechanism outputs a label embedding vector with semantic relevance. In response to the aforementioned problem two, the present invention proposes a WTLBert model based on the traditional Bert model, in which a WTL encoder is newly added. The encoder uses the CLS vector output by the Transformer encoder as input, and the WTL-Attention mechanism is used inside the encoder to calculate the attention score for each word in the text and the label. With the stacking training of multiple Transformer encoders and WTL encoders, the contextual relationship of the text content and the semantic relationship between the text content and the label are mined at a deeper level.

[0010] The further optimized technical solution of the present invention is as follows:

[0011] In step 1, the specific operation of initializing the text word embedding vector is as follows:

[0012] Step 1.1: Preprocess the texts, control the number of words in each text to within 510, and divide the texts into test set and validation set according to a certain ratio;

[0013] Step 1.2: Initialize the test set text word vector, text segmentation vector, and text position vector, and merge the above three vectors into Vec txt , as one of the input vectors of the WTLBert model.

[0014] In step 2, a label forest is constructed, and the embedding vector corresponding to the label is calculated by combining the F-Attention mechanism. The specific operations are as follows:

[0015] Step 2.1: Collect all labels in the test set text and construct a label forest based on the semantic relationship between labels.

[0016] Step 2.2: Based on the constructed label forest, initialize the embedding vector for each label. The first dimension of the vector, e0, indicates which tree the label belongs to. The formula is as follows:

[0017]

[0018] Among them, index i Represents the number of each tree in the forest, t represents the number of trees in the forest; the remaining dimensions are based on Huffman coding, and the insufficient ones are filled with zeros; finally, each label generates an initialization vector Vec i , the formula is as follows:

[0019]

[0020] Among them, lab i represents the label number, and n represents the number of labels;

[0021] Step 2.3: Construct a label-related weight matrix based on the label forest. The formula is as follows:

[0022] M=M+E (5)

[0023] Among them, c represents the number of all labels, P ij represents the probability that labels i and j appear in a tree at the same time, T ij Indicates the number of times two labels appear in a tree at the same time, T i 、T j Respectively represent the number of times labels i and j appear, M ij The elements in the correlation weight matrix represent the semantic weight values ​​of label i and label j, δ represents the relative path length of two labels in the same tree, E represents the identity matrix, and M represents the final generated correlation weight matrix;

[0024] Step 2.4: According to the F-Attention mechanism, calculate the final label embedding vector; linearly map the correlation matrix W for each labeli , the attention score is calculated according to the following formula:

[0025] S ij =(W j -W i )*(W j +W i ) T (6)

[0026]

[0027] Among them, M ij is the element in the tag correlation weight matrix in step 2.3, indicating the semantic correlation degree of the two tags. i ,W j Respectively represent the matrix of linear mapping of labels i and j, S ij It represents the dot product score after the two matrices are fused, S ij ′ represents the attention score of label i for label j calculated according to the F-Attention mechanism. Finally, the final label embedding vector is generated according to the calculated attention score;

[0028] Vec i ′=S ij ′*Vec i (8)

[0029]

[0030] Among them, Vec i Initialize the label vector generated in step 2.2, Vec i ′ represents the vector obtained by integrating the attention score calculation, Vec lab Represents the final normalized label embedding vector;

[0031] In step 2.1, the label forest construction rules are as follows:

[0032] 1) If a node newly added to the forest has no semantic relationship with any tree in the forest, the node is used as the root node of the new tree;

[0033] 2) If a new node added to the forest has a semantic relationship with a tree in the forest, then each time a node is added to the tree, the tree is sorted in a big heap according to the degree of semantic association;

[0034] 3) If a node newly added to the forest has the same semantic relationship with a node in a tree in the forest, then the node is discarded to avoid label redundancy;

[0035] 4) The nodes of the hierarchical traversal results of each tree in the forest must be in descending order of semantic relevance;

[0036] 5) If the trees in the forest have repeated branch and leaf structures, prune these branches and leaves.

[0037] The label embedding vector training method based on the label forest in step 2 above first makes full use of the structural characteristics of the forest, and more intuitively and clearly links semantically similar labels together. In the pruned label forest, if any two labels are in the same tree, it means that there is a semantic correlation between the two labels, and the degree of correlation is related to the relative path of the leaf nodes of the two labels, otherwise there is no semantic correlation between the two labels. Secondly, the embedding vector calculated according to the F-Attention mechanism more accurately reflects the degree of semantic correlation between labels. The F-Attention mechanism combines the semantic weights and attention scores generated according to the label forest to closely link the distance between the label embedding vectors with their semantic similarity, thereby reducing the error of downstream tasks.

[0038] In step 3, the initialized text word embedding vector and the label embedding vector are used as the input of the WTLBert model. The model is iteratively trained through multiple Transformer encoders and WTL encoders to calculate the final aggregated embedding vector. The specific operations are as follows:

[0039] Step 3.1: embed the initialized text word vector Vec obtained in step 1.2 txt , and the label embedding vector Vec generated in step 2.4 lab , input into the WTLBert model;

[0040] Step 3.2: Embed the text word into vector Vec txt Input the Transformer encoder, which outputs the CLS vector containing the contextual semantic relationship;

[0041] Step 3.3: Input the CLS vector output by the Transformer encoder into the WTL encoder to obtain Vec′ txt vector;

[0042] Step 3.4: Vec′ obtained in step 3.3.5 txt The vector is input into the next Transformer encoder, and steps 3.1 to 3.3 are repeated until the feature vector passes through all encoders of the WTLBert model, and finally the WTLBert model outputs the final aggregate embedding vector Vec h .

[0043] In step 3.3, the WTL encoder obtains Vec′ txt The specific operations of vectors are as follows:

[0044] Step 3.3.1: In the WTL encoder, the input CLS vector is mapped to the label feature vector space through the trained fully connected network to obtain the transformed vector h i ′;

[0045] Step 3.3.2: Transformer encoder output CLS vector Vec cls The label embedding vector Vec generated in step 2.4 lab Fusion gets vector α i , the specific formula is as follows:

[0046] α i =Vec cls +Vec lab (10)

[0047]

[0048] Among them, d represents the dimension of the vector, α ij Represents α i The values ​​of each dimension of the vector, Represents α i Mean over vector dimensions;

[0049] Step 3.3.3: Transformer encoder output CLS vector Vec cls And the vector α obtained in step 3.3.2 i Input to the WTL-Attention mechanism, in the WTL-Attention mechanism, the CLS vector is linearly mapped to generate the Query matrix Q w , for α i Vector linear generation Key matrix K α And the Value matrix V α , and then the initial aggregation vector Vec′ is calculated by the attention formula cls , the specific formula is as follows:

[0050] Q′ w =Q w *Vec cls (12)

[0051] K′ α =K α *α i (13)

[0052] V′ α =V α *α i (14)

[0053]

[0054] Among them, Q′ w Indicates Q w Matrix dot product Vec cls The result obtained by vector, K′ α K α Matrix dot multiplication α i The result obtained by vector, V α ′ represents V α Matrix dot multiplication α i The result obtained by vector, g represents the calculated WTL-Attention score, c represents the number of labels, and i represents the label number;

[0055] Step 3.3.4: The aggregate vector Vec′ obtained in step 3.3.3 cls Add the CLS vector output by the Transformer encoder to get Vec″ cls , then for Vec″ cls Take the mean variance to get the normalized vector Vec″′ cls , the specific formula is as follows:

[0056] Vec″ cls =Vec′ cls +Vec cls (17)

[0057]

[0058] Where d is Vec″′ cls is the dimension of the vector, i is the dimension number;

[0059] Step 3.3.5: Vec″′ calculated in step 3.3.4 cls Input to the fully connected layer to extract deeper semantic features, and finally the WTL encoder outputs the vector Vec′ that integrates the label semantics txt .

[0060] The WTLBert model mentioned above takes into account the semantic relationship between text content and labels. Based on the original Bert model, the WTLBert model adds a WTL encoder. The encoder takes the CLS vector containing contextual semantic information output by the Transformer and the label embedding vector calculated according to the label forest as input. The CLS vector is mapped to the label embedding vector space through convolution pooling, and the WTL-Attention mechanism inside the encoder is used to analyze the semantic relevance between each word and multiple labels. Finally, the aggregate vector output by the encoder not only contains the semantic relationship of each word's own context, but also indicates the degree of association between each word and each label. A WTL encoder is configured for each Transformer encoder in the Bert model, and stacking training is repeated. Finally, the vector output by the WTLBert model closely links the contextual semantic relationship of the text content and the semantic relationship between the text content and the label. The vector obtained in this way improves the accuracy of downstream classification tasks.

[0061] The specific operation of step 4 is as follows: embed the aggregated vector Vec h The data is input into the classification network, and the probability of each label is output in the output layer through the Sigmoid activation function. If the probability of a label is greater than the threshold value of 0.5, it means that the label can identify the text; all labels with a probability greater than 0.5 are a multi-label set that identifies the text.

[0062] The present invention performs word segmentation on a text training set, controls the number of words in each training text within 510 words, and initializes an embedding vector for each text; constructs a label forest for all the labeled labels in the text training set according to the semantic relationship between the labels, divides labels with similar semantics into the same tree in the forest, and then prunes the trees with the same branches and leaves in the forest; initializes an embedding vector for each label according to the constructed label forest; generates a label-related weight matrix according to the constructed label forest; calculates the final embedding vector of each label according to F-Attention; inputs the label embedding vector and the text initialization embedding vector into a WTLBert model, and the WTLBert model is iteratively trained through 12 Transformer encoders and 12 WTL encoders to obtain an aggregated embedding vector, which not only considers the semantic relationship of the text context, but also takes into account the semantic relationship between the text content and the label; and finally inputs the aggregated vector into a classification network to obtain a multi-label classification result. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of the present invention.

[0064] Figure 2This is the overall model architecture diagram of multi-label text classification in the present invention.

[0065] Figure 3 This is a diagram of a label embedding module in the present invention.

[0066] Figure 4 This is a diagram of the WTLBert model architecture in the present invention. DETAILED DESCRIPTION

[0067] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and specific operation process are given, but the protection authority of the present invention is not limited to the following embodiments.

[0068] Example 1

[0069] This embodiment proposes an optimization method for news text multi-label classification based on WTLBert, such as Figure 1 and Figure 2 As shown, the following steps are included:

[0070] Step 1: Initialize the news text word embedding vector

[0071] Step 1.1: Preprocess the news text, and the number of words in the text after word segmentation shall not exceed 510. And divide the preprocessed text into a test set and a validation set in a ratio of 8:2.

[0072] Step 1.2: Initialize the text word vector, the text segmentation vector, and the text position vector for the test set text, and merge the above three vectors into Vec txt , as one of the input vectors of the WTLBert model.

[0073] Step 2: Collect the labels in the news text test set and uniformly construct a label forest. Combined with the F-Attention mechanism, calculate the embedding vector corresponding to the label. The complete label embedding process is as follows: Figure 3 shown.

[0074] Step 2.1: Collect all the labels in the test set text and construct a label forest based on the semantic relationship between the labels. The specific construction rules are as follows:

[0075] 1) If a node newly added to the forest has no semantic relationship with any tree in the forest, the node is used as the root node of the new tree;

[0076] 2) If a new node added to the forest has a semantic relationship with a tree in the forest, then each time a node is added to the tree, the tree is sorted in a big heap according to the degree of semantic association;

[0077] 3) If a node newly added to the forest has the same semantic relationship with a node in a tree in the forest, then the node is discarded to avoid label redundancy;

[0078] 4) The nodes of the hierarchical traversal results of each tree in the forest must be in descending order of semantic relevance;

[0079] 5) If the trees in the forest have repeated branch and leaf structures, prune these branches and leaves.

[0080] Step 2.2: Initialize the embedding vector for each label based on the constructed label forest. The first dimension of the vector, e0, indicates which tree the label belongs to. The formula is as follows:

[0081]

[0082] Among them, index i Represents the number of each tree in the forest, t represents the number of trees in the forest; the remaining dimensions are based on Huffman coding, and the insufficient ones are filled with zeros; finally, each label generates an initialization vector Vec i , the formula is as follows:

[0083]

[0084] Among them, lab i represents the label number, and n represents the number of labels.

[0085] Step 2.3: Construct a label-related weight matrix based on the label forest. The formula is as follows:

[0086] M=M+E (5)

[0087] Among them, c represents the number of all labels, P ij represents the probability that labels i and j appear in a tree at the same time, T ij Indicates the number of times two labels appear in a tree at the same time, T i , T j Respectively represent the number of times labels i and j appear, M ij The elements in the correlation weight matrix represent the semantic weight values ​​of label i and label j, δ represents the relative path length of two labels in the same tree, E represents the identity matrix, and M represents the final generated correlation weight matrix;

[0088] Step 2.4: According to the F-Attention mechanism, calculate the final label embedding vector. Linearly map the correlation matrix W for each label i , the attention score is calculated according to the following formula:

[0089] S ij=(W j -W i )*(W j +W i ) T (6)

[0090]

[0091] Among them, M ij is the element in the tag correlation weight matrix in step 2.3, indicating the semantic correlation degree of the two tags. i ,W j Respectively represent the matrix of linear mapping of labels i and j, S ij It represents the dot product score after the two matrices are fused, S ij ′ represents the attention score of label i for label j calculated according to the F-Attention mechanism. Finally, the final label embedding vector is generated according to the calculated attention score;

[0092] Vec i ′=S ij ′*Vec i (8)

[0093]

[0094] Among them, Vec i Initialize the label vector generated in step 2.2, Vec i ′ represents the vector obtained by integrating the attention score calculation, Vec lab Represents the final normalized label embedding vector;

[0095] Step 3: Use the initialized text word embedding vector and label embedding vector as the input of the WTLBert model. The model is iteratively trained through 12 Transformer encoders and 12 WTL encoders to calculate the final aggregate embedding vector. The complete WTLBert model and WTL encoder model are shown in Figure 2. Figure 4 shown.

[0096] Step 3.1: embed the initialized text word vector Vec obtained in step 1.2 txt , and the label embedding vector Vec generated in step 2.4 lab , input into the WTLBert model.

[0097] Step 3.2: Embed the text word into vector Vec txt The Transformer encoder is input and outputs a CLS vector containing contextual semantic relations.

[0098] Step 3.3: Input the CLS vector output by the Transformer encoder into the WTL encoder to obtain Vec′ txt vector.

[0099] Step 3.3.1: In the WTL encoder, the input CLS vector is mapped to the label feature vector space through the trained fully connected network to obtain the transformed vector h′ i ;

[0100] Step 3.3.2: Transformer encoder output CLS vector Vec cls The label embedding vector Vec generated in step 2.4 lab Fusion gets vector α i , the specific formula is as follows:

[0101] α i =Vec cls +Vec lab (10)

[0102]

[0103] Among them, d represents the dimension of the vector, α ij Represents α i The values ​​of each dimension of the vector, Represents α i Mean over vector dimensions;

[0104] Step 3.3.3: Transformer encoder output CLS vector Vec cls And the vector α obtained in step 3.3.2 i Input to the WTL-Attention mechanism, in the WTL-Attention mechanism, the CLS vector is linearly mapped to generate the Query matrix Q w , for α i Vector linear generation Key matrix K α And the Value matrix V α , and then the initial aggregation vector Vec′ is calculated by the attention formula cls , the specific formula is as follows:

[0105] Q′ w =Q w *Vec cls (12)

[0106] K′ α =K α *α i (13)

[0107] V′ α =Vα *α i (14)

[0108]

[0109] Among them, Q′ w Indicates Q w Matrix dot product Vec cls The result obtained by vector, K′ α K α Matrix dot multiplication α i The result obtained by vector represents V α Matrix dot multiplication α i The result obtained by vector, g represents the calculated WTL-Attention score, c represents the number of labels, and i represents the label number;

[0110] Step 3.3.4: The aggregate vector Vec′ obtained in step 3.3.3 cls Add the CLS vector output by the Transformer encoder to get Vec″ cls , then for Vec″ cls Take the mean variance to get the normalized vector Vec″′ cls , the specific formula is as follows:

[0111] Vec″ cls =Vec′ cls +Vec cls (17)

[0112]

[0113] Where d is Vec″′ cls is the dimension of the vector, i is the dimension number;

[0114] Step 3.3.5: Vec″′ calculated in step 3.3.4 cls Input to the fully connected layer to extract deeper semantic features, and finally the WTL encoder outputs the vector Vec′ that integrates the label semantics txt .

[0115] Step 3.4: Vec′ obtained in step 3.3.5 txt The vector is input into the next Transformer encoder, and steps 3.1 to 3.3 are repeated until the feature vector passes through all encoders of the WTLBert model, and finally the WTLBert model outputs the final aggregate embedding vector Vec h .

[0116] Step 4: Embed the aggregate into vector Vec hThe data is input into the classification network, and the probability of each label is output in the output layer through the Sigmoid activation function. If the probability of a label is greater than the threshold value of 0.5, it means that the label can identify the text; all labels with a probability greater than 0.5 are a multi-label set that identifies the text.

[0117] Step 5: When the accuracy of the classification network reaches more than 90% on the validation set, the model is the best model.

[0118] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technology can understand and think of any changes or substitutions within the technical scope disclosed by the present invention, which should be included in the scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An optimization method for multi-label text classification based on WTLBert, characterized in that: The specific steps are as follows: Step 1: Initialize the text word embedding vector; Step 2: Construct a label forest, generate a semantic weight matrix, and obtain the semantic weights between labels. Then, according to the constructed label forest, initialize the embedding vector for each label. Then, combined with the F-Attention mechanism, combine the semantic weights between the two labels in the semantic weight matrix with the attention score to calculate the embedding vector corresponding to each label. Construct a label forest, combine the F-Attention mechanism, and calculate the embedding vector corresponding to the label. The specific operations are as follows: Step 2.1: Collect all labels in the test set text and construct a label forest based on the semantic relationship between labels. Step 2.2: Based on the constructed label forest, initialize the embedding vector for each label. The first dimension of the vector, e0, indicates which tree the label belongs to. The formula is as follows: Among them, index i Represents the number of each tree in the forest, t represents the number of trees in the forest; the remaining dimensions are filled with zeros according to Huffman coding, and finally each label generates an initialization vector Vec i , the formula is as follows: Among them, lab i represents the label number, and n represents the number of labels; Step 2.3: Construct a label-related weight matrix based on the label forest. The formula is as follows: M=M+E (5) Among them, c represents the number of all labels, P ij represents the probability that labels i and j appear in a tree at the same time, T ij Indicates the number of times two labels appear in a tree at the same time, T i , T j Respectively represent the number of times labels i and j appear, M ij The elements in the correlation weight matrix represent the semantic weight values ​​of label i and label j, δ represents the relative path length of two labels in the same tree, E represents the identity matrix, and M represents the final generated correlation weight matrix; Step 2.4: According to the F-Attention mechanism, calculate the final label embedding vector; linearly map the correlation matrix W for each label and calculate the attention score according to the following formula: S ij =(W j -W i )*(W j +W i ) T (6) Among them, M ij is the element in the label correlation weight matrix in step 2.3, indicating the semantic correlation between the two labels, W i ,W j Represents the matrix of linear mapping of labels i and j, S ij It represents the dot product score after the two matrices are fused, S ij ′ represents the attention score of label i for label j calculated according to the F-Attention mechanism. Finally, the final label embedding vector is generated according to the calculated attention score; Thing i ′=S ij ′*Thing i (8) Among them, Vec i Initialize the label vector generated in step 2.2, Vec i ′ represents the vector obtained by integrating the attention score calculation, Vec lab Represents the final normalized label embedding vector; Step 3: The text word embedding vector initialized in step 1 and the label embedding vector obtained in step 2 are used as the input of the WTLBert model. The WTLBert model is iteratively trained through multiple Transformer encoders and multiple WTL encoders to continuously mine the semantic relevance between fine-grained text content and labels, and finally calculates the aggregated embedding vector; Step 4: Input the aggregated embedding vector into the classification network and obtain the final classification network model through multiple iterative training.

2. According to claim 1, a WTLBert-based multi-label text classification optimization method is characterized in that: In step 1, the specific operation of initializing the text word embedding vector is as follows: Step 1.1: Preprocess the texts, control the number of words in each text to within 510, and divide the texts into test set and validation set according to a certain ratio; Step 1.2: Initialize the test set text word vector, text segmentation vector and text position vector, and merge the above three vectors into Vec txt , as one of the input vectors of the WTLBert model.

3. According to claim 2, a WTLBert-based multi-label text classification optimization method is characterized in that: In step 2.1, the label forest construction rules are as follows: 1) If a node newly added to the forest has no semantic relationship with any tree in the forest, the node is used as the root node of the new tree; 2) If a new node added to the forest has a semantic relationship with a tree in the forest, then each time a node is added to the tree, the tree is sorted in a big heap according to the degree of semantic association; 3) If a node newly added to the forest has the same semantic relationship with a node in a tree in the forest, then the node is discarded to avoid label redundancy; 4) The nodes of the hierarchical traversal results of each tree in the forest must be in descending order of semantic relevance; 5) If the trees in the forest have repeated branch and leaf structures, prune these branches and leaves.

4. According to claim 3, a WTLBert-based multi-label text classification optimization method is characterized in that: In step 3, the initialized text word embedding vector and the label embedding vector are used as the input of the WTLBert model. The model is iteratively trained through multiple Transformer encoders and WTL encoders to calculate the final aggregated embedding vector. The specific operations are as follows: Step 3.1: embed the initialized text word vector Vec obtained in step 1.2 txt , and the label embedding vector Vec generated in step 2.4 lab , input into the WTLBert model; Step 3.2: Embed the text word into vector Vec txt Input the Transformer encoder, which outputs the CLS vector containing the contextual semantic relationship; Step 3.3: Input the CLS vector output by the Transformer encoder into the WTL encoder to obtain Vec′ txt vector; Step 3.4: Vec′ txt The vector is input into the next Transformer encoder, and steps 3.1 to 3.3 are repeated until the feature vector passes through all encoders of the WTLBert model, and finally the WTLBert model outputs the final aggregate embedding vector Vec h .

5. The optimization method for multi-label text classification based on WTLBert according to claim 4 is characterized in that: In step 3.3, the WTL encoder obtains Vec′ txt The specific operations of vectors are as follows: Step 3.3.1: In the WTL encoder, the input CLS vector is mapped to the label feature vector space through the trained fully connected network to obtain the transformed vector h′ i ; Step 3.3.2: Transformer encoder output CLS vector Vec cls The label embedding vector Vec generated in step 2.4 lab Fusion gets vector α i , the specific formula is as follows: α i =Thing cls +Thing lab (10) Among them, d represents the dimension of the vector, α ij Represents α i The values ​​of each dimension of the vector, Represents α i Mean over vector dimensions; Step 3.3.3: Transformer encoder output CLS vector Vec cls And the vector α obtained in step 3.3.2 i Input to the WTL-Attention mechanism, in the WTL-Attention mechanism, the CLS vector is linearly mapped to generate the Query matrix Q w , for α i Vector linear generation Key matrix K α And the Value matrix V α , and then the initial aggregation vector Vec′ is calculated by the attention formula cls , the specific formula is as follows: Q′ w =Q w *Vec cls (12) K′ α =K α *a i (13) V′ α =V α *α i (14) Among them, Q′ w Indicates Q w Matrix dot product Vec cls The result obtained by vector, K′ α K α Matrix dot multiplication α i The result obtained by vector, V′ α Indicates V α Matrix dot multiplication α i The result obtained by vector, g represents the calculated WTL-Attention score, c represents the number of labels, and i represents the label number; Step 3.3.4: The aggregate vector Vec′ obtained in step 3.3.3 cls Add the CLS vector output by the Transformer encoder to get Vec″ cls , then for Vec″ cls Take the mean variance to get the normalized vector Vec″′ cls , the specific formula is as follows: Thing cls =Thing′ cls +Thing cls (17) Where d is Vec″ cls is the dimension of the vector, i is the dimension number; Step 3.3.5: Substitute the Vec″ calculated in step 3.3.4 cls Input to the fully connected layer to extract deeper semantic features, and finally the WTL encoder outputs the vector Vec′ that integrates the label semantics txt .

6. The optimization method for multi-label text classification based on WTLBert according to claim 5, characterized in that: The specific operation of step 4 is as follows: embed the aggregated vector Vec h Input to the classification network, and output the probability of each label in the output layer through the Sigmoid activation function. If the probability of a label is greater than the threshold 0.5, it means that the label can identify the text; All labels with a probability greater than 0.5 are the multi-label set that identifies the text.

Images