Text classification method and system based on topic-aware hierarchical multi-attention network

Abstract

The application provides a text classification method and system based on a topic-aware hierarchical multi-attention network, comprising: obtaining text information to be classified; obtaining a classification result according to the obtained text information and a preset text classification model; wherein the text classification model is constructed in a hierarchical structure to form multiple attention mechanisms, and converts sentence-level and document-level inputs into sentence and document encoders respectively; the application takes self-attention as the main building block of the neural network, not only improves the modeling ability of the distance relationship, but also makes the training speed of the self-attention network faster due to the feedforward structure; a hierarchical neural structure is introduced, which converts sentence-level and document-level inputs into sentence and document encoders respectively, and realizes the most advanced classification accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field related to text classification, and particularly relates to a text classification method and system based on a topic-aware hierarchical multi-attention network. BACKGROUND

[0002] Text classification is a basic task in natural language processing (NLP), which aims to automatically assign labels to a given text; traditional text classification methods use word co-occurrence information obtained by bag-of-words (BoW) technology, keyword frequency-inverse document frequency (TF-IDF), or latent Dirichlet allocation topic distribution as features for training classifiers; these methods convert text into digital data, which can be used to automatically perform document summarization, classification, and clustering on a large number of documents.

[0003] The inventors found that the above-mentioned method has dimension sparsity in document representation, which leads to high computational cost, and also ignores the context information in the text; traditional neural networks scan the entire document at once without utilizing the hierarchical features of the document, such as the structural relationship between words and sentences, and the structural relationship between sentences and documents, resulting in low classification accuracy. SUMMARY

[0004] To solve the above-mentioned problems, the application provides a text classification method and system based on a topic-aware hierarchical multi-attention network, which aggregates multiple convolutional layers with different window sizes to enhance feature extraction, and uses self-attention as the main building block of the neural network; self-attention networks can connect words that are far apart through shorter network paths, not only improving the modeling ability of distance relationships, but also training faster due to the feedforward structure; the hierarchical neural structure is introduced, which converts sentence-level and document-level inputs into sentence and document encoders, respectively, and achieves the most advanced classification accuracy.

[0005] In a first aspect, the application provides a text classification method based on a topic-aware hierarchical multi-attention network, comprising:

[0006] obtaining text information to be classified;

[0007] obtaining a classification result according to the obtained text information and a preset text classification model;

[0008] The text classification model is constructed in a hierarchical structure with multiple attention mechanisms, which converts sentence-level and document-level inputs into sentence and document encoders, respectively; first, the multi-head self-attention is combined with the convolutional neural network in the hierarchical structure of the text classification model; then, the sentence-level representation before and after the self-attention block in the document encoder is learned through interaction; finally, the document representation is enriched by combining the topic distribution generated by the topic model.

[0009] Further, the importance of sentence-level and document-level representation is encoded by using soft attention order.

[0010] Further, feature extraction is enhanced by aggregating multiple convolutional layers with different window sizes.

[0011] Further, sentence representations with different levels of abstraction are learned interactively by using a collaborative attention mechanism, and the document-level representation is enriched in the document encoder.

[0012] Further, the feature representation of the word-level and document-level input is expanded by fusing the global co-occurrence information from the topic word distribution and the document topic probability from the document-topic distribution generated by the LDA model.

[0013] Further, the self-attention mechanism focuses on each position in the sequence by comparing each entry with all entries in the same sequence; the output sequence of self-attention contains information about each entry and its relationship with all entries; the query, key and value vectors for each entry position are calculated through a fully connected layer; attention weights are learned from different parts of the sequence.

[0014] Further, the convolutional layer and the pooling layer are used to enable the self-attention network to obtain expressive representations from the query, key and value vectors to enhance the ability of feature extraction.

[0015] In a second aspect, the present application further provides a text classification system based on a topic-aware hierarchical multi-attention network, comprising:

[0016] A data acquisition module configured to acquire text information to be classified;

[0017] A classification module configured to obtain a classification result according to the acquired text information and a preset text classification model;

[0018] The text classification model is constructed in a hierarchical structure, multiple attention mechanisms are combined, and the sentence-level and document-level inputs are converted into sentence and document encoders; first, the multi-head self-attention is combined with the convolutional neural network in the hierarchical structure of the text classification model; then, the sentence-level representation before and after the self-attention block in the document encoder is learned interactively; finally, the document representation is enriched by combining the topic distribution generated by the topic model.

[0019] In a third aspect, the present application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to realize the steps of the text classification method based on the topic-aware hierarchical multi-attention network in the first aspect.

[0020] In a fourth aspect, the application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the steps of the text classification method based on the topic-aware hierarchical multi-attention network in the first aspect.

[0021] Compared with the prior art, the application has the following beneficial effects:

[0022] The application takes self-attention as the main building block of the neural network, the self-attention network can connect words far away through a shorter network path, not only improves the modeling ability of the distance relationship, but also the training speed of the self-attention network is faster due to the feedforward structure; the hierarchical neural structure is introduced, which converts the sentence-level and document-level inputs into sentence and document encoders respectively, and realizes the most advanced classification accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0023] The drawings constituting a part of this embodiment are used to provide further understanding of the embodiment, the illustrative embodiments of the embodiment and the description thereof are used to explain the embodiment, and do not constitute improper limitation on the embodiment.

[0024] Figure 1 It is a network structure schematic diagram of the embodiment 1 of the application;

[0025] Figure 2 It is a convolutional self-attention module of the embodiment 1 of the application. DETAILED DESCRIPTION

[0026] The application will be further described below in combination with the drawings and embodiments.

[0027] It should be pointed out that the following detailed description is all exemplary, and is intended to provide further description of the application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the application belongs.

[0028] Embodiment 1:

[0029] The embodiment provides a text classification method based on a topic-aware hierarchical multi-attention network, which comprises the following steps:

[0030] Obtaining text information to be classified;

[0031] According to the obtained text information and the preset text classification model, a classification result is obtained;

[0032] The text classification model constructs multiple attention mechanisms in a hierarchical structure, converts sentence-level and document-level inputs into sentence and document encoders respectively; first, multi-head self-attention is combined with a convolutional neural network in the hierarchical structure of the text classification model; then, the sentence-level representation before and after the self-attention block in the document encoder is learned through interaction; finally, the document representation is enriched by combining the topic distribution generated by the topic model.

[0033] In this embodiment, self-attention is used as the main building block of the neural network. The self-attention network can connect words that are far apart through a shorter network path, not only improving the modeling ability of the distance relationship, but also making the training speed of the self-attention network faster due to the feedforward structure. A hierarchical neural structure is introduced, which converts sentence-level and document-level inputs into sentence and document encoders respectively, achieving the most advanced classification accuracy

[0034] In this embodiment, a topic-aware hierarchical multi-attention network (T-HMAN) is proposed, which constructs multiple attention mechanisms in a hierarchical structure; unlike other hierarchical models, this model uses a collaborative attention mechanism to interactively learn sentence representations with different levels of abstraction and enriches the document-level representation in the document encoder; in addition, soft attention is used to sequentially summarize the importance of sentence-level and document-level representations, because all parts of a word or sentence are not equally related to the final prediction; at the same time, the feature representation of the word-level and document-level inputs is expanded by fusing global co-occurrence information from the topic word distribution and each document topic probability from the document-topic distribution generated by the LDA model.

[0035] In this embodiment, the overall architecture of T-HMAN is as shown in Figure 1 The hierarchical framework utilizes document structure characteristics, such as semantic features between words and sentences, sentences and documents, and obtains word-level and sentence-level text representations through sentence encoders and document encoders respectively; specifically, the model first encodes the word-level input into sentence-level representation, then calculates the document-level representation using the sentence-level representation as input, and finally passes the document-level representation to Softmax for classification.

[0036] Figure 1 In the formula, we represents word embedding, tw represents topic-word distribution, wp represents word-level position embedding, se represents sentence representation before the self-attention module in the document encoder, sp represents sentence-level position embedding, ce represents sentence representation after the self-attention module in the document encoder, de represents document representation, and dt is the document-topic distribution.

[0037] In this embodiment, when pre-training the topic distribution, it is assumed that D represents a document consisting of m sentence sequences, D = (s1, s2, ..., sm). m S represents a sentence consisting of n words: S = (w1, w2, ... wn) n In this implementation, S is embedded into a distribution space, we = (we1, we2, ..., we...). n ), where we∈Rn×e, n is the sequence length, and e is the dimension of the word embedding; the LDA model simultaneously generates topic-word distributions and document-topic distributions. The former is shared among all documents and contains global word co-occurrence features across the entire corpus; while the latter is a local distribution on a given document topic and is independent of all other documents; these two distributions can serve as additional features for sentence and document encoders in a hierarchical framework. The LDA model is trained before training the network. Let tw = (tw1, tw2, ..., tw... n ) represents the transposed topic distribution, where tw∈Rn×t, and t is the number of topics. The document-topic distribution can be represented as dt=(dt1, dt2, …, dt d ), where dt∈RD×t, and d is the number of documents. In this embodiment, the word embeddings are first concatenated with the transposed topic word distribution as input to the sentence encoder. Similarly, before the final Softmax classifier, the document representation is concatenated with the document topic distribution as an additional feature.

[0038] In this embodiment, unlike the recurrent mechanism, self-attention does not explicitly simulate the relative or absolute position information of the sequence in its structure during position encoding; therefore, element-wise summation is used to sum the word-position embedding wp, wp∈Rn×e and this word embedding to obtain the word-level input w=([we1;tw1]+wp1,…,(we n ;tw n )+wp n Similarly, the sentence-position embedding sp, sp∈Rm×dk, dk represents the model dimension and the sentence embedding se=(se1, se2, ..., se m ), where se i ∈Rm×dk, to obtain sentence-level input s=(se1+sp1,…,se m +sp m). By comparing two position encoding methods: sinusoidal position encoding and absolute position encoding, the former is introduced in the Transformer model by using various frequencies of sine and cosine functions to form a geometric progression, which is then added to the original word embedding. The position embedding is a dk-dimensional vector containing information that cannot be integrated into the model itself, but can be used as an additional feature to enrich the input of the model. The latter generates a vector representation of the absolute position of an entry in the sequence. Then the position embedding is added directly to the original embedding. This method works the same as the sinusoidal position encoding operation, so the randomly initialized embedding is used to learn the absolute position of the word during the training process to achieve fast deployment.

[0039] In this embodiment, the self-attention mechanism focuses on each position in the sequence by comparing each entry with all entries in the same sequence, which enables the model to learn the contextual relationship of the sequence and capture long-term semantic dependencies. Specifically, the output sequence of self-attention contains the information of each entry and its relationship with all entries. In T-HMAN, linear projection is completed by scaled-dot product, and the query (Q), key (K), and value (V) vectors of each entry position are calculated by a fully connected layer. Since the same attention weight is applied to all dk dimensions of the V vector, in order to expand the self-attention capability, the model uses multi-head self-attention. It utilizes h parallel self-attention modules, each of which focuses on a different part of the embedding dimension. Therefore, attention weights can be learned from different parts of the sequence, resulting in a more expressive output representation, rather than using a single attention function to obtain attention weights for all embeddings in one pass.

[0040] In this embodiment, in T-HMAN, the self-attention network is enabled to obtain expressive representations from Q, K, and V vectors by utilizing convolutional layers and pooling layers to enhance the ability of feature extraction, as shown in Figure 2 In the convolutional layer, three different window sizes k = [3, 4, 5] are used to obtain n-gram features from the source sequence, followed by layer normalization. The addition of layer normalization to the convolutional layer can significantly improve model performance, where the purpose of layer normalization is to reduce training time and normalize the activity of neurons for faster training convergence. In the T-HMAN model, layer normalization is stacked after each convolutional layer, followed by max pooling applied to the sequence to obtain the most significant feature representation.

[0041] In this embodiment, as shown in formula (1) and formula (2), a soft attention mechanism is adopted in both sentence-level encoding and document encoding; by calculating a context vector, a normalized attention weight is obtained through Softmax, and the importance of each item is measured through Softmax. The final output is f (soft-attention can calculate the normalized weight through weighted summation, as formula (3). Wherein, W s and b s are weight and bias respectively, c e ∈Rn×dk represents an output multi-thread self-attention block, tanh is a nonlinear function, u represents an input sequence of hidden states, W context ∈R dk represents a context vector, a is a normalized weight, represents the importance of each item in the sequence, and f is the output of soft attention.

[0042] u=tanh(W s c e +b s ) (1)

[0043]

[0044]

[0045] Then the common attention in the document encoder is used to expand the document-level representation. Specifically, given the sentence representation before the document encoder s e ∈Rm×dk and the sentence representation after the document encoder c e ∈Rm×dk, the common attention interactively focuses on the two sentence representations. In this embodiment, first, the matrix L∈Rdk×dk in formula (4) is calculated, and then the matrix is taken as a feature to predict the attention weight in formula (5). Similar to soft attention, the attention weight of the two sentence representations is calculated using Softmax, and the output vector of common attention is calculated through the weighted sum of the two sentence features in formula (6) and formula (7).

[0046] L=tanh(ce T W l se) (4)

[0047]

[0048]

[0049]

[0050] Wherein, W l ∈Rdk×dk represents a weight matrix learned through the network, W s , W d∈ Rp x dk, W hs and W hd ∈ R1 x p is the weight parameter, p represents the hidden size of the common attention, α s , α d ∈ R1 x m is the attention weight, s, c ∈ R1 x dk is the two sentence representations generated by the common attention. Then, this embodiment connects s and c and passes them to the soft attention in the document encoder; finally, the output de of the soft-attention and the document-topic distribution dt are concatenated and fed into the final Softmax function.

[0051] This embodiment proposes a Topic-Aware Hierarchical Multi-Attention Network (T-HMAN) based on topic perception; in order to achieve better modeling capability, the multi-head self-attention is combined with the convolutional neural network in the hierarchical structure; T-HMAN also uses soft attention to encode the importance of sentence-level and document-level representations in sequence; then, through the interaction between the sentence-level representations before and after the self-attention block in the document encoder, the document-level representation is further enriched by using the collaborative attention; finally, the document representation is enriched by combining the topic distribution generated by the topic model.

[0052] In recent years, neural network models have been used to solve various text classification tasks. In order to encode text sequences of different lengths into a fixed vector representation, convolutional neural networks (CNN) and recurrent neural networks (RNN) are the most commonly used neural architectures. Recently, the Transformer, which relies entirely on attention mechanisms, has shown the latest performance of neural machine translation. Since then, attention mechanisms have been successfully applied in multiple fields, including image classification, text summarization, natural language inference, and sentiment analysis. Specifically, self-attention networks are a kind of attention mechanism, which can connect words that are far apart through shorter network paths compared to RNNs, not only improving the modeling ability of distance relationships, but also training faster due to the feedforward structure. It is also found in sentiment analysis research that self-attention outperforms CNN and RNN in classification accuracy. Therefore, in this embodiment, feature extraction is enhanced by aggregating multiple convolutional layers with different window sizes, and self-attention is used as the main building block of T-HMAN. However, traditional neural networks scan the entire document at once and do not take advantage of the hierarchical nature of the document, such as the structural relationship between words and sentences, and the structural relationship between sentences and documents. In order to solve this problem, a hierarchical neural structure is introduced, which converts sentence-level and document-level inputs into sentence and document encoders, respectively, and achieves the most advanced classification accuracy.

[0053] Embodiment 2:

[0054] The embodiment provides a text classification system based on a topic-aware hierarchical multi-attention network, which comprises the following steps:

[0055] A data acquisition module is configured to acquire text information to be classified.

[0056] A classification module is configured to obtain a classification result according to the acquired text information and a preset text classification model.

[0057] The text classification model is constructed in a hierarchical structure and comprises multiple attention mechanisms, and converts sentence-level and document-level inputs into sentence and document encoders. First, the multi-head self-attention is combined with a convolutional neural network in the hierarchical structure of the text classification model. Then, the sentence-level representation before and after the self-attention block in the document encoder is learned through interaction. Finally, the document representation is enriched by combining the topic distribution generated by the topic model.

[0058] The working method of the system is the same as the text classification method based on the topic-aware hierarchical multi-attention network in the embodiment 1, and details are not repeated here.

[0059] Embodiment 3

[0060] The embodiment provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the text classification method based on the topic-aware hierarchical multi-attention network in the embodiment 1 when executing the program.

[0061] Embodiment 4

[0062] The embodiment provides a computer readable storage medium, which stores a computer program, and the program implements the steps in the text classification method based on the topic-aware hierarchical multi-attention network in the embodiment 1 when executed by a processor.

[0063] The above only describes the preferred embodiments of the embodiment and is not used to limit the embodiment. The embodiment can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the embodiment should be included in the protection scope of the embodiment.

Claims

1. A text classification method based on topic-aware hierarchical multi-attention network, characterized in that, The method comprises the following steps: obtaining text information to be classified; obtaining a classification result according to the obtained text information and a preset text classification model; wherein the text classification model is constructed in a hierarchical structure to form multiple attention mechanisms, and converts sentence-level and document-level inputs into sentence and document encoders respectively; first, the multi-head self-attention is combined with the convolutional neural network in the hierarchical structure of the text classification model; then, the sentence-level representations before and after the self-attention block in the document encoder are interactively learned; finally, the document representation is enriched by combining the topic distribution generated by the topic model; the importance of the sentence-level and document-level representations is encoded by using soft attention order; the importance of each entry is measured by Softmax through the calculation of the context vector, and the normalized attention weight is obtained by Softmax; the final output representation is calculated by weighted summation of the normalized weight; The document-level representation is then extended with common attention in the document encoder; given a document encoder s e ∈ Rm×dkprevious sentence representation and document encoder c e ∈ Rm×dkposterior sentence representation, common attention attending to both sentence representations interactively; the attention weight of the two sentence representations is calculated by Softmax, and the output vector of the common attention is calculated by the weighted sum of the two sentence features; then, the two sentence representations are connected and fed into the soft attention in the document encoder; finally, the output of the soft-attention and the document-topic distribution are concatenated and fed into the final Softmax function; the soft attention mechanism is used in the sentence-level encoding and the document encoding; the importance of each entry is measured by Softmax through the calculation of the context vector, and the normalized attention weight is obtained by Softmax; the final output representation f(soft-attention) can be calculated by weighted summation of the normalized weight; where W s and b s are weights and biases, respectively, c e is an output multi-threaded self-attention block represented by, tanh is a non-linear function, u represents an input sequence of hidden states, Wcontext e R dk represents a context vector, a is a normalized weight, represents the importance of each entry in the sequence, and f is the output of the soft attention. （1） （2） （3） The document-level representation is then extended with common attention in the document encoder; specifically, given a document encoder s e The sentence representations before and the document encoder c e The sentence representations after, the common attention interacts on both sentence representations; first, the matrix L in formula (4) is calculated, and then the matrix is used as a feature to predict the attention weight in formula (5); the attention weight of the two sentence representations is calculated by Softmax, and the output vector of the common attention is calculated by the weighted sum of the two sentence features in formula (6) and formula (7); （4） （5） （6） （7） where W l ∈ Rdk×dk denotes the weight matrix learned through the network, W s , W d ∈ Rp×dk, W hs and W hd ∈ R1×p are weight parameters, p denotes the hidden size of common attention, a s , a d ∈ R1×m are attention weights, s, c ∈ R1×dk are two sentence representations generated by common attention; then, s and c are concatenated and passed to soft attention in the document encoder; finally, the output de of soft-attention and the document-topic distribution dt are concatenated and fed into the final Softmax function. 2.The theme-aware based hierarchical multi-attention network based text classification method of claim 1, wherein, the feature extraction is enhanced by aggregating multiple convolution layers with different window sizes. 3.The theme-aware based hierarchical multi-attention network based text classification method of claim 1, wherein, The collaborative attention mechanism is used to interactively learn the sentence representations with different levels of abstraction, and enrich the document-level representation in the document encoder. 4.The method of claim 1, wherein, The feature representation of the word-level and document-level inputs is extended by fusing the global co-occurrence information from the topic word distribution and the document-topic probability from the document-topic distribution generated by the LDA model. 5.The theme-aware based hierarchical multi-attention network based text classification method of claim 1, wherein, The self-attention mechanism focuses on each position in the sequence by comparing each entry with all entries in the same sequence; the output sequence of the self-attention contains the information of each entry and its relationship with all entries; the query, key and value vectors of each entry position are calculated by a fully connected layer; the attention weight is learned from different parts of the sequence. 6.The method of text classification based on topic-aware hierarchical multi-attention network according to claim 5, wherein, The convolutional layer and the pooling layer are used to enable the self-attention network to obtain the expressive representation from the query, key and value vectors, so as to enhance the feature extraction capability.

7. A text classification system based on topic-aware hierarchical multi-attention network, based on the text classification method based on topic-aware hierarchical multi-attention network according to any one of claims 1-6, characterized in that, The method comprises the following steps: a data acquisition module configured to obtain text information to be classified; a classification module configured to obtain a classification result according to the obtained text information and a preset text classification model; The text classification model is constructed in a hierarchical structure, and multiple attention mechanisms are used to convert sentence-level and document-level inputs into sentence and document encoders respectively; first, multi-head self-attention is combined with a convolutional neural network in the hierarchical structure of the text classification model; then, sentence-level representations before and after the self-attention block in the document encoder are learned through interaction; finally, the document representation is enriched by combining the topic distribution generated by the topic model.

8. An electronic device, comprising: A computer program product comprising a memory, a processor and a computer program stored on the memory and loadable into the processor, the processor implementing the steps of the method for text classification based on a topic-aware hierarchical multi-attention network according to any one of claims 1-6 when executing the program.

9. A computer-readable storage medium, characterized in that, A computer program product comprising a memory, a processor and a computer program stored on the memory and loadable into the processor, the processor implementing the steps of the method for text classification based on a topic-aware hierarchical multi-attention network according to any one of claims 1-6 when executing the program.

Images