Text classification network based on BiLSTM and mixed attention ResNet

Abstract

The invention relates to the field of text classification, in particular to a BiLSTM and mixed attention ResNet-based text classification network, a Bi-Hybrid Att-ResNet text classification network model comprises a BiLSTM, mixed attention and ResNet, the BiLSTM of the model is used for establishing a long-term dependency relationship, context representation before and after access and a mixed attention mechanism, key information in BiLSTM output can be captured more effectively, and the text classification network model can be applied to the field of text classification. The method comprises the following steps of: firstly, extracting a sentence by using ResNet, facilitating feature extraction, increasing network depth, accelerating network convergence and capturing n-gram features in the sentence, and finally, classifying the processed context information by using a full connection layer. And the Bi-Hybrid Att-ResNet can be used for capturing the local features and the global semantic information of the text.

Description

Technical Field

[0001] The present invention relates to the field of text classification, and in particular to a text classification network based on BiLSTM and hybrid attention ResNet. Background Art

[0002] Text classification is the most widely used and important field in natural language processing. At present, text classification has been widely used in many fields in real life, such as sentiment analysis, question answering, topic tagging, etc. Early text classification tasks can be completed by manually labeling text data. However, with the rapid development of the Internet, the amount of text data that needs to be processed has increased exponentially. Traditional manual data processing faces huge challenges, and the classification accuracy of this method is easily affected by human factors such as the knowledge reserve and energy of professionals. Therefore, automatic text classification has become very important.

[0003] Automatic text classification methods can be divided into rule-based methods, machine learning-based methods, and deep learning-based methods. The rule-based method classifies texts through a set of pre-defined rules, but defining the rules requires complete domain knowledge. Machine learning-based methods have been proven to be very effective in the field of text classification. Machine learning methods are mostly divided into three steps in the classification process: data preprocessing, feature extraction (such as bag-of-words model and TF-IDF) and classification methods (using Naive Bayes ( Although these methods have performed well in classification tasks, they have obvious limitations in actual use. For example, they rely heavily on feature engineering, which takes a lot of time and money, and are too dependent on domain knowledge, which leads to limited effectiveness of such methods in new classification tasks. In addition, these methods usually ignore the sequence information, contextual information, and semantic information of the words themselves in the text data, which is inconsistent with the process of human understanding of sentences.

[0004] In recent years, deep learning methods have made breakthrough progress in the field of natural language processing. Compared with existing traditional methods, deep learning methods avoid the process of manually designing rules and features. They can automatically mine the essential features of text from samples by learning a deep nonlinear network structure, and can capture the deep semantic representation information of text data, making recognition and classification more accurate and reliable. The most significant results have been achieved by convolutional neural networks (CNN) and recurrent neural networks (RNN).

[0005] CNN can learn local features from data, but it does not have the ability to learn sequential correlation. Compared with CNN, RNN has obvious advantages in learning sequential correlation, but it cannot extract features in parallel. Text classification is actually a modeling task. Due to the characteristics of RNN, RNN is more widely used in text-related tasks. However, for long text data, traditional RNN will have gradient explosion and vanishing problems. Long short-term memory (LSTM) is an RNN architecture with long short-term memory units as hidden units, which can capture long-term dependencies and effectively solve the problem of gradient vanishing and explosion. Because of LSTM's powerful ability to extract high-level text information, it plays an important role in NLP. BiLSTM is a further development of LSTM. BiLSTM combines forward hidden layers and backward hidden layers, and can access the contextual representations before and after. Compared with BiLSTM, LSTM only uses unidirectional contextual information. Therefore, BiLSTM works better than LSTM in NLP tasks.

[0006] For existing text classification tasks, using only BiLSTM cannot meet the requirements. The reason is that the vector representation of text is a high-dimensional vector. When facing high-dimensional vectors, BiLSTM faces the problems of large network parameters, difficult optimization, and insufficient classification accuracy when performing classification tasks. In view of the limitations of existing text classification models, the present invention proposes a text classification network based on BiLSTM and hybrid attention ResNet. Summary of the invention

[0007] The purpose of the present invention is to provide a text classification network based on BiLSTM and hybrid attention ResNet to solve the problems raised in the above background technology.

[0008] To achieve the above purpose, a text classification network based on BiLSTM and hybrid attention ResNet, the Bi-HybridAtt-ResNet text classification network model includes:

[0009] BiLSTM, used to establish long-term dependencies. BiLSTM processes input sentences from two directions, namely from left to right and from right to left, to obtain feature representations with more contextual information. Two LSTMs are used, one of which processes past information and the other processes future information. The outputs of the two LSTMs are combined to form the final output.

[0010] Hybrid attention, used to improve classification accuracy, consists of multi-head attention and coordinate attention, using residual connections;

[0011] ResNet is used to increase network depth and accelerate model convergence. ResNet consists of convolutional layers and pooling layers. The convolutional layers are used for feature extraction, and the pooling layers are used for feature processing. After the data enters ResNet and passes through the pooling layer and multiple layers of convolution, the output enters the fully connected layer.

[0012] Preferably: the multi-head self-attention mechanism accelerates the calculation through matrix dot product, giving a matrix Q with n query vectors, where Key-value The hidden layer value of the encoder is The weight matrix for calculating the attention mechanism is as follows:

[0013]

[0014] where d is the dimension of the query matrix and key values.

[0015] Preferably: The multi-head attention calculation first inputs a matrix vector It uses different initialization matrices to linearly calculate the key values ​​and hidden layers of the encoder h times, and the attention mechanism is executed in parallel each time, and finally obtains d v The output value of the dimension is then connected once and linearly calculated to obtain the final output value. The coefficient matrix corresponding to the i-th head query, key value and encoder hidden layer value is recorded as Then the correlation between the query and the key value is calculated by the scaled dot product, and finally the hybrid semantic representation H is output. i , the mathematical formula is as follows:

[0016]

[0017] Finally, the parallel multi-heads are concatenated through serial operations to form a vector M, and then the final semantic representation Y is obtained through linear calculation. The calculation formula is as follows:

[0018] M=Concat(H1,...,H h )

[0019] Y=MW

[0020] in

[0021] Preferred: The coordinate attention utilizes two one-dimensional global pooling operations to aggregate the input features along the vertical and horizontal directions into two separate direction-aware feature maps, respectively. Then, the two feature maps embedded with specific direction information are encoded into two attention maps, respectively, each of which captures the long-range dependency of the input feature map along a spatial direction. In this way, the position information is retained in the generated attention map. Then, the two attention maps are applied to the input feature map by multiplication to emphasize the representation of interest.

[0022] Preferably: the coordinate attention given input X, using the two spatial ranges (H, 1) or (1, W) of the pooling kernel to encode each channel along the horizontal coordinate and the vertical coordinate respectively, the output of the cth channel at height h can be expressed as:

[0023]

[0024] Similarly, the output of the cth channel with width w can be written as:

[0025]

[0026] The generated feature maps are first concatenated and then sent to the shared 1×1 convolution transformation function F1 to obtain:

[0027]

[0028] where [·,·] represents the concatenation operation along the spatial dimension, δ is a nonlinear activation function, is an intermediate feature map encoding spatial information in the horizontal and vertical directions, r is the reduction ratio used to control the block size, and f is divided into two independent tensors along the spatial dimension. and The other two 1×1 convolution transformations F h and F w Used to respectively h and f w Transformed into a tensor with the same number of channels as the input X, we get:

[0029] g h =σ(F h (f h ))

[0030] g w =σ(F w (f w ))

[0031] σ is the sigmoid function, and the output g h and g w are expanded and used as attention weights respectively, the output of the coordinate attention block Y can be written as:

[0032]

[0033] Preferably, the ResNet is composed of a plurality of residual blocks arranged in sequence, and adjacent convolutional layers are connected by shortcuts to form residual blocks, and the residual blocks are defined as:

[0034] y=F(x,{W i})+x

[0035] Here x and y are the input and output vectors of the layer under consideration, and the function F(x,{W i}) represents the residual mapping to be learned. The operation F+x is performed by shortcut connection and element-by-element addition. In the above formula, the sizes of x and F are equal. If the sizes are not equal, they can be linearly projected W s To match the size:

[0036] y=F(x,{W i})+W s x.

[0037] Preferably: the model uses cross entropy as the loss function, and the loss function can be expressed as follows, respectively for binary classification tasks and multi-classification tasks

[0038]

[0039]

[0040] Where N is the number of training samples, M is the number of categories, i represents the training sample, y i and ic is the label of the sample, p i and p ic is the output of Bi-HybridAtt-ResNet.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. A hybrid attention mechanism based on multi-head attention mechanism and coordinate attention mechanism is proposed. The coordinate attention mechanism is an attention module proposed for image field research. The coordinate attention mechanism used in the image field is used in the text field to more effectively capture the key information in the BiLSTM output, which is conducive to feature extraction.

[0043] 2. The complete residual network is used to extract deeper features of the data and achieve better classification results. The advantage of the residual network is that it can continue to update the parameters of the neural network while extracting deep features, and there will be no problem of stopping updating the parameters of the neural network.

[0044] 3. The Bi-HybridAtt-ResNet model is proposed by modifying and combining the classic model. BiLSTM is used to establish long-term dependencies. The hybrid attention mechanism can more effectively capture the key information in the BiLSTM output, which is conducive to feature extraction. ResNet is used to increase the network depth and accelerate network convergence, achieving better classification results in the field of text classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is the Bi-HybridAtt-ResNet framework of the present invention;

[0046] Figure 2 This is the LSTM structure diagram of the present invention;

[0047] Figure 3 This is the BiLSTM structure diagram of the present invention;

[0048] Figure 4 It is the multi-head attention mechanism of the present invention;

[0049] Figure 5 Scaling dot product attention structure diagram for the present invention;

[0050] Figure 6 It is the coordinate attention structure diagram of the present invention;

[0051] Figure 7 This is the ResNet structure diagram of the present invention;

[0052] Figure 8 This is a residual block structure diagram of the present invention;

[0053] Fig. 9 is the accuracy of each model of the present invention on four data sets;

[0054] Fig.10 It is the accuracy of different models on the AGNews dataset in the ablation experiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the implementation regulations described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Example

[0057] See also Figure 1 , the figure is a model framework diagram of the present invention, a text classification network based on BiLSTM and hybrid attention ResNet, the Bi-HybridAtt-ResNet text classification network model includes:

[0058] BiLSTM, used to establish long-term dependencies. BiLSTM processes input sentences from two directions, namely from left to right and from right to left, to obtain feature representations with more contextual information. Two LSTMs are used, one of which processes past information and the other processes future information. The outputs of the two LSTMs are combined to form the final output.

[0059] Hybrid attention, used to improve classification accuracy, consists of multi-head attention and coordinate attention, using residual connections;

[0060] ResNet is used to increase network depth and accelerate model convergence. ResNet consists of convolutional layers and pooling layers. The convolutional layers are used for feature extraction, and the pooling layers are used for feature processing. After the data enters ResNet and passes through the pooling layer and multiple layers of convolution, the output enters the fully connected layer.

[0061] As a feature learning technology, word embedding aims to map each word in the vocabulary to a real vector of a specific dimension, and finally input the vector into the model for training. Existing word embedding models include Word2Vec, GloVe, FastText, etc. The present invention adopts the 100-d GloVe model. The GloVe model is a method proposed by Stanford University in 2014. By mapping words into vector space, word vectors can contain as much semantic and grammatical information as possible. The GloVe model constructs a word co-occurrence matrix to represent the co-occurrence information of each word in the corpus in matrix form, and then decomposes the matrix to generate a low-dimensional representation vector for each word. GloVe combines global statistical information and local context features, so that the generated word vector reflects both the overall statistical characteristics of the corpus and captures the local context relationship of the word.

[0062] LSTM is an improvement on RNN. It solves the long-distance dependency problem of RNN by introducing a gating mechanism into LSTM. Like RNN, the output of LSTM is affected by the current input and previous output. The input gate, forget gate and output gate, three state gates and a cell state constitute an LSTM unit. The cell state and gate are the key parts of LSTM. Together, they retain the long-distance dependency between input elements. These gates are used to regulate the information flow in and out of LSTM. Figure 2 The architecture of LSTM is shown in Figure 1. Equations (1) to (7) describe the working principle of the input gate, forget gate, and output gate, where:

[0063] x t ,h t : represents the input state and hidden state at time t

[0064] C t : represents the candidate cell state and final cell state at time t

[0065] F t ,I t ,Out t : Represents forget gate, input gate and output gate

[0066] W f ,W i ,W o : Represents the weight parameters of the forget gate, input gate, and output gate

[0067] W c : represents the input unit weight parameter

[0068] b f ,b i ,b o : represents the bias parameters of the forget gate, input gate, and output gate

[0069] σ: represents the sigmoid function

[0070] *,+: represents point-by-point multiplication and addition

[0071] ++: represents the concatenation operator

[0072] u t =h t-1 ++x t (1)

[0073] F t =σ(W f u t +b f ) (2)

[0074] I t =σ(W i u t +b i ) (3)

[0075]

[0076]

[0077] Out t =σ(W o u t +b o ) (6)

[0078] h t =Out t *tanh(C t ) (7)

[0079] The input gate defines which data should be focused on in the current input, while the forget gate determines which data can be retained and deleted from the previous state. The output gate determines the data sent to the next time step. The unidirectional LSTM creates contextual representations by processing the input on the right or left side. However, both the previous and following contexts will have an impact on the semantics of the word. Processing only one-sided input cannot obtain enough semantic information. BiLSTM processes the input sentences from two directions, namely from left to right and from right to left, so that feature representations with more contextual information can be obtained. Figure 3 The structure of BiLSTM is shown, which uses two LSTMs, one of which processes past information and the other processes future information, and combines the outputs of the two LSTMs to form the final output.

[0080] The basic idea of ​​the attention mechanism is to assign different weights according to the relevance and importance of each information element to emphasize those parts that are more critical to the current task or context. The hybrid attention mechanism proposed in this invention consists of multi-head attention and coordinate attention, and uses residual connections to enhance the reuse of features.

[0081] The structure of the multi-head attention mechanism is as follows Figure 4 As shown in the figure, it can be seen that the core part of multi-head attention is the scaled dot product operation, and the structure is as follows Figure 5 As shown in Figure 1, this is different from the traditional attention mechanism, which has only one forward neural network. The multi-head self-attention mechanism accelerates the calculation through matrix dot product. The model gives a matrix Q with n query vectors, where Key-value The hidden layer value of the encoder is The weight matrix for calculating the attention mechanism is as follows:

[0082]

[0083] where d is the dimension of the query matrix and the key value. The two most commonly used attention functions are additive attention and dot product (multiplicative) attention, except for the scaling factor In addition, the dot product attention is the same as the algorithm of the present invention. The additive attention uses a feedforward network with a single hidden layer to calculate the compatibility function. For smaller d values, the performance of the two attentions is similar, and the additive attention even performs better. The reason is that when the d value is larger, the dot product will become larger and the softmax function will have a smaller gradient value. In order to reduce this effect, the dot product is selected and multiplied by

[0084] The multi-head attention calculation first inputs a matrix vector It uses different initialization matrices to linearly calculate the key values ​​and hidden layers of the encoder h times, and the attention mechanism is executed in parallel each time, and finally obtains d v The output value of the dimension is then connected once and linearly calculated to obtain the final output value. The coefficient matrix corresponding to the i-th head query, key value and encoder hidden layer value is recorded as Then the correlation between the query and the key value is calculated by the scaled dot product, and finally the hybrid semantic representation H is output. i , the mathematical formula is as follows:

[0085]

[0086] Finally, the parallel multi-heads are concatenated through serial operations to form a vector M, and then the final semantic representation Y is obtained through linear calculation. The calculation formula is as follows:

[0087] M=Concat(H1,...,H h ) (10)

[0088]

[0089] in

[0090] Multi-head self-attention can enhance the representation ability between sentences, the forward propagation speed is fast, and the multi-head self-attention calculation achieves higher parallelism. These advantages help to extract semantic information faster and more comprehensively.

[0091] Coordinate Attention( Figure 6 ) utilizes two one-dimensional global pooling operations to aggregate the input features along the vertical and horizontal directions into two separate direction-aware feature maps, respectively. Then, the two feature maps embedded with specific direction information are encoded into two attention maps, respectively. Each attention map captures the long-range dependency of the input feature map along one spatial direction. In this way, the position information is retained in the generated attention maps. Then, the two attention maps are applied to the input feature map by multiplication to emphasize the representation of interest.

[0092] The four-dimensional vector B×C×H×W is used in coordinate attention. The data in text tasks are usually represented as a three-dimensional vector B×C×S. In order to use the complete coordinate attention mechanism, we pad the data with one dimension before entering the coordinate attention, turning it from three-dimensional to four-dimensional B×C×S×F, where F represents the fourth dimension filled in this article.

[0093] Coordinate attention transforms global pooling into a one-to-one one-dimensional feature encoding to obtain precise location information. The attention module captures long-range interactions in space through location information. Specifically, given an input X, the two spatial ranges (H, 1) or (1, W) of the pooling kernel are used to encode each channel along the horizontal coordinate and the vertical coordinate respectively. Therefore, the output of the cth channel at height h can be expressed as:

[0094]

[0095] Similarly, the output of the cth channel with width w can be written as:

[0096]

[0097] The above two transformations aggregate features along two directions of space respectively, producing a pair of direction-aware feature maps. These two transformations also allow the attention module to capture long-range dependencies along one spatial direction and retain precise location information along another spatial direction, which helps the network locate objects of interest more accurately. After generating the feature maps, they are concatenated and then sent to the shared 1×1 convolution transformation function F1 to obtain:

[0098]

[0099] where [·,·] represents the concatenation operation along the spatial dimension, δ is a nonlinear activation function, is an intermediate feature map encoding spatial information in the horizontal and vertical directions, r is the reduction ratio used to control the block size, and f is divided into two independent tensors along the spatial dimension. and The other two 1×1 convolution transformations F h and F w Used to respectively h and f w Transformed into a tensor with the same number of channels as the input X, we get:

[0100] g h =σ(F h (f h )) (15)

[0101] g w =σ(F w (f w )) (16)

[0102] σ is the sigmoid function, and the output g h and g w are expanded and used as attention weights respectively, the output of the coordinate attention block Y can be written as:

[0103]

[0104] The coordinate attention module applies attention along the horizontal and vertical directions to the input tensor simultaneously. Each element in the two attention maps reflects whether there is an object of interest in the corresponding row and column. This encoding process enables the coordinate attention to more accurately locate the exact position of the object of interest, thereby helping the entire model to better recognize it.

[0105] Combining multi-head attention and coordinate attention to form a hybrid attention module can combine the advantages of the two attention mechanisms. On the basis of enhancing sentence representation capabilities and improving parallelism, it can more accurately focus on the location of the target of interest and improve the recognition ability of the entire model.

[0106] Simply using BiLSTM cannot extract deep features and achieve better classification results. Using multiple convolutional layers can mine deep features of text. However, when there are too many neural network parameters, gradient vanishing and gradient explosion will occur, causing parameter update stagnation. For this reason, ResNet is introduced, such as Figure 7 As shown in the figure, ResNet consists of convolutional layers and pooling layers. The convolutional layers are used for feature extraction, and the pooling layers are used for feature processing. After the data enters ResNet and passes through the pooling layer and multi-layer convolution, the output enters the fully connected layer. ResNet uses shortcut connections and identity mapping to reconstruct the learning process, redirect the network information flow, and increase the network depth. This architecture improves the representation ability of the model, speeds up the network convergence speed, and effectively solves common problems such as network degradation and gradient disappearance. Adjacent convolutional layers form residual blocks through shortcut connections. Multiple residual blocks are arranged in sequence to form ResNet. The residual blocks are as follows: Figure 8 As shown, it is defined as:

[0107] y=F(x,{W i})+x (18)

[0108] Here x and y are the input and output vectors of the layer under consideration, and the function F(x,{W i}) represents the residual mapping to be learned, and the operation F+x is performed through shortcut connections and element-by-element addition.

[0109] In formula (18), the sizes of x and F must be equal. If the sizes are not equal, they can be obtained by linear projection W s To match the size:

[0110] y=F(x,{W i})+W s x (19)

[0111] Compared with the original mapping, the residual mapping F is more sensitive to changes in the output, has a wider range of parameter adjustment, can speed up the learning speed of the model, and improves network performance.

[0112] Using cross entropy as the loss function can reduce the risk of gradient vanishing during stochastic gradient descent. The loss function can be expressed as follows: (20) for binary classification tasks and (21) for multi-classification tasks

[0113]

[0114]

[0115] Where N is the number of training samples, M is the number of categories, i represents the training sample, y i and ic is the label of the sample, p i and p ic is the output of Bi-HybridAtt-ResNet.

[0116] This embodiment evaluates the proposed Bi-HybridAtt-ResNet model by conducting experimental analysis on various binary and multi-classification datasets. Four widely used datasets are used to evaluate the model of the present invention. These datasets vary in type, size, number of categories, and document length. Table 1 shows the statistics of the datasets. Yelp Review Polarity (Yelp P) is used as the dataset for sentiment analysis, and AG's News (AGNews), DBPedia, and Yahoo! Answers (Yahoo) are used as the datasets for topic classification. The following is a detailed description of the datasets:

[0117] Yelp Review Polarity: The polarity Yelp dataset we use is a subset of the Yelp dataset, which comes from the Yelp dataset challenge in 2015. The polarity dataset has 280,000 training samples and 19,000 test samples per category.

[0118] AGNews: This dataset is constructed by selecting the four largest classes from the AG news article corpus, using only the title and description fields. Each class contains 30,000 training samples and 1,900 test samples.

[0119] DBPedia: DBPedia is a large multilingual knowledge base generated using the most frequently used infoboxes from Wikipedia, with 14 non-repeated classes selected from DBpedia 2014, each containing 40,000 training samples and 5,000 test samples. The fields used for this dataset contain the title and abstract of each Wikipedia article.

[0120] Yahoo! Answers: This dataset comes from the Yahoo! Answers comprehensive question-answering version 1.0 dataset, from which the 10 largest main categories are selected to construct the Yahoo! Answers dataset. Each class contains 140,000 training samples and 5,000 test samples. The fields we use include question title, question content, and best answer.

[0121] The baseline model is as follows:

[0122] TextCNN: A classic CNN-based classifier that uses one-dimensional convolution operations and max pooling to extract features.

[0123] CNN-based models: word-level CNN and character-level CNN.

[0124] BERT: BERT-based pre-trained model with classifier.

[0125] Table 1 Four text classification datasets used in the experiments in this paper.

[0126]

[0127] The commonly used evaluation indicators for text classification tasks are accuracy, precision, recall and F1 score. These indicators are explained in detail below.

[0128] Accuracy refers to the proportion of correct hypotheses generated by the algorithm among various predictions.

[0129]

[0130] Precision is the ratio of the number of samples correctly predicted as positive to the number of samples predicted as positive.

[0131]

[0132] Recall is the number of true positives divided by the number of all relevant samples (including samples that should have been positive).

[0133]

[0134] The F1 score is the harmonic mean of the precision and recall values.

[0135]

[0136] Among them, TP: true positive, TN: true negative, FP: false positive, and FN: false negative.

[0137] The hyperparameters involved are set as follows: the ResNet filter size is 3, the number of BILSTM hidden layers is set to 1, the size is set to 64, the number of attention heads of the multi-head attention is set to 8, the Adam optimizer is used, ReLU is used for all non-linear activation functions, Dropout regularization with a ratio of 0.6 is applied during training, and the learning rate is set to 3×10 -3 , Batchsize is set to 512 and the model is trained for 20 rounds. The experimental environment programming language of the model proposed in this invention is Python 3.8, the deep learning framework is PyTorch1.8.0, and the hardware environment is NVIDIA RTX 4070GPU.

[0138] In addition to the above five benchmark models, the present invention also adds two popular deep learning models, FastText and DPCNN, to conduct comparative experimental analysis with the model proposed in the present invention. The comparison results of the four data sets are presented in the following four tables. The data sets corresponding to these four tables are AGNews (Table 2), DBpedia (Table 3), YelpP (Table 4), and Yahoo (Table 5). Fig. 9 It is a bar chart of the accuracy of all models on four datasets.

[0139] Table 2 Performance of all models on the AGNews dataset, the best results are shown in bold.

[0140]

[0141] Table 3 Performance of all models on the DBpedia dataset, the best results are shown in bold.

[0142]

[0143] Table 4 Performance of all models on the Yelp P dataset, the best results are shown in bold.

[0144]

[0145]

[0146] Table 5 Performance of all models on the Yahoo dataset, the best results are shown in bold.

[0147]

[0148] As shown in the table above and Fig. 9As shown in the figure, the Bi-HybridAtt-ResNet proposed in the present invention outperforms other models on the three datasets of DBpedia, Yelp P and Yahoo, and is second only to BERT on the AGNews dataset, which verifies its effectiveness. Bi-HybridAtt-ResNet integrates BiLSTM, hybrid attention mechanism and ResNet, and in this way makes up for the shortcomings of single RNN and CNN. The model can better capture contextual semantic information and handle long-distance dependencies, and can also give more attention and weight to key information, capture local features, increase the reuse of features, reduce the risk of overfitting, and accelerate convergence.

[0149] In order to verify the rationality of the model configuration proposed in this paper, four ablation experiments were conducted, and the BiLSTM network was improved step by step. First, a multi-head attention module was added after the BILSTM network to form a BiLSTM+Muti-Head network. Next, a coordinate attention module was added to the bottom of the network to form a BiLSTM+Hybridatt network. Finally, ResNet was added to form Bi-HybridAtt-ResNet. All models were trained and tested on the AGNews dataset. The evaluation index data are shown in Table 6.

[0150] The loss function curve and evaluation index data are shown in Table 6 and Fig.10 As shown, from Fig.10 It can be seen that with the continuous improvement of the network model structure, the accuracy of the model on the data set continues to improve, and the fluctuation of the curve is getting smaller and smaller. As shown in Table 6, with the continuous improvement of the network structure, the values ​​of various evaluation indicators are increasing, and the data are relatively stable. Multi-head attention, coordinate attention and ResNet have greatly improved the network structure, enabling the network to extract more information and obtain higher accuracy.

[0151] Table 6 Ablation experiments on the AGNews dataset

[0152]

[0153]

[0154] The above content is a further detailed description of the present invention in combination with specific implementation methods. It cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, some simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the scope of protection determined by the claims submitted for the present invention.

Claims

1. A text classification network based on BiLSTM and hybrid attention ResNet, characterized by: The Bi-HybridAtt-ResNet text classification network model includes: BiLSTM, used to establish long-term dependencies. BiLSTM processes input sentences from two directions, namely from left to right and from right to left, to obtain feature representations with more contextual information. Two LSTMs are used, one of which processes past information and the other processes future information. The outputs of the two LSTMs are combined to form the final output. Hybrid attention, used to improve classification accuracy, consists of multi-head attention and coordinate attention, using residual connections; ResNet is used to increase network depth and accelerate model convergence. ResNet consists of convolutional layers and pooling layers. The convolutional layers are used for feature extraction, and the pooling layers are used for feature processing. After the data enters ResNet and passes through the pooling layer and multiple layers of convolution, the output enters the fully connected layer.

2. The text classification network based on BiLSTM and hybrid attention ResNet according to claim 1, characterized in that: The multi-head self-attention mechanism accelerates the calculation through matrix dot product, giving a matrix Q with n query vectors, where Key-value The hidden layer value of the encoder is The weight matrix for calculating the attention mechanism is as follows: where d is the dimension of the query matrix and key values.

3. The text classification network based on BiLSTM and hybrid attention ResNet according to claim 2, characterized in that: The multi-head attention calculation first inputs a matrix vector It uses different initialization matrices to linearly calculate the key values ​​and hidden layers of the encoder h times, and the attention mechanism is executed in parallel each time, and finally obtains d v The output value of the dimension is then connected once and linearly calculated to obtain the final output value. The coefficient matrix corresponding to the i-th head query, key value and encoder hidden layer value is recorded as Then the correlation between the query and the key value is calculated by the scaled dot product, and finally the hybrid semantic representation H is output. i , the mathematical formula is as follows: H i =Attention(QW i Q ,KW i K ,VW i V ) Finally, the parallel multi-heads are concatenated through serial operations to form a vector M, and then the final semantic representation Y is obtained through linear calculation. The calculation formula is as follows: M=Concat(H1,...,H h ) Y=MW in 4. The text classification network based on BiLSTM and hybrid attention ResNet according to claim 3, characterized in that: The coordinate attention utilizes two one-dimensional global pooling operations to aggregate the input features along the vertical and horizontal directions into two separate direction-aware feature maps, respectively. Then, the two feature maps embedded with specific direction information are encoded into two attention maps, respectively. Each attention map captures the long-range dependency of the input feature map along a spatial direction. In this way, the position information is retained in the generated attention map. Then, the two attention maps are applied to the input feature map by multiplication to emphasize the representation of interest.

5. The text classification network based on BiLSTM and hybrid attention ResNet according to claim 4, characterized in that: The coordinate attention is given an input X, and uses the two spatial ranges (H, 1) or (1, W) of the pooling kernel to encode each channel along the horizontal coordinate and the vertical coordinate respectively. The output of the cth channel at height h can be expressed as: Similarly, the output of the cth channel with width w can be written as: The generated feature maps are first concatenated and then sent to the shared 1×1 convolution transformation function F1 to obtain: f=δ(F1([z h ,z w ])) where [·,·] represents the concatenation operation along the spatial dimension, δ is a nonlinear activation function, is an intermediate feature map encoding spatial information in the horizontal and vertical directions, r is the reduction ratio used to control the block size, and f is divided into two independent tensors along the spatial dimension. and The other two 1×1 convolution transformations F h and F w Used to respectively h and f w Transformed into a tensor with the same number of channels as the input X, we get: g h =σ(F h (f h )) g w =σ(F w (f w )) σ is the sigmoid function, and the output g h and g w are expanded and used as attention weights respectively, the output of the coordinate attention block Y can be written as:

6. The text classification network based on BiLSTM and hybrid attention ResNet according to claim 5, characterized in that: The ResNet is composed of multiple residual blocks arranged in sequence, and adjacent convolutional layers are connected through shortcuts to form residual blocks, which are defined as: y=F(x,{W i })+x Here x and y are the input and output vectors of the layer under consideration, and the function F(x,{W i }) represents the residual mapping to be learned. The operation F+x is performed by shortcut connection and element-by-element addition. In the above formula, the sizes of x and F are equal. If the sizes are not equal, they can be linearly projected W s To match the size: y=F(x,{W i })+W s x。 7. The text classification network based on BiLSTM and hybrid attention ResNet according to claim 6, characterized in that: The model uses cross entropy as the loss function. The loss function can be expressed as follows: binary classification task and multi-classification task Where N is the number of training samples, M is the number of categories, i represents the training sample, y i and ic is the label of the sample, p i and p ic is the output of Bi-HybridAtt-ResNet.

Images