Short text classification method based on fasttext and lstm

Abstract

A short text classification method based on FastText and LSTM, in the offline stage, the mapping of word or word vector is generated by pre-training word vector, the single-layer bidirectional chain long short-term memory network and the hidden layer are trained; in the online stage, the trained neural network model is used to process the input short text and obtain the predicted classification label. Through the extraction link, the long short-term memory network in the recurrent neural network is fused, in the activation stage, the tanh activation function is used for processing, the accuracy of text classification is significantly improved under the condition of ensuring the training efficiency and the size of the model.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of neural network applications, and particularly relates to a short text classification method based on FastText and LSTM. BACKGROUND

[0002] With the development of mobile phone chips, the computing power of mobile phone chips is continuously improved, and the method of deploying a deep learning model on a mobile phone has been widely used, which can guarantee the privacy of users and reduce the time and network consumption of software for communication with a server. However, most of the existing text classification algorithms pursue accuracy, which leads to problems such as long model training time and large model size, and are not suitable for deployment on mobile phones. FastText is an open-source text classification model, which adopts the structure of a continuous bag-of-words model and is composed of an input layer, a hidden layer and an output layer. Since the text often contains many meaningless and low-differentiation high-frequency words, if the text is directly converted into a word vector for classification, the classification efficiency will be affected. Therefore, the common solution is to further operate the word vector in the feature selection step. For Chinese text, FastText is often combined with an n-gram model, but the n-gram model generates a large number of word tables during training, which occupies a large amount of storage space and takes a long time to train, and is not suitable for deployment on mobile phones. The long short-term memory network is an optimized recurrent neural network, which introduces a gate mechanism to control the flow of features and loss, and can solve the long-term dependence problem of the recurrent neural network. SUMMARY

[0003] The application provides a short text classification method based on FastText and LSTM, which solves the problems of long training time, low accuracy and large model size in the prior art, and significantly improves the accuracy of text classification by fusing the long short-term memory network in the recurrent neural network in the feature extraction link, and using the tanh activation function for processing in the activation stage under the condition of ensuring the training efficiency and the model size.

[0004] The application is implemented by the following technical solutions:

[0005] The application relates to a short text classification method based on FastText and LSTM, which generates a mapping of word or word vectors by pre-training word vectors in an offline stage, and trains a single-layer bidirectional chain long short-term memory network and a hidden layer; in an online stage, a trained neural network model is used to process input short text and obtain a predicted classification label.

[0006] This invention relates to a system for implementing the above method, comprising: a text preprocessing unit, a word embedding unit, a neural network unit, and an output classification unit, wherein: the text preprocessing unit performs word segmentation or character segmentation on the input short Chinese text to obtain a word sequence; the word embedding unit performs text-to-word vector conversion on the word sequence using pre-trained word vectors to obtain a word vector matrix; the neural network unit performs feature extraction on the word vector matrix using an LSTM layer and performs non-linear activation on a hidden layer to obtain a feature matrix; and the output classification unit performs normalization processing on the feature matrix using a hierarchical softmax function to obtain a predicted classification label.

[0007] Technical effect

[0008] This invention incorporates a feature selection step into the FastText algorithm model, using a single-layer bidirectional LSTM network to extract features from the word vector matrix. Compared to existing technologies, this improves word vector discriminative power and classification accuracy while maintaining shorter model training time and smaller model footprint. Attached Figure Description

[0009] Figure 1 This is a flowchart of the present invention;

[0010] Figure 2 A schematic diagram of LSTM;

[0011] Figure 3 This is a word vector matrix diagram;

[0012] Figure 4 This is a model diagram of the present invention;

[0013] Figure 5 A schematic diagram of the activation function tanh;

[0014] Figure 6 This is a schematic diagram of a layered Softmax layer.

[0015] Figure 7 The following is a graph showing experimental data from an example. Detailed Implementation

[0016] like Figure 1 As shown, this embodiment relates to a short text classification method based on FastText and LSTM. The parameters used in this method are as follows: each iteration contains 128 samples; the hidden layer size is 256, the sentence length is 32, and the classification labels are 10. The method involved in this embodiment includes the following steps:

[0017] Step 1) Generate pre-trained word vectors: Using the Sogou News dataset, map a character or word to a vector space to form a multi-dimensional vector. Specific operations include:

[0018] 1.1 Tokenization of the sample in words or characters;

[0019] 1.2 Removing duplicate words and encoding each word;

[0020] 1.3 Generating a dictionary, each row contains a word and its corresponding index code.

[0021] Step 2) Convert the input text into a word vector matrix, i.e. divide the text by words or characters, and map all words into a fixed-length word vector As shown in the matrix Figure 3 , where: IndexD(W i ) is the corresponding index of the word W i in the dictionary, Emb is a two-dimensional word vector list, and Emb(i) represents the i-th row vector in Emb.

[0022] As shown in the matrix Figure 3 , the i-th row in the word vector x represents the vector corresponding to the i-th character in the input text. In this embodiment, the input text is converted into a three-dimensional word vector matrix with a size of [128, 32, 300]

[0023] Step 3) Construct and train a single-layer bidirectional chain long short-term memory (LSTM) network for feature extraction.

[0024] As shown in the matrix Figure 4 , the single-layer bidirectional chain long short-term memory network includes a forward processing unit, a reverse processing unit, and an output splicing unit, wherein: the forward processing unit processes the calculation of the forward propagation of the text sequence, the reverse processing unit processes the calculation of the reverse propagation of the text sequence, and the output splicing unit splices the input of the forward processing unit with the output of the reverse processing unit to obtain a feature matrix.

[0025] In this embodiment, the word vector matrix is passed through the LSTM layer to obtain a feature matrix with a size of [128, 32, 600].

[0026] The long short-term memory network includes an input gate, an output gate, a forget gate, and a recurrently connected neuron, wherein: i t = σ(W i X t + U i h t-1 + b i ), C′ t = tanh(W c X t + U c h t-1 + b c ), ft = σ(W f X t + U f h t-1 + b f ), C t = i t tanh(C t + f t C t-1 ), o t = σ(W o X t + U o h t-1 + V o V t + b o ), h t = o t tanh(C t ), i t is the input vector at time t, h t is the output vector at time t, f t is the output matrix of the forget gate, W is the weight matrix of the input vector, U is the weight matrix of the output vector at time t-1, b is the bias weight vector, the function σ is the nonlinear activation function sigmoid, and V is the weight matrix.

[0027] The training of the single-layer bidirectional chain long short-term memory network specifically comprises the following steps.

[0028] Step i) averaging the word vectors: all inputs in the hidden layer are averaged together ave where F kt is the feature representation of the text, n is the number of hidden layer inputs, F i is the feature of a single input, w ave is the training weight, and the feature matrix with a size of [128, 32, 600] is calculated to obtain a feature matrix with a size of [128, 600]

[0029] Step ii) calculating the predicted value using the tanh function as shown in Figure 5 and the hierarchical softmax as shown in Figure 6 to output the classification label, wherein the average of the outputs of tanh is 0, and tanh is similar to y=kx near the origin, and after the tanh function is calculated, a matrix with a size of [128, 256] is obtained; the hierarchical softmax function reduces the time complexity of the softmax function from |v| to log|v|, thereby optimizing the time-consuming problem of the softmax function. The prediction formula of the softmax function is lable=softmax(F ave ), wherein label is the predicted probability distribution, and F The feature representation of the text. After normalization calculation by the hierarchical softmax function, a matrix of size [128, 10] is finally obtained, representing the probability of each of the 128 samples corresponding to 10 labels, and the label with the maximum probability is taken as the sample output label.

[0030] Step iii) training the weight matrix using the gradient descent method: after each iteration, the loss function Loss of the sample is calculated, and the validation data set Loss is calculated,

[0031] In this embodiment, the loss function uses the 0-1 function, that is, whether the labeled label in the sample and the model output label are equal is compared, and the result is 1 if they are equal, otherwise 0.

[0032] In this embodiment, the validation data set Where: loss_total represents the number of samples whose loss function Loss output is 0 in one iteration, and len(data_iter) represents the total number of samples in one iteration. If the loss of the validation data set does not decrease in 1000 iterations, or the test data set is trained for more than 20 times, the model training is ended.

[0033] Through specific actual experiments, using the programming language Python on a computer with an Intel Core i5-11300H processor and 16GB of memory, the THUNews data set, the Sohu news data set, the Toutiao news book and the test are used, the loss function is set to 0.5, the learning rate is set to 0.001, the batch-size is set to 128, the sentence length is set to 32, and the hidden layer size is set to 256. Run the above method to get the experimental results as shown in Figure 7 Compared with the traditional FastText model, the accuracy of the present application is higher. The calculation formula of the evaluation index F1 is In the formula, precision represents the precision, recall represents the recall, and the higher the F1 value, the higher the accuracy of the model

[0034] Compared with the FastText with a vocabulary size of 250,000 and using n-gram, the average training time of the present application is 24 minutes, and the average model size is 11.28MB; the average training time of the FastText using n-gram is 88 minutes, and the model size is 292MB. The model training time of the present application is shorter, and the occupied space is smaller.

[0035] Compared with the prior art, the present application improves the accuracy of text classification while ensuring shorter model training time and smaller model occupied space.

[0036] The above specific embodiments can be partially adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application, the protection scope of the present application is subject to the claims and is not limited by the above specific embodiments, each implementation within the scope is subject to the present application.

Claims

1. A short text classification method based on FastText and LSTM, characterized in that, In the offline phase, word or word pair mappings are generated by pre-trained word vectors, and a single-layer bidirectional chained long short-term memory network and hidden layers are trained. In the online phase, a trained neural network model is used to process the input short text and obtain predicted classification labels, specifically including: Step 1) Generate pre-trained word vectors: Using the Sogou News dataset, map a character or word to a vector space to form a multi-dimensional vector; Step 2) Convert the input text into a word vector matrix, that is, divide the text into words or characters and map all words into a word vector of fixed length. ,in: For words The corresponding index in the dictionary, It is a two-dimensional word vector list. express The row vector of the i-th row in the array; Step 3) Construct and train a single-layer bidirectional chained long short-term memory network for feature extraction; The training of the single-layer bidirectional chained long short-term memory network specifically includes: Step i) Calculate the mean of the word vectors: Calculate the mean of all inputs from the hidden layer together. ,in: Here, n represents the feature representation of the text, and n is the number of hidden layer inputs. Features for a single input For training weights; Step ii) Calculate the predicted value using the tanh function and hierarchical softmax, and output the classification label. The mean of the output of tanh is 0, and tanh is similar to y=kx near the origin. The hierarchical softmax function reduces the time complexity of the softmax function from |v| to log|v|, thus optimizing the problem of the computation time of the softmax function. Step iii) Train the weight matrix using gradient descent: After each iteration, calculate the loss function Loss of the samples, and then calculate the Loss of the validation dataset.

2. The short text classification method based on FastText and LSTM according to claim 1, characterized in that, The single-layer bidirectional chain-type long short-term memory network includes: a forward processing unit, a reverse processing unit, and an output splicing unit; The long short-term memory network includes: an input gate, an output gate, a forgetting gate, and recurrent self-connected neurons, wherein: , , , , , , The input vector at time t, The output vector at time t The output matrix of the forget gate, W is the weight matrix of the input vector, U is the weight matrix of the output vector at time t-1, b is the bias vector, and the function... The non-linear activation function is sigmoid, where V is the weight matrix.

3. The short text classification method based on FastText and LSTM according to claim 2, characterized in that, The aforementioned multidimensional vector is obtained in the following way: 1.1 Segment the samples into words or characters; 1.2 Remove duplicate words and encode each word; 1.3 Generate a dictionary, with each line containing a word and its corresponding index code; In the word vector, the i-th row of x represents the vector corresponding to the i-th character in the input text.

4. The short text classification method based on FastText and LSTM according to claim 1, characterized in that, The loss function is a 0-1 function, which compares whether the labels labeled in the sample are equal to the labels output by the model. If they are equal, the output result is 1, otherwise it is 0. The aforementioned verification dataset ,in: This represents the number of samples for which the Loss function outputs 0 in one iteration. This represents the total number of samples in one iteration.

5. A system for implementing the short text classification method based on FastText and LSTM as described in any one of claims 1-4, characterized in that, include: The system comprises a text preprocessing unit, a word embedding unit, a neural network unit, and an output classification unit. Specifically: the text preprocessing unit segments the input Chinese short text into words or characters to obtain a word sequence; the word embedding unit converts the text to word vectors using pre-trained word vectors to obtain a word vector matrix; and the neural network unit extracts features from the word vector matrix using an LSTM layer and performs non-linear activation through hidden layers to obtain a feature matrix. The output classification unit is normalized using a hierarchical softmax function based on the feature matrix to obtain the predicted classification label.

Images