Text classification method and device based on multi-channel feature representation of convolutional neural network

Abstract

The present invention provides a text classification method and device based on multi-channel feature representation using a convolutional neural network. The method relates to the field of artificial intelligence and can be applied to the financial technology and other technical fields. The text classification method based on multi-channel feature representation using a convolutional neural network includes: inputting the text data to be classified into two word vector models to obtain two word vector matrices; inputting the two word vector matrices into corresponding text information extraction models to obtain multi-channel text feature data; and inputting the multi-channel text feature data into a text classification model to obtain the text type. This invention can enrich text representation and improve text classification accuracy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a text classification method and device based on multi-channel feature representation of a convolutional neural network. Background Art

[0002] Figure 1 is a schematic diagram of the existing text classification scheme. Figure 1 As shown, the existing technology proposes a text classification solution that combines long short-term memory networks and convolutional neural networks. First, the input text is represented as a vector using word embedding. The local features of the text are extracted through a three-layer CNN, and then the full text semantics are integrated. At the same time, the LSTM is used to store the features of historical information in the text to obtain the text contextual semantics. Secondly, the input vector is fused with the output of each CNN layer to reuse the original features. The text contextual semantics and local features of the text are then fused. The resulting feature vector is first passed through a fully connected layer and then a softmax classifier is used to classify the final text feature vector. The dropout mechanism is used before the softmax classifier to prevent overfitting of the algorithm.

[0003] The existing technology has the following shortcomings:

[0004] 1. Text representation uses a single word vector, which cannot fully represent the semantic information in the corpus, resulting in a lack of rich semantic information in the text, affecting the final classification effect;

[0005] 2. The conventional LSTM method is used to extract the temporal features of text. Although it can obtain the long-distance temporal relationship of text, it does not fully consider the potential semantic relationship of temporal features, resulting in unsatisfactory classification results. Summary of the Invention

[0006] The main purpose of the embodiments of the present invention is to provide a text classification method and device based on multi-channel feature representation of convolutional neural networks to enrich text representation and improve text classification accuracy.

[0007] To achieve the above objectives, an embodiment of the present invention provides a text classification method based on multi-channel feature representation of a convolutional neural network, comprising:

[0008] Input the text data to be classified into two word vector models respectively to obtain two word vector matrices;

[0009] Input the two word vector matrices into the corresponding text information extraction model to obtain multi-channel text feature data;

[0010] Input multi-channel text feature data into the text classification model to obtain the text type.

[0011] In one embodiment, two word vector matrices are respectively input into corresponding text information extraction models to obtain multi-channel feature representation data including:

[0012] Inputting the first word vector matrix into a first text information extraction model to obtain first positive order text feature data and first reverse order text feature data;

[0013] Inputting the second word vector matrix into the second text information extraction model to obtain second positive order text feature data and second reverse order text feature data;

[0014] Multi-channel feature representation data is obtained according to the first normal order text feature data, the first reverse order text feature data, the second normal order text feature data, and the second reverse order text feature data.

[0015] In one embodiment, inputting the first word vector matrix into the first text information extraction model to obtain the first forward-order text feature data and the first reverse-order text feature data includes:

[0016] Inputting the first word vector matrix into a first text information extraction model to obtain a first positive order text feature vector and a first reverse order text feature vector respectively;

[0017] Vertically stacking the first positive order text feature vectors in sequence to obtain first positive order text feature data;

[0018] The first reverse-order text feature vectors are stacked vertically in sequence to obtain first reverse-order text feature data.

[0019] In one embodiment, inputting the second word vector matrix into the second text information extraction model to obtain the second forward text feature data and the second reverse text feature data includes:

[0020] Inputting the second word vector matrix into the second text information extraction model to obtain a second positive order text feature vector and a second reverse order text feature vector respectively;

[0021] vertically stacking the second positive order text feature vectors in sequence to obtain second positive order text feature data;

[0022] The second reverse-order text feature vectors are stacked vertically in sequence to obtain second reverse-order text feature data.

[0023] In one embodiment, inputting multi-channel text feature data into a text classification model to obtain a text type includes:

[0024] Input the multi-channel text feature data into the multi-scale convolutional network layer to obtain multi-dimensional text feature data;

[0025] The multi-dimensional text feature data is input into the fully connected output network to obtain the text type.

[0026] In one embodiment, it further includes:

[0027] Perform the following iterative process:

[0028] Input the training text data into the two word vector models respectively to obtain two training word vector matrices;

[0029] The two training word vector matrices are input into the corresponding bidirectional long short-term memory artificial neural network model to obtain multi-channel text feature training data;

[0030] Input the multi-channel text feature training data into the multi-channel convolutional network model to obtain the prediction type;

[0031] Determine the loss function based on the predicted type and the corresponding actual type;

[0032] When the loss function converges, the bidirectional long short-term memory artificial neural network model is determined as the text information extraction model, and the multi-channel convolutional network model is determined as the text classification model. Otherwise, the bidirectional long short-term memory artificial neural network model and the multi-channel convolutional network model are updated according to the loss function.

[0033] An embodiment of the present invention further provides a text classification device based on multi-channel feature representation of a convolutional neural network, comprising:

[0034] The word vector matrix module is used to input the text data to be classified into two word vector models respectively to obtain two word vector matrices;

[0035] The multi-channel text feature data module is used to input the two word vector matrices into the corresponding text information extraction model to obtain multi-channel text feature data;

[0036] The text type module is used to input multi-channel text feature data into the text classification model to obtain the text type.

[0037] In one embodiment, the multi-channel text feature data module includes:

[0038] A first text feature data unit is used to input the first word vector matrix into the first text information extraction model to obtain first positive order text feature data and first reverse order text feature data;

[0039] A second text feature data unit is used to input the second word vector matrix into the second text information extraction model to obtain second positive order text feature data and second reverse order text feature data;

[0040] The multi-channel feature representation data unit is used to obtain multi-channel feature representation data according to the first normal order text feature data, the first reverse order text feature data, the second normal order text feature data and the second reverse order text feature data.

[0041] In one embodiment, the first text feature data unit includes:

[0042] A first text feature vector subunit is used to input the first word vector matrix into the first text information extraction model to obtain a first positive order text feature vector and a first reverse order text feature vector;

[0043] A first positive-sequence text feature data subunit, configured to vertically stack first positive-sequence text feature vectors in order to obtain first positive-sequence text feature data;

[0044] The first reverse order text feature data subunit is used to vertically stack the first reverse order text feature vectors in sequence to obtain first reverse order text feature data.

[0045] In one embodiment, the second text feature data unit includes:

[0046] A second text feature vector subunit is used to input the second word vector matrix into the second text information extraction model to obtain a second positive order text feature vector and a second reverse order text feature vector respectively;

[0047] A second positive-sequence text feature data subunit is used to vertically stack the second positive-sequence text feature vectors in order to obtain second positive-sequence text feature data;

[0048] The second reverse order text feature data subunit is used to vertically stack the second reverse order text feature vectors in sequence to obtain second reverse order text feature data.

[0049] In one embodiment, the text type module includes:

[0050] A multi-dimensional text feature data unit, used to input multi-channel text feature data into a multi-scale convolutional network layer to obtain multi-dimensional text feature data;

[0051] The text type unit is used to input multi-dimensional text feature data into the fully connected output network to obtain the text type.

[0052] In one embodiment, it further includes:

[0053] The training word vector matrix module is used to input the training text data into two word vector models respectively to obtain two training word vector matrices;

[0054] The text feature training data module is used to input the two training word vector matrices into the corresponding bidirectional long short-term memory artificial neural network model to obtain multi-channel text feature training data;

[0055] The prediction type module is used to input the multi-channel text feature training data into the multi-channel convolutional network model to obtain the prediction type;

[0056] The loss function module is used to determine the loss function based on the predicted type and the corresponding actual type;

[0057] The iterative module is used to determine the bidirectional long short-term memory artificial neural network model as the text information extraction model and the multi-channel convolutional network model as the text classification model when the loss function converges; otherwise, the bidirectional long short-term memory artificial neural network model and the multi-channel convolutional network model are updated according to the loss function.

[0058] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of the text classification method based on multi-channel feature representation of a convolutional neural network are implemented.

[0059] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the text classification method based on multi-channel feature representation of a convolutional neural network are implemented.

[0060] An embodiment of the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the text classification method based on multi-channel feature representation of a convolutional neural network.

[0061] The text classification method and device based on multi-channel feature representation of convolutional neural network in an embodiment of the present invention input the text data to be classified into two word vector models respectively to obtain two word vector matrices, which are then input into corresponding text information extraction models to obtain multi-channel text feature data, and then the multi-channel text feature data is input into the text classification model to obtain the text type, which can enrich the text representation and improve the accuracy of text classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0063] Figure 1It is a schematic diagram of the existing text classification scheme;

[0064] Figure 2 4 is a flowchart of a text classification method based on multi-channel feature representation of a convolutional neural network in an embodiment of the present invention;

[0065] Figure 3 is a schematic diagram of a text classification method based on multi-channel feature representation of a convolutional neural network in an embodiment of the present invention;

[0066] Figure 4 This is a flow chart of S102 in an embodiment of the present invention;

[0067] Figure 5 This is a flow chart of S201 in an embodiment of the present invention;

[0068] Figure 6 This is a flow chart of S202 in an embodiment of the present invention;

[0069] Figure 7 This is a flow chart of S103 in an embodiment of the present invention;

[0070] Figure 8 is a flowchart of creating a text information extraction model and a text classification model in an embodiment of the present invention;

[0071] Figure 9 is a flowchart of creating a convolutional neural network text classification model in an embodiment of the present invention;

[0072] Figure 10 is a schematic diagram of multi-channel text feature representation in an embodiment of the present invention;

[0073] Figure 11 is a schematic diagram of a multi-scale convolutional network layer in an embodiment of the present invention;

[0074] Figure 12 1 is a structural block diagram of a text classification device based on multi-channel feature representation of a convolutional neural network in an embodiment of the present invention;

[0075] Figure 13 It is a structural block diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0077] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0078] The key terms involved in the present invention are as follows:

[0079] Convolutional Neural Network (CNN): A type of feedforward neural network with a deep structure that includes convolution calculations. It is one of the representative algorithms of deep learning.

[0080] Long Short-Term Memory (LSTM): A time-recurrent neural network designed to address the long-term dependency issues of general RNNs (recurrent neural networks). Therefore, RNNs all have a chain-like structure of repeated neural network modules.

[0081] Word2Vec and Glove: A group of related models for generating word vectors. These models are shallow, two-layer neural networks trained to reconstruct linguistic word text.

[0082] Word embedding: A general term for a set of language modeling and feature learning techniques in natural language processing, in which words or phrases from a vocabulary are mapped to vectors of real numbers. Conceptually, it involves mathematical embedding from a one-dimensional space for each word to a continuous vector space with lower dimensions.

[0083] Given that the existing technology does not fully consider the potential semantic relationship of temporal features, resulting in unsatisfactory classification results, an embodiment of the present invention provides a text classification method based on multi-channel feature representation of convolutional neural networks, which uses pre-trained word vectors from third parties from different sources to enrich text representation: one is the Word2Vec model based on the prediction method to better characterize local information, and the other is the Glove model based on the counting method to better utilize global information. The word vector representation obtained by fusing the above two models can enrich the text semantic information; then, two bidirectional long short-term memory networks are used to extract forward and reverse temporal information features from the text sequences represented by word vectors from different sources. In order to obtain the context dependency of the text, the prior art splices the feature vectors output in forward and reverse order at each moment as the final output result at each moment; unlike the prior art, the present invention stacks the outputs of each moment vertically in order to form a multi-channel method, which is specifically implemented by stacking the output vectors of each forward moment and the output vectors of each reverse moment, so that two channels can be formed. Two different word vector representation methods can form four channels, further enriching the text semantic information while retaining the temporal features of the text; then the multi-channel feature representation of the text is subjected to a multi-scale convolution operation, fully considering the information before and after the current moment, mining the potential semantic relationship between the temporal features, obtaining rich text features, and sending them to the subsequent fully connected layer and Softmax classifier to improve the classification accuracy. The present invention is described in detail below with reference to the accompanying drawings.

[0084] Figure 2 This is a flowchart of a text classification method based on multi-channel feature representation of a convolutional neural network in an embodiment of the present invention. Figure 3 Schematic diagram of a text classification method based on multi-channel feature representation of a convolutional neural network in an embodiment of the present invention. Figure 2-Figure 3 As shown in Figure 2, text classification methods based on multi-channel feature representation of convolutional neural networks include:

[0085] S101: Input the text data to be classified into two word vector models respectively to obtain two word vector matrices.

[0086] In the specific implementation, two trained word vector models are used to map the classified text data. Each word is mapped to a d-dimensional vector, and each sample sentence is mapped to a vector matrix. Where W1 represents the word vector model trained by Word2Vec, and W2 represents the word vector model trained by Glove. W1, W2∈R d*v , d is the word vector dimension, |v| represents the size of the dictionary (Internet vocabulary). The word vector matrix output by the Word2Vec word vector model is J1=W1V; the word vector matrix output by the Glove word vector model is J2=W2V, V∈R v*n, is the bag-of-words representation of the text data to be classified, and n is the length of the text data to be classified.

[0087] S102: Input the two word vector matrices into the corresponding text information extraction models respectively to obtain multi-channel text feature data.

[0088] Figure 8 This is a flowchart of creating a text information extraction model and a text classification model in an embodiment of the present invention. Figure 9 This is a flow chart of creating a convolutional neural network text classification model in an embodiment of the present invention. Figure 8-Figure 9 As shown, before executing S102, the process further includes:

[0089] Perform the following iterative process:

[0090] S601: Input the training text data into two word vector models respectively to obtain two training word vector matrices.

[0091] In practice, the training text data can include three long text datasets and three short text datasets. These six datasets are standard English datasets. AG is a dataset for classifying news article types; Yelp_P is a dataset for classifying user review sentiment, with only two categories: positive and negative; Yelp_F is also a user review dataset for sentiment classification, a more fine-grained sentiment classification dataset with five categories; SST-2 is a dataset for classifying movie review sentiment, divided into two categories: negative and positive; TREC is a dataset for classifying question types, with six categories: abbreviations, entities, people, places, and numbers; and MR is a dataset for classifying film reviews, divided into two categories: positive and negative. To reduce the interference of noise information, we first remove a series of prepositions that are prevalent in the text but have no actual meaning; secondly, we use regular expressions to remove a series of useless special characters; then, we restore the part of speech of abbreviations such as "I'm"; and finally, we convert all characters to lowercase. After preprocessing, the length of the original text is not uniform. In order to optimize the model training, the text length is set to the default value and adjusted by the hyperparameter max_length. The part of the sample sentence length that is shorter than the default value is marked with a specific word " <pda>" is padded to a uniform length. If the sample sentence length is longer than the default value, the excess part will be truncated.

[0092] S602: Input the two training word vector matrices into the corresponding bidirectional long short-term memory artificial neural network model respectively to obtain multi-channel text feature training data.

[0093] The core idea of ​​the Long Short-Term Memory (LSTM) network is to forget historical information and filter input information by introducing an adaptive "gate" mechanism. The three gate structures are called the input gate, forget gate, and output gate. The formulas involved are as follows:

[0094]

[0095] where i t 、f t 、o t They represent the input gate at time t, the forget gate at time t, and the output gate at time t. At each moment, the forget gate controls the degree of forgetting at the previous moment (t-1 moment), and the input gate controls the candidate memory information at time t. The degree of writing into long-term memory, the output gate controls how short-term memory is affected by long-term memory. For this layer, the input at time t is a training word vector matrix x t , the hidden layer state vector at time t-1 is h t-1 . Calculate the forget gate f t , select the information to be forgotten at time t: f t =σ(W f ·[h t-1 ,c t ]+b f ); Calculate memory gate i t , select the information i to be remembered at time t t =σ(W i ·[h t-1 ,c t ]+b i ) and candidate memory information at time t: Then, the cell state c at the previous moment (t-1 moment) t-1 , information to be forgotten at any time t , the information to be remembered at time t t and candidate memory information at time t Update the cell state at time t Calculate the output gate o at time t t =σ(W o ·[h t-1 ,c t ]+b o ) and the hidden layer state h at time t t =o t *tanh(C t ). i t 、f t and are all the same as the hidden layer h at the previous moment t-1 And the training word vector matrix x at time t t Related functions. And i t 、f t It is the sigmoid function, with a value range of [0, 1]; is the tanh function, with a value range of [-1, 1], W i is the input gate weight, b i is the input gate bias, W f is the forget gate weight, b f is the forget gate bias, W o is the output gate weight, b o is the output gate bias, W C is the memory weight, b C is the memory bias. Through a forward LSTM network, an output y is obtained at each moment, y = h t The core structure of the bidirectional long short-term memory neural network is the same as that of the LSTM, but it considers the text context information from both positive and negative directions at the same time.

[0096] S603: Input the multi-channel text feature training data into the multi-channel convolutional network model to obtain a prediction type.

[0097] S604: Determine a loss function according to the predicted type and the corresponding actual type.

[0098] In specific implementation, the dropout mechanism can be used to prevent the algorithm from overfitting. The experiment uses cross entropy as the objective function, minimizes the cross entropy loss function to optimize the network model, uses the gradient descent algorithm for parameter optimization, and introduces L2 regularization in the objective function to achieve better algorithm fitting results. The cross entropy loss function is as follows:

[0099]

[0100] Among them, loss is the loss function, D is the size of the training text data, C is the number of sample categories, y is the probability that the p-th training text data predicts category q, is the probability that the actual category of the p-th training text data is q, λ||θ|| 2 is the regularization term, λ is the penalty term, and θ is the parameter.

[0101] S605: Determine whether the loss function converges.

[0102] S606: When the loss function converges, the bidirectional long short-term memory artificial neural network model is determined as the text information extraction model, and the multi-channel convolutional network model is determined as the text classification model.

[0103] Among them, the convolutional neural network text classification model is the MC-CNN model, which consists of a text information extraction model and a text classification model.

[0104] S607: When the loss function has not converged, updating the bidirectional long short-term memory artificial neural network model and the multi-channel convolutional network model according to the loss function.

[0105] Figure 4 This is a flowchart of S102 in an embodiment of the present invention. Figure 10 Schematic diagram of multi-channel text feature representation in an embodiment of the present invention. Figure 4 and Figure 10 As shown, after the bidirectional long short-term memory network, the output results of l moments are obtained in positive and reverse order respectively, and the feature vectors of these l moments are stacked vertically in order to form a multi-channel feature representation. S102 includes:

[0106] S201: Inputting the first word vector matrix into the first text information extraction model to obtain first positive order text feature data and first reverse order text feature data.

[0107] Figure 5 This is a flow chart of S201 in an embodiment of the present invention. Figure 5 As shown, S201 includes:

[0108] S301: Input the first word vector matrix into the first text information extraction model to obtain a first positive order text feature vector and a first reverse order text feature vector.

[0109] S302: vertically stacking the first positive-order text feature vectors in order to obtain first positive-order text feature data.

[0110] In specific implementation, the first positive sequence text feature data can be obtained by the following formula:

[0111]

[0112] in, is the first positive sequence text feature data, is the first positive sequence text feature vector at the first moment, is the first positive sequence text feature vector at the second moment, is the first positive sequence text feature vector at the lth moment.

[0113] S303: vertically stacking the first reverse-order text feature vectors in order to obtain first reverse-order text feature data.

[0114] In specific implementation, the first reverse text feature data can be obtained by the following formula:

[0115]

[0116] in, is the first reverse text feature data, is the first reverse text feature vector at the first moment, is the first reverse text feature vector at the second moment, is the first reverse text feature vector at the lth moment.

[0117] S202: Input the second word vector matrix into the second text information extraction model to obtain second forward text feature data and second reverse text feature data.

[0118] Figure 6 This is a flow chart of S202 in an embodiment of the present invention. Figure 6 As shown, S202 includes:

[0119] S401: Input the second word vector matrix into the second text information extraction model to obtain a second forward text feature vector and a second reverse text feature vector.

[0120] S402: vertically stacking the second positive-order text feature vectors in order to obtain second positive-order text feature data.

[0121] In specific implementation, the second positive sequence text feature data can be obtained by the following formula:

[0122]

[0123] in, is the second positive sequence text feature data, is the second positive sequence text feature vector at the first moment, is the second positive sequence text feature vector at the second moment, is the second positive sequence text feature vector at the lth moment.

[0124] S403: vertically stacking the second reverse-order text feature vectors in order to obtain second reverse-order text feature data.

[0125] In specific implementation, the second reverse text feature data can be obtained by the following formula:

[0126]

[0127] in, is the second reverse text feature data, is the second reverse text feature vector at the first moment, is the second reverse text feature vector at the second moment, is the second reverse text feature vector at the lth moment.

[0128] S203: Obtain multi-channel feature representation data according to the first normal order text feature data, the first reverse order text feature data, the second normal order text feature data, and the second reverse order text feature data.

[0129] The first forward-order text feature data, the first reverse-order text feature data, the second forward-order text feature data, and the second reverse-order text feature data are all stacked vertically in sequence to form a feature matrix. Each matrix represents a feature channel, thereby forming a multi-channel feature representation of the text. This achieves the purpose of retaining the latent semantic information of the temporal features while obtaining the text context dependency.

[0130] S103: Input the multi-channel text feature data into a text classification model to obtain a text type.

[0131] In specific implementation, S103 is based on the multi-channel feature representation of the text, enhanced by multi-scale convolution operations, while considering the ability of the current contextual information of the text to further explore the potential semantic information of the text's temporal features.

[0132] Figure 7 This is a flow chart of S103 in an embodiment of the present invention. Figure 11 Schematic diagram of a multi-scale convolutional network layer in an embodiment of the present invention. Figure 7 and Figure 11 As shown, S103 includes:

[0133] S501: Input the multi-channel text feature data into the multi-scale convolutional network layer to obtain multi-dimensional text feature data.

[0134] Under the action of convolution kernels of different window sizes, the dimensions of the feature vector output after the convolution operation are different. In specific implementation, the multi-dimensional text feature data is obtained through the following formula:

[0135]

[0136] Among them, a i is the i-th word feature, W is the weight of the multi-scale convolutional network layer, b is the bias of the multi-scale convolutional network layer, f represents the nonlinear activation function, H i:l is the matrix consisting of the feature vectors of the i-th word to the l-th word in the multi-channel text feature data. Therefore, after convolution processing on a sequence of length l, the following word feature matrix is ​​obtained:

[0137] A=[a1,a2,…,a l ];

[0138] Therefore, after processing with K convolution kernels of different window sizes, K word feature vectors with different dimensions [A1, A2, ..., A K ], the most important features corresponding to each convolution kernel of different sizes are extracted through the maximum pooling operation, namely:

[0139]

[0140] Finally, after processing by the multi-scale convolutional network layer, K-dimensional text feature data is obtained:

[0141]

[0142] Among them, Y is K-dimensional text feature data, is a one-dimensional word feature vector processed by maximum pooling, is a two-dimensional word feature vector processed by maximum pooling, is the K-dimensional word feature vector after maximum pooling.

[0143] S502: Input the multi-dimensional text feature data into the fully connected output network to obtain the text type.

[0144] In specific implementation, the multi-dimensional text feature data is passed through a three-layer fully connected network and then softmax classification is performed to obtain the text type.

[0145] Figure 2 The execution subject of the text classification method based on multi-channel feature representation of convolutional neural network shown in the figure can be a computer. Figure 2 As can be seen from the process shown, the text classification method based on multi-channel feature representation of convolutional neural network in an embodiment of the present invention inputs the text data to be classified into two word vector models respectively to obtain two word vector matrices, which are then input into corresponding text information extraction models to obtain multi-channel text feature data, and then the multi-channel text feature data is input into the text classification model to obtain the text type, which can enrich the text representation and improve the accuracy of text classification.

[0146] The specific process of the embodiment of the present invention is as follows:

[0147] 1. Input the training text data into two word vector models respectively to obtain two training word vector matrices.

[0148] 2. Input the two training word vector matrices into the corresponding bidirectional long short-term memory artificial neural network model to obtain multi-channel text feature training data.

[0149] 3. Input the multi-channel text feature training data into the multi-channel convolutional network model to obtain the prediction type.

[0150] 4. Determine the loss function based on the predicted type and the corresponding actual type.

[0151] 5. When the loss function converges, the bidirectional long short-term memory artificial neural network model is determined as the text information extraction model, and the multi-channel convolutional network model is determined as the text classification model. Otherwise, the bidirectional long short-term memory artificial neural network model and the multi-channel convolutional network model are updated according to the loss function, and the process returns to step 1.

[0152] 6. Input the text data to be classified into two word vector models respectively to obtain two word vector matrices.

[0153] 7. Input the first word vector matrix into the first text information extraction model to obtain the first positive order text feature vector and the first reverse order text feature vector respectively.

[0154] 8. Stack the first positive order text feature vectors vertically in order to obtain first positive order text feature data, and stack the first reverse order text feature vectors vertically in order to obtain first reverse order text feature data.

[0155] 9. Input the second word vector matrix into the second text information extraction model to obtain a second positive order text feature vector and a second reverse order text feature vector respectively.

[0156] 10. Vertically stack the second positive order text feature vectors in sequence to obtain second positive order text feature data, and vertically stack the second reverse order text feature vectors in sequence to obtain second reverse order text feature data.

[0157] 11. Obtain multi-channel feature representation data based on the first normal order text feature data, the first reverse order text feature data, the second normal order text feature data, and the second reverse order text feature data.

[0158] 12. Input the multi-channel text feature data into the multi-scale convolutional network layer to obtain multi-dimensional text feature data.

[0159] 13. Input the multidimensional text feature data into the fully connected output network to obtain the text type.

[0160] Table 1

[0161]

[0162] Table 1 is a comparison table of the classification results of the MC-CNN model used in the present invention and other models. As shown in Table 1, the DSCNN model first extracts features from temporal information through LSTM, and then uses CNN to extract spatial features, the purpose of which is to solve the shortcoming of LSTM that cannot consider future time information; the VeryDeep-CNN model uses a 29-layer CNN network to obtain long-distance text dependencies; the textCNN model is the first to use convolutional neural networks to solve natural language processing tasks, using only convolutional layers to extract text information features to complete text classification tasks; the fastText model is a simple and efficient linear classification model that can achieve massive text classification in a very short time, supporting billions of data volumes, and performs average pooling on the word vectors of all words to vectorize the text and thus complete the classification task; the WE model is similar to the fastText model, except that average pooling is replaced by maximum pooling; the core idea of ​​the MCTC-dense model is to use convolution kernels of different sizes in different time dimensions to extract text information of different scales. At the same time, the features extracted by convolution kernels of different sizes can also be used to resolve dependencies between long sequences.

[0163] In Table 1, MR is a dataset for classifying movie reviews, which is divided into two categories: positive and negative. SST-2 is a dataset for classifying movie review sentiment, which is divided into two categories: negative and positive. TREC is a dataset for classifying question types, which specifically includes six categories: abbreviations, descriptions, entities, people, places, and values. AG is a dataset for classifying news article types. Yelp_F is a dataset for user review sentiment classification, which is a more fine-grained sentiment classification dataset with five categories. Yelp_P is a dataset for classifying user review sentiment, which only has two categories: positive and negative. By comparing with the results of the above models, especially on long text datasets, the present invention has achieved the best effect. Therefore, it can be shown that the present invention can not only obtain long-term text dependencies but also mine the potential semantic information of text sequences, providing a good solution for text classification tasks.

[0164] Table 2

[0165]

[0166] Table 2 compares the classification results of the MC-CNN model used in this paper and its own variant models. As shown in Table 2, MC-CNN-w2v is a model that uses only Word2vec pre-trained word vectors as input for classification; MC-CNN-glo is a model that uses only Glove pre-trained word vectors for classification; and SC-CNN is a model that assigns different weights to four channels and then fuses them into a single channel.

[0167] Judging from the final results, the MC-CNN model adopted by the present invention performs better than its own variant model on these six data sets. Although the improvement effect is not very obvious, it can still reflect the advantages of the multi-channel feature representation proposed by the present invention as a whole. When the convolution processing is performed after the algorithm, the algorithm can simultaneously consider the information before and after the current moment, and can also mine the potential semantic information of the text sequence. Therefore, the present invention still achieves the best performance.

[0168] In summary, the present invention uses third-party pre-trained word vectors from different sources to enrich text representation, thereby enriching text semantic information, and at the same time uses two different bidirectional long short-term memory networks to extract temporal features in forward and reverse order on the text sequences represented by the above two word vectors to obtain the current moment text context information. Unlike the prior art that splices the forward and reverse outputs as the final result, the present invention stacks the outputs of each moment vertically in order to form a multi-channel method, and uses multi-scale convolution operations after the multi-channel feature representation to further explore the potential semantic relationship between temporal features while fully considering the information before and after the current moment, thereby solving the problem of insufficient semantic information in text representation and the inability to fully consider the information before and after the current moment and the potential semantic information of temporal features, providing a good solution for text classification tasks.

[0169] Based on the same inventive concept, an embodiment of the present invention also provides a text classification device based on multi-channel feature representation of convolutional neural networks. Since the principle of solving the problem of this device is similar to that of the text classification method based on multi-channel feature representation of convolutional neural networks, the implementation of this device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0170] Figure 12 This is a structural block diagram of a text classification device based on multi-channel feature representation of a convolutional neural network in an embodiment of the present invention. Figure 12 As shown, the text classification device based on convolutional neural network multi-channel feature representation includes:

[0171] The word vector matrix module is used to input the text data to be classified into two word vector models respectively to obtain two word vector matrices;

[0172] The multi-channel text feature data module is used to input the two word vector matrices into the corresponding text information extraction model to obtain multi-channel text feature data;

[0173] The text type module is used to input multi-channel text feature data into the text classification model to obtain the text type.

[0174] In one embodiment, the multi-channel text feature data module includes:

[0175] A first text feature data unit is used to input the first word vector matrix into the first text information extraction model to obtain first positive order text feature data and first reverse order text feature data;

[0176] A second text feature data unit is used to input the second word vector matrix into the second text information extraction model to obtain second positive order text feature data and second reverse order text feature data;

[0177] The multi-channel feature representation data unit is used to obtain multi-channel feature representation data according to the first normal order text feature data, the first reverse order text feature data, the second normal order text feature data and the second reverse order text feature data.

[0178] In one embodiment, the first text feature data unit includes:

[0179] A first text feature vector subunit is used to input the first word vector matrix into the first text information extraction model to obtain a first positive order text feature vector and a first reverse order text feature vector;

[0180] A first positive-sequence text feature data subunit, configured to vertically stack first positive-sequence text feature vectors in order to obtain first positive-sequence text feature data;

[0181] The first reverse order text feature data subunit is used to vertically stack the first reverse order text feature vectors in sequence to obtain first reverse order text feature data.

[0182] In one embodiment, the second text feature data unit includes:

[0183] A second text feature vector subunit is used to input the second word vector matrix into the second text information extraction model to obtain a second positive order text feature vector and a second reverse order text feature vector respectively;

[0184] A second positive-sequence text feature data subunit is used to vertically stack the second positive-sequence text feature vectors in order to obtain second positive-sequence text feature data;

[0185] The second reverse order text feature data subunit is used to vertically stack the second reverse order text feature vectors in sequence to obtain second reverse order text feature data.

[0186] In one embodiment, the text type module includes:

[0187] A multi-dimensional text feature data unit, used to input multi-channel text feature data into a multi-scale convolutional network layer to obtain multi-dimensional text feature data;

[0188] The text type unit is used to input multi-dimensional text feature data into the fully connected output network to obtain the text type.

[0189] In one embodiment, it further includes:

[0190] The training word vector matrix module is used to input the training text data into two word vector models respectively to obtain two training word vector matrices;

[0191] The text feature training data module is used to input the two training word vector matrices into the corresponding bidirectional long short-term memory artificial neural network model to obtain multi-channel text feature training data;

[0192] The prediction type module is used to input the multi-channel text feature training data into the multi-channel convolutional network model to obtain the prediction type;

[0193] The loss function module is used to determine the loss function based on the predicted type and the corresponding actual type;

[0194] The iterative module is used to determine the bidirectional long short-term memory artificial neural network model as the text information extraction model and the multi-channel convolutional network model as the text classification model when the loss function converges; otherwise, the bidirectional long short-term memory artificial neural network model and the multi-channel convolutional network model are updated according to the loss function.

[0195] In summary, the text classification device based on multi-channel feature representation of convolutional neural network in an embodiment of the present invention inputs the text data to be classified into two word vector models respectively to obtain two word vector matrices, which are then input into corresponding text information extraction models to obtain multi-channel text feature data, and then the multi-channel text feature data is input into the text classification model to obtain the text type, which can enrich the text representation and improve the accuracy of text classification.

[0196] An embodiment of the present invention also provides a specific implementation of a computer device that can implement all steps in the text classification method based on multi-channel feature representation of a convolutional neural network in the above embodiment. Figure 13 This is a block diagram of the computer device in the embodiment of the present invention, see Figure 13 , the computer equipment specifically includes the following:

[0197] Processor 1301 and memory 1302 .

[0198] The processor 1301 is configured to call the computer program in the memory 1302. When the processor executes the computer program, all steps of the text classification method based on multi-channel feature representation of a convolutional neural network in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0199] Input the text data to be classified into two word vector models respectively to obtain two word vector matrices;

[0200] Input the two word vector matrices into the corresponding text information extraction model to obtain multi-channel text feature data;

[0201] Input multi-channel text feature data into the text classification model to obtain the text type.

[0202] In summary, the computer device of an embodiment of the present invention inputs the text data to be classified into two word vector models respectively to obtain two word vector matrices, which are then input into corresponding text information extraction models to obtain multi-channel text feature data, and then inputs the multi-channel text feature data into a text classification model to obtain text types, which can enrich text representation and improve text classification accuracy.

[0203] An embodiment of the present invention also provides a computer-readable storage medium capable of implementing all steps of the text classification method based on multi-channel feature representation of a convolutional neural network in the above embodiment. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all steps of the text classification method based on multi-channel feature representation of a convolutional neural network in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0204] Input the text data to be classified into two word vector models respectively to obtain two word vector matrices;

[0205] Input the two word vector matrices into the corresponding text information extraction model to obtain multi-channel text feature data;

[0206] Input multi-channel text feature data into the text classification model to obtain the text type.

[0207] In summary, the computer-readable storage medium of an embodiment of the present invention inputs the text data to be classified into two word vector models respectively to obtain two word vector matrices, which are then input into corresponding text information extraction models to obtain multi-channel text feature data, and then the multi-channel text feature data is input into a text classification model to obtain a text type, which can enrich the text representation and improve the accuracy of text classification.

[0208] An embodiment of the present invention also provides a computer program product capable of implementing all steps of the text classification method based on multi-channel feature representation of a convolutional neural network in the above embodiment. The computer program product includes a computer program / instructions. When the computer program / instructions are executed by a processor, all steps of the text classification method based on multi-channel feature representation of a convolutional neural network in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0209] Input the text data to be classified into two word vector models respectively to obtain two word vector matrices;

[0210] Input the two word vector matrices into the corresponding text information extraction model to obtain multi-channel text feature data;

[0211] Input multi-channel text feature data into the text classification model to obtain the text type.

[0212] In summary, the computer program product of an embodiment of the present invention inputs the text data to be classified into two word vector models respectively to obtain two word vector matrices, which are then input into corresponding text information extraction models to obtain multi-channel text feature data, and then inputs the multi-channel text feature data into a text classification model to obtain text types, which can enrich text representation and improve text classification accuracy.

[0213] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0214] Those skilled in the art will also appreciate that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of the two. To clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.

[0215] The various illustrative logic blocks, units, or devices described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0216] The steps of the methods or algorithms described in the embodiments of the present invention may be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module may be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. For example, the storage medium may be connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium may also be integrated into the processor. The processor and storage medium may be provided in an ASIC, which may be provided in a user terminal. Alternatively, the processor and storage medium may also be provided in different components in the user terminal.

[0217] In one or more exemplary designs, the above-mentioned functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of the three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted in the form of one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one location to another. Storage media can be any available medium that can be accessed by a general or special computer. For example, such computer-readable media can include but are not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other forms that can be read by a general or special computer, or a general or special processor. In addition, any connection can be appropriately defined as a computer-readable medium. For example, if the software is transmitted from a website, server or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless methods such as infrared, wireless, and microwave, it is also included in the definition of computer-readable media. The disks and discs mentioned above include compact disks, laser disks, optical disks, DVDs, floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs typically reproduce data optically with lasers. Combinations of the above may also be included in computer-readable media.< / pda>

Claims

1. A text classification method based on multi-channel feature representation of convolutional neural network, characterized in that: include: Input the text data to be classified into two word vector models respectively to obtain two word vector matrices; Input the two word vector matrices into corresponding text information extraction models respectively to obtain multi-channel text feature data; Inputting the multi-channel text feature data into a text classification model to obtain a text type; The method of inputting the two word vector matrices into corresponding text information extraction models to obtain multi-channel text feature data includes: inputting the first word vector matrix into the first text information extraction model to obtain first normal order text feature data and first reverse order text feature data; inputting the second word vector matrix into the second text information extraction model to obtain second normal order text feature data and second reverse order text feature data; and obtaining multi-channel feature representation data based on the first normal order text feature data, the first reverse order text feature data, the second normal order text feature data, and the second reverse order text feature data. Among them, the first normal order text feature data, the first reverse order text feature data, the second normal order text feature data and the second reverse order text feature data are all feature matrices formed by vertically stacking in sequence, and each matrix represents a feature channel; based on the multi-channel feature representation data, text context dependency and temporal feature potential semantic information are obtained.

2. The text classification method based on convolutional neural network multi-channel feature representation according to claim 1 is characterized in that Inputting the first word vector matrix into the first text information extraction model to obtain first positive order text feature data and first reverse order text feature data includes: Inputting the first word vector matrix into the first text information extraction model to obtain a first forward-order text feature vector and a first reverse-order text feature vector; vertically stacking the first positive-order text feature vectors in sequence to obtain the first positive-order text feature data; The first reverse-order text feature vectors are stacked vertically in sequence to obtain the first reverse-order text feature data.

3. The text classification method based on convolutional neural network multi-channel feature representation according to claim 1 is characterized in that Inputting the second word vector matrix into the second text information extraction model to obtain second positive order text feature data and second reverse order text feature data includes: Inputting the second word vector matrix into the second text information extraction model to obtain a second forward-order text feature vector and a second reverse-order text feature vector respectively; vertically stacking the second normal-order text feature vectors in sequence to obtain the second normal-order text feature data; The second reverse-order text feature vectors are stacked vertically in sequence to obtain the second reverse-order text feature data.

4. The text classification method based on convolutional neural network multi-channel feature representation according to claim 1 is characterized in that Inputting the multi-channel text feature data into a text classification model to obtain text types includes: Inputting the multi-channel text feature data into a multi-scale convolutional network layer to obtain multi-dimensional text feature data; The multidimensional text feature data is input into a fully connected output network to obtain the text type.

5. The text classification method based on convolutional neural network multi-channel feature representation according to claim 1 is characterized in that: Also includes: Perform the following iterative process: Input the training text data into the two word vector models respectively to obtain two training word vector matrices; Inputting the two training word vector matrices into corresponding bidirectional long short-term memory artificial neural network models respectively to obtain multi-channel text feature training data; Inputting the multi-channel text feature training data into a multi-channel convolutional network model to obtain a prediction type; Determine a loss function based on the predicted type and the corresponding actual type; When the loss function converges, the bidirectional long short-term memory artificial neural network model is determined as the text information extraction model, and the multi-channel convolutional network model is determined as the text classification model; otherwise, the bidirectional long short-term memory artificial neural network model and the multi-channel convolutional network model are updated according to the loss function.

6. A text classification device based on multi-channel feature representation of convolutional neural network, characterized in that: include: The word vector matrix module is used to input the text data to be classified into two word vector models respectively to obtain two word vector matrices; The multi-channel text feature data module is used to input the two word vector matrices into the corresponding text information extraction model to obtain multi-channel text feature data; A text type module, configured to input the multi-channel text feature data into a text classification model to obtain a text type; The multi-channel text feature data module includes: a first text feature data unit, which is used to input the first word vector matrix into the first text information extraction model to obtain first normal order text feature data and first reverse order text feature data; a second text feature data unit, which is used to input the second word vector matrix into the second text information extraction model to obtain second normal order text feature data and second reverse order text feature data; a multi-channel feature representation data unit, which is used to obtain multi-channel feature representation data according to the first normal order text feature data, the first reverse order text feature data, the second normal order text feature data and the second reverse order text feature data; Among them, the first normal order text feature data, the first reverse order text feature data, the second normal order text feature data and the second reverse order text feature data are all feature matrices formed by vertically stacking in sequence, and each matrix represents a feature channel; based on the multi-channel feature representation data, text context dependency and temporal feature potential semantic information are obtained.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the text classification method based on convolutional neural network multi-channel feature representation are implemented as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the text classification method based on convolutional neural network multi-channel feature representation according to any one of claims 1 to 5 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the text classification method based on convolutional neural network multi-channel feature representation are implemented.

Images