Short text classification method and apparatus

Abstract

The application provides a short text classification method and device, and belongs to the field of artificial intelligence. The short text classification method comprises the following steps: determining a model combination according to a business stage and a sample number of a training data set, wherein the model combination comprises at least one neural network model; training the models in the model combination; inputting a text to be processed into the trained model combination; and outputting a classification result. The technical scheme of the application can accurately classify short texts.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of short text classification, and particularly to a short text classification method and device. BACKGROUND

[0002] With the rapid development of information technology, mobile, communication and other telecommunication service users and microblog, WeChat, headlines, Zhihu and other network users will generate a large amount of short text content every day, and it is extremely important to use classification technology to analyze public opinion, sentiment and evaluation feedback of these short text content.

[0003] The current short text classification algorithm includes traditional machine learning (Nave Bayes, Decision Tree, SVM, etc.), deep learning (CNN, RNN, Bert, etc.), each of which has advantages and disadvantages, and a single algorithm cannot simultaneously consider high precision and recall rate, low time consumption, low variance and other indicators. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a short text classification method and device capable of accurately classifying short text.

[0005] To solve the above technical problems, the embodiments of the present application provide the technical solutions as follows:

[0006] In one aspect, a short text classification method is provided, comprising:

[0007] determining a model combination according to a business stage and a sample number of a training data set, the model combination comprising at least one neural network model;

[0008] training the models in the model combination;

[0009] inputting a text to be processed into the trained model combination, and outputting a classification result.

[0010] In some embodiments, determining the model combination according to the business stage and the sample number of the training data set and training the models in the model combination comprises:

[0011] training only a Bert matching model when the sample number of the training data set is less than a first threshold value in a cold start stage;

[0012] training the Bert matching model, a convolutional neural network (CNN) model and a recurrent neural network (RNN) model when the sample number of the training data set is greater than or equal to the first threshold value and less than a second threshold value in a middle-late stage of the business, and training a Bert classification model, a CNN model and an RNN model when the sample number of the training data set is greater than or equal to the second threshold value.

[0013] In some embodiments, the first threshold value is 3000-7000, and the second threshold value is 30000-70000.

[0014] In some embodiments, the inputting the to-be-processed text into the trained model combination and outputting a classification result comprises:

[0015] If the number of samples of the currently used training data set is less than the first threshold value, a Bert matching model is used to find a text most similar to the to-be-processed text in the training corpus, and the category corresponding to the text is taken as the classification result.

[0016] If the number of samples of the currently used training data set is greater than or equal to the first threshold value and less than the second threshold value, the to-be-processed text is input into a CNN model to obtain a first prediction result, and the to-be-processed text is input into an RNN model to obtain a second prediction result, if the first prediction result is the same as the second prediction result, the first prediction result or the second prediction result is taken as the classification result, if the first prediction result is not the same as the second prediction result, a Bert matching model is used to find a first text and a second text most similar to the to-be-processed text in the training corpus of the categories represented by the first prediction result and the second prediction result respectively, and a first score corresponding to the first text and a second score corresponding to the second text, if the first score is greater than or equal to the second score, the category to which the first text belongs is taken as the classification result, if the first score is less than the second score, the category to which the second text belongs is taken as the classification result.

[0017] If the number of samples of the currently used training data set is greater than or equal to the second threshold value, the to-be-processed text is input into a CNN model to obtain a first prediction result, and the to-be-processed text is input into an RNN model to obtain a second prediction result, if the first prediction result is the same as the second prediction result, the first prediction result or the second prediction result is taken as the classification result, if the first prediction result is not the same as the second prediction result, a third prediction result is obtained through a Bert classification model, and the third prediction result is taken as the classification result.

[0018] In some embodiments,

[0019] The Bert matching model is a Sentence-Bert model, a first corpus sample is randomly selected from a training data set, a second corpus sample is obtained by replacing the stop words of the first corpus sample, the first corpus sample and the second corpus sample are taken as positive samples, corpus unrelated to the first corpus sample is selected from the training data set as negative samples, the ratio of positive samples to negative samples is 1: N, where N is an integer not less than 5, and the training is performed in a fine-tune manner, and the formula of the loss function is:

[0020] max(||sa-sp||-||sa-sn||+∈,0)

[0021] Wherein, sa represents the first corpus sample, sn represents the negative sample, sp represents the second corpus sample, "||·||" represents the cosine distance, and represents the distance boundary.

[0022] In some embodiments,

[0023] The sample quantity ratio of the training set, the validation set and the test set of the CNN model is 8:1:1, the embedding layer of the CNN model uses word vectors as input, the embedding layer adopts two channels of static and dynamic word vectors connected with the convolution layer, and the training result of Chinese Wikipedia of word2vector or fast-text is used as the static channel, the convolution layer adopts three convolution kernels of 2, 3 and 4, and the pooling layer adopts the maximum pooling method.

[0024] In some embodiments,

[0025] The RNN model is a bidirectional long short-term memory Bi-LSTM model based on an attention mechanism, the sample quantity ratio of the training set, the validation set and the test set of the Bi-LSTM model based on the attention mechanism is 8:1:1, the embedding layer of the RNN model uses word vectors as input, and there are 3 hidden layers, each hidden layer contains 256 lstm units.

[0026] In some embodiments, the sample quantity ratio of the training set, the validation set and the test set of the Bert classification model is 8:1:1, and fine-tune training is performed on the basis of the Chinese-Bert-Wwm model.

[0027] The embodiment of the present application also provides a short text classification device, which comprises:

[0028] A processing module is configured to determine a model combination according to a business phase and a sample quantity of a training data set, wherein the model combination comprises at least one neural network model.

[0029] A training module is configured to train the models in the model combination.

[0030] A prediction module is configured to input a to-be-processed text into the trained model combination and output a classification result.

[0031] In some embodiments, the training module is specifically configured to:

[0032] In the cold start phase, only train the Bert matching model when the sample quantity of the training data set is less than a first threshold value.

[0033] In the later stage of the business, when the sample number of the training data set is greater than or equal to the first threshold value and less than the second threshold value, the Bert matching model, the CNN model and the RNN model are trained; when the sample number of the training data set is greater than or equal to the second threshold value, the Bert classification model, the CNN model and the RNN model are trained.

[0034] In some embodiments,

[0035] The first threshold value is 3000-7000, and the second threshold value is 30000-70000.

[0036] In some embodiments, the prediction module is specifically configured to:

[0037] If the sample number of the currently used training data set is less than the first threshold value, the Bert matching model is used to find a text most similar to the to-be-processed text in the training corpus, and the category corresponding to the text is taken as the classification result;

[0038] If the sample number of the currently used training data set is greater than or equal to the first threshold value and less than the second threshold value, the to-be-processed text is input into the CNN model to obtain a first prediction result, and the to-be-processed text is input into the RNN model to obtain a second prediction result, if the first prediction result is the same as the second prediction result, the first prediction result or the second prediction result is taken as the classification result; if the first prediction result is not the same as the second prediction result, the Bert matching model is used to find a first text and a second text most similar to the to-be-processed text in the training corpus of the categories represented by the first prediction result and the second prediction result respectively, and a first score corresponding to the first text and a second score corresponding to the second text, if the first score is greater than or equal to the second score, the category to which the first text belongs is taken as the classification result, if the first score is less than the second score, the category to which the second text belongs is taken as the classification result;

[0039] If the sample number of the currently used training data set is greater than or equal to the second threshold value, the to-be-processed text is input into the CNN model to obtain a first prediction result, and the to-be-processed text is input into the RNN model to obtain a second prediction result, if the first prediction result is the same as the second prediction result, the first prediction result or the second prediction result is taken as the classification result; if the first prediction result is not the same as the second prediction result, a third prediction result is obtained through the Bert classification model, and the third prediction result is taken as the classification result.

[0040] In some embodiments,

[0041] The Bert matching model is a Sentence-Bert model, a first corpus sample is randomly selected from a training data set, a second corpus sample is obtained after stop words of the first corpus sample are replaced, the first corpus sample and the second corpus sample are used as positive samples, corpus unrelated to the first corpus sample is selected from the training data set as negative samples, the ratio of positive samples to negative samples is 1:N, wherein N is an integer not less than 5, training is performed in a fine-tune manner, and the formula of the loss function is:

[0042] max(||sa-sp||-||sa-sn||+∈,0)

[0043] Wherein, sa represents the first corpus sample, sn represents the negative sample, sp represents the second corpus sample, "||·||" represents the cosine distance, and ∈ represents the distance boundary.

[0044] In some embodiments,

[0045] The sample quantity ratio of the training set, the validation set and the test set of the CNN model is 8:1:1, the embedding layer of the CNN model uses word vectors as input, the embedding layer adopts two channels of static and dynamic word vectors connected with the convolution layer, and the training result of Chinese Wikipedia of word2vector or fast-text is used as the static channel, the convolution layer adopts three convolution kernels of 2, 3 and 4, and the pooling layer adopts the maximum pooling method.

[0046] In some embodiments,

[0047] The RNN model is a bidirectional long short-term memory Bi-LSTM model based on an attention mechanism, the sample quantity ratio of the training set, the validation set and the test set of the Bi-LSTM model based on the attention mechanism is 8:1:1, the embedding layer of the RNN model uses word vectors as input, and 3 hidden layers each containing 256 lstm units.

[0048] In some embodiments, the sample quantity ratio of the training set, the validation set and the test set of the Bert classification model is 8:1:1, and fine-tune training is performed on the basis of the Chinese-Bert-Wwm model.

[0049] The embodiment of the application also provides a short text classification device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor; when the processor executes the program, the short text classification method described above is realized.

[0050] In some embodiments, the processor is configured to determine a model combination according to a business stage and a sample number of a training data set, the model combination comprising at least one neural network model; train the models in the model combination; and input the text to be processed into the trained model combination to output a classification result.

[0051] In some embodiments, the processor is specifically configured to train only the Bert matching model when the sample number of the training data set is less than a first threshold value in a cold start stage; train the Bert matching model, the CNN model and the RNN model when the sample number of the training data set is greater than or equal to the first threshold value and less than a second threshold value in a later stage of the business; and train the Bert classification model, the CNN model and the RNN model when the sample number of the training data set is greater than or equal to the second threshold value.

[0052] In some embodiments, the first threshold value is 3000-7000, and the second threshold value is 30000-70000.

[0053] In some embodiments, the processor is specifically configured to use the Bert matching model to find a text most similar to the text to be processed in the training corpus if the sample number of the currently used training data set is less than the first threshold value, and use the category corresponding to the text as the classification result.

[0054] If the sample number of the currently used training data set is greater than or equal to the first threshold value and less than the second threshold value, input the text to be processed into the CNN model to obtain a first prediction result, input the text to be processed into the RNN model to obtain a second prediction result, if the first prediction result is the same as the second prediction result, use the first prediction result or the second prediction result as the classification result, if the first prediction result is not the same as the second prediction result, use the Bert matching model to find a first text and a second text most similar to the text to be processed in the training corpus of the categories represented by the first prediction result and the second prediction result respectively, and a first score corresponding to the first text and a second score corresponding to the second text, if the first score is greater than or equal to the second score, use the category to which the first text belongs as the classification result, if the first score is less than the second score, use the category to which the second text belongs as the classification result; if the sample number of the currently used training data set is greater than or equal to the second threshold value, input the text to be processed into the CNN model to obtain a first prediction result, input the text to be processed into the RNN model to obtain a second prediction result, if the first prediction result is the same as the second prediction result, use the first prediction result or the second prediction result as the classification result, if the first prediction result is not the same as the second prediction result, obtain a third prediction result through the Bert classification model, and use the third prediction result as the classification result.

[0055] In some embodiments,

[0056] The Bert matching model is a Sentence-Bert model, a first corpus sample is randomly selected from a training data set, a second corpus sample is obtained after stop words of the first corpus sample are replaced, the first corpus sample and the second corpus sample are used as positive samples, corpus unrelated to the first corpus sample is selected from the training data set as negative samples, the ratio of positive samples to negative samples is 1:N, wherein N is an integer not less than 5, training is performed in a fine-tune manner, and the formula of a loss function is:

[0057] max(||sa-sp||-||sa-sn||+∈,0)

[0058] Wherein, sa represents the first corpus sample, sn represents the negative sample, sp represents the second corpus sample, "||·||" represents a cosine distance, and ∈ represents a distance boundary.

[0059] In some embodiments,

[0060] The sample quantity ratio of the training set, the validation set and the test set of the CNN model is 8:1:1, the embedding layer of the CNN model uses word vectors as input, the embedding layer adopts two channels of static and dynamic word vectors connected with a convolution layer, and the training result of Chinese Wikipedia of word2vector or fast-text is used as a static channel, the convolution layer adopts three convolution kernels of 2, 3 and 4, and the pooling layer adopts a maximum pooling method.

[0061] In some embodiments,

[0062] The RNN model is a bidirectional long short-term memory Bi-LSTM model based on an attention mechanism, the sample quantity ratio of the training set, the validation set and the test set of the Bi-LSTM model based on the attention mechanism is 8:1:1, the embedding layer of the RNN model uses word vectors as input, and there are 3 hidden layers, each hidden layer contains 256 lstm units.

[0063] In some embodiments, the sample quantity ratio of the training set, the validation set and the test set of the Bert classification model is 8:1:1, and fine-tune training is performed on the basis of a Chinese-Bert-Wwm model.

[0064] The embodiment of the application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps in the short text classification method.

[0065] The embodiment of the application has the following beneficial effects:

[0066] In the above scheme, the model combination is determined according to the business stage and the sample number of the training data set, for example, when the sample number of the training data set is extremely small, a text matching model is used to find the category of the most similar text in the training corpus as the classification result; when the sample number of the training data set is generally abundant, two more complex classification models are used to vote, and if the results are consistent, the result is output and the process is ended, and if the results are inconsistent, the final result is determined by the text matching model; when the sample number of the training data set is sufficient, two more complex classification models are used to vote, and if the results are consistent, the result is output and the process is ended, and if the results are inconsistent, the final result is determined by the stronger classification model. The technical scheme of the embodiment can pay attention to the problems in different stages of the text classification application scene; the use of the text matching model instead of the classification model in the cold start stage can effectively avoid the problem of model overfitting when the data set is scarce; the use of the voting method instead of the bagging idea of weighted average in the multi-model fusion can improve the precision and recall rate while effectively reducing the prediction time. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figures 1-2 A flowchart of a short text classification method according to an embodiment of the application is shown in FIG. 1.

[0068] Figure 3 A structural diagram of a short text classification device according to an embodiment of the application is shown in FIG. 2.

[0069] Figure 4 A composition diagram of a short text classification device according to an embodiment of the application is shown in FIG. 3. DETAILED DESCRIPTION

[0070] To make the technical problems, technical schemes and advantages to be solved by the embodiments of the application more clear, specific embodiments will be described in detail below with reference to the drawings.

[0071] Current multi-model fusion short text classification schemes mainly include two types: 1. The bagging idea is adopted to independently train multiple classifiers, and the weighted average of the results of the classifiers is used as the final result in the prediction stage; 2. A fusion loss function is set for multiple models for joint training to output a classification result. These schemes can reduce the variance of the prediction result, but still have the following problems: 1. Different schemes are not flexibly adopted according to the number of data sets, but a unified fusion strategy is adopted, which leads to limited improvement of the precision and recall rate; 2. The prediction time is higher, not only the high time consumption of the complex model with a large number of parameters (for example: Bert classification), but also the prediction time of the simple model (for example: CNN, RNN, etc.); 3. The effect is poor in the cold start stage, and the classification error caused by the overfitting of multiple models is amplified due to the small number of data sets in the cold start stage.

[0072] The embodiment of the present application provides a short text classification method and device, which can accurately classify short texts.

[0073] The embodiment of the present application provides a short text classification method, as shown in the formula (1), comprising the steps of: Figure 1

[0074] Step 101: determining a model combination according to a business stage and a sample quantity of a training data set, wherein the model combination comprises at least one neural network model;

[0075] Step 102: training the model in the model combination;

[0076] Step 103: inputting a text to be processed into the trained model combination, and outputting a classification result.

[0077] In the embodiment, the model combination is determined according to the business stage and the sample quantity of the training data set. For example, when the sample quantity of the training data set is extremely small, a text matching model is used to find the category of the most similar text in the training corpus as the classification result; when the sample quantity of the training data set is generally rich, two relatively complex classification models are used for voting, and when the results are consistent, the results are output and the process is ended, and when the results are inconsistent, the final result is determined by the text matching model; when the sample quantity of the training data set is rich enough, two relatively complex classification models are used for voting, and when the results are consistent, the results are output and the process is ended, and when the results are inconsistent, the final result is determined by a stronger classification model. The technical scheme of the embodiment can pay attention to the problems in different stages of the text classification application scene; the text matching model is used to replace the classification model in the cold start stage, which can effectively avoid the problem that the model is prone to overfitting when the data set is scarce; the voting method is used instead of the weighted average of the bagging idea when the multiple models are fused, which can improve the precision recall rate while effectively reducing the prediction time consumption.

[0078] In some embodiments, when the model combination is determined according to the business stage and the sample quantity of the training data set, the training of the model in the model combination comprises the following steps.

[0079] In the cold start stage, only the Bert matching model is trained when the sample quantity of the training data set is less than a first threshold value.

[0080] In the middle and late stage of the business, when the sample quantity of the training data set is greater than or equal to the first threshold value and less than a second threshold value, the Bert matching model, the CNN model and the RNN model are trained; when the sample quantity of the training data set is greater than or equal to the second threshold value, the Bert classification model, the CNN model and the RNN model are trained.

[0081] In some embodiments, the first threshold value is 3000-7000, and the second threshold value is 30000-70000. ​

[0082] In some embodiments, the inputting the to-be-processed text into the trained model combination outputs a classification result, including:

[0083] If the number of samples of the currently used training data set is less than the first threshold value, a Bert matching model is used to find a text most similar to the to-be-processed text in the training corpus, and the category corresponding to the text is taken as the classification result;

[0084] If the number of samples of the currently used training data set is greater than or equal to the first threshold value and less than the second threshold value, the to-be-processed text is input into a CNN model to obtain a first prediction result, and the to-be-processed text is input into an RNN model to obtain a second prediction result, if the first prediction result is the same as the second prediction result, the first prediction result or the second prediction result is taken as the classification result, if the first prediction result is not the same as the second prediction result, a Bert matching model is used to find a first text and a second text most similar to the to-be-processed text in the training corpus of the categories represented by the first prediction result and the second prediction result respectively, and a first score corresponding to the first text and a second score corresponding to the second text, if the first score is greater than or equal to the second score, the category to which the first text belongs is taken as the classification result, if the first score is less than the second score, the category to which the second text belongs is taken as the classification result;

[0085] If the number of samples of the currently used training data set is greater than or equal to the second threshold value, the to-be-processed text is input into a CNN model to obtain a first prediction result, and the to-be-processed text is input into an RNN model to obtain a second prediction result, if the first prediction result is the same as the second prediction result, the first prediction result or the second prediction result is taken as the classification result, if the first prediction result is not the same as the second prediction result, a third prediction result is obtained through a Bert classification model, and the third prediction result is taken as the classification result.

[0086] In this embodiment, the Bert matching model is used in the cold start stage, the CNN, RNN and Bert matching model fusion are used in the middle stage, and the CNN, RNN and Bert classification model fusion are used in the later stage. To a certain extent, the problem that the commonly used classification model in the cold start stage is easy to overfit can be solved, and the problem that the complex model is time-consuming and high can be solved while improving the classification effect. Taking the first threshold value as 3000-7000 and the second threshold value as 30000-70000 as an example, the prediction stage process is as shown in Figure 2 The method comprises the following steps:

[0087] When the number of samples of the training data set is less than the minimum threshold (for example: 5000, depending on the specific business situation), only the Bert matching model is trained, when the number of samples of the training data set is greater than the minimum threshold (5000) and less than the second low threshold (for example: 50000, depending on the specific business situation), three models of Bert matching, CNN, RNN are trained, when the number of data sets is greater than the second low threshold (50000), three models of CNN, RNN and Bert classification are trained.

[0088] The Bert matching model is a Sentence-Bert model, a first corpus sample is randomly selected from the original data set (such as a training data set), a second corpus sample is obtained after stop words of the first corpus sample are replaced, the first corpus sample and the second corpus sample are used as positive samples, corpus unrelated to the first corpus sample is selected from the training data set as negative samples, the ratio of positive samples to negative samples is 1: N (N >= 5), fine-tune is used for training, the loss function is shown in formula 1, "sa" represents the first corpus text selected from the training set, "sn" represents the unrelated negative sample, "sp" represents the second corpus sample, and represents the distance from the boundary, that is, the distance between sa and sp is at least greater than the distance between sa and sn, "||·||" represents the cosine distance, and the optimization goal is to make the distance between "sa" and "sp" closer and the distance between "sa" and "sn" farther. After training, the obtained matching model is stored, and the vector value of all original data sets is calculated according to the matching model and stored.

[0089] Formula 1: max (||sa-sp||-||sa-sn||+∈, 0)

[0090] The sample number ratio of the training set, the validation set and the test set of the CNN model is 8:1:1, the embedding layer of the CNN model uses word vectors as input, adopts double-channel connection, and the training result of Chinese Wikipedia of word2vector or fast-text is used as a static channel, the convolution layer adopts 2, 3 and 4 convolution kernels, and the pooling layer adopts the maximum pooling method.

[0091] The RNN model is a bidirectional long short-term memory Bi-LSTM model based on an attention mechanism, the sample number ratio of the training set, the validation set and the test set of the Bi-LSTM model based on the attention mechanism is 8:1:1, the embedding layer of the RNN model uses word vectors as input, 3 hidden layers, and each hidden layer contains 256 lstm units.

[0092] The sample quantity ratio of the training set, the validation set and the test set in the Bert classification model is 8:1:1, and the fine-tune training is performed on the basis of the Chinese-Bert-Wwm model.

[0093] When predicting the classification result, when the sample quantity of the training data set is less than the minimum threshold (5000), the Bert matching model is used to find a text most similar to the text to be processed in the entire training corpus, and the category corresponding to the text is regarded as the final classification result.

[0094] When the sample quantity of the training data set is greater than the minimum threshold (5000) and less than the second minimum threshold (50000), the first prediction result and the second prediction result are obtained through the CNN model and the RNN model respectively, and if the results are consistent (the first prediction result is equal to the second prediction result), the first prediction result or the second prediction result is directly output as the final classification result. If the results are inconsistent (the first prediction result is not equal to the second prediction result), the Bert matching model is used to match the first text and the second text most similar to the input text and the corresponding scores first score and second score in the first prediction result and the second prediction result category original corpus respectively, and the category of the text with the higher score is regarded as the final classification result.

[0095] When the sample quantity of the training data set is greater than 50000, the first prediction result and the second prediction result are obtained through the CNN model and the RNN model respectively, and if the results are consistent, the final result is directly output. If the results are inconsistent, the third prediction result is obtained by using the Bert classification model, and the third prediction result is the final classification result.

[0096] The embodiment of the application also provides a short text classification device, as shown in the figure, comprising: Figure 3 As shown in the figure, comprising:

[0097] The processing module 11 is used for determining a model combination according to the business stage and the sample quantity of the training data set, and the model combination comprises at least one neural network model;

[0098] The training module 12 is used for training the model in the model combination;

[0099] The prediction module 13 is used for inputting the text to be processed into the trained model combination, and outputting a classification result.

[0100] In the embodiment, the model combination is determined according to the business stage and the sample number of the training data set. For example, when the sample number of the training data set is extremely small, a text matching model is used to find the category of the most similar text in the training corpus as the classification result; when the sample number of the training data set is generally rich, two more complex classification models are used for voting, and the final result is determined by the text matching model when the results are inconsistent; when the sample number of the training data set is sufficient, two more complex classification models are used for voting, and the final result is determined by the stronger classification model when the results are inconsistent. The technical scheme of the embodiment can pay attention to the problems in different stages of the text classification application scene; the text matching model is used instead of the classification model in the cold start stage, which can effectively avoid the problem that the model is prone to overfitting when the data set is scarce; the voting method is used instead of the bagging idea of weighted average in the multi-model fusion, which can improve the precision recall rate while effectively reducing the prediction time consumption.

[0101] In some embodiments, the training module 12 is specifically configured to:

[0102] In the cold start stage, only the Bert matching model is trained when the sample number of the training data set is less than the first threshold value.

[0103] In the middle and late stage of the business, the Bert matching model, the CNN model and the RNN model are trained when the sample number of the training data set is greater than or equal to the first threshold value and less than the second threshold value; the Bert classification model, the CNN model and the RNN model are trained when the sample number of the training data set is greater than or equal to the second threshold value.

[0104] In some embodiments, the first threshold value is 3000-7000, and the second threshold value is 30000-70000.

[0105] In some embodiments, the prediction module 13 is specifically configured to:

[0106] If the sample number of the currently used training data set is less than the first threshold value, the Bert matching model is used to find the most similar text to the text to be processed in the training corpus, and the category corresponding to the text is used as the classification result.

[0107] If the number of samples in the current training dataset is greater than or equal to the first threshold and less than the second threshold, the CNN model and the RNN model are used to obtain the first prediction result and the second prediction result, respectively. If the first prediction result and the second prediction result are the same, the first prediction result or the second prediction result is used as the classification result. If the first prediction result and the second prediction result are different, the BERT matching model is used to find the first text and the second text that are most similar to the text to be processed in the training corpus of the categories represented by the first prediction result and the second prediction result, respectively, as well as the first score corresponding to the first text and the second score corresponding to the second text. If the first score is greater than or equal to the second score, the category to which the first text belongs is used as the classification result. If the first score is less than the second score, the category to which the second text belongs is used as the classification result.

[0108] If the number of samples in the current training dataset is greater than or equal to the second threshold, the first prediction result and the second prediction result are obtained by the CNN model and the RNN model, respectively. If the first prediction result and the second prediction result are the same, the first prediction result or the second prediction result is used as the classification result. If the first prediction result and the second prediction result are different, the third prediction result is obtained by the BERT classification model, and the third prediction result is used as the classification result.

[0109] In some embodiments, the BERT matching model is a Sentence-BERT model. A first corpus sample is randomly selected from the training dataset. Stop words in the first corpus sample are replaced to obtain a second corpus sample. The first and second corpus samples are used as positive samples. Corpus samples unrelated to the first corpus sample are selected from the training dataset as negative samples. The ratio of positive to negative samples is 1:N, where N is an integer not less than 5. Fine-tuning is used for training, and the loss function is calculated as follows:

[0110] max(||sa-sp||-||sa-sn||+∈,0)

[0111] Where sa represents the sample from the first corpus, sn represents the negative sample, sp represents the sample from the second corpus, "||·||" represents the cosine distance, and ∈ represents the distance boundary.

[0112] In some embodiments, the sample quantity ratio of the training set, the validation set and the test set of the CNN model is 8:1:1, the embedding layer of the CNN model uses word vectors as input, adopts double-channel connection, and the Chinese Wikipedia training result of word2vector or fast-text is used as a static channel, the convolution layer adopts three convolution kernels of 2, 3 and 4, and the pooling layer adopts the maximum pooling method.

[0113] In some embodiments, the RNN model is a bidirectional long short-term memory (Bi-LSTM) model based on an attention mechanism, the sample quantity ratio of the Bi-LSTM model based on the attention mechanism is 8:1:1, the embedding layer of the RNN model uses word vectors as input, and three hidden layers each containing 256 lstm units.

[0114] In some embodiments, the sample quantity ratio of the training set, the validation set and the test set of the Bert classification model is 8:1:1, and the fine-tune training is performed on the basis of the Chinese-Bert-Wwm model.

[0115] The embodiment of the present application also provides a short text classification device, as shown in the accompanying drawings, comprising a memory 21, a processor 22, and a computer program stored in the memory 21 and executable on the processor 22; the processor 22 implements the short text classification method as described above when executing the program. Figure 4

[0116] In some embodiments, the processor 22 is configured to determine a model combination according to the business stage and the sample quantity of the training data set, the model combination comprising at least one neural network model; train the models in the model combination; input the text to be processed into the trained model combination, and output a classification result.

[0117] In some embodiments, the processor 22 is specifically configured to train only the Bert matching model when the sample quantity of the training data set is less than a first threshold value in the cold start stage; train the Bert matching model, the CNN model and the RNN model when the sample quantity of the training data set is greater than or equal to the first threshold value and less than a second threshold value in the later stage of the business; and train the Bert classification model, the CNN model and the RNN model when the sample quantity of the training data set is greater than or equal to the second threshold value.

[0118] In some embodiments, the first threshold value is 3000-7000, and the second threshold value is 30000-70000.

[0119] ​In some embodiments, the processor 22 is specifically configured to, if the number of samples of the currently used training data set is less than the first threshold value, use a Bert matching model to find a text most similar to the to-be-processed text in a training corpus, and take the category corresponding to the text as the classification result; if the number of samples of the currently used training data set is greater than or equal to the first threshold value and less than the second threshold value, obtain first prediction result and second prediction result by using a CNN model and an RNN model respectively, if the first prediction result is the same as the second prediction result, take the first prediction result or the second prediction result as the classification result; if the first prediction result is not the same as the second prediction result, use the Bert matching model to find first text and second text most similar to the to-be-processed text in the training corpus of the category represented by the first prediction result and the second prediction result respectively, and find a first score corresponding to the first text and a second score corresponding to the second text, if the first score is greater than or equal to the second score, take the category to which the first text belongs as the classification result, if the first score is less than the second score, take the category to which the second text belongs as the classification result; if the number of samples of the currently used training data set is greater than or equal to the second threshold value, obtain first prediction result and second prediction result by using a CNN model and an RNN model respectively, if the first prediction result is the same as the second prediction result, take the first prediction result or the second prediction result as the classification result; if the first prediction result is not the same as the second prediction result, obtain prediction result by using a Bert classification model, and take the third prediction result as the classification result.

[0120] In some embodiments, the Bert matching model adopts a Sentence-Bert algorithm, randomly selects a first corpus sample from a training data set, replaces the stop words of the first corpus sample to obtain a second corpus sample, takes the first corpus sample and the second corpus sample as positive samples, selects corpus unrelated to the first corpus sample from the training data set as negative samples, the ratio of positive samples to negative samples is 1: N, where N is an integer not less than 5, and the training is performed in a fine-tune manner, and the formula of the loss function is:

[0121] max(||sa-sp||-||sa-sn||+∈,0)

[0122] Wherein, sa represents the first corpus sample, sn represents the negative sample, sp represents the second corpus sample, "||·||" represents the cosine distance, and ∈ represents the distance boundary.

[0123] In some embodiments, the sample quantity ratio of the training set, the validation set and the test set of the CNN model is 8:1:1, the embedding layer of the CNN model uses word vectors as input, adopts double-channel connection, and the Chinese Wikipedia training result of word2vector or fast-text is used as a static channel, the convolution layer adopts three convolution kernels of 2, 3 and 4, and the pooling layer adopts the maximum pooling method.

[0124] In some embodiments, the RNN model is a bidirectional long short-term memory (Bi-LSTM) model based on an attention mechanism, the sample quantity ratio of the training set, the validation set and the test set of the Bi-LSTM model based on the attention mechanism is 8:1:1, the embedding layer of the RNN model uses word vectors as input, and three hidden layers each containing 256 lstm units.

[0125] In some embodiments, the sample quantity ratio of the training set, the validation set and the test set of the Bert classification model is 8:1:1, and the fine-tune training is performed on the basis of the Chinese-Bert-Wwm model.

[0126] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps in the short text classification method.

[0127] The computer readable medium includes permanent and non-permanent, removable and non-removable media, and can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage, or any other non-transmission medium that can be used to store information that can be accessed by a computing device to be detected. According to the definition in this paper, the computer readable medium does not include transitory computer readable media such as modulated data signals and carriers.

[0128] The above is the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A short text classification method, characterized in that, include: The model combination is determined based on the current business stage and the number of samples in the training dataset, and the model combination includes at least one neural network model; The models in the model combination are trained; The text to be processed is input into the trained model, and the model is combined to output the classification result. The process of combining the trained model with the text to be processed and outputting classification results includes: If the number of samples in the current training dataset is less than the first threshold, the Bert matching model is used to find the text most similar to the text to be processed in the training corpus, and the category corresponding to that text is taken as the classification result. If the number of samples in the current training dataset is greater than or equal to the first threshold and less than the second threshold, the text to be processed is input into a CNN model to obtain a first prediction result, and the text to be processed is input into an RNN model to obtain a second prediction result. If the first prediction result and the second prediction result are the same, the first prediction result or the second prediction result is used as the classification result. If the first prediction result and the second prediction result are different, the BERT matching model is used to find the first text and the second text that are most similar to the text to be processed in the training corpus of the categories represented by the first prediction result and the second prediction result, respectively, as well as the first score corresponding to the first text and the second score corresponding to the second text. If the first score is greater than or equal to the second score, the category to which the first text belongs is used as the classification result. If the first score is less than the second score, the category to which the second text belongs is used as the classification result. If the number of samples in the current training dataset is greater than or equal to the second threshold, the text to be processed is input into the CNN model to obtain a first prediction result, and the text to be processed is input into the RNN model to obtain a second prediction result. If the first prediction result and the second prediction result are the same, the first prediction result or the second prediction result is used as the classification result; if the first prediction result and the second prediction result are different, a third prediction result is obtained through the BERT classification model, and the third prediction result is used as the classification result.

2. The short text classification method according to claim 1, characterized in that, Determine the model combination based on the current business stage and the number of samples in the training dataset, and train the models in the model combination, including: During the cold start phase, only the BERT matching model is trained when the number of samples in the training dataset is less than the first threshold. In the later stages of the business, when the number of samples in the training dataset is greater than or equal to the first threshold and less than the second threshold, the BERT matching model, the convolutional neural network (CNN) model, and the recurrent neural network (RNN) model are trained; when the number of samples in the training dataset is greater than or equal to the second threshold, the BERT classification model, the CNN model, and the RNN model are trained.

3. The short text classification method according to claim 2, characterized in that, The first threshold is 3,000-7,000 entries, and the second threshold is 30,000-70,000 entries.

4. The short text classification method according to claim 2, characterized in that, The BERT matching model is a Sentence-BERT model. A first corpus sample is randomly selected from the training dataset. Stop words in the first corpus sample are replaced to obtain a second corpus sample. The first and second corpus samples are used as positive samples. Corpus samples unrelated to the first corpus sample are selected from the training dataset as negative samples. The ratio of positive to negative samples is 1:N, where N is an integer not less than 5. Fine-tuning is used for training, and the loss function is calculated as follows: max(||in-sp||-||in-sn|| + ,0) Where sa represents the sample from the first corpus, sn represents the negative sample, sp represents the sample from the second corpus, and "||·||" represents the cosine distance. Indicates the distance from the boundary.

5. The short text classification method according to claim 2, characterized in that, The ratio of the number of samples in the training set, validation set, and test set of the CNN model is 8:1:

1. The embedding layer of the CNN model uses word vectors as input. The embedding layer uses two channels, static and dynamic word vectors, connected to the convolutional layer. The training results of Chinese Wikipedia using word2vector or fast-text are used as static channels. The convolutional layer uses three types of convolutional kernels: 2, 3, and 4. The pooling layer uses the max pooling method.

6. The short text classification method according to claim 2, characterized in that, The RNN model is a Bi-LSTM model based on an attention mechanism. The ratio of the number of samples in the training set, validation set, and test set of the Bi-LSTM model based on the attention mechanism is 8:1:

1. The embedding layer of the RNN model uses word vectors as input, and there are 3 hidden layers, each containing 256 LSTM units.

7. The short text classification method according to claim 2, characterized in that, The ratio of the number of samples in the training set, validation set, and test set of the BERT classification model is 8:1:1, and fine-tuning training is performed on the basis of the Chinese-Bert-Wwm model.

8. A short text classification device, characterized in that, include: The processing module is used to determine the model combination based on the current business stage and the number of samples in the training dataset, wherein the model combination includes at least one neural network model; The training module is used to train the models in the model combination; The prediction module is used to combine the text to be processed into the trained model and output the classification result; The prediction module is specifically used to find the text most similar to the text to be processed in the training corpus if the number of samples in the current training dataset is less than a first threshold, and take the category corresponding to the text as the classification result. If the number of samples in the current training dataset is greater than or equal to the first threshold and less than the second threshold, the text to be processed is input into a CNN model to obtain a first prediction result, and the text to be processed is input into an RNN model to obtain a second prediction result. If the first prediction result and the second prediction result are the same, the first prediction result or the second prediction result is used as the classification result. If the first prediction result and the second prediction result are different, the BERT matching model is used to find the first text and the second text that are most similar to the text to be processed in the training corpus of the categories represented by the first prediction result and the second prediction result, respectively, as well as the first score corresponding to the first text and the second score corresponding to the second text. If the first score is greater than or equal to the second score, the category to which the first text belongs is used as the classification result. If the first score is less than the second score, the category to which the second text belongs is used as the classification result. If the number of samples in the current training dataset is greater than or equal to the second threshold, the text to be processed is input into the CNN model to obtain a first prediction result, and the text to be processed is input into the RNN model to obtain a second prediction result. If the first prediction result and the second prediction result are the same, the first prediction result or the second prediction result is used as the classification result; if the first prediction result and the second prediction result are different, a third prediction result is obtained through the BERT classification model, and the third prediction result is used as the classification result.

9. A short text classification device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; characterized in that, When the processor executes the program, it implements the short text classification method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the short text classification method as described in any one of claims 1-7.

Images