Text classification method and device, text classification model training method and device and medium

Abstract

The invention provides a text classification and text classification model training method and device and a medium, and the method comprises the steps: carrying out the data processing according to original text data, and obtaining a training sample set and a verification sample set; constructing an initial model; and performing model training on the initial model through the training sample set, and performing model evaluation through the verification sample set to obtain a trained text classification model. On one hand, the word frequency information is added, and the word frequency information is deeply mined, so that the model characterization capability can be improved. On the other hand, weights are provided for different features through the attention network, and the feature values are updated based on the weights. And finally, making a decision through the depth model estimation layer. And the depth model pre-estimation layer predicts a result by integrating various kinds of information. Compared with a traditional mode which does not consider contribution differences of different characteristics, the method has certain advantages. Therefore, the method can effectively improve the accuracy of the text data classification result.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and particularly to a text classification method, a text classification model training method, an apparatus and a medium. Background Art

[0002] Text classification is a classic task in NLP (Natural Language Processing) applications. Usually, according to whether the collected text data has an order relationship, it is divided into two cases: ordered text data and unordered text data. This application mainly focuses on the unordered text classification scenario. The unordered text classification scenario is relatively common. In practice, due to the difficulty of data collection, it is usually impossible to obtain the sequence relationship of text data, resulting in the situation of unordered text data. Usually, only the words of the text are obtained, and the grammatical dependency relationship or the sequential relationship between words is not obtained.

[0003] For example, in the user sentiment classification based on labeled data, the user has attached many text labels to a certain object, but these labels have no order relationship and other scenarios.

[0004] For the unordered text classification scenario, it is regarded as a binary classification problem based on unordered text, and machine learning or deep learning methods are mostly applied, such as methods like DNN (Deep-Learning Neural Network) and NFM (Neural factorization machines). For such methods, usually, words are encoded into word IDs, and then each record is constructed into data composed of word IDs through data processing. By combining feature crossing and a deep neural network, the input record and the target are modeled.

[0005] Among them, the NFM model is a classic deep learning method for binary classification. Usually, a dictionary is first generated for all text records, and each word has an ID (Identity document), called a word ID. Then, the ID is used as the input to the NFM model, and the cross features of the word IDs are obtained through BI (Bilinear Interpol, bilinear interpolation) bilinear crossing. Then, the final probability is estimated for the cross features through a DNN.

[0006] However, the existing text binary classification model based on unordered words only considers the ID information of words, resulting in a relatively low accuracy of the classification result. Summary of the Invention

[0007] The technical problem to be solved by the present application is to provide a text classification method, a text classification model training method, an apparatus and a medium for solving the problems existing in the prior art in view of the above deficiencies of the prior art.

[0008] In a first aspect, the present application provides a method for training a text classification model, the method

[0009] comprises:

[0010] S1. Perform data processing on the original text data to obtain a training sample set and a validation sample set, wherein the training sample set and the validation sample set contain word ids and word frequency information;

[0011] S2. Construct an initial model, the initial model includes an input layer, an embedding layer, a bilinear interpolation Bi interaction layer, a concatenation layer, an attention layer, and a deep model prediction layer;

[0012] wherein, the embedding layer is used to perform dimensionality reduction processing on the word ids and word frequency information input into the input layer to obtain embedding vectors;

[0013] The Bi interaction layer is used to obtain the interaction features of the word ids and the interaction features of the word frequency information according to the embedding vectors;

[0014] The concatenation layer is used to perform feature concatenation on the interaction features of the word ids and the interaction features of the word frequency information to obtain concatenated features;

[0015] The attention layer is used to generate weight values for the concatenated features based on the attention mechanism, and perform feature update based on the weight values to obtain updated features;

[0016] The deep model prediction layer is used to obtain a classification prediction result according to the updated features;

[0017] S3. Train the initial model with the training sample set and evaluate the model with the validation sample set to obtain a trained text classification model.

[0018] In some embodiments, the calculation formula of the embedding layer is:

[0019] e_id i = w_id i x_id i

[0020] e_value i = w_id i x_value i

[0021] e = [e_id, e_value]

[0022] e_id = [e_id 1 , e_id 2 ,..., e_id N

[0023] ​e_value = [e_value 1 , e_value 2 ,..., e_value N

[0024] Wherein, x_id i is the i-th word, w_id i is the embedding weight matrix of the word id, e_id i is the embedding vector of the i-th word id, x_value i is the word frequency of the i-th word id, e_value i is the embedding vector of the word frequency of the i-th word id, and e is the embedding vector, including the embedding vector e_id of the word id and the embedding vector e_value of the word frequency.

[0025] In some embodiments, the calculation formula of the Bi interaction layer is:

[0026]

[0027]

[0028] Wherein, f_id BI is the interaction feature of the word id, f_value BI is the interaction feature of the word frequency information, x_id i is the i-th word, e_id i is the embedding vector of the i-th word id, x_value i is the word frequency of the i-th word id, e_value i is the embedding vector of the word frequency of the i-th word id, and BN() represents batch normalization processing of the result.

[0029] In some embodiments, the calculation formula of the splicing layer is:

[0030] f con = Concat(f_id BI , f_vlue BI , e_id, e_value, x_value)

[0031] f con = Concat(f_id BI , f_vlue BI , x_value)

[0032] Wherein, fcon is the splicing feature, f_id BI is the interaction feature of the word id, f_value BI ​For the interaction feature of word frequency information, e_id is the embedding vector of the word id, e_value is the embedding vector of the word frequency, and x_value is the word frequency information of the word id.

[0033] In some embodiments, the calculation formula of the attention layer is:

[0034] a i ' = HReLU(Wf con,i +b)

[0035]

[0036] f′ con,i = a′ i *f con,i

[0037] Where α’ is the initial weight, fcon is the concatenated feature, α is the final weight value, and f’con is the updated feature.

[0038] In some embodiments, the calculation formula of the deep model prediction layer is:

[0039] f (l+1) = s(W (l) f (l) +b (l) )

[0040]

[0041] Where l is the hidden layer; σ is the activation function, f (l) 、b (l) and W (l) are the feature, bias, and weight of the l-th layer respectively, f (l +1) represents the feature value of the (l + 1)-th layer, l last is the last hidden layer, y dnn is the classification prediction result, Dense is the fully connected layer, and unit is the number of nodes.

[0042] In some embodiments, S1 includes:

[0043] S11. Generate a data matrix according to the original text data;

[0044] S12. Set the word vector length N, count the TOP N words with the highest word frequency for each text data dimension, generate the TOP keywords for each text, and obtain the word vector matrix;

[0045] S13. Mark the data in the word vector matrix and split the marked data to obtain the training sample set and the validation sample set.

[0046] In some embodiments, S11 includes:

[0047] S111, performing word segmentation processing and filtering processing on the original text data to obtain candidate words;

[0048] S112, performing ID encoding on the candidate words to generate a text dictionary;

[0049] S113, counting the word frequency of each candidate word in the original text data in the text dictionary, and performing normalization processing on the word frequency data to obtain a data matrix.

[0050] In a second aspect, the present application provides a text classification method, the method including:

[0051] Obtaining text data to be classified;

[0052] Classifying the text data through a text classification model to obtain a text classification result;

[0053] Wherein, the text classification model is trained by the text classification model training method according to any one of claims 1-8.

[0054] In a third aspect, the present application provides a text classification model training device, the device including:

[0055] A data processing module, configured to perform data processing according to the original text data to obtain a training sample set and a validation sample set, wherein the training sample set and the validation sample set contain word ids and word frequency information;

[0056] A model construction module, configured to construct an initial model, the initial model including an input layer, an embedding layer, a bilinear interpolation Bi interaction layer, a splicing layer, an attention layer, and a deep model prediction layer;

[0057] Wherein, the embedding layer is used to perform dimensionality reduction processing on the word ids and word frequency information input into the input layer to obtain an embedding vector;

[0058] The Bi interaction layer is used to obtain the interaction features of the word ids and the interaction features of the word frequency information according to the embedding vector;

[0059] The splicing layer is used to perform feature splicing on the interaction features of the word ids and the interaction features of the word frequency information to obtain a spliced feature;

[0060] The attention layer is used to generate a weight value of the spliced feature based on the attention mechanism, and perform feature update based on the weight value to obtain an updated feature;

[0061] The deep model prediction layer is used to obtain a classification prediction result according to the updated feature;

[0062] A training and evaluation module, configured to train the initial model through the training sample set and evaluate the model through the validation sample set to obtain a trained text classification model.

[0063] In a fourth aspect, the present application provides a text classification device, which includes:

[0064] A data acquisition module, configured to acquire text data to be classified;

[0065] A text classification module, configured to classify the text data through the text classification model to obtain a text classification result;

[0066] Wherein, the text classification model is trained by the above-mentioned text classification model training method.

[0067] In a fifth aspect, the present application provides a text classification model training device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to implement the above-mentioned text classification model training method.

[0068] In a sixth aspect, the present application provides a text classification device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to implement the above-mentioned text classification method.

[0069] In a seventh aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the above-mentioned text classification model training method and / or text classification method.

[0070] The text classification and text classification model training methods, devices, and media provided by this application. Specifically, data processing is performed on the original text data to obtain a training sample set and a validation sample set. Among them, the training sample set and the validation sample set contain word IDs and word frequency information; an initial model is constructed. The initial model includes an input layer, an embedding layer, a bilinear interpolation Bi interaction layer, a splicing layer, an attention layer, and a deep model prediction layer; among them, the embedding layer is used to perform dimensionality reduction processing on the word IDs and word frequency information input to the input layer to obtain embedding vectors; the Bi interaction layer is used to obtain the interaction features of word IDs and the interaction features of word frequency information based on the embedding vectors; the splicing layer is used to splice the interaction features of word IDs and the interaction features of word frequency information to obtain spliced features; the attention layer is used to generate weight values for the spliced features based on the attention mechanism and update the features based on the weight values to obtain updated features; the deep model prediction layer is used to obtain a classification prediction result based on the updated features; the initial model is trained through the training sample set, and the model is evaluated through the validation sample set to obtain a trained text classification model. Compared with the prior art, this application has the following advantages: 1) This application adds word frequency information, deeply mines the word frequency information, and improves the model representation ability. This application uses word frequency information as input to obtain different representation information of various word frequencies, including the original input form and the form after embedding transformation. After the original NFM obtains interaction information based on BI, it only obtains high-order features through DNN based on the BI interaction information and does not consider the information of the original input. Compared with the prior art, this application not only introduces word frequency information but also applies the original information of word frequency as the input of the deep model. 2) Considering that the contribution of different forms of feature expression to model prediction is different, this application provides weights for different features through an attention network and updates the feature values based on the weights. Finally, a decision is made through the deep model prediction layer. The deep model prediction layer comprehensively considers various information to predict the result. It has a certain advantage over the traditional method that does not consider the contribution difference of different features. Therefore, the method of this application can effectively improve the accuracy of text data classification results. Description of the Drawings

[0071] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0072] Figure 1 Schematic diagram of the text classification model training method provided by an embodiment of this application;

[0073] Figure 2 Schematic diagram of the initial model provided by an embodiment of this application;

[0074] Figure 3 Schematic diagram of the text classification method provided by an embodiment of the present application;

[0075] Figure 4 Schematic diagram of the text classification model training device provided by an embodiment of the present application;

[0076] Figure 5 Schematic diagram of the text classification device provided by an embodiment of the present application;

[0077] Figure 6 Another schematic diagram of the text classification model training device provided by an embodiment of the present application;

[0078] Figure 7 Another schematic diagram of the text classification device provided by an embodiment of the present application.

[0079] Through the above-mentioned drawings, specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Detailed implementation manners

[0080] To enable those skilled in the art to better understand the technical solutions of the present application, the embodiments of the present application will be further described in detail below in conjunction with the drawings.

[0081] It can be understood that the specific embodiments and drawings described herein are only for explaining the present application, rather than limiting the present application.

[0082] It can be understood that, without conflict, the various embodiments and the various features in the embodiments of the present application can be combined with each other.

[0083] It can be understood that, for the convenience of description, only the parts related to the present application are shown in the drawings of the present application, and the parts unrelated to the present application are not shown in the drawings.

[0084] It can be understood that each unit and module involved in the embodiments of the present application may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units and modules may also be integrated into one entity structure.

[0085] It can be understood that the terms "first", "second", etc. in the embodiments of the present application are used to distinguish different objects, or to distinguish different processes for the same object, rather than to describe a specific order of the objects.

[0086] It can be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present application may occur in an order different from that marked in the drawings.

[0087] It can be understood that in the flowcharts and block diagrams of the present application, the possible system architectures, functions, and operations of the systems, devices, equipment, and methods according to the embodiments of the present application are shown. Among them, each block in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified function. Moreover, each block or combination of blocks in the block diagram and flowchart can be implemented by a hardware-based system for implementing the specified function, or by a combination of hardware and computer instructions.

[0088] It can be understood that the units and modules involved in the embodiments of the present application can be implemented in software or in hardware. For example, the units and modules can be located in the processor.

[0089] In the prior art, in the scenario of unordered text classification, there are the following two disadvantages in the binary text classification model based on unordered words:

[0090] 1) Commonly used processing methods usually only use the ID information of words, and apply less or insufficiently the information of word frequency (the number of times a word appears), without deeply exploring the value of word frequency information in the binary classification scenario based on unordered words. Word frequency information is very important. For example, in search terms, words with a large number of searches are relatively more important, representing the user's preference and having a stronger correlation with the estimation target.

[0091] 2) Commonly used processing methods do not consider that the weights of different features are different. For example, in the NFM method, the interaction features obtained by the BI method directly enter the DNN for processing, without considering that the contributions of these interaction features to the final model are different.

[0092] In view of the problems existing in the commonly used models in the binary text classification scenario based on unordered words mentioned above, the present application proposes a new method for training a text classification model based on unordered words and a text classification method based on this model. The method of the present application deeply explores the word frequency information and strengthens the feature expression of the word frequency information for the above-mentioned disadvantages; at the same time, considering that the contributions of different features to the final estimation are different, a weight generation method is designed to effectively improve the model effect. This method can be applied to various unordered word text classification scenarios, such as the risk control scenario based on search terms, sentiment analysis based on comment words, etc.

[0093] Before describing the embodiments of the present application, the following explanations are made for the noun terms involved in the embodiments of the present application:

[0094] (1) Text classification: refers to performing a machine learning classification task on text data. It includes binary classification and multi-classification tasks. Text data is usually data with text, such as search terms, comment records, articles, etc.

[0095] (2) Multilayer Perceptron: The Multilayer Perceptron (MLP), also known as the Artificial Neural Network (ANN), can have multiple hidden layers in the middle in addition to the input and output layers, and the layers are fully connected. The simplest MLP contains only one hidden layer, that is, a three-layer structure. During the prediction process of the input information of the MLP, feed-forward calculation is adopted. Feed-forward is different from the cyclic structure and only propagates unidirectionally, so it is also called a fully connected feed-forward neural network.

[0096] (3) NFM model: A commonly used model for predicting the click-through rate of advertisements. Based on the interaction features of the FM model, this model uses the DNN model to deeply process the interaction features to predict the click-through rate of advertisements. Although this method uses deep learning, it is only based on the interaction features of the FM model and does not consider the weights of the features after interaction.

[0097] (4) Attention mechanism: In deep learning, it is a method to measure the contributions of different inputs, obtain different attention scores through specific calculation methods, and process them differently according to the scores.

[0098] The following uses specific embodiments to elaborate in detail on the technical solutions of this application and how the technical solutions of this application solve the above technical problems. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below in conjunction with the accompanying drawings.

[0099] This application provides a text classification and text classification model training method. The working process of this method can be implemented by an electronic device, such as a computer, a handheld intelligent terminal, etc. For the convenience of explanation, in the embodiments of this application, the method execution subject is a computer for elaboration.

[0100] Figure 1 It is a schematic diagram of the text classification model training method provided by the embodiments of this application. As Figure 1 shown, this application provides a text classification model training method, and the method includes S1 - S3, specifically as follows:

[0101] S1. Perform data processing on the original text data to obtain a training sample set and a validation sample set, where the training sample set and the validation sample set contain word ids and word frequency information;

[0102] Among them, the original text data refers to the sample data used for model training. This step mainly refers to processing the original text data and converting it into data that can be used for text classification modeling, including a training sample set and a validation sample set. Both sample sets contain word ids and word frequency information.

[0103] Optionally, S1 includes S11 - S13, specifically as follows::

[0104] S11. Generate a data matrix based on the original text data;

[0105] Specifically, first perform word segmentation processing and filtering processing on the original text data to obtain candidate words;

[0106] Among them, word segmentation processing refers to segmenting the original text data such as retrieval phrase or review text to obtain independent words.

[0107] Filtering processing refers to performing candidate word selection processing, that is, filtering unimportant words and retaining important words (usually retaining keywords with higher occurrence frequencies), and using the selected words as candidate words to generate a data matrix for the next step of processing.

[0108] S112. Perform ID encoding on the candidate words to generate a text dictionary;

[0109] Specifically, encode all segmented words. The encoding method can be numbered in sequence or can use a custom method to generate a text dictionary, and each word corresponds to a unique ID. Assume the length of the text dictionary is words_number, that is, there are words_number words in total, then the ID encoding will be from 1 to words_number, and all words in the data matrix need to participate in the encoding.

[0110] S113. Count the word frequencies of each candidate word in the original text data in the text dictionary, and perform normalization processing on the word frequency data to obtain a data matrix.

[0111] Specifically, count the number of times each candidate word appears corresponding to each text data; and perform normalization processing on the word frequency data so that the word frequency value is between 0 and 1. Finally, generate a data matrix X(data_id; id1, id2,..., idn, value1, value2,..., valuen), where data_id represents the data record id, id and value represent the ID and frequency of the word respectively, and n represents that the user has n words.

[0112] S12. Set the word vector length N, count the TOP N words with the highest word frequencies for each text data dimension, generate the TOP keywords for each text, and obtain a word vector matrix;

[0113] Specifically, set the word vector length N, count the TOP N words with the highest word frequencies for each text data dimension, and generate the TOP keywords for each text. N can be defined according to the actual situation. For example, if N is assumed to be 256, that is, the number of key words for each text takes the top 256 retrieval word vectors. Since the number of words n corresponding to different text data varies, some are more and some are less. For the sake of uniformity, for users with n greater than N, the first N are intercepted from high to low according to the frequency; for users with the number of retrieval words n less than N, they are uniformly padded to N with 0. Finally, a word vector matrix X(data_id, id1, id2, …, idN; value1, value2, …, valueN) is generated, where the dimension of the X matrix is M * 2N, where M represents the number of samples and N represents the length of the word vector.

[0114] S13. Mark the data in the word vector matrix, and split the marked data to obtain the training sample set and the validation sample set.

[0115] Specifically, mark the text data in the word vector matrix, mark the positive samples as 1 and the negative samples as 0. For example, in the credit risk control scenario, based on retrieval to predict whether a user is overdue, usually an overdue default user is regarded as a positive sample and marked as 1; a non-default user is a negative sample and marked as 0. For example, in the sentiment analysis scenario based on text data, positive is 1 and negative is 0. Finally, the data format output by data preparation is X(label, id1, id2, …, idN; value1, value2, …, valueN). Where label represents the sample mark; id represents the retrieval word number, with a total of N; value represents the word frequency corresponding to id, with a total of N.

[0116] Then, split the collected sample data according to a certain ratio to construct a training sample set and a validation sample set. Usually, the split ratio is 0.7:0.3. The training sample set is used for model training, and the validation sample set is used for model effect evaluation.

[0117] S2. Construct an initial model.

[0118] Figure 2 is a schematic diagram of the initial model provided by the embodiment of the present application. As Figure 2 shown, the initial model includes an input layer, an embedding layer, a bilinear interpolation Bi interaction layer, a splicing layer, an attention layer, and a deep model prediction layer;

[0119] Among them, the embedding layer is used to perform dimensionality reduction on the word IDs and word frequency information input to the input layer to obtain embedding vectors; the Bi-interaction layer is used to obtain the interaction features of word IDs and the interaction features of word frequency information according to the embedding vectors; the splicing layer is used to splice the interaction features of word IDs and the interaction features of word frequency information to obtain spliced features; the attention layer is used to generate weight values for the spliced features based on the attention mechanism and perform feature update based on the weight values to obtain updated features; the deep model prediction layer is used to obtain a classification prediction result according to the updated features.

[0120] Optionally, the input layer mainly inputs the data X of the model, and represents the word encoding (ID) and word frequency (value) information in it as Xid(x id1 , x id2 , …, x idi , …, x idN ) and Xvalue(x value1 , x value2 , …, x valuei , …, x valueN ). N is the selected length of the word vector.

[0121] Optionally, since the length of the text dictionary is words_number, which is usually large, that is, it contains many words, the input data is mapped to a certain low-dimensional space through the embedding layer to form dense vectors. In this way, in addition to dimensionality reduction, the distance between the embedding vectors can also be used to describe the similarity between words.

[0122] In some embodiments, the calculation formula of the embedding layer is:

[0123] e_id i = w_id i x_id i (1)

[0124] e_value i = w_id i x_value i (2)

[0125] e = [e_id, e_value](3)

[0126] e_id = [e_id 1 , e_id 2 ,..., e_id N (4)

[0127] e_value = [e_value 1 , e_value 2 ,..., e_valueN (5)

[0128] Among them, x_id i is the i-th word, w_id i is the embedding weight matrix of the word id, e_id i is the embedding vector of the i-th word id, x_value i is the word frequency of the i-th word id, e_value i is the embedding vector of the word frequency of the i-th word id, and e is the embedding vector, including the embedding vector e_id of the word id and the embedding vector e_value of the word frequency.

[0129] Specifically, the input of the embedding layer is the data output in the data preparation stage, represented by Xid(x id1 , x id2 , …, x idi , …, x idN ). Among them, x idi is the i-th word, and there are N words in total. The calculation formula of the embedding layer is as shown in Equation 1, where w_id i is the embedding weight matrix of the word id, and e_id i is the embedding vector of the i-th word id. The dimension of this embedding vector e_id i is d-dimensional, which is a hyperparameter.

[0130] Through the word id, the embedding matrix is obtained, and the embedding vector e_id of the ID can be obtained. Based on the weight w_id i of the word id, for the word frequency x_value i corresponding to each word id, its embedding vector e_value i can also be obtained through the embedding matrix. The calculation formula is as shown in Formula 2, where x_value i represents the word frequency corresponding to the i-th word, and w_id represents the embedding weight corresponding to the i-th word id.

[0131] Finally, through the embedding matrix, the embedding vector e can be obtained, as shown in Formula 3, including the embedding vector e_id of the word id, as shown in Formula 4, and the embedding vector of the word frequency, as shown in Formula 5. Among them, the dimension of the embedding vector is d, so the dimensions of the embedding vectors e_id and e_value of the word id and the word frequency are both N * d. The next step is that the embedding vector e will be used as the input of the Bi interaction layer.

[0132] In some embodiments, the calculation formula of the Bi interaction layer is:

[0133]

[0134]

[0135] Among them, f_id BI is the interaction feature of the word id, f_value BI is the interaction feature of the word frequency information, x_id i is the i-th word, e_id i is the embedding vector of the i-th word id, x_value i is the word frequency of the i-th word id, e_value i is the embedding vector of the word frequency of the i-th word id, and BN() represents performing BatchNormal batch normalization processing on the result.

[0136] Specifically, the Bi interaction layer is implemented using the BI cross algorithm. The second-order cross feature of the embedding vector is obtained through the BI interaction algorithm, including the interaction features of the word id and the word frequency. The calculation formula of the BI interaction feature refers to the calculation method of the NFM algorithm and adds a BN operation. As shown in Formulas 6 and 7, when the embedding vector is e_id, the interaction feature f_id of the word id is obtained BI , and when the embedding vector is e_vlaue, the interaction feature f_value of the word frequency is obtained BI .

[0137] In some embodiments, the calculation formula of the splicing layer is:

[0138] f con =Concat(f_id BI , f_vlue BI , e_id, e_value, x_value)(8)

[0139] f con =Concat(f_id BI , f_vlue BI , x_value)(9)

[0140] Among them, fcon is the splicing feature, f_id BI is the interaction feature of the word id, f_value BI is the interaction feature of the word frequency information, e_id is the embedding vector of the word id, e_value is the embedding vector of the word frequency, and x_value is the word frequency information of the word id.

[0141] Specifically, the existing method only uses the features after the BI interaction of word IDs as the input for the next DNN processing, without considering word frequency information, and the model effect is relatively poor. This application comprehensively considers the information of IDs and word frequencies. In the feature concatenation layer, it fuses various information such as the interactive features of word IDs and word frequencies, and the first-order word frequency information. The concatenated features are denoted as fcon, as shown in Equation 8. The concatenated features include the interactive feature f_idBI of word IDs and the embedding vector e_id, the interactive feature f_valueBI of word frequencies and the embedding vector e_value, and the word frequency information x_value.

[0142] In addition, through feature selection experiments, it is found that the concatenated features processed according to Equation 9 have the same effect as Equation 8. That is, the calculation formula of the concatenation layer can also only retain the interactive features f_idBI, f_valueBI and word frequencies.

[0143] In some embodiments, the calculation formula of the attention layer is:

[0144] a′ i = HReLU(Wf con,i + b) (10)

[0145]

[0146] f′ con,i = a′ i * f con,i (12)

[0147] where α’ is the initial weight, f con is the concatenated feature, α is the final weight value, and f’ con is the updated feature.

[0148] Specifically, weight values are generated for all the features fcon based on the attention mechanism, and the corresponding features are updated based on the weight values. The purpose of the attention mechanism is to consider that the contributions of different inputs are different, and greater weights need to be assigned to the parts with greater contributions to highlight their contributions. The attention mechanism includes various methods, such as the attention network (AN, Attention Network) based on the MLP network or the multi-head self-attention (MHSA, multiply head self-attention). The input of the attention layer is: the combined feature f con obtained by the concatenation layer, which contains representation information of different dimensions, and its representation ability is relatively stronger than that using only the interactive features of word IDs. The output of the attention layer is the weight values corresponding to different features.

[0149] Taking the attention mechanism network (AN) as an example, as shown in the calculation of the initial weights in Equation 10, where α’ is the initial weight, which is obtained through three-layer fully connected calculation. The number of nodes in the first two layers can be adjusted, and the number of nodes in the last layer is 1. At the same time, to reduce overfitting, a Dropout layer can be added after the first two layers. The number of fully connected layers selected can be adjusted, denoted by H. The input f of Equation 10 con has a dimension of Batch_size*k*1, and the dimension of the output result a’ is also Batch_size*k*1. Batch_size is the number of samples in each batch; k is the number of different sub-prediction results.

[0150] As shown in Equation 11, the initial weight α’ is normalized through Softmax to obtain the final weight, denoted as α, which is used as the output result. Then, as shown in Equation 12, the original feature fcon is updated based on the attention weights to obtain the updated feature f’con, which is used as the input to the depth model prediction layer.

[0151] In some embodiments, the calculation formula of the depth model prediction layer is:

[0152] f (l+1) =s(W (l) f (l) +b (l) ) (13)

[0153]

[0154] where l is the hidden layer; σ is the activation function, f (l) 、b (l) and W (l) are the features, biases, and weights of the l-th layer respectively, f (l +1) represents the feature value of the (l + 1)-th layer, l last is the last hidden layer, y dnn is the classification prediction result, Dense is the fully connected layer, and unit is the number of nodes.

[0155] Specifically, the depth model prediction layer mainly obtains the high-order interaction features of the data and is implemented by using a classical forward multi-hidden layer fully connected network. The feature calculation of each hidden layer is as shown in Equation 13. As shown in Equation 14, the last layer l last is followed by a fully connected layer with 1 node, and its output is used as the sub-model prediction result ydnn, where Dense represents the fully connected layer and unit is the number of nodes. To overcome overfitting, dropout is added to each layer.

[0156] S3. Train the initial model using the training sample set and evaluate the model using the validation sample set to obtain a trained text classification model.

[0157] Specifically, after data preparation and model design are both completed, the model training and evaluation stage can be entered.

[0158] Among them, the training sample set in the data preparation stage is used for model training.

[0159] Optionally, the mini-batch gradient descent method is used for training, and the data size of each batch can be adjusted according to the running environment.

[0160] Optionally, considering that the scenario belongs to text classification, the cross-entropy loss function is used. As shown in Formula 15, is the predicted value, yi is the true value, and N is the number of training samples.

[0161]

[0162] Optionally, the learning rate is guaranteed to be dynamically adjustable by designing a learning rate decay function, so that the learning rate can be set larger in the early stage of training and adjusted smaller appropriately in the later stage.

[0163] Optionally, to improve the model training accuracy, the training set is trained for multiple rounds.

[0164] In addition, the validation set in the data preparation stage is used for model evaluation.

[0165] Optionally, the evaluation metrics selected are Logloss and Auc. Logloss is the cross-entropy loss function, and Auc is the area under the Roc curve. As the model is trained, Logloss becomes smaller and Auc becomes larger, indicating that the model performance is getting better.

[0166] Optionally, for every 1 round of training on the training set dataset, an evaluation is performed on the validation set dataset, that is, the evaluation metrics are calculated.

[0167] Optionally, the early stopping mechanism is adopted for model evaluation, that is: when the change in the model performance evaluation metrics is small for several consecutive rounds of model training, the early stopping condition is triggered and the model training stops; if the model performance continues to change, the training continues until the pre-designed number of training rounds is reached.

[0168] In some embodiments, a model result post-processing stage is further included.

[0169] Specifically, model result post-processing mainly refers to saving and displaying the results after model training and evaluation stop. This stage includes:

[0170] (1) Result saving. Mainly save the relevant files of the trained model results and the relevant metrics for model validation, such as model evaluation metrics like Logloss and Auc, which are used to guide the improvement of the model in the later stage.

[0171] (2) Result visualization. Display the model training evaluation results and the intermediate features during model training in the form of graphs or other visualizations, so that it is possible to clearly obtain information such as which features are involved in modeling and the model effect.

[0172] This application proposes a new method for training a text classification model improved based on the NFM model. The trained text classification model can be applied to various scenarios such as text-based credit risk prediction and sentiment analysis. This method is designed based on the NFM model, comprehensively considering the ID information and word frequency information of words, deeply mining the word frequency information, applying various feature expressions; and fusing the features of various expression forms through the attention mechanism, and using them as the input of the deep model prediction layer for result prediction.

[0173] Figure 3 It is a schematic diagram of the text classification method provided by the embodiments of this application. As Figure 3 shown, this application provides a text classification method, and the method includes:

[0174] S4. Obtain the text data to be classified;

[0175] S5. Classify the text data through the text classification model to obtain the text classification result;

[0176] Among them, the text classification model is trained by the text classification model training method described in the above embodiments.

[0177] This embodiment uses the text classification model training method described in the above embodiments to train a text classification model for text classification. Compared with the prior art, it has the following advantages:

[0178] 1) This application adds word frequency information, deeply mines the word frequency information, and improves the model representation ability. This application takes the word frequency information as the input to obtain different representation information of various word frequencies, including the original input form and the form after embedding transformation. After the original NFM obtains the interaction information based on BI, it only obtains high-order features through DNN based on the BI interaction information, without considering the information of the original input. Compared with the prior art, this application not only introduces the word frequency information, but also applies the original information of the word frequency as the input of the deep model.

[0179] 2) It comprehensively considers that different forms of expression of features contribute differently to model prediction. In this application, an attention network is used to provide weights for different features and update the feature values based on the weights. Finally, a decision is made through the deep model prediction layer. The deep model prediction layer comprehensively considers various information to predict the result. It has certain advantages compared with the traditional method that does not consider the contribution differences of different features.

[0180] In summary, the method of this application can effectively improve the accuracy of text data classification results.

[0181] It should be understood that although the steps in the flowchart in the above embodiments are displayed in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and they can be executed in other orders. Moreover, at least a part of the steps in the figure may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0182] Figure 4 The following is a schematic diagram of the text classification model training device provided by the embodiments of this application. As Figure 4 shown, this application provides a text classification model training device, and the device includes:

[0183] A data processing module 11, which is configured to perform data processing on the original text data to obtain a training sample set and a validation sample set. Among them, the training sample set and the validation sample set contain word ids and word frequency information;

[0184] A model construction module 12, which is configured to construct an initial model. The initial model includes an input layer, an embedding layer, a bilinear interpolation Bi interaction layer, a splicing layer, an attention layer, and a deep model prediction layer;

[0185] Among them, the embedding layer is used to perform dimensionality reduction processing on the word ids and word frequency information input into the input layer to obtain embedding vectors;

[0186] The Bi interaction layer is used to obtain the interaction features of word ids and the interaction features of word frequency information according to the embedding vectors;

[0187] The splicing layer is used to splice the interaction features of word ids and the interaction features of word frequency information to obtain splicing features;

[0188] The attention layer is used to generate weight values for the spliced features based on the attention mechanism, and update the features based on the weight values to obtain updated features;

[0189] The deep model prediction layer is used to obtain a classification prediction result according to the updated features;

[0190] The training and evaluation module 13 is configured to train the initial model through the training sample set and evaluate the model through the validation sample set to obtain a trained text classification model.

[0191] Regarding the definition of the text classification model training device, reference can be made to the definition of the text classification model training method in the above embodiments of the present application, which will not be elaborated herein.

[0192] Figure 5 is a schematic diagram of the text classification device provided by the embodiment of the present application, as Figure 5 shown, the present application provides a text classification device, and the device includes:

[0193] The data acquisition module 14 is configured to acquire text data to be classified;

[0194] The text classification module 15 is configured to classify the text data through a text classification model to obtain a text classification result;

[0195] Among them, the text classification model is trained by the above text classification model training method.

[0196] Regarding the definition of the text classification device, reference can be made to the definition of the text classification method in the above embodiments of the present application, which will not be elaborated herein.

[0197] Figure 6 is another schematic diagram of the text classification model training device provided by the embodiment of the present application, as Figure 6 shown, in some embodiments, the present application provides a text classification model training device, including a memory 22 and a processor 21. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the text classification model training method in the above embodiments of the present application.

[0198] Figure 7 is another schematic diagram of the text classification device provided by the embodiment of the present application, as Figure 7 shown, in some embodiments, the present application provides a text classification device, including a memory 23 and a processor 24. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the text classification method in the above embodiments of the present application.

[0199] Among them, the memory is connected to the processor. The memory can be a flash memory, a read-only memory, or other memories, and the processor can be a central processing unit or a single-chip microcomputer.

[0200] In some embodiments, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the text classification model training method and / or the text classification method in the above various embodiments of the present application.

[0201] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, computer program modules, or other data. The computer-readable storage medium includes, but is not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory, or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), digital versatile disc (DVD), or other optical disc storage, magnetic cassette, tape, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0202] It can be understood that the above embodiments are merely exemplary embodiments adopted to illustrate the principle of the present application. However, the present application is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present application, and these modifications and improvements are also regarded as the protection scope of the present application.

Claims

1. A method for training a text classification model, characterized in that, the method includes: S1. Perform data processing on the original text data to obtain a training sample set and a validation sample set, where the training sample set and the validation sample set contain word IDs and word frequency information; S2. Construct an initial model, where the initial model includes an input layer, an embedding layer, a bilinear interpolation Bi interaction layer, a concatenation layer, an attention layer, and a deep model prediction layer; wherein, the embedding layer is used to perform dimensionality reduction processing on the word IDs and word frequency information input into the input layer to obtain embedding vectors; the Bi interaction layer is used to obtain the interaction features of word IDs and the interaction features of word frequency information according to the embedding vectors; the concatenation layer is used to perform feature concatenation on the interaction features of word IDs and the interaction features of word frequency information to obtain concatenated features; the attention layer is used to generate weight values for the concatenated features based on the attention mechanism, and perform feature update based on the weight values to obtain updated features; the deep model prediction layer is used to obtain a classification prediction result according to the updated features; S3. Train the initial model with the training sample set and evaluate the model with the validation sample set to obtain a trained text classification model.

2. The method for training a text classification model according to claim 1, characterized in that, the calculation formula of the embedding layer is: e_id i = w_id i x_id i e_value i = w_id i x_value i e = [e_id, e_value] e_id = [e_id 1 , e_id 2 ,..., e_id N ​ e_value = [e_value 1 , e_value 2 ,..., e_value N ​ Among them, x_id i is the i-th word, w_id i is the embedding weight matrix of the word id, e_id i is the embedding vector of the i-th word id, x_value i is the word frequency of the i-th word id, e_value i is the embedding vector of the word frequency of the i-th word id, e is the embedding vector, including the embedding vector e_id of the word id and the embedding vector e_value of the word frequency.

3. The method for training a text classification model according to claim 1, characterized in that, the calculation formula of the Bi interaction layer is: Among them, f_id BI is the interaction feature of the word id, f_value BI is the interaction feature of the word frequency information, x_id i is the i-th word, e_id i is the embedding vector of the i-th word id, x_value i is the word frequency of the i-th word id, e_value i is the embedding vector of the word frequency of the i-th word id, and BN() represents BatchNormal batch normalization processing on the result.

4. The method for training a text classification model according to claim 1, characterized in that, the calculation formula of the concatenation layer is: f con = Concat(f_id BI , f_vlue BI , e_id, e_value, x_value) f con = Concat(f_id BI , f_vlue BI , x_value) Among them, fcon is the concatenated feature, and f_id BI is the interaction feature of the word id, and f_value BI is the interaction feature of the word frequency information, e_id is the embedding vector of the word id, e_value is the embedding vector of the word frequency, and x_value is the word frequency information of the word id.

5. The method for training a text classification model according to claim 1, characterized in that, the calculation formula of the attention layer is: α′ i = HReLU(Wf con,i + b) f′ con,i = a′ i * f con,i where α’ is the initial weight, fcon is the concatenated feature, α is the final weight value, and f’con is the updated feature.

6. The method for training a text classification model according to claim 1, characterized in that, the calculation formula of the deep model prediction layer is: f (l+1) = s(W (l) f (l) + b (l) ) Among them, l is the hidden layer; σ is the activation function, f (l) , b (l) and W (l) are the feature, bias, and weight of the l-th layer respectively, and f (l+1) represents the eigenvalue of the (l + 1)-th layer. l last is the last hidden layer, y dnn is the classification prediction result, Dense is the fully connected layer, and unit is the number of nodes.

7. The method for training a text classification model according to any one of claims 1-6, characterized in that, S1 includes: S11. Generate a data matrix according to the original text data; S12. Set the word vector length N, count the TOP N words with the highest word frequency for each text data dimension, generate the TOP keywords for each text, and obtain a word vector matrix; S13. Perform marking processing on the data in the word vector matrix, and split the marked data to obtain the training sample set and the validation sample set.

8. The method for training a text classification model according to claim 7, characterized in that, S11 includes: S111. Perform word segmentation processing and filtering processing on the original text data to obtain candidate words; S112. Perform ID encoding on the candidate words to generate a text dictionary; S113. Count the word frequencies of each candidate word in the original text data in the text dictionary, and perform normalization processing on the word frequency data to obtain a data matrix.

9. A text classification method, characterized in that, the method includes: obtaining text data to be classified; classifying the text data through a text classification model to obtain a text classification result; wherein, the text classification model is trained by the text classification model training method according to any one of claims 1-8.

10. A text classification model training device, characterized in that, the device includes: a data processing module configured to perform data processing on original text data to obtain a training sample set and a validation sample set, wherein the training sample set and the validation sample set contain word ids and word frequency information; a model construction module configured to construct an initial model, the initial model including an input layer, an embedding layer, a bilinear interpolation Bi interaction layer, a splicing layer, an attention layer, and a deep model prediction layer; wherein, the embedding layer is used to perform dimensionality reduction processing on the word ids and word frequency information input into the input layer to obtain embedding vectors; the Bi interaction layer is used to obtain the interaction features of word ids and the interaction features of word frequency information according to the embedding vectors; the splicing layer is used to perform feature splicing on the interaction features of word ids and the interaction features of word frequency information to obtain splicing features; the attention layer is used to generate weight values for the splicing features based on an attention mechanism, and perform feature update based on the weight values to obtain updated features; the deep model prediction layer is used to obtain a classification prediction result according to the updated features; a training and evaluation module configured to perform model training on the initial model through the training sample set and perform model evaluation through the validation sample set to obtain a trained text classification model.

11. A text classification device, characterized in that, the device includes: a data acquisition module configured to acquire text data to be classified; a text classification module configured to classify the text data through a text classification model to obtain a text classification result; wherein, the text classification model is trained by the text classification model training method according to any one of claims 1-8.

12. A text classification model training device, characterized in that, it includes a memory and a processor, a computer program is stored in the memory, and the processor is configured to run the computer program to implement the text classification model training method according to any one of claims 1-8.

13. A text classification device, characterized in that, it includes a memory and a processor, a computer program is stored in the memory, and the processor is configured to run the computer program to implement the text classification method according to claim 9.

14. A computer-readable storage medium, characterized in that, a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the text classification model training method according to any one of claims 1-8 and / or the text classification method according to claim 9.

Images