A text classification method, device, equipment and storage medium

Abstract

The present application relates to the field of artificial intelligence technology, and discloses a text classification method, device, equipment and storage medium, the method comprising: obtaining text data to be classified; extracting keywords from the text data to be classified to obtain word information, encoding the word information to obtain corresponding position information; embedding the word information and position information respectively, and merging the vectors obtained after the embedding process to obtain a text matrix; the text matrix is ​​processed by a pre-trained classification model to obtain a first label probability distribution and a second label probability distribution, and the pre-trained classification model includes a masked multi-head attention structure; based on the first label probability distribution and the second label probability distribution, determine the category to which the text data to be classified belongs. The present application also relates to blockchain technology, and the category data corresponding to the text data to be classified is stored in the blockchain. The present application achieves the goal of improving the classification accuracy while generating new labels.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a text classification method, apparatus, device and storage medium. Background Art

[0002] At present, with the development of the Internet industry and the advancement of science and technology, the information knowledge of all walks of life is growing explosively. In order to meet the diverse needs of users under the background of massive information, it is urgent to effectively manage text information. Therefore, various text classification technologies have been rapidly developed and have become a research hotspot and core technology in the fields of data mining and data retrieval, and have played a significant role in the development of enterprise and social information technology. In the existing technology, text classification is often carried out by machine learning methods such as logistic regression, naive Bayes or random forest, or by using traditional text classification methods, such as TF-IDF, vector space model, etc. The methods in the existing technology are all based on existing labels for classification, and there is no ability to generate new labels, that is, there is a problem that text can only be classified according to existing labels when classifying text. Summary of the invention

[0003] The present application provides a text classification method, apparatus, device and storage medium to solve the problem in the prior art that when classifying text, only existing tags can be used to classify the text.

[0004] To solve the above problems, this application provides a text classification method, including:

[0005] Obtain the text data to be classified;

[0006] Extract keywords from the text data to be classified to obtain word information, encode the word information to obtain corresponding position information;

[0007] The word information and the position information are respectively embedded, and the vectors obtained after the embedding process are merged to obtain a text matrix;

[0008] The text matrix is ​​processed by a pre-trained classification model to obtain a first label probability distribution and a second label probability distribution, wherein the pre-trained classification model includes a masked multi-head attention structure;

[0009] Based on the first label probability distribution and the second label probability distribution, the category to which the to-be-classified text data belongs is determined.

[0010] Furthermore, the text matrix is ​​processed by a pre-trained classification model including:

[0011] Normalizing the text matrix through a normalization layer in the pre-trained classification model to obtain a first matrix;

[0012] The first matrix is ​​processed by performing information extraction through the masked multi-head attention structure in the pre-trained classification model to obtain a second matrix containing context information of the text data to be classified;

[0013] Performing a residual connection on the text matrix and the second matrix to obtain a third matrix, and then normalizing the third matrix through a normalization layer in the pre-trained classification model to obtain a fourth matrix;

[0014] Mapping the fourth matrix through a feedforward network layer in the pre-trained classification model to obtain a fifth matrix;

[0015] The fifth matrix is ​​processed by the activation function layer in the pre-trained classification model to obtain a sixth matrix, and the sixth matrix is ​​residually connected with the fourth matrix to obtain a seventh matrix, and the seventh matrix is ​​normalized by the normalization layer in the pre-trained classification model to obtain an eighth matrix;

[0016] Performing two linear transformations on the eighth matrix to obtain a first label matrix and a second label matrix;

[0017] The first label matrix and the second label matrix are mapped through the Softmax layer in the classification model to obtain the first label probability distribution and the second label probability distribution.

[0018] Furthermore, the first matrix is ​​processed by performing information extraction through the masked multi-head attention structure in the pre-trained classification model, and the second matrix containing the context information of the text data to be classified is obtained, including:

[0019] Multiplying the first matrix with the parameter matrices of multiple batches obtained after pre-training to obtain multiple batches of Q matrices, K matrices, and V matrices;

[0020] By performing a dot multiplication of the Q matrix of each batch with the K matrix, dividing a first result obtained by the dot multiplication by the square root of the corresponding dimension of the Q matrix to obtain a second result, adding the second result to the mask matrix and then performing Softmax calculation to obtain a weight matrix, wherein the mask matrix is ​​constructed based on the word information;

[0021] Multiplying the weight matrix by the V matrix of the corresponding batch to obtain a ninth matrix of each batch;

[0022] The second matrix is ​​obtained by concatenating the ninth matrices of all batches and performing a linear transformation on the concatenated matrices.

[0023] Furthermore, determining the category to which the to-be-classified text data belongs based on the first label probability distribution and the second label probability distribution includes:

[0024] Obtaining a label corresponding to a maximum probability value in the first label probability distribution, and determining whether the maximum probability value is greater than or equal to a preset value;

[0025] If the maximum probability value is less than the preset value, the label corresponding to the maximum probability value in the second label probability distribution is used as the category to which the text data to be classified belongs, and the label corresponding to the maximum probability value in the second label probability distribution is saved in the label vocabulary;

[0026] If the maximum probability value is greater than or equal to the preset value, the label corresponding to the maximum probability value in the first label probability distribution is used as the category to which the text data to be classified belongs.

[0027] Furthermore, extracting keywords from the text data to be classified includes:

[0028] Using the Jieba word segmentation method to perform word segmentation processing on the text data to be classified to obtain a plurality of corresponding words;

[0029] A keyword extraction algorithm is used to extract keywords from the multiple words, a preset ratio of keywords is extracted, and masks are used to replace the keywords in the multiple words.

[0030] Furthermore, the keyword extraction of the text data to be classified to obtain word information, encoding according to the word information, and obtaining corresponding position information includes:

[0031] Based on the weights of the keywords obtained by the keyword extraction algorithm, the keywords are sorted and an identifier is set before each keyword to obtain keyword information;

[0032] Merging the information consisting of multiple words including the mask with the keyword information to obtain word information;

[0033] According to the word information, position encoding is performed to obtain corresponding position information.

[0034] Furthermore, the mask matrix is ​​constructed based on the word information and includes:

[0035] Based on the information consisting of the multiple words including the mask, the contents of the information are mutually associated, and first data is constructed according to the associated relationship;

[0036] Based on the content in the keyword information, the information transmitted in the previous text in the word information is received to construct the second data;

[0037] The first data and the second data are concatenated, and the empty positions after the concatenation are filled with filling content to obtain the mask matrix.

[0038] In order to solve the above problems, the present application also provides a text classification device, the device comprising:

[0039] An acquisition module is used to acquire text data to be classified;

[0040] An information extraction module is used to extract keywords from the text data to be classified to obtain word information, encode the word information, and obtain corresponding position information;

[0041] A vectorization module, used for embedding the word information and the position information respectively, and merging the vectors obtained after the embedding process to obtain a text matrix;

[0042] A classification module, used for processing the text matrix through a pre-trained classification model to obtain a first label probability distribution and a second label probability distribution, wherein the pre-trained classification model includes a masked multi-head attention structure;

[0043] An output module is used to determine the category to which the to-be-classified text data belongs based on the first label probability distribution and the second label probability distribution.

[0044] In order to solve the above problem, the present application also provides a computer device, including:

[0045] at least one processor; and,

[0046] a memory communicatively connected to the at least one processor; wherein,

[0047] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the text classification method as described above.

[0048] In order to solve the above problem, the present application also provides a non-volatile computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the text classification method as described above is implemented.

[0049] A text classification method, apparatus, device, and storage medium provided in the embodiments of the present application have at least the following beneficial effects compared with the prior art:

[0050] By acquiring text data to be classified, performing keyword extraction on the text data to be classified to obtain word information, and then performing position encoding according to the word information to obtain corresponding position information, by obtaining the word information and position information, the subsequent classification model can better understand the text to be classified; by embedding the word information and position information respectively, and merging the vectors obtained after the embedding process to obtain a text matrix, the text matrix is ​​input into a pre-trained classification model for processing to obtain a first label probability distribution and a second label probability distribution, the first label probability distribution is the probability of an existing label, and the second label probability is the probability of a newly generated label for the text; based on the first label probability distribution and the second label probability distribution, the category to which the text data to be classified belongs is determined, which can improve the classification accuracy while not only having the ability to classify into specified labels, but also having the ability to generate labels, thereby improving the fault tolerance of labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the scheme in the present application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 A flowchart of a text classification method provided in one embodiment of the present application;

[0053] Figure 2 A schematic diagram of the overall network structure provided for an embodiment of the present application;

[0054] Figure 3 A schematic diagram of a mask matrix provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of a module of a text classification device provided in one embodiment of the present application;

[0056] Figure 5 A schematic diagram of the structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0058] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly or implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0059] This application provides a text classification method. Figure 1 and Figure 2 As shown, Figure 1 A flowchart of a text classification method provided in an embodiment of the present application is shown below. Figure 2 A schematic diagram of the overall network structure provided for one embodiment of the present application.

[0060] In this embodiment, the text classification method includes:

[0061] S1. Obtain text data to be classified;

[0062] Specifically, the text data to be classified is obtained from a database, or the text data to be classified is directly received from a user.

[0063] When obtaining the text data to be classified from the database, a call request needs to be sent to the database, and the call request carries a signature verification token; the signature verification result returned by the database is received, and when the signature verification result is passed, the text data to be classified in the database is called.

[0064] And also get the existing tags.

[0065] The security of the data is ensured by encrypting the database and verifying the signature when extracting the text data to be classified from the database.

[0066] S2, extracting keywords from the text data to be classified to obtain word information, encoding the word information to obtain corresponding position information;

[0067] Specifically, keywords are extracted by using a keyword extraction algorithm, and the keywords in the text to be classified are replaced by masks, and the extracted keywords are combined into keyword information, and the text to be classified containing the mask and the keyword information are merged and spliced ​​to obtain word information; and the word information is then encoded to obtain the corresponding position information.

[0068] Furthermore, extracting keywords from the text data to be classified includes:

[0069] Using the Jieba word segmentation method to perform word segmentation processing on the text data to be classified to obtain a plurality of corresponding words;

[0070] A keyword extraction algorithm is used to extract keywords from the multiple words, a preset ratio of keywords is extracted, and masks are used to replace the keywords in the multiple words.

[0071] Specifically, firstly, the text data to be processed is segmented by Jieba word segmentation, and the text data to be processed is divided into separate characters, thereby obtaining a plurality of words; then, a keyword extraction algorithm is used to extract keywords from the plurality of words. In the present application, the keyword extraction algorithm is the TF-IDF algorithm; the core keywords in the text are extracted by the TF-IDF algorithm, and the preset ratio is 10%-15%, that is, the keywords are sorted according to their weights, and the first 10%-15% with high weights are extracted. After the extraction is completed, a mask is used to replace the original position of the keyword, thereby obtaining a plurality of words containing the mask and the extracted keywords.

[0072] For example, there are w1, w2, w3, w4, w5, w6, w7...w T Multiple words, among which two keywords consisting of w2, w3 and w6, w7 are extracted, and multiple words containing masks are obtained: w1, [MASK], w4, w5, [MASK] ... w T And the extracted keywords w2, w3, w6, w7.

[0073] By preprocessing the classified text data for keyword extraction, it is convenient to obtain word information and position information later.

[0074] Furthermore, the keyword extraction of the text data to be classified to obtain word information, encoding according to the word information, and obtaining corresponding position information includes:

[0075] Based on the weights of the keywords obtained by the keyword extraction algorithm, the keywords are sorted and an identifier is set before each keyword to obtain keyword information;

[0076] Merging the information consisting of multiple words including the mask with the keyword information to obtain word information;

[0077] According to the word information, position encoding is performed to obtain corresponding position information.

[0078] Specifically, after the keywords are extracted, they are sorted according to their weights, and an identifier [START] is set before each keyword. For example, two groups of keywords w2, w3 and w6, w7 are sorted according to their weights, which are 2 and 1 respectively; and the identifier [START] is added to obtain keyword information [START], w6, w7, [START], w2, w3;

[0079] The information consisting of multiple words including the mask is combined with the keyword information to obtain word information w1, [MASK], w4, w5, [MASK] ... w T , [START], w6, w7, [START], w2, w3, and position encoding based on word information to obtain two groups of codes. The first group is encoded according to the real position to obtain 1, 2, 3, 4, 5 ... T, 5, 5, 5, 2, 2, 2; for the encoding of keywords, its encoding is consistent with the mask encoding of the keyword in the position of multiple words, that is, w6, w7 corresponds to the encoding of the second mask, that is, 5; w2, w3 corresponds to the encoding of the first mask, that is, 2. The second group is encoded according to the identifier start, and 0, 0, 0, 0, 0 ... 0, 1, 2, 3, 1, 2, 3 is obtained.

[0080] The word information is obtained by concatenating the information and the position information is obtained based on the word information. The introduction of the position information facilitates the subsequent classification model to recognize the position and also facilitates the subsequent acquisition of the mask matrix based on the word information.

[0081] S3, embedding the word information and position information respectively, and merging the vectors obtained after the embedding process to obtain a text matrix;

[0082] Specifically, the word information and position information are processed separately through the vector conversion model. For example, the pre-trained language model Bert is used to embed the word information to obtain the vector x1, with a dimension of [L, D], where L is the length of the input data and D is the dimension of the vector after embedding. The position information is also embedded to obtain x2 and x3, respectively, with the same dimension as x1, [L, D]. x1, x2 and x3 are merged to obtain the text matrix.

[0083] S4. The text matrix is ​​processed by a pre-trained classification model to obtain a first label probability distribution and a second label probability distribution, wherein the pre-trained classification model includes a masked multi-head attention structure;

[0084] Specifically, a masked multi-head attention structure is introduced into the classification model to extract the contextual information of the text. The text matrix is ​​processed by a pre-trained classification model to obtain a first label probability distribution and a second label probability distribution. The first label probability distribution is the probability obtained for the existing labels, and the second label probability is the probability of the newly generated labels obtained close to the text matrix.

[0085] Furthermore, the text matrix is ​​processed by a pre-trained classification model including:

[0086] Normalizing the text matrix through a normalization layer in the pre-trained classification model to obtain a first matrix;

[0087] The first matrix is ​​processed by performing information extraction through the masked multi-head attention structure in the pre-trained classification model to obtain a second matrix containing context information of the text data to be classified;

[0088] Performing a residual connection on the text matrix and the second matrix to obtain a third matrix, and then normalizing the third matrix through a normalization layer in the pre-trained classification model to obtain a fourth matrix;

[0089] Mapping the fourth matrix through a feedforward network layer in the pre-trained classification model to obtain a fifth matrix;

[0090] The fifth matrix is ​​processed by the activation function layer in the pre-trained classification model to obtain a sixth matrix, and the sixth matrix is ​​residually connected with the fourth matrix to obtain a seventh matrix, and the seventh matrix is ​​normalized by the normalization layer in the pre-trained classification model to obtain an eighth matrix;

[0091] Performing two linear transformations on the eighth matrix to obtain a first label matrix and a second label matrix;

[0092] The first label matrix and the second label matrix are mapped through the Softmax layer in the classification model to obtain the first label probability distribution and the second label probability distribution.

[0093] Specifically, first, the text matrix is ​​normalized by a normalization layer, namely, a Norm (normalization) layer, which can improve the convergence speed and accuracy of the model to a certain extent, and obtain the first matrix x4. Specifically

[0094] Where μ represents the mean of the text matrix, and σ represents the standard deviation of the text matrix.

[0095] The first matrix is ​​processed by the masked multi-head attention structure in the pre-trained classification model to extract information, and a second matrix x5 containing context information of the text data to be classified is obtained. The masked multi-head attention structure is basically the same as the encoder part of Bert, such as Figure 3 As shown in the figure, the difference is that a mask matrix is ​​added to cover up future information, that is, only the information transmitted by the previous text can be received, thereby converting a single encoder / decoder into a generative model.

[0096] In the present application, after obtaining the second matrix x5, Dropout processing can also be performed to effectively suppress overfitting.

[0097] The text matrix and the second matrix x5 after Dropout processing are residually connected to obtain a third matrix, and then the third matrix is ​​normalized through the Norm layer in the pre-trained classification model to obtain a fourth matrix x6.

[0098] Where μ′ represents the mean after the first residual connection, σ′ represents the standard deviation after the first residual connection, and Add&Norm: residual connection is used to improve the gradient vanishing problem.

[0099] The normalized fourth matrix x6 is input into the feedforward network in the classification model, namely the FNN (feedforward neural network) network, to obtain the fifth matrix x7. The FNN network is a neural network that undergoes two layers of linear mapping. The specific calculation formula is: x7 = f(W2·f(W1·x6+b1)+b2), where W1, W2, b1, b2 are training parameters, which are the parameters that have converged after pre-training the classification model.

[0100] The fifth matrix x7 is processed by the activation function layer in the pre-trained classification model to obtain the sixth matrix, and the sixth matrix is ​​residually connected with the fourth matrix x6 to obtain the seventh matrix, and the seventh matrix is ​​normalized by the Norm layer in the pre-trained classification model to obtain the eighth matrix x8; the specific algorithm is: Wherein μ″ represents the mean after the second residual connection, σ″ represents the standard deviation after the second residual connection, and the dimension of the eighth matrix x8 is [L1, D], where L1 represents the number of words to be generated.

[0101] The eighth matrix x8 is subjected to two linear transformations to obtain a first label matrix x9 and a second label matrix x 10 ; The dimension of the first label matrix x9 is [L2, D1], where L2 represents the number of generated words and D1 represents the number of labels. The second label matrix x10 The dimension is [L3, D2], where L3 represents the number of generated words and D2 represents the number of vocabularies.

[0102] The first label matrix and the second label matrix are mapped through the Softmax layer in the classification model to obtain the first label probability distribution and the second label probability distribution, and the dimensions of the first label probability distribution and the second label probability distribution are respectively the same as the first label matrix x9 and the second label matrix x 10 Consistent.

[0103] The Norm layer, the masked multi-head attention structure, the first add&Norm layer, the FNN layer, the GeLU layer and the second add&Norm layer in the pre-trained classification model can be regarded as a group. In other embodiments of the present application, multiple groups can be set in the classification model to obtain better results, such as Figure 2 As shown, multiple Model layers can be set.

[0104] The data is processed through each layer in the classification model to obtain the first label matrix x9 and the second label matrix x 10 , to achieve accurate classification of classified data, and the classification of classified data is not limited to existing labels, but new labels can be generated.

[0105] Furthermore, the first matrix is ​​processed by performing information extraction through the masked multi-head attention structure in the pre-trained classification model to obtain a second matrix containing context information of the text data to be classified, including:

[0106] Multiplying the first matrix with the parameter matrices of multiple batches obtained after pre-training to obtain multiple batches of Q matrices, K matrices, and V matrices;

[0107] By performing a dot multiplication of the Q matrix of each batch with the K matrix, dividing a first result obtained by the dot multiplication by the square root of the corresponding dimension of the Q matrix to obtain a second result, adding the second result to the mask matrix and then performing Softmax calculation to obtain a weight matrix, wherein the mask matrix is ​​constructed based on the word information;

[0108] Multiplying the weight matrix by the V matrix of the corresponding batch to obtain a ninth matrix of each batch;

[0109] The second matrix is ​​obtained by concatenating the ninth matrices of all batches and performing a linear transformation on the concatenated matrices.

[0110] Specifically, before the first training of the classification model, the parameter matrices of multiple batches are randomly generated, or they can be randomly generated according to normal distribution or uniform distribution. During the continuous training of the classification model, the parameter matrices of multiple batches continuously change and converge until the classification model training is completed and the parameter matrices of multiple batches tend to be stable. In the subsequent use of the trained classification model, the stable parameter matrices of multiple batches can be directly used.

[0111] By multiplying the first matrix x4 by the parameter matrices of multiple batches, we can obtain the three matrices Q, K, and V of multiple batches. The formula is Q = x4·W Q , K = x4·W K ,V=x4·W V , where W Q , W K , W V That is the parameter matrix. Then use the three matrices Q, K, and V to calculate the proportion of each word in the input text. The weight of the word that only receives the above information is 0. The specific calculation formula is:

[0112] Where A represents the weight matrix, d k Represents the Q, K or V matrix dimension, and then multiplies the weight matrix with the V matrix of the corresponding batch to obtain the ninth matrix of each batch, concatenates the ninth matrices of all batches, and then performs a linear transformation on the concatenated matrix to obtain the second matrix x5.

[0113] By introducing the Masked Multi-Head Attention structure, we can extract the contextual information of the text, and add a mask to cover the future information, which can achieve better classification of the text and improve the accuracy of text classification.

[0114] Furthermore, performing two linear transformations on the eighth matrix to obtain a first label matrix and a second label matrix includes:

[0115] After averaging each word vector in the eighth matrix, a linear transformation is performed to obtain the first label matrix;

[0116] Perform a linear transformation on each word vector in the eighth matrix to obtain the second label matrix.

[0117] Specifically, since each label word is composed of multiple characters, the vectors of each character before the identifier [START] are averaged and then linearly transformed to obtain the first label matrix x9; the other one only performs linear transformation processing, and directly linearly transforms each character vector in the eighth matrix to obtain the second label matrix x 10 .

[0118] By performing two different linear transformations, two different label matrices are obtained to achieve classification of existing labels and newly generated labels.

[0119] Furthermore, before performing residual connection between the text matrix and the second matrix, the method further includes:

[0120] The second matrix is ​​processed through a Dropout layer in the classification model to suppress overfitting of the classification model.

[0121] Specifically, by inserting the Dropout layer before the residual connection, the overfitting of the classification model can be effectively suppressed.

[0122] Furthermore, the mask matrix is ​​constructed based on the word information to include:

[0123] Based on the information consisting of the multiple words including the mask, the contents of the information are mutually associated, and first data is constructed according to the associated relationship;

[0124] Based on the content in the keyword information, the information transmitted in the previous text in the word information is received to construct the second data;

[0125] The first data and the second data are concatenated, and the empty positions after the concatenation are filled with filling content to obtain the mask matrix.

[0126] Specifically, Figure 3 As shown, since the word information is divided into two parts, one part is the multiple word parts containing masks, and the other part is the keyword information part. For the multiple word parts containing masks, they can be associated with each other, that is, context information can be extracted, and the first data can be constructed according to the associated information; for the keyword information part, future information needs to be masked, that is, the keyword information part is transmitted in one direction, and only the information transmitted by the previous text is received, so the second data is constructed in this way, the first data and the second data are spliced, and the empty positions after splicing are filled with filling content to obtain the mask matrix. Further, the first data and the second data parts are directly represented by 0, and the filling content part is represented by the server label, that is, -∞.

[0127] By constructing a mask matrix to cover up future information, a single encoder / decoder is converted into a generative model.

[0128] S5. Determine the category to which the to-be-classified text data belongs based on the first label probability distribution and the second label probability distribution.

[0129] Specifically, since the first label probability distribution is a probability distribution based on existing labels, and the second label probability distribution is a probability distribution of newly generated labels, labels are selected based on the first label probability distribution and the second label probability distribution. When the first label probability distribution does not meet the requirements, labels are selected based on the second label probability distribution.

[0130] Furthermore, determining the category to which the to-be-classified text data belongs based on the first label probability distribution and the second label probability distribution includes:

[0131] Obtaining a label corresponding to a maximum probability value in the first label probability distribution, and determining whether the maximum probability value is greater than or equal to a preset value;

[0132] If the maximum probability value is less than the preset value, the label corresponding to the maximum probability value in the second label probability distribution is used as the category to which the text data to be classified belongs, and the label corresponding to the maximum probability value in the second label probability distribution is saved in the label vocabulary;

[0133] If the maximum probability value is greater than or equal to the preset value, the label corresponding to the maximum probability value in the first label probability distribution is used as the category to which the text data to be classified belongs.

[0134] Specifically, since the first label probability distribution is a probability distribution based on existing labels, it is determined whether the maximum probability value in the first label probability distribution is greater than or equal to a preset value. If it is greater than or equal to the preset value, the label corresponding to the maximum probability value in the first label probability distribution is directly used as the category to which the text data to be classified belongs. If it is less than the preset value, the label corresponding to the maximum probability value in the second label probability distribution is directly extracted as the category corresponding to the text data to be classified, and the label is saved in the label word list. In this application, the preset value is 0.5.

[0135] By comparing the first label probability distribution with a preset value, different labels are obtained according to the comparison result, ensuring that the label closest to the text data to be classified is selected, thereby improving the effectiveness of text classification.

[0136] It should be emphasized that in order to further ensure the privacy and security of the data, all data of the category of the text data to be classified can also be stored in a node of a blockchain.

[0137] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0138] The text classification method of this embodiment obtains text data to be classified, extracts keywords from the text data to be classified, obtains word information, and then performs position encoding based on the word information to obtain corresponding position information. By obtaining the word information and position information, the subsequent classification model can better understand the text to be classified; the word information and position information are respectively embedded, and the vectors obtained after the embedding process are merged to obtain a text matrix, and the text matrix is ​​input into a pre-trained classification model for processing to obtain a first label probability distribution and a second label probability distribution, wherein the first label probability distribution is the probability of an existing label, and the second label probability is the probability of a newly generated label for the text; based on the first label probability distribution and the second label probability distribution, the category to which the text data to be classified belongs is determined. While ensuring the classification accuracy, the method not only has the function of classifying into specified labels, but also has the ability to generate labels, thereby improving the fault tolerance of labeling.

[0139] This embodiment also provides a text classification device, such as Figure 4 The figure shows a functional module diagram of the text classification device of the present application.

[0140] The text classification device 100 described in the present application can be installed in an electronic device. According to the functions to be implemented, the text classification device 100 may include an acquisition module 101, an information extraction module 102, a vectorization module 103, a classification module 104 and an output module 105. The module described in the present application may also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0141] In this embodiment, the functions of each module / unit are as follows:

[0142] An acquisition module 101 is used to acquire text data to be classified;

[0143] Specifically, the acquisition module 101 acquires the text data to be classified from a database, or directly receives the text data to be classified input by a user.

[0144] The information extraction module 102 is used to extract keywords from the text data to be classified to obtain word information, encode the word information, and obtain corresponding position information;

[0145] Specifically, the information extraction module 102 extracts keywords by using a keyword extraction algorithm, replaces the keywords in the text to be classified with a mask, and combines the extracted keywords into keyword information, merges and concatenates the text to be classified containing the mask with the keyword information to obtain word information; and then encodes the word information to obtain corresponding position information.

[0146] Furthermore, the information extraction module 102 includes a word segmentation submodule and an extraction and replacement submodule;

[0147] The word segmentation submodule is used to perform word segmentation processing on the text data to be classified using the Jieba word segmentation to obtain a corresponding plurality of words;

[0148] The extraction and replacement submodule is used to extract keywords from the multiple words using a keyword extraction algorithm, extract keywords with a preset ratio, and replace the keywords in the multiple words with masks.

[0149] Specifically, first, the word segmentation submodule performs word segmentation processing on the text data to be processed through the Jieba word segmentation, and divides the text data to be processed into separate characters, thereby obtaining a plurality of words; then the extraction and replacement submodule uses the keyword extraction algorithm to extract keywords from the plurality of words. In the present application, the keyword extraction algorithm is the TF-IDF algorithm; the extraction and replacement submodule extracts the core keywords in the text through the TF-IDF algorithm, and the preset ratio is 10%-15%, that is, the keywords are sorted according to their weights, and the first 10%-15% with high weights are extracted. After the extraction is completed, the original position of the keyword is replaced with a mask, thereby obtaining a plurality of words containing the mask and the extracted keywords.

[0150] The word segmentation submodule and the extraction and replacement submodule are used to pre-process the classified text data for keyword extraction, so as to facilitate the subsequent acquisition of word information and position information.

[0151] Furthermore, the information extraction module 102 includes a key information submodule, a word information submodule and a coding submodule;

[0152] The key information submodule is used to sort the keywords based on the weights of the keywords obtained by the keyword extraction algorithm, and set an identifier before each keyword to obtain keyword information;

[0153] The word information submodule is used to merge the information consisting of multiple words including the mask with the keyword information to obtain the word information;

[0154] The encoding submodule is used to perform position encoding according to the word information to obtain corresponding position information.

[0155] Specifically, after extracting the keywords, the key information submodule sorts the keywords according to their weights and sets an identifier [START] in front of each keyword; the word information submodule merges the information consisting of multiple words including the mask with the keyword information to obtain the word information; the encoding submodule performs position encoding based on the word information to obtain two groups of codes, the first group is encoded according to the actual position, and the second group is encoded according to the identifier start.

[0156] The key information submodule, the word information submodule and the encoding submodule are used to splice the information to obtain the word information and obtain the position information based on the word information. The introduction of the position information facilitates the subsequent classification model to recognize the position and also facilitates the subsequent acquisition of the mask matrix based on the word information.

[0157] A vectorization module 103, used for embedding the word information and the position information respectively, and merging the vectors obtained after the embedding process to obtain a text matrix;

[0158] Specifically, the vectorization module 103 processes the word information and position information separately through the vector conversion model. For example, the pre-trained language model Bert is used to embed the word information to obtain a vector x1 with a dimension of [L, D], where L is the length of the input data and D is the dimension of the vector after embedding. The position information is also embedded to obtain x2 and x3, respectively, with the same dimension as x1, [L, D]. x1, x2 and x3 are merged to obtain a text matrix.

[0159] A classification module 104, configured to process the text matrix through a pre-trained classification model to obtain a first label probability distribution and a second label probability distribution, wherein the pre-trained classification model includes a masked multi-head attention structure;

[0160] Specifically, the classification module 104 extracts the contextual information of the text by introducing a masked multi-head attention structure into the classification model. The text matrix is ​​processed by a pre-trained classification model to obtain a first label probability distribution and a second label probability distribution. The first label probability distribution is the probability obtained for the existing labels, and the second label probability is the probability of the newly generated labels obtained close to the text matrix.

[0161] Further, the classification module 104 includes a normalization submodule, an information extraction submodule, a first connection submodule, a mapping submodule, a second connection submodule, a linear transformation submodule and an activation submodule;

[0162] The normalization submodule is used to normalize the text matrix through the normalization layer in the pre-trained classification model to obtain a first matrix;

[0163] The information extraction submodule is used to perform information extraction processing on the first matrix through the masked multi-head attention structure in the pre-trained classification model to obtain a second matrix containing context information of the text data to be classified;

[0164] The first connection submodule is used to perform residual connection on the text matrix and the second matrix to obtain a third matrix, and then normalize the third matrix through a normalization layer in the classification model to obtain a fourth matrix;

[0165] The mapping submodule is used to perform mapping processing on the fourth matrix through the feedforward network layer in the pre-trained classification model to obtain a fifth matrix;

[0166] The second connection submodule is used for processing the fifth matrix through the activation function layer in the pre-trained classification model to obtain a sixth matrix, and performing residual connection on the sixth matrix and the fourth matrix to obtain a seventh matrix, and normalizing the seventh matrix through the normalization layer in the pre-trained classification model to obtain an eighth matrix;

[0167] The linear transformation submodule is used to perform two linear transformations on the eighth matrix to obtain a first label matrix and a second label matrix;

[0168] The activation submodule is used to map the first label matrix and the second label matrix through the Softmax layer in the classification model to obtain the first label probability distribution and the second label probability distribution.

[0169] Specifically, first, the text matrix is ​​normalized by the normalization layer of the normalization submodule, namely, the Norm (normalization) layer, which can improve the convergence speed and accuracy of the model to a certain extent, and obtain the first matrix x4. Specifically

[0170] Where μ represents the mean of the text matrix, and σ represents the standard deviation of the text matrix.

[0171] The information extraction submodule extracts information from the first matrix through the masked multi-head attention structure in the pre-trained classification model to obtain a second matrix x5 containing context information of the text data to be classified. The masked multi-head attention structure is basically the same as the encoder part of Bert, except that a mask matrix is ​​added to cover up future information, that is, only information transmitted by the previous text can be received, thereby converting a single encoder / decoder into a generative model.

[0172] In the present application, after obtaining the second matrix x5, Dropout processing can also be performed to effectively suppress overfitting.

[0173] The first connection submodule performs residual connection on the text matrix and the second matrix x5 after Dropout processing to obtain a third matrix, and then normalizes the third matrix through the Norm layer in the pre-trained classification model to obtain a fourth matrix x6. Where μ′ represents the mean after the first residual connection, σ′ represents the standard deviation after the first residual connection, and Add&Norm: residual connection is used to improve the gradient vanishing problem.

[0174] The mapping submodule inputs the normalized fourth matrix x6 into the feedforward network in the classification model, namely the FNN (feedforward neural network) network, to obtain the fifth matrix x7. The FNN network is a neural network that undergoes two layers of linear mapping, and the specific calculation formula is:

[0175] x7=f(W2·f(W1·x6+b1)+b2)

[0176] Wherein W1, W2, b1, b2 are training parameters, and the training parameters are the parameters that have converged after pre-training the classification model.

[0177] The second connection submodule processes the fifth matrix x7 through the activation function layer in the pre-trained classification model to obtain a sixth matrix, and performs residual connection on the sixth matrix and the fourth matrix x6 to obtain a seventh matrix, and normalizes the seventh matrix through the Norm layer in the pre-trained classification model to obtain an eighth matrix x8; the specific algorithm is:

[0178] Wherein μ″ represents the mean after the second residual connection, σ″ represents the standard deviation after the second residual connection, and the dimension of the eighth matrix x8 is [L1, D], where L1 represents the number of words to be generated.

[0179] The linear transformation submodule performs two linear transformations on the eighth matrix x8 to obtain a first label matrix x9 and a second label matrix x 10 ; The dimension of the first label matrix x9 is [L2, D1], where L2 represents the number of generated words and D1 represents the number of labels. The second label matrix x 10 The dimension is [L3, D2], where L3 represents the number of generated words and D2 represents the number of vocabularies.

[0180] The activation submodule maps the first label matrix and the second label matrix through the Softmax layer in the classification model to obtain the first label probability distribution and the second label probability distribution, and the dimensions of the first label probability distribution and the second label probability distribution are respectively the same as the first label matrix x9 and the second label matrix x 10 Consistent.

[0181] Through the coordination of the normalization submodule, the information extraction submodule, the first connection submodule, the mapping submodule, the second connection submodule, the linear transformation submodule and the activation submodule, the first label matrix x9 and the second label matrix x 10 , to achieve accurate classification of classified data, and the classification of classified data is not limited to existing labels, but new labels can be generated.

[0182] Furthermore, the information extraction submodule includes a first matrix multiplication unit, a calculation unit, a second matrix multiplication unit and a splicing transformation unit;

[0183] The first matrix multiplication unit is used to multiply the first matrix with multiple batches of parameter matrices obtained after pre-training to obtain multiple batches of Q matrices, K matrices and V matrices;

[0184] The calculation unit is used to obtain a second result by performing a dot multiplication of the Q matrix of each batch with the K matrix, dividing a first result obtained by the dot multiplication by the square root of the corresponding dimension of the Q matrix, adding the second result to a mask matrix and then performing a Softmax calculation to obtain a weight matrix, wherein the mask matrix is ​​constructed based on the word information;

[0185] The second matrix multiplication unit is used to multiply the weight matrix with the V matrix of the corresponding batch to obtain a ninth matrix of each batch;

[0186] The splicing transformation unit is used to obtain the second matrix by splicing the ninth matrices of all batches and performing linear transformation on the spliced ​​matrices.

[0187] Specifically, the first matrix multiplication unit obtains multiple batches of Q, K, and V matrices by multiplying the first matrix x4 by multiple batches of parameter matrices, and the formula is Q = x4·W Q , K = x4·W K ,V=x4·W V , where W Q , W K , W V That is, the parameter matrix. The calculation unit then uses the three matrices Q, K, and V to calculate the proportion of each word in the input text, where the weight of the word that only receives the above information is 0. The specific calculation formula is:

[0188] Where A represents the weight matrix, d k Represents the Q, K or V matrix dimension. The second matrix multiplication unit then multiplies the weight matrix with the V matrix of the corresponding batch to obtain the ninth matrix of each batch. The ninth matrices of all batches are spliced. The splicing transformation unit performs a linear transformation on the spliced ​​matrix to obtain the second matrix x5.

[0189] Through the cooperation of the first matrix multiplication unit, the calculation unit, the second matrix multiplication unit and the splicing transformation unit, a masked multi-head attention structure, namely, Masked Multi-Head Attention, is introduced to extract the context information of the text, and a mask is added to cover the future information, which can achieve better classification of the text and improve the accuracy of text classification.

[0190] Further, the linear transformation submodule includes a first linear change unit and a second linear change unit;

[0191] The first linear transformation unit is used to average each word vector in the eighth matrix and then perform linear transformation to obtain the first label matrix;

[0192] The second linear transformation unit is used to perform a linear transformation on each word vector in the eighth matrix to obtain the second label matrix.

[0193] Specifically, the first linear transformation unit, since each label word is composed of multiple characters, averages the vectors of each character before the identifier [START], and then performs a linear transformation to obtain the first label matrix x9; the second linear transformation unit only performs a linear transformation process, and directly performs a linear transformation on each character vector in the eighth matrix to obtain the second label matrix x 10 .

[0194] Two different linear transformation methods are performed by the first linear change unit and the second linear change unit to obtain two different label matrices, thereby realizing the classification of existing labels and newly generated labels.

[0195] Furthermore, the classification module 104 also includes a Dropout submodule;

[0196] The Dropout submodule is used to process the second matrix through the Dropout layer in the classification model to suppress overfitting of the classification model.

[0197] The Dropout submodule is used to effectively suppress overfitting of the classification model.

[0198] Still further, the computing unit includes a first data construction subunit, a second data construction subunit and a splicing subunit;

[0199] The first data construction subunit is used to construct the first data based on the information composed of the multiple words including the mask, the contents of the information being mutually associated, and according to the associated relationship;

[0200] The second data construction subunit is used to receive the information transmitted in the previous text of the word information based on the content in the keyword information to construct the second data;

[0201] The splicing subunit is used to splice the first data and the second data, and fill the empty positions after splicing with filling content to obtain the mask matrix.

[0202] Specifically, since the word information is divided into two parts, one part is the multiple word parts containing masks, and the other part is the keyword information part, the first data construction subunit can associate the multiple word parts containing masks with each other, that is, it can extract context information and construct the first data according to the associated information; the second data construction subunit needs to mask the future information for the keyword information part, that is, the keyword information part is transmitted in one direction and only receives the information transmitted in the previous text, so the second data is constructed in this way, and the splicing subunit splices the first data and the second data, and fills the empty positions after splicing with the filling content to obtain the mask matrix. Furthermore, the first data and the second data parts are directly represented by 0, and the filling content part is represented by the server label, that is, -∞.

[0203] A mask matrix is ​​constructed by the cooperation of the first data construction subunit, the second data construction subunit and the splicing subunit to cover up future information and transform a single encoder / decoder into a generative model.

[0204] The output module 105 is used to determine the category to which the to-be-classified text data belongs based on the first label probability distribution and the second label probability distribution.

[0205] Specifically, since the first label probability distribution is a probability distribution based on existing labels, and the second label probability distribution is a probability distribution of newly generated labels, the output module 105 selects labels based on the first label probability distribution and the second label probability distribution. When the first label probability distribution does not meet the requirements, the label is selected according to the second label probability distribution.

[0206] Furthermore, the output module 105 includes a label determination submodule, an extraction and storage submodule, and a label extraction submodule;

[0207] The label judgment submodule is used to obtain the label corresponding to the maximum probability value in the first label probability distribution, and judge whether the maximum probability value is greater than or equal to a preset value;

[0208] The extraction and storage submodule is used for, if the maximum probability value is less than the preset value, taking the label corresponding to the maximum probability value in the second label probability distribution as the category to which the text data to be classified belongs, and saving the label corresponding to the maximum probability value in the second label probability distribution into a label vocabulary;

[0209] The label extraction submodule is used to use the label corresponding to the maximum probability value in the first label probability distribution as the category to which the text data to be classified belongs if the maximum probability value is greater than or equal to the preset value.

[0210] Specifically, the label judgment submodule determines whether the maximum probability value in the first label probability distribution is greater than or equal to a preset value, since the first label probability distribution is a probability distribution obtained based on existing labels. When the maximum probability value in the first label probability distribution is greater than or equal to the preset value, the label extraction submodule directly uses the label corresponding to the maximum probability value in the first label probability distribution as the category to which the text data to be classified belongs. When the maximum probability value in the first label probability distribution is less than the preset value, the extraction and storage submodule directly extracts the label corresponding to the maximum probability value in the second label probability distribution as the category corresponding to the text data to be classified, and saves the label to the label word list.

[0211] Through the cooperation of the label judgment submodule, the extraction and preservation submodule and the label extraction submodule, the first label probability distribution is compared with the preset value, and different labels are obtained according to the comparison results to ensure that the label closest to the text data to be classified is selected, thereby improving the effectiveness of text classification.

[0212] By adopting the above-mentioned device, the text classification device 100, through the coordinated use of the acquisition module 101, the information extraction module 102, the vectorization module 103, the classification module 104 and the output module 105, can improve the classification accuracy while not only having the ability to classify into specified labels, but also having the ability to generate labels, thereby improving the fault tolerance of labeling.

[0213] The present application also provides a computer device. Figure 5 , Figure 5 This is a basic structural block diagram of the computer device in this embodiment.

[0214] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0215] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0216] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the text classification method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0217] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the text classification method.

[0218] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0219] This embodiment implements the steps of the text classification method of the above embodiment when the processor executes computer-readable instructions stored in the memory, by acquiring the text data to be classified, extracting keywords from the text data to be classified, obtaining word information, and then performing position encoding according to the word information to obtain corresponding position information, so that the subsequent classification model can better understand the text to be classified by obtaining the word information and position information; by embedding the word information and position information respectively, and merging the vectors obtained after the embedding process to obtain a text matrix, the text matrix is ​​input into a pre-trained classification model for processing, and a first label probability distribution and a second label probability distribution are obtained, wherein the first label probability distribution is the probability of an existing label, and the second label probability is the probability of a newly generated label for the text; based on the first label probability distribution and the second label probability distribution, the category to which the text data to be classified belongs is determined, so that the classification accuracy can be improved while not only having the ability to classify into specified labels, but also having the ability to generate labels, thereby improving the fault tolerance of labeling.

[0220] The embodiment of the present application also provides a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to execute the steps of the text classification method as described above, by acquiring text data to be classified, extracting keywords from the text data to be classified, obtaining word information, and then performing position encoding according to the word information to obtain corresponding position information, so that the subsequent classification model can better understand the text to be classified by obtaining the word information and position information; by embedding the word information and position information respectively, and merging the vectors obtained after the embedding process to obtain a text matrix, the text matrix is ​​input into a pre-trained classification model for processing, and a first label probability distribution and a second label probability distribution are obtained, wherein the first label probability distribution is the probability of an existing label, and the second label probability is the probability of a newly generated label for the text; based on the first label probability distribution and the second label probability distribution, the category to which the text data to be classified belongs is determined, so that the classification accuracy can be improved while not only having the ability to classify into specified labels, but also having the ability to generate labels, thereby improving the fault tolerance of labeling.

[0221] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0222] The text classification apparatus, computer device, and computer-readable storage medium of the above-mentioned embodiments of the present application have the same technical effects as the text classification method of the above-mentioned embodiments, and will not be elaborated here.

[0223] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A text classification method, characterized in that: The method comprises: Obtain text data to be classified; Extract keywords from the text data to be classified to obtain word information, encode the word information to obtain corresponding position information; The word information and the position information are respectively embedded, and the vectors obtained after the embedding process are merged to obtain a text matrix; The text matrix is ​​processed by a pre-trained classification model to obtain a first label probability distribution and a second label probability distribution, wherein the pre-trained classification model includes a masked multi-head attention structure; Based on the first label probability distribution and the second label probability distribution, determining the category to which the to-be-classified text data belongs; The processing of the text matrix by the pre-trained classification model includes: Normalizing the text matrix through a normalization layer in the pre-trained classification model to obtain a first matrix; The first matrix is ​​processed by performing information extraction through the masked multi-head attention structure in the pre-trained classification model to obtain a second matrix containing context information of the text data to be classified; Performing a residual connection on the text matrix and the second matrix to obtain a third matrix, and then normalizing the third matrix through a normalization layer in the pre-trained classification model to obtain a fourth matrix; Mapping the fourth matrix through a feedforward network layer in the pre-trained classification model to obtain a fifth matrix; The fifth matrix is ​​processed by the activation function layer in the pre-trained classification model to obtain a sixth matrix, and the sixth matrix is ​​residually connected with the fourth matrix to obtain a seventh matrix, and the seventh matrix is ​​normalized by the normalization layer in the pre-trained classification model to obtain an eighth matrix; Performing two linear transformations on the eighth matrix to obtain a first label matrix and a second label matrix; Mapping the first label matrix and the second label matrix through the Softmax layer in the classification model to obtain the first label probability distribution and the second label probability distribution; The first label probability distribution is the probability of existing labels, and the second label probability is the probability of newly generated labels for the text.

2. The text classification method according to claim 1, characterized in that: The first matrix is ​​processed by performing information extraction through the masked multi-head attention structure in the pre-trained classification model to obtain a second matrix containing context information of the text data to be classified, including: Multiplying the first matrix with the parameter matrices of multiple batches obtained after pre-training to obtain multiple batches of Q matrices, K matrices, and V matrices; By performing a dot multiplication of the Q matrix of each batch with the K matrix, dividing a first result obtained by the dot multiplication by the square root of the corresponding dimension of the Q matrix to obtain a second result, adding the second result to the mask matrix and then performing Softmax calculation to obtain a weight matrix, wherein the mask matrix is ​​constructed based on the word information; Multiplying the weight matrix by the V matrix of the corresponding batch to obtain a ninth matrix of each batch; The second matrix is ​​obtained by concatenating the ninth matrices of all batches and performing a linear transformation on the concatenated matrices.

3. The text classification method according to claim 1, characterized in that: The determining, based on the first label probability distribution and the second label probability distribution, of the category to which the to-be-classified text data belongs comprises: Obtaining a label corresponding to a maximum probability value in the first label probability distribution, and determining whether the maximum probability value is greater than or equal to a preset value; If the maximum probability value is less than the preset value, the label corresponding to the maximum probability value in the second label probability distribution is used as the category to which the text data to be classified belongs, and the label corresponding to the maximum probability value in the second label probability distribution is saved in the label vocabulary; If the maximum probability value is greater than or equal to the preset value, the label corresponding to the maximum probability value in the first label probability distribution is used as the category to which the text data to be classified belongs.

4. The text classification method according to claim 1, characterized in that: The extracting keywords from the text data to be classified includes: Using the Jieba word segmentation method to perform word segmentation processing on the text data to be classified to obtain a plurality of corresponding words; A keyword extraction algorithm is used to extract keywords from the multiple words, a preset ratio of keywords is extracted, and masks are used to replace the keywords in the multiple words.

5. The text classification method according to claim 4, characterized in that: The step of extracting keywords from the text data to be classified to obtain word information, encoding the word information, and obtaining corresponding position information includes: Based on the weights of the keywords obtained by the keyword extraction algorithm, the keywords are sorted and an identifier is set before each keyword to obtain keyword information; Merging the information consisting of multiple words including the mask with the keyword information to obtain word information; According to the word information, position encoding is performed to obtain corresponding position information.

6. The text classification method according to claim 5, characterized in that: The mask matrix is ​​constructed based on the word information and includes: Based on the information composed of the multiple words including the mask, the contents of the information are mutually associated, and first data is constructed according to the associated relationship; Based on the content in the keyword information, the information transmitted in the previous text in the word information is received to construct the second data; The first data and the second data are concatenated, and the empty positions after the concatenation are filled with filling content to obtain the mask matrix.

7. A text classification device, characterized in that: The device comprises: An acquisition module is used to acquire text data to be classified; An information extraction module, used for extracting keywords from the text data to be classified to obtain word information, and encoding the word information to obtain corresponding position information; A vectorization module, used for embedding the word information and the position information respectively, and merging the vectors obtained after the embedding process to obtain a text matrix; A classification module, used for processing the text matrix through a pre-trained classification model to obtain a first label probability distribution and a second label probability distribution, wherein the pre-trained classification model includes a masked multi-head attention structure; An output module, configured to determine the category to which the to-be-classified text data belongs based on the first label probability distribution and the second label probability distribution; Wherein, the classification module includes a normalization submodule, an information extraction submodule, a first connection submodule, a mapping submodule, a second connection submodule, a linear transformation submodule and an activation submodule; The normalization submodule is used to normalize the text matrix through the normalization layer in the pre-trained classification model to obtain a first matrix; The information extraction submodule is used to perform information extraction processing on the first matrix through the masked multi-head attention structure in the pre-trained classification model to obtain a second matrix containing context information of the text data to be classified; The first connection submodule is used to perform residual connection on the text matrix and the second matrix to obtain a third matrix, and then normalize the third matrix through a normalization layer in the classification model to obtain a fourth matrix; The mapping submodule is used to perform mapping processing on the fourth matrix through the feedforward network layer in the pre-trained classification model to obtain a fifth matrix; The second connection submodule is used for processing the fifth matrix through the activation function layer in the pre-trained classification model to obtain a sixth matrix, and performing residual connection on the sixth matrix and the fourth matrix to obtain a seventh matrix, and normalizing the seventh matrix through the normalization layer in the pre-trained classification model to obtain an eighth matrix; The linear transformation submodule is used to perform two linear transformations on the eighth matrix to obtain a first label matrix and a second label matrix; The activation submodule is used to map the first label matrix and the second label matrix through the Softmax layer in the classification model to obtain the first label probability distribution and the second label probability distribution; The first label probability distribution is the probability of existing labels, and the second label probability is the probability of newly generated labels for the text.

8. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer-readable instructions, and the processor implements the text classification method according to any one of claims 1 to 6 when executing the computer-readable instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the text classification method according to any one of claims 1 to 6 is implemented.

Images