A text classification method, device and equipment and readable storage medium

Abstract

The application discloses a text classification method, which calls a trained model comprising a mask language model decoder, the semantic representor and the classifier, and performs parameter optimization training on the semantic representor and the mask language model decoder classifier, so that the semantic representation ability of the semantic representor and the mask language model decoder classifier is trained through the mask language model decoder, more effective semantic representation vectors can be provided for the classifier part, the semantic representation ability of the model is improved, the convergence speed of the model on the semantic classification data set in actual application is faster, the semantic representor and the mask language model decoder classifier obtained through training have stronger classification ability and better generalization, and therefore the classification effect on the text to be classified is improved. The application also discloses a text classification device, equipment and readable storage medium, which have corresponding technical effects.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a text classification method, apparatus, device, and readable storage medium. Background Technology

[0002] Text classification is a fundamental problem in the field of natural language processing, with wide applications in industry. For example, in financial risk control, text classification can predict the likelihood of a user defaulting on a loan based on various textual information (search history, browsing logs, etc.). However, in practical applications, especially in risk control, the ratio of positive to negative samples is extremely skewed, reaching one to tens or even hundreds of times the original ratio. Under such extreme class imbalance, models often suffer from severe overfitting, making it difficult to guarantee the accuracy of the actual recognition process.

[0003] Currently, the main methods for solving the class imbalance problem in the field of natural language processing include resampling, reweighting, and transfer learning. However, in practical applications, resampling methods suffer from low robustness and low generalization, reweighting methods suffer from low universality and poor applicability, and transfer learning suffers from stringent application conditions and high costs.

[0004] In conclusion, how to ensure the generalization ability of a model while giving it stronger classification capabilities is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a text classification method, apparatus, device, and readable storage medium, so that the model has stronger classification ability while ensuring the model's generalization ability.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] A text classification method, comprising:

[0008] Obtain the numerical sequence of the text to be classified;

[0009] The semantic representation unit is invoked to perform vector transformation on the numerical sequence, generating a semantic vector;

[0010] The classifier is invoked to perform semantic classification calculations based on the semantic vector to obtain the text category;

[0011] The semantic representation and the classifier are obtained by recognition training based on the training model; the training model includes: a masked language model decoder, the semantic representation, and the classifier; the masked language model decoder and the classifier are connected in parallel to the output of the semantic representation.

[0012] Optionally, the training method for the semantic representation and the classifier includes:

[0013] After symbol replacement of some characters in the first sample data, a digitization conversion is performed to obtain variable samples. Based on the variable samples, the classifier and the masked language model decoder are jointly trained to obtain a first classifier and a first masked language model decoder. The joint training includes: category prediction training of the classifier and character prediction training of the masked language model decoder.

[0014] Based on the first sample data, the parameters of the first classifier and the semantic representative are optimized and trained to obtain the second classifier and the second semantic representative.

[0015] The second classifier is retrained based on the second sample data to obtain the third classifier;

[0016] Output the third classifier and the second semantic representative.

[0017] Optionally, the step of jointly training the classifier and the masked language model decoder based on the variable samples includes:

[0018] The variable samples are input into the training model;

[0019] Obtain the predicted classification output by the classifier and the predicted sequence output by the masked language model decoder;

[0020] Calculate the loss between the predicted classification and the corresponding classification label, and use it as the classification loss;

[0021] Calculate the loss between the predicted sequence and the corresponding actual sequence in the first sample data, and use it as the sequence loss;

[0022] The model parameters of the classifier and the masked language model decoder are optimized and adjusted based on the classification loss and the sequence loss.

[0023] Optionally, calculating the loss between the predicted sequence and the corresponding actual sequence in the first sample data includes:

[0024] Calculate the loss between the replaced character in the predicted sequence and the corresponding actual character in the first sample data.

[0025] Optionally, retraining the second classifier based on the second sample data includes:

[0026] The model parameters of the second classifier are randomly initialized to obtain the initialized classifier.

[0027] The data corresponding to each category are extracted proportionally and evenly from the sample data and used as the second sample data;

[0028] The second sample data is digitally preprocessed and then input into the training model;

[0029] The model parameters of the classifier are optimized and adjusted based on the loss between the predicted classification output by the classifier and the corresponding actual classification.

[0030] A text classification device, comprising:

[0031] Sequence acquisition unit, used to acquire the numerical sequence of the text to be classified;

[0032] The vector transformation unit is used to call the semantic representation unit to perform vector transformation on the digital sequence and generate a semantic vector.

[0033] The semantic classification unit is used to call the classifier to perform semantic classification calculations based on the semantic vector to obtain the text category;

[0034] The semantic representation called in the vector transformation unit and the classifier called in the semantic classification unit are obtained by the training unit through recognition training based on the training model; the training model called by the training unit includes: a masked language model decoder, the semantic representation, and the classifier; the masked language model decoder and the classifier are connected in parallel to the output of the semantic representation.

[0035] Optionally, the training unit includes:

[0036] The first training subunit is used to perform symbol replacement on some characters in the first sample data and then perform digital conversion to obtain variable samples. Based on the variable samples, the classifier and the masked language model decoder are jointly trained to obtain the first classifier and the first masked language model decoder. The joint training includes: category prediction training of the classifier and character prediction training of the masked language model decoder.

[0037] The second training subunit is used to perform parameter optimization training on the first classifier and the semantic representation based on the first sample data to obtain a second classifier and a second semantic representation.

[0038] The third training subunit is used to retrain the second classifier based on the second sample data to obtain the third classifier.

[0039] The model output subunit is used to output the third classifier and the second semantic representation.

[0040] Optionally, the first training subunit specifically includes:

[0041] The variable input subunit is used to input the variable samples into the training model;

[0042] The result acquisition subunit is used to acquire the predicted classification output by the classifier and the predicted sequence output by the masked language model decoder.

[0043] The classification loss calculation subunit is used to calculate the loss between the predicted classification and the corresponding classification label, which is the classification loss.

[0044] The sequence loss calculation subunit is used to calculate the loss between the predicted sequence and the corresponding actual sequence in the first sample data, as the sequence loss.

[0045] The parameter adjustment subunit is used to optimize and adjust the model parameters of the classifier and the masked language model decoder based on the classification loss and the sequence loss.

[0046] A computer device, comprising:

[0047] Memory, used to store computer programs;

[0048] A processor is used to implement the steps of the above-described text classification method when executing the computer program.

[0049] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described text classification method.

[0050] The method provided in this invention calls a training model including a masked language model decoder, a semantic representation, and a classifier. It optimizes the parameters of the semantic representation and the masked language model decoder classifier. By training the semantic representation capability through the masked language model decoder, more effective semantic representation vectors can be provided to the classifier, thereby improving the model's semantic representation capability. This results in faster convergence of the model on semantic classification datasets in practical applications. It ensures that the trained semantic representation and masked language model decoder classifier have stronger classification capabilities while also maintaining their generalization ability, thus improving the classification effect on the text to be classified.

[0051] Accordingly, embodiments of the present invention also provide a text classification apparatus, device, and readable storage medium corresponding to the above-described text classification method, which have the aforementioned technical effects, and will not be elaborated further here. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating the implementation of a text classification method in an embodiment of the present invention.

[0054] Figure 2 This is a schematic diagram of a classifier structure in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of a masked language model decoder structure in an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram illustrating the joint training of a classification and masking language model in an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of overall model fine-tuning training in an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of classifier fine-tuning training in an embodiment of the present invention;

[0059] Figure 7 This is a schematic diagram of text classification in a practical application process according to an embodiment of the present invention;

[0060] Figure 8 This is a schematic diagram of the structure of a text classification device according to an embodiment of the present invention;

[0061] Figure 9 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0062] The core of this invention is to provide a text classification method with strong classification and generalization capabilities.

[0063] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] Currently, the main methods for solving the class imbalance problem in the field of natural language processing are as follows:

[0065] (1) Resampling

[0066] Resampling is divided into undersampling of samples from a high-proportion class and oversampling of samples from a low-proportion class. Essentially, it uses various methods to ensure that the proportion of different classes in the input samples during model training is roughly the same. For example, undersampling involves collecting samples from the majority class with a lower probability and samples from the minority class with a higher probability during training; oversampling can augment the minority class samples through data augmentation, such as creating similar samples by randomly inserting, deleting, or replacing individual characters in text data. This method has been widely researched and applied in natural language processing, image processing, machine learning, and other fields.

[0067] However, this method reduces the model's semantic representation ability by artificially altering the original distribution of samples. Specifically, undersampling prevents the majority class samples in the true distribution from being fully learned, while oversampling repeatedly teaches the minority class samples. Both of these factors result in weak robustness and generalization of the model.

[0068] (2) Reweighting

[0069] It is generally believed that under class imbalance, the model's insufficient fit to minority class samples during training is the cause of overfitting. Reweighting involves assigning different weights to different classes in the loss algorithm, with lower weights for majority class samples and higher weights for minority class samples. This reduces the influence of majority class samples on the model during training and strengthens the influence of minority class samples, thus offsetting the effects of class imbalance.

[0070] The biggest problem with this type of method is that it requires manual assignment of different weights to samples of different categories. The weights may vary for different tasks and model structures, making it difficult to apply universally and resulting in poor general applicability.

[0071] (3) Transfer learning

[0072] In the field of natural language processing, transfer learning generally refers to the paradigm of pre-training followed by fine-tuning. In text classification problems, the model can be viewed as consisting of two parts: a semantic representation and a classifier. The semantic representation represents textual information as a semantic vector, and the classifier calculates the category based on this semantic vector. A model pre-trained on a large amount of data possesses strong semantic representation capabilities; it only requires training on a small amount of labeled data to effectively represent samples of different categories, thus alleviating the problem of class imbalance.

[0073] However, pre-trained models usually have limitations, such as restricting the length of the input sequence and the input dictionary set, and are not necessarily applicable to the problems to be solved. For example, open-source pre-trained Chinese models such as BERT and RoBERTa limit the maximum sequence length to 512, while many text classification tasks involve long texts of thousands or even tens of thousands of words (for example, when classifying people through search records, it is necessary to combine the search records of a user over a period of time, and the sample length is generally in the thousands). The pre-trained model cannot be directly used; moreover, it is very time-consuming and costly to select and train a suitable pre-trained model by oneself.

[0074] In view of this, how to solve the overfitting problem of the text classification model under the condition of class imbalance, while avoiding the low robustness and low generalization of the resampling method model, the low generality of the reweighting method, the applicability and high cost of transfer learning, etc. are the main objectives of this invention. For this reason, this invention proposes a text classification method, please refer to Figure 1 , Figure 1 is a flowchart of a text classification method in an embodiment of this invention. The method includes the following steps:

[0075] S101. Obtain the digital sequence of the text to be classified;

[0076] The obtained digital sequence refers to the digital form sequence of the text to achieve digital recognition. Specifically, the text can be converted into a digital sequence by segmenting or splitting the Chinese sequence (i.e., the text to be classified) and then converting the word sequence into a digital id sequence according to the dictionary. Taking a text as an example to introduce the generation process of the digital sequence. For example, if the text to be classified is "What's the weather like today", after splitting the words of "What's the weather like today", the sequence ["今", "天", "天", "气", "怎", "么", "样"] is obtained. Then, according to the id of each word in the dictionary, the sequence can be converted into a digital id sequence [5, 100, 100, 20, 101, 489, 57]. In this embodiment, only the above-mentioned generation method of the digital sequence is used as an example for introduction, and other generation methods can be referred to the introduction of this embodiment and will not be elaborated here.

[0077] S102. Call the semantic representer to perform vector transformation on the digital sequence and generate a semantic vector;

[0078] The semantic representer converts the input digital sequence into a set of vectors through complex calculations as semantic vectors (which can also be called semantic representation vectors).

[0079] Depending on the type of recognition task, different sequence modeling models can be selected as semantic representations. In this embodiment, the type of semantic representation used is not limited. For example, LSTM, Transformer, and other models can be used for vector transformation of short texts, while Longformer can be used for vector transformation of long texts (the main optimization scenario of this invention). The output sequence and input sequence in the semantic representation are of equal length. For example, if the input length is n IDs, the output must be an n*h semantic representation matrix, where h is a hyperparameter of the model and is not restricted. Each h-dimensional vector represents the semantic meaning of the input ID at its corresponding position, containing rich contextual information.

[0080] S103. Call the classifier to perform semantic classification calculation based on the semantic vector to obtain the text category;

[0081] The classifier calculates the category based on the semantic vector. For example... Figure 2 As shown, a classifier may specifically include:

[0082] Pooling layer: Transforms the n*h semantic matrix into an h-dimensional vector, which represents the semantic vector of the entire sequence. Here, we directly take the first row of the n*h semantic matrix.

[0083] Dropout: Sets the values ​​in an h-dimensional vector to zero with a certain probability p, where p = 0.1.

[0084] Dense layer + tanh: Multiply the h-dimensional vector by the parameter matrix [h, h], and then apply the tanh activation function. The formula for the tanh activation function is as follows:

[0085]

[0086] Dropout: Sets the values ​​in an h-dimensional vector to zero with a certain probability p, where p = 0.1.

[0087] Dense layer + softmax: Multiply the h-dimensional vector by the parameter matrix [h,c], where c is the number of classes. The softmax function converts it into a c-dimensional vector, with each dimension having a value between 0 and 1, representing the probability that the sample belongs to the corresponding class.

[0088] The above is an introduction to a common structure of a classifier. This embodiment only uses one example. Figure 2 The structure shown is used as an example to introduce the classifier function. Other structures can be described in the same way as this embodiment, and will not be repeated here.

[0089] It's important to note that traditional text classification training models only include semantic representations and classifiers. Sample data is directly input into these models for training, resulting in a practically applicable text classification model (i.e., a model composed of semantic representations and classifiers). However, due to the imbalance in the ratio of positive to negative samples and limitations in applicable objects, traditional methods cannot achieve comprehensive and accurate classification. To address this, the training model for the semantic representations and classifiers used in this method further includes, in addition to the semantic representations and classifiers, a masked language model decoder.

[0090] The structure of a masked language model decoder is as follows: Figure 3 As shown, dense layer + gleu: multiplies the input n*h matrix by an h*h matrix and performs a non-linear mapping using the gleu function, the formula of which is as follows:

[0091]

[0092] Layer normalization operation.

[0093] dense+softmax: Transforms the input n*h matrix into an n*d matrix, where d is the size of the dictionary. The (i,j)th element represents the probability that the i-th input position is the j-th symbol in the dictionary (assuming the table below counts from 1).

[0094] In the training model, the masked language model decoder is connected in parallel with the classifier to the output of the semantic representation. Based on the classifier and the semantic representation, the semantic representation ability is trained through the masked language model decoder, which can provide more effective semantic representation vectors for the classifier. This makes the model converge faster on the classification dataset, and while having stronger classification ability, it also ensures its generalization ability, thereby improving the classifier's performance.

[0095] The specific process of calling the training model containing the masked language model decoder to optimize the semantic representation and classifier is not limited in this embodiment. The training process can be set according to the actual optimization object. This embodiment will not elaborate on this.

[0096] Based on the above description, the technical solution provided by the embodiments of the present invention calls a training model including a masked language model decoder, a semantic representation, and a classifier. The semantic representation and the masked language model decoder classifier are trained with optimized parameters. By training the semantic representation capability through the masked language model decoder, more effective semantic representation vectors can be provided to the classifier, thereby improving the model's semantic representation capability. This results in faster convergence of the model on semantic classification datasets in practical applications, ensuring that the trained semantic representation and masked language model decoder classifier have stronger classification capabilities while also maintaining their generalization ability, thus improving the classification effect on the text to be classified.

[0097] It should be noted that, based on the above embodiments, the present invention also provides corresponding improvements. In the preferred / improved embodiments, steps that are the same as or corresponding to those in the above embodiments can be referred to each other, and the corresponding beneficial effects can also be referred to each other; however, these will not be elaborated upon in the preferred / improved embodiments herein.

[0098] The above embodiments do not limit the specific process of calling the training model containing the masked language model decoder to optimize the training of the semantic representative and the classifier. To deepen understanding, this embodiment proposes a three-step training method, which can significantly enhance the representation ability of the semantic representative on the dataset.

[0099] In this embodiment, the training process is divided into three progressive steps: joint training of the classification and masked language models, overall fine-tuning, and classifier fine-tuning. After these three training steps, the final model for text classification prediction is obtained. Specifically, the training methods for the semantic representation and classifier can be implemented according to the following steps:

[0100] (1) Symbol replacement is performed on some characters in the first sample data to obtain variable samples, and the classifier and mask language model decoder are jointly trained based on the variable samples to obtain the first classifier and the first mask language model decoder.

[0101] like Figure 4The following shows a schematic diagram of the joint training of a classification and masked language model. The sample text data of the original input is first segmented into a sequence of words or characters. For each character in the sequence, with a certain probability (such as 15%), it is replaced by a special symbol "[MASK]" (using the [MASK] symbol is a common practice in the field, and other characters can also be selected. Refer to the introduction of related technologies for details and are not limited here), and a special symbol "[CLS]" is added at the head of the sequence to use its semantic representation as the representation of the entire text sequence and send it to the classifier for classification. Joint training refers to the combination of two tasks. The first task is the category prediction training of the classifier, that is, predicting the category label of the original input. The second task is the character prediction training of the masked language model decoder, which refers to predicting the original characters replaced by the "[MASK]" symbol in the input.

[0102] As Figure 4 shown, assume the original input is "What's the weather like today", and the category it is finally classified into is denoted as "Label 1". After sentence processing, it becomes the sequence ["[CLS]", "今", "[MASK]", "天", "[MASK]", "怎", "么", "样"]. According to the id corresponding to the character in the dictionary, the character sequence is converted into an id sequence [0, 5, 1, 100, 1, 101, 489, 57]. The second task needs to predict what the symbols replaced by "[MASK]" are. In this example, the replaced characters are "天" and "气", and the corresponding ids are 100 and 20 respectively. Label 2 can be denoted as [-1, -1, 100, -1, 20, -1, -1, -1], where "-1" represents that prediction is not required at this position.

[0103] Optionally, a joint training step is as follows:

[0104] (1.1) Input the variable sample into the training model;

[0105] After the id sequence is input into the model, the output of the classifier part is denoted as Output 1, which is a c-dimensional vector, where c is the number of categories, and each value represents the probability that the sample belongs to a certain category.

[0106] (1.2) Obtain the predicted classification output by the classifier and the predicted sequence output by the masked language model decoder;

[0107] (1.3) Calculate the loss between the predicted classification and the corresponding classification label as the classification loss;

[0108] Calculate the loss (cross-entropy) between Output 1 and Label 1, denoted as Loss 1.

[0109] (1.4) Calculate the loss between the predicted sequence and the corresponding actual sequence in the first sample data as the sequence loss;

[0110] The decoder part of the masked language model outputs an [n, d]-dimensional matrix, denoted as Output 2, where n is the length of the input sequence and d is the dictionary size. This is equivalent to predicting the original input at each position. A cross-entropy loss is calculated between Output 2 and Label 2. To avoid interference from characters not substituted in the loss calculation, the loss for positions marked "-1" in Label 2 can be set to 0. The total loss for this part is denoted as Loss 2.

[0111] (1.5) Optimize and adjust the model parameters of the classifier and the masked language model decoder based on the classification loss and sequence loss.

[0112] The model parameters of the classifier and the masked language model decoder are optimized and adjusted based on the classification loss (i.e., loss 1) and the sequence loss (i.e., loss 2). The specific process of optimizing model parameters based on the loss values ​​can be found in the implementation of related technologies, and will not be elaborated in this embodiment.

[0113] (2) Optimize the parameters of the first classifier and semantic representation based on the first sample data to obtain the second classifier and the second semantic representation.

[0114] Figure 5 This is a schematic diagram of the overall fine-tuning process. In this step, only the semantic representation and classifier are trained, as shown in the gray area in the diagram.

[0115] After serializing the original input, a special symbol "[CLS]" is added to the header, and then the character sequence is converted into an ID sequence input model.

[0116] The cross-entropy loss is calculated using the classifier's output 1 and label 1 for training. For details, please refer to the above description of the joint training steps, which will not be repeated here.

[0117] (3) Retrain the second classifier based on the second sample data to obtain the third classifier;

[0118] The first two training steps yield a semantic representative that is effective on the dataset. In this step, only the classifier part needs to be trained, such as... Figure 6 This diagram illustrates a fine-tuning training method for a classifier, where the parameters of other parts of the model can remain fixed. For the separate training of the classifier, the classifier parameters are fine-tuned based on the loss between the classifier's output and the actual classification labels. For details, please refer to the implementation of relevant technologies.

[0119] Alternatively, one implementation step for retraining the second classifier based on the second sample data is as follows:

[0120] (3.1) Randomly initialize the model parameters of the second classifier to obtain the initialized classifier;

[0121] First, re-initialize the parameters of the classifier part randomly.

[0122] (3.2) Extract the data corresponding to each category from the sample data in a balanced manner according to the proportion, and use it as the second sample data;

[0123] During training, undersampling can be used to maintain the proportion of different categories of data in the training data at approximately equal ratios, thereby ensuring the training effect of the classifier.

[0124] (3.3) The second sample data is digitally preprocessed and then input into the training model;

[0125] After sample serialization, a special symbol "[CLS]" is added to the header, and then the character sequence is converted into an ID sequence for input into the model. The (cross-entropy) loss is calculated using the classifier's output 1 and label 1 for training.

[0126] (3.4) Optimize and adjust the model parameters of the classifier based on the loss between the predicted classification output by the classifier and the corresponding actual classification.

[0127] This embodiment only uses the training process of the above classifier as an example for introduction. Other training methods can refer to the introduction of this embodiment, and will not be repeated here.

[0128] (4) Output the third classifier and the second semantic representation as the optimized text classification model.

[0129] After training, only the semantic representation and classifier of the model are retained for text classification prediction. For a sample to be predicted, it is serialized and the special symbol "[CLS]" is added to the header. Then, the character sequence is converted into an ID sequence and input into the model. The output of the classifier is the prediction result, such as... Figure 7 The diagram shown illustrates a text classification process in a practical application.

[0130] Based on the above introduction, in the three-step training method proposed in this embodiment, the classification and masked language model tasks mutually promote each other in the first step of training. The masked language model task enables the model to converge faster on the classification dataset, while the classification task is equivalent to imposing a constraint boundary on the representation space of the semantic representation, making it exist in a flow space that is favorable to the classification task. In this way, the model can have stronger classification ability while ensuring the model's generalization ability.

[0131] In the first step, training the classifier requires joint training of the masked language model decoder. The input needs to have some characters replaced with the special character [MASK]. The masked language model decoder is then trained to predict the original symbol at the [MASK] position, thus enhancing the semantic representation's capabilities. However, introducing [MASK] is essentially "noise" for the classifier. Therefore, in the second step, the [MASK] special character needs to be removed, and the original data is used to fine-tune both the classifier and the semantic representation, bringing the entire model to a better state. It should be noted that [MASK] is not needed in the second and third training steps.

[0132] In the third step, the classifier is trained separately after data resampling, which decouples the classifier from the semantic representation. The classifier focuses more on finding the segmentation surface in the semantic space, thereby improving the model's classification ability.

[0133] Corresponding to the above method embodiments, this invention also provides a text classification device, and the text classification device described below can be referred to in correspondence with the text classification method described above.

[0134] See Figure 8 As shown, the device includes the following modules:

[0135] The sequence acquisition unit 110 is mainly used to acquire the numerical sequence of the text to be classified;

[0136] Vector transformation unit 120 is mainly used to call the semantic representation unit to perform vector transformation on the numerical sequence and generate semantic vectors;

[0137] The semantic classification unit 130 is mainly used to call the classifier to perform semantic classification calculations based on the semantic vector to obtain the text category;

[0138] The semantic representation called in the vector transformation unit and the classifier called in the semantic classification unit are obtained by the training unit 140 through recognition training based on the training model. The training model called by the training unit includes: mask language model decoder, semantic representation, and classifier. The mask language model decoder and the classifier are connected in parallel to the output of the semantic representation.

[0139] In one specific embodiment of the present invention, the training unit 140 specifically includes:

[0140] The first training subunit is used to perform symbol replacement on some characters in the first sample data and then perform digitization to obtain variable samples. Based on the variable samples, the classifier and the masked language model decoder are jointly trained to obtain the first classifier and the first masked language model decoder. The joint training includes: training the classifier for category prediction and training the masked language model decoder for character prediction.

[0141] The second training subunit is used to perform parameter optimization training on the first classifier and semantic representation based on the first sample data to obtain the second classifier and the second semantic representation.

[0142] The third training subunit is used to retrain the second classifier based on the second sample data to obtain the third classifier.

[0143] The model output subunit is used to output the third classifier and the second semantic representation.

[0144] In one specific embodiment of the present invention, the first training subunit specifically includes:

[0145] The variable input subunit is used to input variable samples into the training model;

[0146] The result acquisition subunit is used to acquire the predicted classification output by the classifier and the predicted sequence output by the masked language model decoder.

[0147] The classification loss calculation subunit is used to calculate the loss between the predicted classification and the corresponding classification label, which is used as the classification loss.

[0148] The sequence loss calculation subunit is used to calculate the loss between the predicted sequence and the corresponding actual sequence in the first sample data, which is used as the sequence loss.

[0149] The parameter tuning subunit is used to optimize and adjust the model parameters of the classifier and the masked language model decoder based on the classification loss and sequence loss.

[0150] Corresponding to the above method embodiments, this invention also provides a computer device. The computer device described below and the text classification method described above can be referred to each other.

[0151] The computer device includes:

[0152] Memory, used to store computer programs;

[0153] A processor is used to implement the steps of the text classification method in the above method embodiments when executing a computer program.

[0154] For details, please refer to Figure 9This is a schematic diagram illustrating the specific structure of a computer device provided in this embodiment. The computer device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer application programs 342 or data 344. The memory 332 can be temporary or persistent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the data processing device. Furthermore, the central processing unit 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the computer device 301.

[0155] Computer device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0156] The steps in the text classification method described above can be implemented using the architecture of a computer device.

[0157] Corresponding to the above method embodiments, this invention also provides a readable storage medium. The readable storage medium described below corresponds to and can be referred to in relation to the text classification method described above.

[0158] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the text classification method described in the above method embodiments.

[0159] Specifically, the readable storage medium can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other readable storage medium capable of storing program code.

[0160] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the invention.

Claims

1. A method of text classification, characterized by, The method comprises the following steps: obtaining a digital sequence of a text to be classified; calling a semantic representation device to perform vector conversion on the digital sequence to generate a semantic vector; calling a classifier to perform semantic classification calculation according to the semantic vector to obtain a text category; wherein the semantic representation device and the classifier are obtained by identifying and training a training model; the training model comprises a mask language model decoder, the semantic representation device and the classifier; the mask language model decoder and the classifier are connected in parallel to the output end of the semantic representation device; and the training method of the semantic representation device and the classifier comprises the following steps: performing symbol replacement on part of the characters in the first sample data and then performing digital conversion to obtain a variable sample; performing joint training on the classifier and the mask language model decoder according to the variable sample to obtain a first classifier and a first mask language model decoder; wherein the joint training comprises class prediction training of the classifier and character prediction training of the mask language model decoder; performing parameter optimization training on the first classifier and the semantic representation device according to the first sample data to obtain a second classifier and a second semantic representation device; retraining the second classifier according to second sample data to obtain a third classifier; outputting the third classifier and the second semantic representation device.

2. The text classification method of claim 1, wherein, The joint training of the classifier and the mask language model decoder according to the variable sample comprises the following steps: inputting the variable sample into the training model; obtaining a predicted classification output by the classifier and a predicted sequence output by the mask language model decoder; calculating the loss between the predicted classification and the corresponding classification label as a classification loss; calculating the loss between the predicted sequence and the corresponding actual sequence in the first sample data as a sequence loss; optimizing and adjusting the model parameters of the classifier and the mask language model decoder according to the classification loss and the sequence loss.

3. The text classification method of claim 2, wherein, The calculation of the loss between the predicted sequence and the corresponding actual sequence in the first sample data comprises the following steps: calculating the loss between the replaced characters in the predicted sequence and the corresponding actual characters in the first sample data.

4. The text classification method of claim 1, wherein, The retraining of the second classifier according to the second sample data comprises the following steps: randomly initializing the model parameters of the second classifier to obtain an initialized classifier; extracting the data corresponding to each category from the sample data in proportion to balance the proportion of each category, as the second sample data; inputting the second sample data into the training model after digital preprocessing; optimizing and adjusting the model parameters of the classifier according to the loss between the predicted classification output by the classifier and the corresponding actual classification.

5. A text classification apparatus characterized by comprising: The method comprises the following steps: a sequence acquisition unit for obtaining a digital sequence of a text to be classified; a vector conversion unit for calling a semantic representation device to perform vector conversion on the digital sequence to generate a semantic vector; a semantic classification unit for calling a classifier to perform semantic classification calculation according to the semantic vector to obtain a text category; The semantic representor called in the vector conversion unit and the classifier called in the semantic classification unit are identified and trained according to a training model by a training unit; the training model called by the training unit comprises a mask language model decoder, the semantic representor and the classifier; the mask language model decoder and the classifier are connected in parallel to an output end of the semantic representor; the training unit comprises: A first training subunit is configured to perform symbol replacement on part of characters in first sample data and then perform digitization conversion to obtain variable samples, perform joint training on the classifier and the mask language model decoder according to the variable samples, and obtain a first classifier and a first mask language model decoder; wherein the joint training comprises class prediction training on the classifier and character prediction training on the mask language model decoder; A second training subunit is configured to perform parameter optimization training on the first classifier and the semantic representor according to the first sample data, and obtain a second classifier and a second semantic representor; A third training subunit is configured to perform retraining on the second classifier according to second sample data, and obtain a third classifier; A model output subunit is configured to output the third classifier and the second semantic representor.

6. The text classification apparatus according to claim 5, characterized by The first training subunit specifically comprises: A variable input subunit is configured to input the variable samples to the training model; A result acquisition subunit is configured to acquire a predicted classification output by the classifier and a predicted sequence output by the mask language model decoder; A classification loss calculation subunit is configured to calculate a loss between the predicted classification and a corresponding classification label as a classification loss; A sequence loss calculation subunit is configured to calculate a loss between the predicted sequence and a corresponding actual sequence in the first sample data as a sequence loss; A parameter adjustment subunit is configured to optimize and adjust model parameters of the classifier and the mask language model decoder according to the classification loss and the sequence loss.

7. A computer device, comprising: The memory is configured to store a computer program; The processor is configured to implement steps of the text classification method in any one of claims 1 to 4 when executing the computer program. The computer program is stored on the readable storage medium, and when executed by the processor, the computer program implements steps of the text classification method in any one of claims 1 to 4.

8. A readable storage medium, characterized by, ​

Images