Task text classification method and device, electronic equipment and computer program product
By extracting text sample features using the BERT model and oversampling them to generate data-augmented sample sentence vectors, the problem of weak model generalization ability caused by the small amount of text data in the task is solved, and the generalization ability of the model is improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MOBILE GROUP JIANGSU
- Filing Date
- 2022-04-19
- Publication Date
- 2026-06-02
AI Technical Summary
In task-oriented dialogue scenarios, the limited and short amount of task text data results in weak generalization ability of the trained intent classification model.
The text classification model is trained by extracting features from text samples using the BERT model, obtaining sample sentence vectors, and generating data-augmented sample sentence vectors through oversampling.
The amount of text data used in the training tasks was increased, which improved the generalization ability of the text classification model.
Smart Images
Figure CN116975258B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, specifically to a task text classification method, apparatus, electronic device, and computer program product. Background Technology
[0002] In the field of machine learning, higher-performing neural networks often require larger data scales. If the sample data of different categories in a classification task differ greatly, the generalization ability of the trained model will tend to be weaker.
[0003] In task-oriented dialogue scenarios, task text data can be used to train models for classifying the intent of task-oriented dialogues. However, task text data is characterized by its small amount and short duration, resulting in insufficient data for training the model and thus weak generalization ability of the trained intent classification model. Summary of the Invention
[0004] This application provides a task text classification method, apparatus, electronic device, and computer program product to solve the technical problem in related technologies where insufficient task text data is available for training the model, resulting in weak generalization ability of the trained intent classification model.
[0005] In a first aspect, embodiments of this application provide a task text classification method, including:
[0006] Obtain the target sentence vector corresponding to the target task text;
[0007] The target sentence vector is classified by intent using a text classification model to obtain the target classification corresponding to the target sentence vector;
[0008] The text classification model is a model trained on data-augmented sample sentence vectors. The data-augmented sample sentence vectors are sentence vectors obtained by oversampling the sample sentence vectors. The sample sentence vectors are sentence vectors obtained by extracting features from text samples based on the BERT model.
[0009] In one embodiment, the method further includes:
[0010] Based on the BERT model, feature extraction is performed on the text sample to obtain the output values corresponding to the first N layers of the BERT model, where N is a positive integer and N is greater than or equal to 2.
[0011] Based on the output values of the first N layers of the BERT model, the sample sentence vector corresponding to the text sample is determined, and the number of the sample sentence vectors is greater than or equal to 2.
[0012] Oversample the sample sentence vector corresponding to the text sample to obtain the data-enhanced sample sentence vector corresponding to the text sample;
[0013] Based on the data-enhanced sample sentence vectors corresponding to the text samples, the model is trained to obtain the text classification model.
[0014] In one embodiment, oversampling the sample sentence vector corresponding to the text sample to obtain the data-enhanced sample sentence vector corresponding to the text sample includes:
[0015] The data-enhanced sample sentence vector is obtained based on the average value of the characters corresponding to two or more sample sentence vectors and / or based on the random replacement between the first sample sentence vector and the second sample sentence vector;
[0016] Wherein, the first sample sentence vector and the second sample sentence vector are any two sentence vectors from the sample sentence vectors corresponding to the text sample.
[0017] In one embodiment, obtaining the data-enhanced sample sentence vector based on the average character value corresponding to two or more of the sample sentence vectors and / or based on random replacement between the first and second sample sentence vectors includes:
[0018] Determine one or more sets of the sample sentence vectors, wherein any set of the sample sentence vectors includes two or more of the sample sentence vectors;
[0019] Based on the average character value corresponding to the sentence vector of each group of sample sentence vectors, determine the data-enhanced sample sentence vector corresponding to each group of sample sentence vectors.
[0020] In one embodiment, obtaining the data-enhanced sample sentence vector based on the average character value corresponding to two or more of the sample sentence vectors and / or based on random replacement between the first and second sample sentence vectors includes:
[0021] Determine one or more random replacement ratios;
[0022] Based on each of the random replacement ratios, the first sample sentence vector and the second sample sentence vector are randomly replaced to obtain the data-enhanced sample sentence vector corresponding to each of the random replacement ratios.
[0023] In one embodiment, obtaining the target sentence vector corresponding to the target task text includes:
[0024] Based on a 12-layer BERT model, the target sentence vector corresponding to the target task text is obtained.
[0025] In one embodiment, the step of performing intent classification on the target sentence vector using a text classification model to obtain the target classification corresponding to the target sentence vector includes:
[0026] Based on the target number of convolutional kernels, text features are extracted from the target sentence vector to obtain the convolution result corresponding to the target sentence vector;
[0027] Perform a pooling operation on the convolution result corresponding to the target sentence vector to obtain the pooling result corresponding to the target sentence vector;
[0028] Based on a fully connected network, the pooling results corresponding to the target sentence vector are classified to obtain the target classification corresponding to the target sentence vector.
[0029] Secondly, embodiments of this application provide a task text classification device, comprising:
[0030] The first acquisition module is used to acquire the target sentence vector corresponding to the target task text;
[0031] The second acquisition module is used to perform intent classification on the target sentence vector using a text classification model to obtain the target classification corresponding to the target sentence vector;
[0032] The text classification model is trained based on data-augmented sample sentence vectors, which are obtained by oversampling the sample sentence vectors. The sample sentence vectors are obtained by feature extraction of text samples based on the BERT model.
[0033] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the method described in the first aspect.
[0034] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.
[0035] The task text classification method, apparatus, electronic device, and computer program product provided in this application extract features from text samples using the BERT model, thereby obtaining sample sentence vectors corresponding to the text samples. By oversampling the sample sentence vectors, data-enhanced sample sentence vectors corresponding to the text samples can be obtained, increasing the number of sentence vectors corresponding to the text samples. This, in turn, increases the amount of task text data used for model training, thereby improving the generalization ability of the text classification model. Attached Figure Description
[0036] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0037] Figure 1 This is one of the flowcharts illustrating the task text classification method provided in this application embodiment;
[0038] Figure 2 This is a second flowchart illustrating the task text classification method provided in this application embodiment;
[0039] Figure 3 This is a schematic diagram of the convolutional pooling operation provided in an embodiment of this application;
[0040] Figure 4 This is a schematic diagram of the structure of the task text classification device provided in the embodiments of this application;
[0041] Figure 5 The present application provides a schematic diagram of the structure of an electronic device. Detailed Implementation
[0042] To facilitate a clearer understanding of the various embodiments of this application, some relevant background knowledge will be introduced as follows.
[0043] In the field of natural language processing, text data augmentation methods mainly include: Easy Data Augmentation for Text Classification Tasks (EDA) and reverse translation methods; among which:
[0044] (1) EDA methods, which achieve data augmentation by adding noise, such as synonym replacement, random insertion, or random deletion, etc.
[0045] Synonym replacement, where synonyms are very similar in word vector representation, results in two texts with high similarity after replacement, but in essence, it does not effectively expand the dataset.
[0046] Random insertion, without considering stop words, will insert some low-value information into the sample data, and random insertion will also cause chaos in the original semantic and word order structure. The insertion of synonyms does not focus on the keywords of the sample, and it is also more limited in expanding the diversity of the sample.
[0047] Random deletion has the drawbacks of random insertion. If the randomly deleted information is key information in the sample statement, it will also lead to problems with the correctness of the sample label.
[0048] It is evident that simple data augmentation methods require manual construction of the thesaurus and continuous maintenance, resulting in significant costs. As the data sample size gradually increases, these methods show insufficient support in experiments, and both synonym replacement and random deletion reduce the model's generalization ability.
[0049] (2) Reverse translation method: This method first uses a machine to translate Chinese into English and then translates English back into Chinese. It can increase the diversity of samples to a certain extent, but this method relies too much on the effect of machine translation. Moreover, current machine translation also makes a lot of mistakes, and the process is difficult to automate.
[0050] To overcome the above-mentioned shortcomings, this application provides a task text classification method, apparatus, electronic device, and computer program product. By oversampling the sample sentence vectors, the number of sentence vectors corresponding to the text samples can be increased, thereby increasing the amount of task text data used for model training.
[0051] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0052] Figure 1 This is one of the flowcharts illustrating the task text classification method provided in this application embodiment, such as... Figure 1 The present application provides a task text classification method, the execution subject of which can be an electronic device, and the method may include:
[0053] Step 101: Obtain the target sentence vector corresponding to the target task text;
[0054] Specifically, in order to perform intent classification on the target task text, the target sentence vector corresponding to the target task text can be obtained first, and then the target sentence vector corresponding to the target task text can be input into the text classification model for intent classification processing.
[0055] Optionally, the target task text can be the task text in a task-oriented dialogue scenario.
[0056] Step 102: Using a text classification model, perform intent classification on the target sentence vector to obtain the target classification corresponding to the target sentence vector;
[0057] The text classification model is a model trained on data-augmented sample sentence vectors. The data-augmented sample sentence vectors are sentence vectors obtained by oversampling the sample sentence vectors. The sample sentence vectors are sentence vectors obtained by extracting features from text samples based on the BERT model.
[0058] Specifically, after obtaining the target sentence vector corresponding to the target task text, the target sentence vector can be classified according to intent based on the text classification model. The probability of the target sentence vector under multiple intent classifications can be obtained by the text classification model, and then the intent classification with the highest probability can be determined as the target classification corresponding to the target sentence vector.
[0059] It is understood that the text classification model can be a model obtained by training with data-enhanced sample sentence vectors. The data-enhanced sample sentence vectors can be sample data obtained by expanding the original text sample data. The number of data-enhanced sample sentence vectors is greater than the number of original text sample data. Compared with the module obtained by training with the original text sample data, the model obtained by training with data-enhanced sample sentence vectors in this embodiment has better generalization ability.
[0060] Optionally, in order to obtain data-enhanced sample sentence vectors, features can be extracted from text samples based on the BERT model, thereby obtaining sample sentence vectors corresponding to the text samples, and then oversampling can be performed on the sample sentence vectors corresponding to the text samples to obtain data-enhanced sample sentence vectors corresponding to the text samples.
[0061] Optionally, it can be applied to short text classification scenarios. The target task text can be a target short text, and then the target sentence vector corresponding to the target short text can be obtained. Then, the intent classification can be performed on the target sentence vector, and the target classification corresponding to the target short text can be obtained.
[0062] The task text classification method provided in this application uses the BERT model to extract features from text samples, thereby obtaining sample sentence vectors corresponding to the text samples. By oversampling the sample sentence vectors, data-enhanced sample sentence vectors corresponding to the text samples can be obtained, which can increase the number of sentence vectors corresponding to the text samples, thereby increasing the amount of task text data used for model training and improving the generalization ability of the text classification model.
[0063] Optionally, the method further includes:
[0064] Based on the BERT model, feature extraction is performed on the text sample to obtain the output values corresponding to the first N layers of the BERT model, where N is a positive integer and N is greater than or equal to 2.
[0065] Based on the output values of the first N layers of the BERT model, the sample sentence vector corresponding to the text sample is determined, and the number of the sample sentence vectors is greater than or equal to 2.
[0066] Oversample the sample sentence vector corresponding to the text sample to obtain the data-enhanced sample sentence vector corresponding to the text sample;
[0067] Based on the data-enhanced sample sentence vectors corresponding to the text samples, the model is trained to obtain the text classification model.
[0068] Specifically, to obtain a trained text classification model, features can be extracted from text samples using the BERT model. The output values corresponding to the first N layers of the BERT model can be obtained. Then, two or more output values from the first N layers of the BERT model can be used as the sample sentence vectors corresponding to the text samples. Then, the sample sentence vectors corresponding to the text samples can be oversampled to obtain the data-augmented sample sentence vectors corresponding to the text samples. Finally, the model can be trained based on the data-augmented sample sentence vectors corresponding to the text samples to obtain the trained text classification model.
[0069] Optionally, it can be applied to short text classification scenarios. The text sample can be a short text sample. Based on the BERT model, feature extraction can be performed on the short text sample to obtain the output values corresponding to the first N layers of the BERT model, where N is a positive integer and N is greater than or equal to 2.
[0070] Optionally, sentence vectors (embeddings) can be generated from the input task-oriented dialogue corpus based on a 12-layer pre-trained Chinese BERT model. The first 6 layers of the 12-layer embedding generated by the BERT model can be selected for embedding-based oversampling to obtain data-enhanced sample sentence vectors corresponding to the text samples.
[0071] Understandably, the BERT model generates sentence vectors at different layers for data augmentation without losing basic semantic information. Furthermore, by oversampling the sentence vectors corresponding to the text samples, automated sample data augmentation can be achieved without manual maintenance.
[0072] Therefore, by extracting features from text samples using the BERT model, we can obtain the corresponding sample sentence vectors. By oversampling the sample sentence vectors, we can obtain the data-enhanced sample sentence vectors corresponding to the text samples, thereby increasing the number of sentence vectors corresponding to the text samples. This can increase the amount of task text data used for model training, and thus improve the generalization ability of the text classification model.
[0073] Optionally, the step of oversampling the sample sentence vector corresponding to the text sample to obtain the data-enhanced sample sentence vector corresponding to the text sample includes:
[0074] The data-enhanced sample sentence vector is obtained based on the average value of the characters corresponding to two or more sample sentence vectors and / or based on the random replacement between the first sample sentence vector and the second sample sentence vector;
[0075] Wherein, the first sample sentence vector and the second sample sentence vector are any two sentence vectors from the sample sentence vectors corresponding to the text sample.
[0076] Optionally, data-enhanced sample sentence vectors can be obtained based on the average value of characters corresponding to two or more sample sentence vectors.
[0077] Optionally, data-enhanced sample sentence vectors can be obtained based on random replacement between the first sample sentence vector and the second sample sentence vector.
[0078] Understandably, the oversampling process can be achieved by averaging the characters (tokens) in multiple sentence embeddings and / or by randomly replacing the characters (tokens) in two sentence embeddings to generate new corpus at the sentence embedding level.
[0079] Optionally, data-enhanced sample sentence vectors can be obtained based on the average value of characters corresponding to two or more sample sentence vectors, and data-enhanced sample sentence vectors can be obtained based on random replacement between the first sample sentence vector and the second sample sentence vector.
[0080] Understandably, by processing the sample sentence vectors at the character (token) level, sample data can be quickly expanded, providing multiple sentence vector (embedding) calculation and generation methods, resulting in a large amount of data expansion.
[0081] Therefore, by using various sentence vector calculation and generation methods, sample data can be quickly expanded, the number of sentence vectors corresponding to text samples can be increased, and the amount of task text data used for model training can be increased, thereby improving the generalization ability of the text classification model.
[0082] Optionally, obtaining the data-enhanced sample sentence vector based on the average character values corresponding to two or more of the sample sentence vectors and / or based on random replacement between the first and second sample sentence vectors includes:
[0083] Determine one or more sets of the sample sentence vectors, wherein any set of the sample sentence vectors includes two or more of the sample sentence vectors;
[0084] Based on the average character value corresponding to the sentence vector of each group of sample sentence vectors, determine the data-enhanced sample sentence vector corresponding to each group of sample sentence vectors.
[0085] Specifically, for the sample sentence vectors corresponding to the text samples, one or more sets of sample sentence vectors can be determined, and then the data-enhanced sample sentence vectors corresponding to each set of sample sentence vectors can be determined based on the average value of the characters corresponding to the sentence vectors of each set of sample sentence vectors.
[0086] Therefore, by using various sentence vector calculation and generation methods, sample data can be quickly expanded, the number of sentence vectors corresponding to text samples can be increased, and the amount of task text data used for model training can be increased, thereby improving the generalization ability of the text classification model.
[0087] Optionally, obtaining the data-enhanced sample sentence vector based on the average character values corresponding to two or more of the sample sentence vectors and / or based on random replacement between the first and second sample sentence vectors includes:
[0088] Determine one or more random replacement ratios;
[0089] Based on each of the random replacement ratios, the first sample sentence vector and the second sample sentence vector are randomly replaced to obtain the data-enhanced sample sentence vector corresponding to each of the random replacement ratios.
[0090] Specifically, one or more random replacement ratios can be determined for the sample sentence vector corresponding to the text sample. Then, based on each random replacement ratio, the first sample sentence vector and the second sample sentence vector can be randomly replaced to obtain the data-enhanced sample sentence vector corresponding to each random replacement ratio.
[0091] Therefore, by using various sentence vector calculation and generation methods, sample data can be quickly expanded, the number of sentence vectors corresponding to text samples can be increased, and the amount of task text data used for model training can be increased, thereby improving the generalization ability of the text classification model.
[0092] Optionally, Figure 2 This is a second flowchart illustrating the task text classification method provided in this application embodiment, as shown below. Figure 2 As shown, the training process of the text classification model may include steps 201 to 205, wherein:
[0093] Step 201, Text data acquisition and preprocessing;
[0094] Optionally, query statements can be collected and written for various task query scenarios, and data can be preprocessed and cleaned (including filtering stop words and abnormal symbols) so that the preprocessed output format can be consistent with the format required by the BERT model input.
[0095] Understandably, text samples can be obtained through text data collection and preprocessing, and the format of these text samples is consistent with the input format required by the BERT model.
[0096] Step 202: Based on the BERT model, generate 12 layers of sentence vectors and extract the sentence vectors from the first 6 layers;
[0097] Optionally, the BERT model can be used to generate embeddings for the input text corpus (i.e., the text samples obtained in step 201). A 12-layer Chinese pre-trained BERT model can be selected to generate sentence vectors, and the sentence vectors output by the first 6 layers can be selected as sample sentence vectors. These sample sentence vectors can be used as input for step 203.
[0098] It is understandable that the output value of each transformer encoder layer in the BERT model can be used as a sentence vector. Compared with the output of the later layers, the output of the earlier layers has not yet fully learned the semantic features and is more inclined to word-level embeddings. Therefore, the sentence vectors output by the first 6 layers can be selected as sample sentence vectors.
[0099] Optionally, the BERT model performs word segmentation on Chinese text at the character level. The sentence vector generated by each of the first 6 layers of the BERT model has a dimension of [b, m, 768], where b is the input batch, m is the maximum sentence length, and 768 is the embedding dimension of each character (token). The maximum sentence length also includes two special characters [cls] and [sep], which represent the beginning and end of the sentence, respectively. These two character vectors can be removed, leaving the remaining real text character vectors, and then the sample sentence vector generated by each layer can be obtained. The sample sentence vector has a dimension of [b, m-2, 768].
[0100] Optionally, when the input batch b is 1 (i.e., when a single text sample is input), the dimension of the sample sentence vector generated at each layer is [m-2, 768].
[0101] It is understandable that the input batch b represents the number of samples input to the model each time it is trained. For example, when b is 32, it means that 32 text samples are input at a time for parallel training.
[0102] Optionally, after processing in step 202, the dimension of the sample sentence vector generated by each layer of the BERT model is [m-2, 768], and thus the dimension of the sample sentence vector generated by the first 6 layers of the BERT model is [6, m-2, 768].
[0103] Step 203: Perform oversampling based on OTM;
[0104] Optionally, after processing in step 202, a text sample can obtain a total of 6-dimensional sample sentence vectors [6, m-2, 768], where each dimension of the sample sentence vector [m-2, 768] can represent the text sample. Then, oversampling on token embedding (OTM) can be used to oversample each dimension of the sample sentence vector [m-2, 768]. OTM can be oversampling by calculating the average value of characters, or it can be oversampling by randomly replacing two sample sentence vectors.
[0105] Optionally, the oversampling for calculating the character average may specifically include: based on the correspondence between the above-mentioned 6-dimensional sample sentence vector [6, m-2, 768] and each layer in the BERT model, the 6-dimensional sample sentence vector [6, m-2, 768] can be marked as the first to sixth layers. Then, sample sentence vectors corresponding to two or more layers can be extracted from the first to the sixth layers, the character average can be calculated, and the data-augmented sample sentence vector can be determined.
[0106] For example, by extracting sample sentence vectors from two corresponding layers (e.g., layer 1 and layer 4, layer 2 and layer 5, layer 3 and layer 6) from layers 1 to 6, and calculating the character average, the data-augmented sample sentence vectors can be determined. The character average can be calculated using the following formula:
[0107]
[0108] In the above formula for the average character value, a and b represent the sample sentence vectors of the two layers extracted.
[0109] Optionally, by extracting the corresponding sample sentence vectors from two layers from the first to the sixth layer and calculating the character average, three or more data augmentation processes can be performed, resulting in three or more data-augmented sample sentence vectors.
[0110] Optionally, by extracting sample sentence vectors corresponding to three layers from the first to the sixth layer (e.g., the first, third, and fifth layers, or the second, fourth, and sixth layers), and calculating the character average, two or more data augmentation processes can be performed, resulting in two or more data-augmented sample sentence vectors.
[0111] Optionally, by extracting sample sentence vectors corresponding to four layers from the first to the sixth layer (e.g., the first, second, third, and fourth layers, or the third, fourth, fifth, and sixth layers), and calculating the character average, two or more data augmentation processes can be performed to obtain two or more data-augmented sample sentence vectors.
[0112] Optionally, by extracting two layers of corresponding sample sentence vectors and calculating the character average, at least three data-enhanced sample sentence vectors can be obtained. By extracting three layers of corresponding sample sentence vectors and calculating the character average, at least two data-enhanced sample sentence vectors can be obtained. By extracting four layers of corresponding sample sentence vectors and calculating the character average, at least two data-enhanced sample sentence vectors can be obtained. Thus, after processing the above 6-dimensional sample sentence vectors [6, m-2, 768] by calculating the character average, at least seven data-enhanced sample sentence vectors can be obtained.
[0113] It is understandable that the data-enhanced sample sentence vectors obtained through step 203 can maintain the same dimension as the sample sentence vectors obtained in step 202. That is, the dimension of each data-enhanced sample sentence vector obtained in step 203 can be [m-2, 768].
[0114] Optionally, oversampling by randomly replacing the two sample sentence vectors can specifically include: data-enhanced sample sentence vectors based on the random replacement between the first sample sentence vector and the second sample sentence vector, wherein the first sample sentence vector and the second sample sentence vector can be sentence vectors corresponding to any two dimensions of the above-mentioned 6-dimensional sample sentence vectors [6, m-2, 768].
[0115] Optionally, a random replacement ratio can be determined, and then, based on the random replacement ratio, the first sample sentence vector and the second sample sentence vector can be randomly replaced to obtain the data-enhanced sample sentence vector corresponding to the random replacement ratio.
[0116] It is understandable that the random replacement ratio can represent the proportion of the number of replaced tokens to the total number of tokens in the sample sentence vector.
[0117] Understandably, the 6-layer sentence vector generated by the BERT model is produced through its self-attention mechanism and feedforward neural network. Based on the self-attention mechanism, each character vector (token embedding) carries semantic information of other characters (tokens). Token-level random replacement can be performed using the sentence vectors generated from samples. The replacement method can be executed based on two conditions, where:
[0118] The first condition is to ensure that the position of the replaced token remains unchanged within its respective sample.
[0119] The second condition is to randomly replace tokens according to the proportion of the number of replaced tokens to the total number of tokens in the sample sentence vector (e.g., 10%, 20%, 30%, etc.).
[0120] Optionally, three random replacement ratios can be determined: 10%, 20%, and 30%. Then, based on each random replacement ratio, the first sample sentence vector and the second sample sentence vector can be randomly replaced to obtain the data-enhanced sample sentence vector corresponding to the 10% random replacement ratio, the data-enhanced sample sentence vector corresponding to the 20% random replacement ratio, and the data-enhanced sample sentence vector corresponding to the 30% random replacement ratio.
[0121] Understandably, given a set of 3 random replacement ratios, two sample sentence vectors (i.e., the first sample sentence vector and the second sample sentence vector) can be extracted each time. Randomly replacing these two sample sentence vectors can yield at least 3 data-enhanced sample sentence vectors.
[0122] Optionally, OTM can be a combination of oversampling that calculates the average value of characters and oversampling that randomly replaces two sample sentence vectors. By oversampling that calculates the average value of characters, at least 7 data-enhanced sample sentence vectors can be obtained. By oversampling that randomly replaces two sample sentence vectors, at least 3 data-enhanced sample sentence vectors can be obtained. That is, the above 6-dimensional sample sentence vector [6, m-2, 768] corresponding to a text sample can be oversampled by OTM to obtain at least 10 data-enhanced sample sentence vectors.
[0123] Understandably, for a given text sample, at least 10 data-enhanced sample sentence vectors can be obtained, which can fully expand the sample data. The expanded sample data can be used to train the text classification model and improve the generalization ability of the trained model.
[0124] Step 204, one-dimensional convolutional text feature extraction;
[0125] Optionally, one-dimensional convolutional text feature extraction may include a convolutional feature extraction process and a pooling feature condensation process.
[0126] Optionally, Figure 3 This is a schematic diagram of the convolutional pooling operation provided in the embodiments of this application, as shown below. Figure 3 As shown, step 203 can obtain multiple data-enhanced sample sentence vectors corresponding to the text sample. The dimension of each data-enhanced sample sentence vector is [m-2, 768]. For the above convolutional feature extraction process, the convolution kernel size can be set to [2, 768] and the number of input channels can be 1. Text features can be extracted from the data-enhanced sample sentence vectors using 10 convolution kernels. The convolution result corresponding to the data-enhanced sample sentence vector can be obtained. The vector dimension of the convolution result can be [10, m-3, 1].
[0127] Optionally, such as Figure 3 As shown, for the above pooling feature condensation process, max pooling can be performed on the convolution result corresponding to the data augmented sample sentence vector to obtain the pooling result corresponding to the data augmented sample sentence vector. The vector dimension of the pooling result can be [10, 1, 1].
[0128] Understandably, max pooling can reduce the dimensionality of feature vectors while maintaining certain invariant properties. During this process, a Rectified Linear Unit (RELU) can be chosen as the activation function, which can alleviate gradient vanishing caused by deep neural networks to some extent. One-dimensional convolutional text feature extraction can extract the features of the input sentence vector to the greatest extent.
[0129] Step 205: FC combines with Softmax to output the intent category.
[0130] Optionally, after step 204, the pooling result corresponding to the data-enhanced sample sentence vector can be obtained, with a dimension of [10, 1, 1]. After the dimension is flattened, it can be input into a fully connected network layer (FC). The fully connected network layer can map the distributed feature representation obtained by the convolutional layer in step 204 to the sample label space. The mapping process can be based on matrix-vector multiplication and feature vector weighting.
[0131] Optionally, the dimension of the matrix output by FC can be [b, n], where b is the batch size and n is the number of text label categories. The position of the maximum value in each row of the matrix can represent the category of the sample.
[0132] Alternatively, the matrix output by the fully connected (FC) system can be converted into a true probability distribution using the softmax function, which can be expressed by the following formula:
[0133]
[0134] Where h(x,y) i h(x,y) is an element value of the matrix output by FC. j ) represents an element value of the matrix output by FC, and n represents the number of text label categories.
[0135] Alternatively, a multi-class cross-entropy loss function can be used during training, which can be expressed by the following formula:
[0136]
[0137] Where n is the number of text label categories.
[0138] It is understandable that after multiple rounds of training on the model using sample data augmented with OTM data augmentation, the various test metrics on the test set can be better than those of the model trained without sample data augmentation.
[0139] Therefore, by using various sentence vector calculation and generation methods, sample data can be quickly expanded, the number of sentence vectors corresponding to text samples can be increased, and the amount of task text data used for model training can be increased, thereby improving the generalization ability of the text classification model.
[0140] Optionally, obtaining the target sentence vector corresponding to the target task text includes:
[0141] Based on a 12-layer BERT model, the target sentence vector corresponding to the target task text is obtained.
[0142] Specifically, in order to perform intent classification on the target task text, a 12-layer BERT model can be used to obtain the target sentence vector corresponding to the target task text. Then, the target sentence vector corresponding to the target task text can be input into the text classification model for intent classification processing.
[0143] Specifically, after obtaining the target sentence vector corresponding to the target task text, the target sentence vector can be classified according to intent based on the text classification model. The probability of the target sentence vector under multiple intent classifications can be obtained by the text classification model, and then the intent classification with the highest probability can be determined as the target classification corresponding to the target sentence vector.
[0144] Optionally, the target task text is input into the BERT model. The sentence vector generated by each of the first 6 layers of the BERT model has a dimension of [b, m, 768], where b is the input batch, m is the maximum sentence length, and 768 is the embedding dimension of each character (token). The maximum sentence length here also includes two special characters [cls] and [sep], which represent the beginning and end of the sentence, respectively. These two character vectors can be removed, leaving the remaining real text character vectors. Thus, the dimension of the target sentence vector generated by each layer can be obtained as [b, m-2, 768]. The output of a certain layer in the first 6 layers of the BERT model can be selected as the target sentence vector corresponding to the target task text.
[0145] Optionally, when the input batch b is 1 (i.e., when a target task text is input), the dimension of the target sentence vector generated at each layer is [m-2, 768].
[0146] It is understandable that the input batch b represents the number of target task texts input into the BERT model for each target classification. For example, when b is 32, it means that 32 target task texts are input at a time for parallel target classification.
[0147] Optionally, a target task text is input into the BERT model. The target sentence vector generated by each layer of the BERT model has a dimension of [m-2, 768]. The output of one of the first 6 layers of the BERT model can be selected as the target sentence vector corresponding to the target task text.
[0148] Therefore, by extracting features from text samples using the BERT model, we can obtain the corresponding sample sentence vectors. By oversampling the sample sentence vectors, we can obtain the data-enhanced sample sentence vectors corresponding to the text samples, thereby increasing the number of sentence vectors corresponding to the text samples. This can increase the amount of task text data used for model training, and thus improve the generalization ability of the text classification model.
[0149] Optionally, the step of performing intent classification on the target sentence vector using a text classification model to obtain the target classification corresponding to the target sentence vector includes:
[0150] Based on the target number of convolutional kernels, text features are extracted from the target sentence vector to obtain the convolution result corresponding to the target sentence vector;
[0151] Perform a pooling operation on the convolution result corresponding to the target sentence vector to obtain the pooling result corresponding to the target sentence vector;
[0152] Based on a fully connected network, the pooling results corresponding to the target sentence vector are classified to obtain the target classification corresponding to the target sentence vector.
[0153] Specifically, after obtaining the target sentence vector corresponding to the target task text, text features can be extracted from the target sentence vector based on the target number of convolution kernels to obtain the convolution result corresponding to the target sentence vector. Then, pooling operation can be performed on the convolution result corresponding to the target sentence vector to obtain the pooling result corresponding to the target sentence vector. Finally, the pooling result corresponding to the target sentence vector can be classified through a fully connected network to obtain the target classification corresponding to the target sentence vector.
[0154] Optionally, a target sentence vector corresponding to a target task text can be obtained using the BERT model. The dimension of the target sentence vector can be [m-2, 768], such as... Figure 3 As shown, the convolution kernel size can be set to [2, 768], the number of input channels is 1, and the text features of the target sentence vector can be extracted by 10 convolution kernels. The convolution result corresponding to the target sentence vector can be obtained, and the vector dimension of the convolution result can be [10, m-3, 1].
[0155] Optionally, such as Figure 3 As shown, max pooling can be performed on the convolution result corresponding to the target sentence vector to obtain the pooling result corresponding to the target sentence vector. The vector dimension of the pooling result can be [10, 1, 1].
[0156] Understandably, max pooling can reduce the dimensionality of feature vectors while maintaining certain invariant properties. During this process, ReLU can be chosen as the activation function, which can alleviate gradient vanishing caused by deep neural networks to some extent and can extract the features of the input sentence vector to the greatest extent.
[0157] Optionally, after obtaining the pooling result corresponding to the target sentence vector, the dimension of the pooling result can be [10, 1, 1]. After the dimension is flattened, it can be input into a fully connected network layer (FC). The fully connected network layer can map the distributed feature representation of the pooling result to the sample label space. The mapping process can be based on matrix-vector multiplication and feature vector weighting.
[0158] Optionally, the dimension of the matrix output by FC can be [b, n], where b is the batch size and n is the number of text label categories. The position of the maximum value in each row of the matrix can represent the category of the target task text.
[0159] Alternatively, the matrix output by the fully connected (FC) system can be converted into a true probability distribution using the softmax function, which can be expressed by the following formula:
[0160]
[0161] Where h(x,y) i h(x,y) is an element value of the matrix output by FC. j ) represents an element value of the matrix output by FC, and n represents the number of text label categories.
[0162] The task text classification method provided in this application uses the BERT model to extract features from text samples, thereby obtaining sample sentence vectors corresponding to the text samples. By oversampling the sample sentence vectors, data-enhanced sample sentence vectors corresponding to the text samples can be obtained, which can increase the number of sentence vectors corresponding to the text samples, thereby increasing the amount of task text data used for model training and improving the generalization ability of the text classification model.
[0163] The task text classification device provided in the embodiments of this application will be described below. The task text classification device described below can be referred to in correspondence with the task text classification method described above.
[0164] Figure 4 This is a schematic diagram of the structure of the task text classification device provided in the embodiments of this application, as shown below. Figure 4 As shown, the device includes: a first acquisition module 401 and a second acquisition module 402, wherein:
[0165] The first acquisition module 401 is used to acquire the target sentence vector corresponding to the target task text;
[0166] The second acquisition module 402 is used to perform intent classification on the target sentence vector through a text classification model to obtain the target classification corresponding to the target sentence vector;
[0167] The text classification model is trained based on data-augmented sample sentence vectors, which are obtained by oversampling the sample sentence vectors. The sample sentence vectors are obtained by feature extraction of text samples based on the BERT model.
[0168] The task text classification device provided in this application embodiment extracts features from text samples using the BERT model, thereby obtaining sample sentence vectors corresponding to the text samples. By oversampling the sample sentence vectors, data-enhanced sample sentence vectors corresponding to the text samples can be obtained, increasing the number of sentence vectors corresponding to the text samples. This, in turn, increases the amount of task text data used for model training, thereby improving the generalization ability of the text classification model.
[0169] Optionally, the device further includes a training module, the training module being used for:
[0170] Based on the BERT model, feature extraction is performed on the text sample to obtain the output values corresponding to the first N layers of the BERT model, where N is a positive integer and N is greater than or equal to 2.
[0171] Based on the output values of the first N layers of the BERT model, the sample sentence vector corresponding to the text sample is determined, and the number of the sample sentence vectors is greater than or equal to 2.
[0172] Oversample the sample sentence vector corresponding to the text sample to obtain the data-enhanced sample sentence vector corresponding to the text sample;
[0173] Based on the data-enhanced sample sentence vectors corresponding to the text samples, the model is trained to obtain the text classification model.
[0174] Optionally, the training module is specifically used for:
[0175] The data-enhanced sample sentence vector is obtained based on the average value of the characters corresponding to two or more sample sentence vectors and / or based on the random replacement between the first sample sentence vector and the second sample sentence vector;
[0176] Wherein, the first sample sentence vector and the second sample sentence vector are any two sentence vectors from the sample sentence vectors corresponding to the text sample.
[0177] Optionally, the training module is specifically used for:
[0178] Determine one or more sets of the sample sentence vectors, wherein any set of the sample sentence vectors includes two or more of the sample sentence vectors;
[0179] Based on the average character value corresponding to the sentence vector of each group of sample sentence vectors, determine the data-enhanced sample sentence vector corresponding to each group of sample sentence vectors.
[0180] Optionally, the training module is specifically used for:
[0181] Determine one or more random replacement ratios;
[0182] Based on each of the random replacement ratios, the first sample sentence vector and the second sample sentence vector are randomly replaced to obtain the data-enhanced sample sentence vector corresponding to each of the random replacement ratios.
[0183] Optionally, the first acquisition module is specifically used for:
[0184] Based on a 12-layer BERT model, the target sentence vector corresponding to the target task text is obtained.
[0185] Optionally, the second acquisition module is specifically used for:
[0186] Based on the target number of convolutional kernels, text features are extracted from the target sentence vector to obtain the convolution result corresponding to the target sentence vector;
[0187] Perform a pooling operation on the convolution result corresponding to the target sentence vector to obtain the pooling result corresponding to the target sentence vector;
[0188] Based on a fully connected network, the pooling results corresponding to the target sentence vector are classified to obtain the target classification corresponding to the target sentence vector.
[0189] The task text classification device provided in this application embodiment extracts features from text samples using the BERT model, thereby obtaining sample sentence vectors corresponding to the text samples. By oversampling the sample sentence vectors, data-enhanced sample sentence vectors corresponding to the text samples can be obtained, increasing the number of sentence vectors corresponding to the text samples. This, in turn, increases the amount of task text data used for model training, thereby improving the generalization ability of the text classification model.
[0190] Figure 5 The structural schematic diagram of the electronic device provided in the embodiments of this application is as follows: Figure 5 As shown, the electronic device may include a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call a computer program in the memory 530 to execute the steps of a task text classification method, such as including:
[0191] Obtain the target sentence vector corresponding to the target task text;
[0192] The target sentence vector is classified by intent using a text classification model to obtain the target classification corresponding to the target sentence vector;
[0193] The text classification model is a model trained on data-augmented sample sentence vectors. The data-augmented sample sentence vectors are sentence vectors obtained by oversampling the sample sentence vectors. The sample sentence vectors are sentence vectors obtained by extracting features from text samples based on the BERT model.
[0194] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0195] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the task text classification method provided in the above embodiments, such as including:
[0196] Obtain the target sentence vector corresponding to the target task text;
[0197] The target sentence vector is classified by intent using a text classification model to obtain the target classification corresponding to the target sentence vector;
[0198] The text classification model is a model trained on data-augmented sample sentence vectors. The data-augmented sample sentence vectors are sentence vectors obtained by oversampling the sample sentence vectors. The sample sentence vectors are sentence vectors obtained by extracting features from text samples based on the BERT model.
[0199] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to perform the steps of the methods provided in the above embodiments, such as including:
[0200] Obtain the target sentence vector corresponding to the target task text;
[0201] The target sentence vector is classified by intent using a text classification model to obtain the target classification corresponding to the target sentence vector;
[0202] The text classification model is a model trained on data-augmented sample sentence vectors. The data-augmented sample sentence vectors are sentence vectors obtained by oversampling the sample sentence vectors. The sample sentence vectors are sentence vectors obtained by extracting features from text samples based on the BERT model.
[0203] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).
[0204] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0205] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0206] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A task text classification method, characterized in that, include: Obtain the target sentence vector corresponding to the target task text; The target sentence vector is classified by intent using a text classification model to obtain the target classification corresponding to the target sentence vector; The text classification model is a model trained on data-augmented sample sentence vectors. The data-augmented sample sentence vectors are sentence vectors obtained by oversampling the sample sentence vectors. The sample sentence vectors are sentence vectors obtained by extracting features from text samples based on the BERT model. The method further includes: Based on the BERT model, feature extraction is performed on the text sample to obtain the output values corresponding to the first N layers of the BERT model, where N is a positive integer and N is greater than or equal to 2. Based on the output values of the first N layers of the BERT model, the sample sentence vector corresponding to the text sample is determined, and the number of the sample sentence vectors is greater than or equal to 2. Oversample the sample sentence vector corresponding to the text sample to obtain the data-enhanced sample sentence vector corresponding to the text sample; Based on the data-enhanced sample sentence vectors corresponding to the text samples, the model is trained to obtain the text classification model; The step of oversampling the sample sentence vector corresponding to the text sample to obtain the data-enhanced sample sentence vector corresponding to the text sample includes: The data-enhanced sample sentence vector is obtained by averaging the characters corresponding to two or more sample sentence vectors and / or by randomly replacing the first sample sentence vector with the second sample sentence vector. Wherein, the first sample sentence vector and the second sample sentence vector are any two sentence vectors from the sample sentence vectors corresponding to the text sample.
2. The task text classification method according to claim 1, characterized in that, The step of obtaining the data-enhanced sample sentence vector based on the average of the characters corresponding to two or more sample sentence vectors, and / or based on random replacement between the first and second sample sentence vectors, includes: Determine one or more sets of the sample sentence vectors, wherein any set of the sample sentence vectors includes two or more of the sample sentence vectors; Based on the average value of the characters corresponding to the sentence vectors of each group of sample sentence vectors, the data-enhanced sample sentence vectors corresponding to each group of sample sentence vectors are determined.
3. The task text classification method according to claim 1, characterized in that, The step of obtaining the data-enhanced sample sentence vector based on the average of the characters corresponding to two or more sample sentence vectors, and / or based on random replacement between the first and second sample sentence vectors, includes: Determine one or more random replacement ratios; Based on each of the random replacement ratios, the first sample sentence vector and the second sample sentence vector are randomly replaced to obtain the data-enhanced sample sentence vector corresponding to each of the random replacement ratios.
4. The task text classification method according to any one of claims 1-3, characterized in that, The step of obtaining the target sentence vector corresponding to the target task text includes: Based on a 12-layer BERT model, the target sentence vector corresponding to the target task text is obtained.
5. The task text classification method according to any one of claims 1-3, characterized in that, The step of performing intent classification on the target sentence vector using a text classification model to obtain the target classification corresponding to the target sentence vector includes: Based on the target number of convolutional kernels, text features are extracted from the target sentence vector to obtain the convolution result corresponding to the target sentence vector; Perform a pooling operation on the convolution result corresponding to the target sentence vector to obtain the pooling result corresponding to the target sentence vector; Based on a fully connected network, the pooling results corresponding to the target sentence vector are classified to obtain the target classification corresponding to the target sentence vector.
6. A task text classification device, characterized in that, include: The first acquisition module is used to acquire the target sentence vector corresponding to the target task text; The second acquisition module is used to perform intent classification on the target sentence vector using a text classification model to obtain the target classification corresponding to the target sentence vector; The text classification model is trained based on data-augmented sample sentence vectors, which are obtained by oversampling the sample sentence vectors. The sample sentence vectors are obtained by feature extraction of text samples based on the BERT model. The task text classification device further includes a training module, which is used for: Based on the BERT model, feature extraction is performed on the text sample to obtain the output values corresponding to the first N layers of the BERT model, where N is a positive integer and N is greater than or equal to 2. Based on the output values of the first N layers of the BERT model, the sample sentence vector corresponding to the text sample is determined, and the number of the sample sentence vectors is greater than or equal to 2. Oversample the sample sentence vector corresponding to the text sample to obtain the data-enhanced sample sentence vector corresponding to the text sample; Based on the data-enhanced sample sentence vectors corresponding to the text samples, the model is trained to obtain the text classification model; The training module is specifically used for: The data-enhanced sample sentence vector is obtained by averaging the characters corresponding to two or more sample sentence vectors and / or by randomly replacing the first sample sentence vector with the second sample sentence vector. Wherein, the first sample sentence vector and the second sample sentence vector are any two sentence vectors from the sample sentence vectors corresponding to the text sample.
7. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the task text classification method according to any one of claims 1 to 5.
8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the task text classification method according to any one of claims 1 to 5.