An Intent Classification Method Based on Label Semantic Inference

Abstract

The present invention provides an intent classification method based on label semantic inference, comprising the following steps: S1: constructing a corpus dataset and expanding it using a back-translation method to obtain an expanded corpus dataset; S2: performing data cleaning to obtain an enhanced corpus dataset; S3: performing label semantic inference to obtain label semantics corresponding to each label; S4: training a pre-constructed text classification model to obtain a trained text classification model; S5: using the trained text classification model to extract shallow and deep features of the corpus data and label semantics from the text to be classified, respectively, and performing feature fusion to obtain an intent classification result. The present invention provides an intent classification method based on label semantic inference, which can make intent classification results more accurate and solves the problem that current text classification methods have difficulty in accurately classifying customer intent.

Description

Technical Field

[0001] The present invention relates to the technical field of classification and recognition, and more specifically, to an intent classification method based on label semantic inference. Background Art

[0002] Customer service staff need to identify customer intent based on conversations to provide high-quality service. However, customer service corpus data is highly repetitive, and the text data generated by conversations is colloquial, with multiple phrases, flexible language forms, and ambiguous intent. This can result in dozens of processing decisions, making intent identification a difficult task.

[0003] Traditional customer service robots achieve intent classification by building a corpus and using keyword matching and semantic similarity matching methods. However, with the increase in business volume, this method has become difficult to meet customer needs.

[0004] In recent years, artificial intelligence (AI) has been introduced for customer service. AI-based text classification methods can classify corpus data into corresponding intent categories. However, customer service corpus datasets are often small, such as those for financial customer service. This insufficient sample size makes it difficult for models to learn data features. Furthermore, the high degree of colloquialism leads to text redundancy in the data, and the short length of the corpus generated by the service makes it difficult to accurately classify customer intent using current text classification methods. Summary of the Invention

[0005] In order to overcome the technical defect that current text classification methods are difficult to accurately classify customer intentions, the present invention provides an intention classification method based on tag semantic inference.

[0006] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0007] An intent classification method based on label semantic inference includes the following steps:

[0008] S1: Construct a labeled corpus dataset and use the back-translation method to expand the corpus dataset to obtain an expanded corpus dataset;

[0009] S2: Perform classification prediction on the expanded corpus data to obtain labels for the expanded corpus data, perform data cleaning based on the labels, and obtain an enhanced corpus dataset;

[0010] S3: perform label semantic inference based on the similarity between the corpus data in the enhanced corpus dataset to obtain the label semantics corresponding to each label;

[0011] S4: Use the enhanced corpus dataset and the label semantics corresponding to each label to train the pre-built text classification model to obtain a trained text classification model;

[0012] S5: Use the trained text classification model to extract shallow and deep features of the corpus data and label semantics from the text to be classified, and perform feature fusion to obtain the intent classification result.

[0013] In the above scheme, the data set is expanded through the back-translation method. The expanded corpus data includes noise data, which can improve the robustness of the trained text classification model. The dirty data containing uncontrollable noise in the data set is cleaned to form a high-quality data set, which helps to improve the intent classification accuracy of the text classification model. At the same time, label semantics are inferred based on the corpus data to obtain label semantics, and external information is provided to the text classification model through label semantics. Then, the shallow features and deep features of the corpus data and label semantics are extracted respectively, and feature fusion is performed to make the intent classification results more accurate.

[0014] Preferably, the method of expanding the data of the corpus dataset by back-translation is as follows: expanding the data volume of the corpus dataset by back-translation of Japanese, Korean and English respectively.

[0015] In the above scheme, since Japanese and Korean contain Chinese characters and these two languages ​​are similar to Chinese, the back-translation method of Japanese and Korean is used. At the same time, there are a small number of English samples in the dataset, and English is also used for the back-translation method.

[0016] Preferably, step S2 is specifically: using the BERT classifier to classify the corpus data X i Expanded corpus data X i ′ Perform classification prediction and get X i ′ Label Y i ′ , change Y i ′ With X i Label Y i Compare and keep X with the same label i ′ , realize data cleaning, and combine the corpus data set and the retained X i ′ The enhanced corpus dataset is obtained by merging.

[0017] In the above solution, a semi-supervised learning method is introduced to automatically clean data that has features that are very different from the original text.

[0018] Preferably, in label semantic inference, solving the label semantic inference problem is transformed into finding the problem of minimizing the distance L between the corpus data and the label center, that is:

[0019]

[0020] Among them, n represents the total number of labels in the enhanced corpus dataset, D j Represents the label C in the enhanced corpus dataset i All the corpus data, Z i Indicates C i The label center;

[0021] By optimizing and solving the minimum distance between the corpus data features under the current label category and the total features of the current category, the corpus data with the minimum distance is finally set as the label semantics of the current category.

[0022] Preferably, in label semantics inference:

[0023] The distance between each corpus data and each category center is obtained by the following steps:

[0024] S3.1: Segment the corpus data in the enhanced corpus dataset and calculate the word frequency to obtain high-frequency words;

[0025] S3.2: Create a feature matrix for each piece of corpus data based on the high-frequency words, and perform PCA dimensionality reduction processing;

[0026] S3.3: Use the K-Means algorithm to perform clustering and calculate the distance between each corpus data and each category center;

[0027] The piece of corpus data closest to the category center in each category is used as the label semantics of the corresponding category.

[0028] Preferably, the text classification model obtains shallow features through BERT encoding and deep features through downstream tasks.

[0029] Preferably, obtaining shallow features through BERT encoding includes the following steps:

[0030] The first BERT encoder of the text classification model is used to encode the corpus data in the text. The first BERT encoder includes 12 BertLayer layers connected in sequence. The encoding outputs of the first and sixth BertLayer layers in the first BERT encoder are extracted respectively, corresponding to the first shallow feature a1 and the second shallow feature a2 of the corpus data;

[0031] The label semantics of the text are obtained through label semantic inference, and the obtained label semantics are directly spliced. The second BERT encoder of the text classification model is used to encode the spliced ​​label semantics. The second BERT encoder includes 12 BertLayer layers connected in sequence. The encoding outputs of the first BertLayer layer and the sixth BertLayer layer in the second BERT encoder are extracted respectively, and the first shallow feature b1 and the second shallow feature b2 of the label are obtained accordingly.

[0032] Preferably, deep features are obtained through downstream tasks including:

[0033] The first, second, and third convolutional neural networks of the text classification model are used to process the contextual semantic feature h1 output by the first BERT encoder, obtaining three corresponding convolution results. The three convolution results are concatenated with h1 to obtain the deep feature a3 of the corpus data.

[0034] The label semantic feature h2 output by the second BERT encoder is input into the BiLSTM of the text classification model. The output of BiLSTM is used as the input of the multi-head attention mechanism of the text classification model, and the output of the multi-head attention mechanism is used as the deep feature b3 of the label.

[0035] Preferably, the convolution kernel sizes of the first convolutional neural network, the second convolutional neural network, and the third convolutional neural network are (2,768), (3,768), and (4,768), respectively.

[0036] Preferably, the hidden layer dimension of the BiLSTM output is set to 1024, the multi-head attention mechanism is set to 4 heads and the hidden layer dimension is 852.

[0037] Preferably, the features are fused through dot product:

[0038] e1=1*1

[0039] e2=2*2

[0040] e3=3*3

[0041] After fusion, e1, e2, and e3 are concatenated. The concatenated results are then sequentially input into the linear layer and Softmax layer of the text classification model to obtain the final output. The maximum value of the final output is taken as the intent classification result.

[0042] Among them, e1 represents the result of the first shallow feature fusion, e2 represents the result of the second shallow feature fusion, and e3 represents the result of the deep feature fusion.

[0043] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0044] The present invention provides an intent classification method based on label semantic inference. The data set is expanded by the back-translation method. The expanded corpus data includes noise data, which can improve the robustness of the trained text classification model. The dirty data containing uncontrollable noise in the data set is cleaned to form a high-quality data set, which helps to improve the intent classification accuracy of the text classification model. At the same time, label semantics are inferred based on the corpus data to obtain label semantics, and external information is provided to the text classification model through the label semantics. Then, the shallow features and deep features of the corpus data and label semantics are extracted respectively, and feature fusion is performed to make the intent classification result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of the steps for implementing the technical solution of the present invention;

[0046] Figure 2 This is a flowchart of inferring tag semantics in the present invention;

[0047] Figure 3 Schematic diagram of the framework of the text classification model in the present invention. DETAILED DESCRIPTION

[0048] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0049] In order to better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size;

[0050] It is understandable to those skilled in the art that some well-known structures and descriptions thereof may be omitted in the drawings.

[0051] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0052] Example 1

[0053] like Figure 1 As shown in FIG, an intent classification method based on label semantic inference includes the following steps:

[0054] S1: Construct a labeled corpus dataset and use the back-translation method to expand the corpus dataset to obtain an expanded corpus dataset;

[0055] S2: Perform classification prediction on the expanded corpus data to obtain labels for the expanded corpus data, perform data cleaning based on the labels, and obtain an enhanced corpus dataset;

[0056] S3: perform label semantic inference based on the similarity between the corpus data in the enhanced corpus dataset to obtain the label semantics corresponding to each label;

[0057] S4: Use the enhanced corpus dataset and the label semantics corresponding to each label to train the pre-built text classification model to obtain a trained text classification model;

[0058] S5: Use the trained text classification model to extract shallow and deep features of the corpus data and label semantics from the text to be classified, and perform feature fusion to obtain the intent classification result.

[0059] In the specific implementation process, the data set is expanded through the back-translation method. The expanded corpus data includes noise data, which can improve the robustness of the trained text classification model. The dirty data containing uncontrollable noise in the data set is cleaned to form a high-quality data set, which helps to improve the intent classification accuracy of the text classification model. At the same time, label semantics are inferred based on the corpus data to obtain label semantics, and external information is provided to the text classification model through label semantics. Then, the shallow features and deep features of the corpus data and label semantics are extracted respectively, and feature fusion is performed to make the intent classification results more accurate.

[0060] Example 2

[0061] This embodiment provides an intent classification method based on tag semantic inference. The difference from Example 1 is that, based on Example 1, this embodiment provides a specific description of steps S1 to S3.

[0062] More specifically, the back-translation method is used to expand the data of the corpus dataset: the back-translation method of Japanese, Korean and English is used to expand the data volume of the corpus dataset.

[0063] In the specific implementation process, since Japanese and Korean contain Chinese characters and these two languages ​​are similar to Chinese, the back-translation method of Japanese and Korean is used. At the same time, there are a small number of English samples in the dataset, and English is also used for the back-translation method.

[0064] More specifically, step S2 is as follows: use the BERT classifier to classify the corpus data X i Expanded corpus data X i ′ Perform classification prediction and get X i ′ Label Y i ′ , change Y i ′ With X i Label Y i Compare and keep X with the same label i ′ , realize data cleaning, and combine the corpus data set and the retained X i ′ The enhanced corpus dataset is obtained by merging.

[0065] In the specific implementation process, the semi-supervised learning method is introduced. Here, the BERT classifier is used to predict X i ′ The probability category of each unlabeled corpus data is selected with the category with the maximum predicted probability. The predicted data set is Da={X i ′ ,Y i ′}, the predicted label Y i ′ and the label Y of the original corpus data i Compare and keep the same labels (Y i ′ =Y i )'s expanded corpus data X i ′ , achieving dirty data cleaning. The retained expanded corpus data and the corpus dataset together constitute the enhanced corpus dataset as a new training set, which can more effectively guide the text classification model to classify the text into the correct category.

[0066] More specifically, in label semantic inference, the problem of solving label semantic inference is transformed into the problem of finding the minimum distance L between the corpus data and the label center, that is:

[0067]

[0068] Among them, n represents the total number of labels in the enhanced corpus dataset, D j Represents the label C in the enhanced corpus dataset i All the corpus data, Z i Indicates C i The label center;

[0069] By optimizing and solving the minimum distance between the corpus data features under the current label category and the total features of the current category, the corpus data with the minimum distance is finally set as the label semantics of the current category.

[0070] In the specific implementation process, short text is a text of relatively short length, generally referring to a text form of no more than 160 characters. Label semantics is generally the category meaning expressed by short text. Taking the customer service corpus dataset in the financial field as an example, each category is not represented by an actual semantic label, and the length of the sample characters in the dataset is distributed between 4-8 words, which belongs to a short text dataset. Therefore, such a dataset is suitable for using text samples as category labels. Label embedding technology can effectively improve text classification performance. This embodiment uses the similarity between texts as a factor to distinguish categories based on the fact that the text similarity in the same category is small and the text similarity in different categories is large. It converts the problem of solving label inference semantics into the problem of finding the minimum distance L between features, which can provide the text classification model with external information from label features, helping to improve classification accuracy.

[0071] Example 3

[0072] This embodiment provides an intent classification method based on tag semantic inference, which differs from Embodiment 2 in that the specific implementation of tag semantic inference is different.

[0073] More specifically, in label semantic inference:

[0074] The distance between each corpus data and each category center is obtained by the following steps:

[0075] S3.1: Segment the corpus data in the enhanced corpus dataset and calculate the word frequency to obtain high-frequency words;

[0076] S3.2: Create a feature matrix for each piece of corpus data based on the high-frequency words, and perform PCA dimensionality reduction processing;

[0077] S3.3: Use the K-Means algorithm to perform clustering and calculate the distance between each corpus data and each category center;

[0078] The piece of corpus data closest to the category center in each category is used as the label semantics of the corresponding category.

[0079] In the specific implementation process, the K-means clustering method is used to set the sample with the minimum distance as the label of the inferred sample, thereby reducing the solution time of the optimization algorithm.

[0080] Example 4

[0081] This embodiment provides an intent classification method based on tag semantic inference. The difference from Example 1 is that, based on Example 1, this embodiment provides a specific description of steps S4 to S5.

[0082] More specifically, the text classification model obtains shallow features through BERT encoding and deep features through downstream tasks.

[0083] Before BERT encoding, the given input length n of text w1w2…w n Processed into the BERT encoder input format W:

[0084] W=[CLS]w1w2…w n [SEP]

[0085] Then map the input format W to get the output v:

[0086] v=InputRepresentation(W)

[0087] Among them, w1w…w n Represents the weight of each word vector of the given text, [CLS] indicates that the current feature is used for the classification model, [SEP] indicates the sentence symbol, and InputRepresentation indicates the input representation format specified by the BERT encoder.

[0088] The output v is encoded by BERT to obtain the contextual semantic feature h:

[0089] h=BERT(v)

[0090] More specifically, obtaining shallow features through BERT encoding includes the following steps:

[0091] The first BERT encoder of the text classification model is used to encode the corpus data in the text. The first BERT encoder includes 12 BertLayer layers connected in sequence. The encoding outputs of the first and sixth BertLayer layers in the first BERT encoder are extracted respectively, corresponding to the first shallow feature a1 and the second shallow feature a2 of the corpus data;

[0092] The label semantics of the text are obtained through label semantic inference, and the obtained label semantics are directly spliced. The second BERT encoder of the text classification model is used to encode the spliced ​​label semantics. The second BERT encoder includes 12 BertLayer layers connected in sequence. The encoding outputs of the first BertLayer layer and the sixth BertLayer layer in the second BERT encoder are extracted respectively, and the first shallow feature b1 and the second shallow feature b2 of the label are obtained accordingly.

[0093] More specifically, Figure 3 As shown in Figure 2, deep features are obtained through downstream tasks including:

[0094] The first convolutional neural network (Conv1), the second convolutional neural network (Conv2), and the third convolutional neural network (Conv3) of the text classification model are used to process the contextual semantic feature h1 output by the first BERT encoder, and three convolution results are obtained respectively. The three convolution results are concatenated with h1 to obtain the deep feature a3 of the corpus data;

[0095] The label semantic feature h2 output by the second BERT encoder is input into the BiLSTM of the text classification model, and the output of BiLSTM is used as the multi-head attention mechanism of the text classification model.

[0096] ) input, and the output of the multi-head attention mechanism is used as the deep feature b3 of the label.

[0097] More specifically, the convolution kernel sizes of the first convolutional neural network, the second convolutional neural network, and the third convolutional neural network are (2,768), (3,768), and (4,768), respectively.

[0098] More specifically, the hidden layer dimension of the BiLSTM output is set to 1024, and the multi-head attention mechanism is set to 4 heads and the hidden layer dimension is 852.

[0099] In the specific implementation process, the multi-head attention mechanism adopts the structure in Transformer, where each head may focus on different parts of the input and can represent more complex functions than simple weighted averages.

[0100] More specifically, features are fused through dot product:

[0101] e1=1*1

[0102] e2=2*2

[0103] e3=3*3

[0104] After fusion, concatenate e1, e2, and e3. Then, input the concatenated results into the linear layer and Softmax layer of the text classification model in sequence to obtain the final output. The maximum value of the final output is taken as the intent classification result.

[0105] Among them, e1 represents the result of the first shallow feature fusion, e2 represents the result of the second shallow feature fusion, and e3 represents the result of the deep feature fusion.

[0106] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. An intent classification method based on label semantic inference, characterized in that: The following steps are involved: S1: Construct a labeled corpus dataset and use the back-translation method to expand the corpus dataset to obtain an expanded corpus dataset; S2: Perform classification prediction on the expanded corpus data to obtain labels for the expanded corpus data, perform data cleaning based on the labels, and obtain an enhanced corpus dataset; S3: perform label semantic inference based on the similarity between the corpus data in the enhanced corpus dataset to obtain the label semantics corresponding to each label; S4: Use the enhanced corpus dataset and the label semantics corresponding to each label to train the pre-built text classification model to obtain a trained text classification model; S5: Use the trained text classification model to extract shallow and deep features of the corpus data and label semantics from the text to be classified, and perform feature fusion to obtain the intent classification result.

2. The method for intent classification based on label semantic inference according to claim 1, characterized in that: The back-translation method is used to expand the data of the corpus dataset. Specifically, the back-translation method of Japanese, Korean and English is used to expand the data volume of the corpus dataset.

3. The method for intent classification based on label semantic inference according to claim 1, characterized in that: Step S2 is as follows: Use the BERT classifier to classify the corpus data X i The expanded corpus data X′ i Perform classification prediction and obtain X′ i The label Y′ i , change Y′ i With X i Label Y i Compare and keep X′ with the same label i , realize data cleaning, and combine the corpus dataset and the retained X′ i The enhanced corpus dataset is obtained by merging.

4. The method for intent classification based on label semantic inference according to claim 1, characterized in that: In label semantic inference, the problem of solving label inference semantics is transformed into the problem of finding the minimum distance L between the corpus data and the label center, that is: Among them, n represents the total number of labels in the enhanced corpus dataset, D j Represents the label C in the enhanced corpus dataset i All the corpus data, Z i Indicates C i The label center; By optimizing and solving the minimum distance between the corpus data features under the current label category and the total features of the current category, the corpus data with the minimum distance is finally set as the label semantics of the current category.

5. The method for intent classification based on label semantic inference according to claim 1, characterized in that: In tag semantic speculation: The distance between each corpus data and each category center is obtained by the following steps: S3.1: Segment the corpus data in the enhanced corpus dataset and calculate the word frequency to obtain high-frequency words; S3.2: Create a feature matrix for each piece of corpus data based on the high-frequency words, and perform PCA dimensionality reduction processing; S3.3: Use the K-Means algorithm to perform clustering and calculate the distance between each corpus data and each category center; The piece of corpus data closest to the category center in each category is used as the label semantics of the corresponding category.

6. The method for intent classification based on label semantic inference according to claim 1, characterized in that: The text classification model obtains shallow features through BERT encoding and deep features through downstream tasks.

7. The method for intent classification based on label semantic inference according to claim 6, characterized in that: Obtaining shallow features through BERT encoding involves the following steps: The first BERT encoder of the text classification model is used to encode the corpus data in the text. The first BERT encoder includes 12 BertLayer layers connected in sequence. The encoding outputs of the first and sixth BertLayer layers in the first BERT encoder are extracted respectively, corresponding to the first shallow feature a1 and the second shallow feature a2 of the corpus data; The label semantics of the text are obtained through label semantic inference, and the obtained label semantics are directly spliced. The second BERT encoder of the text classification model is used to encode the spliced ​​label semantics. The second BERT encoder includes 12 BertLayer layers connected in sequence. The encoding outputs of the first BertLayer layer and the sixth BertLayer layer in the second BERT encoder are extracted respectively, and the first shallow feature b1 and the second shallow feature b2 of the label are obtained accordingly.

8. The method for intent classification based on label semantic inference according to claim 7, characterized in that: Acquiring deep features through downstream tasks includes: The first, second, and third convolutional neural networks of the text classification model are used to process the contextual semantic feature h1 output by the first BERT encoder, obtaining three corresponding convolution results. The three convolution results are concatenated with h1 to obtain the deep feature a3 of the corpus data. The label semantic feature h2 output by the second BERT encoder is input into the BiLSTM of the text classification model. The output of BiLSTM is used as the input of the multi-head attention mechanism of the text classification model, and the output of the multi-head attention mechanism is used as the deep feature b3 of the label.

9. The method for intent classification based on label semantic inference according to claim 8, characterized in that: The convolution kernel sizes of the first convolutional neural network, the second convolutional neural network, and the third convolutional neural network are (2,768), (3,768), and (4,768), respectively; The hidden layer dimension of BiLSTM output is set to 1024, and the multi-head attention mechanism is set to 4 heads and the hidden layer dimension is 852.

10. The method for intent classification based on label semantic inference according to claim 8, characterized in that: Fusion of features via dot product: e1＝1*1 e2＝2*2 e3＝3*3 After fusion, e1, e2, and e3 are concatenated. The concatenated results are then sequentially input into the linear layer and Softmax layer of the text classification model to obtain the final output. The maximum value of the final output is taken as the intent classification result. Among them, e1 represents the result of the first shallow feature fusion, e2 represents the result of the second shallow feature fusion, and e3 represents the result of the deep feature fusion.

Images