A text intent classification method, system and computer readable storage medium

Abstract

The present invention discloses a text intent classification method, system and computer-readable storage medium, comprising the following steps: converting the vocabulary in the input corpus information into word vectors to obtain a word vector matrix; encoding the position of the word through relative position encoding based on the word vector matrix to obtain a position vector matrix; calculating the relative position attention weight of the input corpus information according to the relative position relationship of the position vector matrix; and calculating the correlation of the position vector matrix; splicing the relative position attention weight with the self-attention weight, and constructing a linear layer to obtain a linear mapping vector matrix; summing the linear vector matrix and the position vector matrix and taking the average value to obtain a mean vector matrix; classifying the mean vector matrix to obtain a text intent classification result. The present invention solves the problem that the prior art ignores the addition of the structured information of the text into the attention mechanism, and the accuracy of text intent recognition is low.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a text intent classification method, system and computer-readable storage medium. Background Art

[0002] Since the most important application of artificial intelligence in the field of natural language processing is the dialogue system, the most important of which is the intent recognition task. Therefore, many researchers have devoted themselves to the study of text classification and proposed many effective methods. The text intent classification in the existing technology can be mainly divided into the following two types:

[0003] 1. Traditional method: First preprocess the text, then perform feature extraction and feature engineering, and finally use the classifier model to classify intent.

[0004] 2. Deep learning method: first use word vectors as text representation, and then use deep neural networks to perform intent classification. In the existing technology, there are two mainstream methods for using deep learning methods to classify text intent. One is to build a convolutional network based on CNN, and the other is based on RNN.

[0005] The existing algorithms all ignore the addition of text structured information to the attention mechanism. At most, the existing technologies combine word-level attention and sentence-level attention to understand text. However, due to the inconsistency of sentence lengths, the difference in the relative positions of words has a great impact on the intent categories of two similar sentences. In this case, the text intent classification methods in the existing technologies cannot achieve accurate intent recognition. Summary of the invention

[0006] The main purpose of the present invention is to provide a text intent classification method and a computer-readable storage medium, aiming to solve the technical problem in the prior art that the accuracy of text intent recognition is low when the difference in relative positions of words has a great impact on the intent categories of two similar sentences due to inconsistent sentence lengths.

[0007] To achieve the above-mentioned purpose, the present invention provides a text intent classification method, which comprises the following steps: step a. converting the vocabulary in the input corpus information into word vectors to obtain a word vector matrix; step b. encoding the position of the word through relative position encoding based on the word vector matrix to obtain a position vector matrix; step c. calculating the relative position attention weight of the input corpus information according to the relative position relationship of the position vector matrix; and calculating the correlation of the position vector matrix to represent the self-attention weight; step d. concatenating the relative position attention weight with the self-attention weight, and constructing a linear layer, the linear layer is used to perform a linear transformation on the concatenated result to obtain a linear mapping vector matrix; step e. summing the linear vector matrix and the position vector matrix, and taking the average value to obtain a mean vector matrix; step f. classifying the mean vector matrix to obtain a text intent classification result.

[0008] Optionally, the input corpus information is obtained by collecting conversation data as original corpus information and performing data processing on the original corpus information.

[0009] Optionally, the data processing includes the following steps: Step 1: preprocess the original corpus information and construct a character table, which is a list of all Chinese characters contained in the original corpus information; digitize each character in the original corpus information based on the character table to obtain digitized original corpus information; Step 2: classify the sentence texts of the digitized original corpus information to obtain the intent category of each sentence text, read the sentence texts and their corresponding categories in batches; and segment the sentence texts to obtain corresponding annotation labels; Step 3: concatenate the annotation labels with the sentence text length and text mask to form the input corpus information.

[0010] Optionally, the preprocessing in step 1 further includes removing special characters, stop words and emoticons.

[0011] Optionally, step a is implemented by one of a BERT pre-trained model, a skip-gram model or a cbow model.

[0012] Optionally, in step a, the maximum length of the sentences in the input corpus information is set to n, the word vector dimension is set to k, and the word vector matrix is ​​an n*k matrix; the maximum length of the sentences n is set to 32 or 64, and the word vector dimension k is set to 128 or 256.

[0013] Optionally, the step c comprises the following steps:

[0014] c1. Calculate the relative position relationship between Q and K according to the following formula:

[0015] pos(Q),pos(K)=cosine(Q·Q),cosine(K),

[0016] Among them, Q represents the word vector matrix, K represents the first weight sequence vector;

[0017] c2. Calculate the relative position attention weight according to the relative position relationship. The calculation formula of the relative position attention weight is as follows:

[0018]

[0019] Among them, Attention(Q,K,V) represents the attention weight of the relative position, d represents the dimension of the vector, V is obtained by taking the cosine value based on K, representing the second weight sequence vector, and k represents the dimension of the square root;

[0020] c3. Perform a dot product operation on Q and K to obtain the correlation of the position vector matrix.

[0021] Optionally, the step f is to perform feature integration on the mean vector matrix through a fully connected layer, then perform classification processing through a Softmax layer, and finally output the text intent classification result corresponding to the input corpus information.

[0022] In addition to the above-mentioned text intent classification method, the present invention also provides a text intent classification system, including: a text representation module: inputting the input corpus information into the text representation module to obtain a word vector matrix; a position attention module: based on the word vector matrix, encoding the position of the word through relative position encoding to obtain a position vector matrix; according to the relative position relationship of the position vector matrix, calculating the relative position attention weight of the corpus information; a self-attention module: based on the position vector matrix, calculating the correlation of the position vector matrix to represent the self-attention weight; a linear transformation module: concatenating the relative position attention weight with the attention weight, and constructing a linear layer, the linear layer is used to perform a linear transformation on the concatenated result to obtain a linear mapping vector matrix; a text intent classification module: classifying the mean vector matrix to obtain a text intent classification result.

[0023] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a text intent classification program is stored, and when the text intent classification program is executed by a processor, the steps of text intent classification are implemented.

[0024] The beneficial effects of the present invention are:

[0025] (1) The present invention can better learn and integrate the relative position relationship between text characters without constraints on the positions of text characters through relative position attention weights. No matter where the characters are in the text, the present invention can learn the connection between the context and the more critical characters in the current attention mechanism, and perform more accurate text intent classification;

[0026] (2) By superimposing the position encoding and the word vector matrix, the position features are further extracted, which effectively improves the accuracy of text intent classification. In addition, the relative position attention weight is combined with the attention weight to construct a linear layer. On the one hand, it learns the supervision information, and on the other hand, it is actually a vote or weighted sum of the two attentions, which better increases the role of attention and has a significant optimization effect on semantic learning.

[0027] (3) Data processing of the original corpus information to obtain input corpus information can effectively improve the efficiency of subsequent text intent classification;

[0028] (4) By setting the maximum sentence length n and the word vector dimension k, the problem of inefficient batch input caused by inconsistent sentence lengths can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0030] Figure 1 The figure is a simplified flowchart of the text intent classification method of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0032] like Figure 1 As shown, a text classification method of the present invention comprises the following steps:

[0033] A text intent classification method comprises the following steps: step a. converting words in input corpus information into word vectors to obtain a word vector matrix; step b. encoding the word positions through relative position encoding based on the word vector matrix to obtain a position vector matrix; step c. calculating the relative position attention weight of the input corpus information according to the relative position relationship of the position vector matrix; and calculating the correlation of the position vector matrix to represent the self-attention weight; step d. concatenating the relative position attention weight with the self-attention weight, and constructing a linear layer, the linear layer is used to perform a linear transformation on the concatenated result to obtain a linear mapping vector matrix; step e. summing the linear vector matrix and the position vector matrix, and taking the average value to obtain a mean vector matrix; step f. classifying the mean vector matrix to obtain a text intent classification result.

[0034] The present invention can better learn and integrate the relative position relationship between text characters without constraints through relative position attention weights. No matter where the characters are located in the text, the present invention can learn the connection between the context and the more critical characters in the current attention mechanism, and perform more accurate text intent classification.

[0035] The present invention can achieve an average improvement of more than 3% in labeled data samples in the medical vertical field, with significant effects.

[0036] In this embodiment, step a is implemented by one of the BERT pre-training model, the skip-gram model or the cbow model. The specific method to be used can be selected according to actual needs.

[0037] In this embodiment, the maximum length of sentences in the input corpus information is set to n, the word vector dimension is set to k, and the word vector matrix is ​​an n*k matrix; the maximum length of sentences n is set to 32 or 64, and the word vector dimension k is set to 128 or 256.

[0038] By setting the maximum sentence length n and the word vector dimension k, the problem of inefficient batch input caused by inconsistent sentence lengths can be avoided.

[0039] In this embodiment, the input corpus information is obtained by collecting conversation data as original corpus information and performing data processing on the original corpus information; performing data processing on the original corpus information to obtain the input corpus information can effectively improve the efficiency of subsequent text intent classification.

[0040] Specifically, data processing includes the following steps: Step 1: preprocess the original corpus information and construct a character table, which is a list of all Chinese characters contained in the original corpus information; digitize each character in the original corpus information based on the character table to obtain digitized original corpus information; Step 2: classify the sentence text of the digitized original corpus information for text intent, obtain the intent category of each sentence text, read the sentence text and its corresponding category in batches; and segment the sentence text to obtain the corresponding annotation label; Step 3: concatenate the annotation label with the sentence text length and text mask to form the input corpus information.

[0041] In this embodiment, the preprocessing in step 1 also includes removing special characters, stop words and emoticons.

[0042] In this embodiment, the corpus of the infertility department of a hospital is taken as an example to illustrate the digitization and vectorization of the input corpus information. Please refer to the digitization and vectorization processing results of the input corpus information in Table 1.

[0043]

[0044] Table 1

[0045] After completing the digitization and vectorization of the input corpus information, proceed to step c. Calculate the relative position attention weight of the input corpus information according to the relative position relationship of the position vector matrix; and calculate the correlation of the position vector matrix to represent the self-attention weight;

[0046] In this embodiment, step c comprises the following steps:

[0047] c1. Calculate the relative position relationship between Q and K according to the following formula:

[0048] pos(Q),pos(K)=cosine(Q·Q),cosine(K),

[0049] Among them, Q represents the word vector matrix, K represents the first weight sequence vector;

[0050] c2. Calculate the relative position attention weight according to the relative position relationship. The calculation formula of the relative position attention weight is as follows:

[0051]

[0052] Among them, Attention(Q,K,V) represents the attention weight of the relative position, d represents the dimension of the vector, V is obtained by taking the cosine value based on K, representing the second weight sequence vector, and k represents the dimension of the square root;

[0053] c3. Perform a dot product operation on Q and K to obtain the correlation of the position vector matrix.

[0054] The present invention further extracts position features by superimposing position encoding and word vector matrix, effectively improving the accuracy of text intent classification; and, after merging the relative position attention weight and the attention weight, constructs a linear layer, which, on the one hand, learns supervision information, and on the other hand, is actually a vote or weighted sum of the two attentions, which better increases the role of attention and produces a significant optimization effect on semantic learning.

[0055] In this embodiment, the splicing step in step d is actually to perform a dot product operation on the relative position attention weight and the self-attention weight.

[0056] In this embodiment, step f is to perform feature integration on the mean vector matrix through a fully connected layer, then perform classification processing through a Softmax layer, and finally output the text intent classification result corresponding to the input corpus information.

[0057] The present invention proposes a text classification method that can better learn and integrate the relative position relationship between text characters without constraints between the positions of text characters. Therefore, the text intent classification system can solve the problem that no matter where the characters are in the entire text, the connection between the context and the more critical characters in the current attention mechanism can be learned, and more accurate intent recognition can be performed.

[0058] In this embodiment, a text intent classification system is also provided, including: a text representation module: inputting input corpus information into the text representation module to obtain a word vector matrix;

[0059] Position attention module: Based on the word vector matrix, the word position is encoded through relative position encoding to obtain the position vector matrix; according to the relative position relationship of the position vector matrix, the relative position attention weight of the corpus information is calculated;

[0060] Self-attention module: Based on the position vector matrix, the correlation of the position vector matrix is ​​calculated to represent the self-attention weight;

[0061] Linear transformation module: concatenates the relative position attention weight with the attention weight and constructs a linear layer. The linear layer is used to perform a linear transformation on the concatenated result to obtain a linear mapping vector matrix.

[0062] Text intent classification module: classifies the mean vector matrix to obtain the text intent classification result.

[0063] In the present invention, the relative position of the key semantic information of the text can be learned and aligned through the projection attention module. In this embodiment, when the input corpus information is input into the text representation module in step a, the input corpus information and its corresponding intent category are input one by one, which improves the accuracy of intent recognition.

[0064] And, a computer-readable storage medium stores a text intent classification program, which implements the steps of text intent classification when executed by a processor.

[0065] In this article, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0066] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention, through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A text intent classification method, characterized in that: The steps include: Step a. Convert the words in the input corpus information into word vectors to obtain a word vector matrix; Step b. Based on the word vector matrix, encode the position of the word through relative position encoding to obtain a position vector matrix; Step c. Calculating the relative position attention weight of the input corpus information according to the relative position relationship of the position vector matrix; And calculate the correlation of the position vector matrix to represent the self-attention weight; Step d. concatenate the relative position attention weight and the self-attention weight, and construct a linear layer, wherein the linear layer is used to perform a linear transformation on the concatenated result to obtain a linear mapping vector matrix; Step e. performing a sum operation on the linear mapping vector matrix and the position vector matrix, and taking an average value to obtain a mean vector matrix; Step f. classifying the mean vector matrix to obtain a text intent classification result; The step c comprises the following steps: c1. Calculate the relative position relationship between Q and K according to the following formula: ， Among them, Q represents the word vector matrix, K represents the first weight sequence vector; c2. Calculate the relative position attention weight according to the relative position relationship. The calculation formula of the relative position attention weight is as follows: ； Among them, Attention(Q,K,V) represents the attention weight of the relative position, d represents the dimension of the vector, V is obtained by taking the cosine value based on K, representing the second weight sequence vector, and k represents the dimension of the square root; c3. Perform a dot product operation on Q and K to obtain the correlation of the position vector matrix.

2. A text intent classification method according to claim 1, characterized in that: The input corpus information is obtained by collecting conversation data as original corpus information and performing data processing on the original corpus information.

3. A text intent classification method according to claim 2, characterized in that: The data processing comprises the following steps: Step 1: preprocessing the original corpus information and constructing a character table, wherein the character table is a list consisting of all Chinese characters contained in the original corpus information; digitizing each word in the original corpus information based on the word table to obtain digitized original corpus information; Step 2: Classify the sentence texts of the digitized original corpus information to obtain the intent category of each sentence text, read the sentence texts and their corresponding categories in batches; and segment the sentence texts to obtain corresponding annotation labels; Step 3: Concatenate the annotation label with the sentence text length and text mask to form the input corpus information.

4. A text intent classification method according to claim 3, characterized in that: The preprocessing in step 1 also includes removing special characters, stop words and emoticons.

5. A text intent classification method according to claim 3, characterized in that: The step a is implemented by using one of the BERT pre-training model, the skip-gram model or the cbow model.

6. A text intent classification method according to claim 1, characterized in that: In the step a, the maximum length of the sentences in the input corpus information is set to n, the word vector dimension is set to k, and the word vector matrix is ​​an n*k matrix; The maximum sentence length n is set to 32 or 64, and the word vector dimension k is set to 128 or 256.

7. A text intent classification method according to claim 1, characterized in that: The step f is to perform feature integration on the mean vector matrix through a fully connected layer, and then perform classification processing through a Softmax layer, and finally output the text intent classification result corresponding to the input corpus information.

8. A text intent classification system, characterized in that: include: Text representation module: Input the input corpus information into the text representation module for word vector conversion to obtain a word vector matrix; Position attention module: Based on the word vector matrix, the word position is encoded by relative position encoding to obtain a position vector matrix; according to the relative position relationship of the position vector matrix, the relative position attention weight of the corpus information is calculated, which specifically includes the following steps: According to the following formula, the relative position relationship between Q and K is calculated: ， Among them, Q represents the word vector matrix, K represents the first weight sequence vector; The relative position attention weight is calculated according to the relative position relationship, and the calculation formula of the relative position attention weight is as follows: ； Among them, Attention(Q,K,V) represents the attention weight of the relative position, d represents the dimension of the vector, V is obtained by taking the cosine value based on K, representing the second weight sequence vector, and k represents the dimension of the square root; Self-attention module: based on the position vector matrix, calculate the correlation of the position vector matrix to represent the self-attention weight, specifically: perform a dot product operation on Q and K to obtain the correlation of the position vector matrix; Linear transformation module: concatenates the relative position attention weight with the attention weight and constructs a linear layer, which is used to perform a linear transformation on the concatenated result to obtain a linear mapping vector matrix; Text intent classification module: summing the linear mapping vector matrix and the position vector matrix, taking the average value to obtain a mean vector matrix; classifying the mean vector matrix to obtain a text intent classification result.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a text intent classification program, and when the text intent classification program is executed by a processor, the text intent classification steps as described in any one of claims 1 to 7 are implemented.

Images