Traffic text intention recognition method based on knowledge prompt

Abstract

The invention relates to a telephone traffic text intention recognition method based on knowledge prompt, and relates to the technical field of computer science and natural language processing. The method comprises the steps that a prompt template is constructed, relational words are predicted through the template and text context, the dependency relationship between vocabularies is fully learned, and a model decision boundary is optimized; performing text semantic feature extraction, performing representation learning on the traffic text, performing feature extraction on the preprocessed text data, converting the text into a numerical form by utilizing a word embedding technology, constructing coding vectors, and splicing two coding vectors to output traffic text representation; and target function optimization: performing relation prediction by using the semantic features of the text and the semantic features of the prediction words, measuring the similarity of probability distribution by calculating the cross entropy between the prediction relation and the real relation, and training a text classification model by taking the similarity as the target function. The method has the effects of extracting the key information from the incoming call dialogue record actively initiated by the client and automatically identifying the specific service intention of the client.

Description

Technical Field

[0001] The present application relates to the technical fields of computer science and natural language processing, and in particular to a method for identifying the intent of a conversation text based on knowledge prompts. Background Art

[0002] With the booming development of artificial intelligence technology and 5G communications, the communication between enterprises and customers is increasingly dependent on electronic channels, such as email, online chat, social media, etc. The large amount of text data generated by these channels contains rich customer intent information. How to quickly and accurately identify the real needs of customers has become an important research topic. Traditional text intent recognition methods often rely on keyword matching or simple statistical models. These methods are difficult to achieve high accuracy when facing complex and changeable actual application scenarios. More and more companies are beginning to use automated tools to assist customer service, such as machine learning and deep learning methods. Due to their powerful data processing and pattern recognition capabilities, they have high application value in the field of customer service.

[0003] Early text intent recognition mainly used machine learning methods, which were suitable for simple call text classification scenarios. Machine learning methods used statistical language features, rule design and other methods to extract text semantic features, and then used supervised algorithms such as support vector machines, decision trees, and Bayesian algorithms to construct classifiers for classification. These methods can be further divided into three types: those based on word frequency statistics, alignment, and logical rules. However, based on the research results of domain knowledge and general algorithms, the shortcomings of existing call text intent recognition methods mainly include the following points: 1) Simple rule matching has limited coverage: rule matching methods can only process text that has been explicitly encoded, and for unforeseen expressions or subtle differences in expressions, rule matching may fail, resulting in some intent being missed; 2) Insufficient extraction of contextual semantic features: existing methods do not adequately consider the importance of information at different positions and lengths in the text, resulting in the extraction of features. The features contain noise information that is irrelevant to intent recognition, which makes the model susceptible to interference and difficult to accurately distinguish different types of customer intent, especially when dealing with complex or multi-intent texts, which is prone to misclassification; 3) Ignoring the semantics of category text: Existing methods regard intent recognition as a simple binary classification problem, that is, simplifying customer intent into a binary classification of "yes" or "no", ignoring the rich semantic information in the actual intent category. Since real customer intent often has specific semantic descriptions (such as "technical support", "product consultation", etc.), ignoring these relational semantics will cause the model to be unable to fully explore the intrinsic connection between text features and intent categories, thereby affecting the overall training effect of the model and its accuracy in practical applications. Therefore, how to accurately extract text semantic features and introduce relational semantics to train the intent classification model has become a key issue that needs to be urgently solved in the call text intent recognition method. Summary of the invention

[0004] In order to extract key information from incoming call conversation records initiated by customers and automatically identify customers' specific service intentions, the present application provides a method for identifying call text intentions based on knowledge prompts.

[0005] The method for identifying the intent of a conversation text based on knowledge prompts provided in this application adopts the following technical solutions:

[0006] The method of identifying the intent of traffic text based on knowledge prompts, prompting template construction, transforming the relationship classification problem into a relationship word prediction problem, using templates and text context to predict relationship words, fully learning the dependency relationship between words, and optimizing the model decision boundary;

[0007] Extract text semantic features, learn the representation of the conversation text, extract features from the preprocessed text data, convert the text into numerical form using word embedding technology, build a coding vector, build a prompt encoder using BiLSTM and a feedforward neural network, and then use BERT encoding to concatenate two coding vectors to output the conversation text representation; extract key semantic features from the text by building a multi-scale convolutional attention network;

[0008] Objective function optimization uses text semantic features and predicted word semantic features to predict relationships, measures the similarity of probability distribution by calculating the cross entropy between predicted relationships and true relationships, and uses this as the objective function to train the text classification model.

[0009] Preferably, the input of the prompt template is a historical traffic training text set X = {X1, ..., X N} and BERT vocabulary S, the output is a text sequence DT, where N is the number of samples in the training sample set, and the prompt module construction includes the following sub-steps:

[0010] Extract m pseudo-cue words {u1, ..., u m} as a soft prompt template ST, extract x i The co-occurrence word set C = {g1, ..., g C}, calculate the TF-IDF value of each co-occurring word in the sample. The calculation methods of TF, IDF, and TF-IDF are shown in formulas (1), (2), and (3), respectively:

[0011]

[0012] Among them, g i is the i-th co-occurring word, n i It is the word g i In the sample x i The number of occurrences in n k is the number of words in the sample, TF(gi , x i ) is the co-occurring word g i The frequency of words;

[0013]

[0014] Where N is the number of words, g n Represents the word g i The +1 operation is to avoid the denominator being 0. IDF(g i , X) is the co-occurring word g i The inverse document frequency of

[0015] TF-IDF(g i , x i , X) = TF(g i , x i )*IDF(g i ,X)#(3)

[0016] TF-IDF(g i , x i , X) is used to measure the co-occurrence of words in sample x i The importance of

[0017] The co-occurring words are sorted according to the TF-IDF value, and the sorted co-occurring word set C is used to concatenate with placeholders such as [CLS], [SEP], and [MASK] to construct hard prompts. Then, the hard prompt structured text is used to obtain the text sequence HT to be processed, and ST and HT are concatenated into a text sequence DT to be input into the next module. The construction method of the module is shown in formula (4):

[0018] DT={[CLS],g1,…,g c ,ST,[MASK],x i ,[SEP]}#(4)

[0019] Among them, [CLS], [SEP], and [MASK] are placeholder characters, representing classification tokens, sentence separation tokens, and mask tokens, respectively. They are used to capture the information of the entire input sequence, identify the boundaries of sentences, and predict missing words based on the context, thereby helping the model learn richer semantic information; g i is the i-th co-occurring word in the sample, c is the number of co-occurring words, and ST is the soft prompt template.

[0020] Preferably, the input of the text semantic feature extraction is a text sequence DT, and the output is a text representation E t , the text semantic feature extraction includes the following steps:

[0021] A prompt encoder is constructed using BiLSTM and a feedforward neural network, and the prompt encoder is used to encode the soft prompt ST to obtain a soft prompt identifier E S , the input of each word in the soft prompt is taken as a time step, and for each time step t (t = 1, 2, 3...), the forward long short-term memory neural network outputs the hidden state The backward long short-term memory network outputs the hidden state Concatenate the two hidden state vectors to get the feature vector of each character The soft prompt word vector encoding method is shown in formula (5):

[0022]

[0023] The soft prompt feature vector is obtained by merging the forward and reverse end output vectors of the BiLSTM layer. The soft prompt feature vector is shown in formula (6), where l is the length of the soft prompt template:

[0024]

[0025] H s The soft prompt representation E is obtained through a feedforward neural network S , the soft prompt indicates that the calculation is as shown in formula (7):

[0026] E S =tanh(W T H S +b)#(7)

[0027] Among them, W T is the weight, b is the bias term, W T and b are trainable parameter weights of the feedforward neural network, and the activation function is the tanh function;

[0028] Use BERT to encode the HT text sequence to get the text sequence representation E ht , input {ht1, ht2, …, ht n}, output The BERT encoding method is shown in formula (8):

[0029]

[0030] in, is the sequence hidden state vector of the BERT embedding layer, and n is the length of the HT text sequence;

[0031] After encoding the characters in HT one by one, concatenate all the character vectors to get the text sequence representation E ht , the splicing algorithm is shown in formula (9):

[0032]

[0033] The soft prompt indicates E S and text sequence representation E ht The spliced ​​sequence is represented by E O , the vector concatenation algorithm is shown in formula (10):

[0034] E O =E S ⊕E ht #(10)

[0035] Among them, ⊕ represents the spelling operation, E S is the representation vector of the soft prompt ST, E ht is the representation vector of the text sequence HT;

[0036] Using the Transformer-based BERT model to represent the sequence E O Perform feature extraction again to obtain the text representation E t .

[0037] Preferably, the text semantic feature extraction also includes constructing a multi-scale convolutional attention network training.

[0038] Preferably, the step of training the multi-scale convolutional attention network comprises the following steps:

[0039] Convolution kernels of different scales are used to extract the feature map of text semantics, and K filters are used to filter E t ∈R L×D Convolution operation is performed to obtain K feature maps, and zero padding is used to make the feature maps consistent in scale. The convolution calculation method is shown in formula (11):

[0040] Ui=f(w i *E t +b i ), i∈[1,K]#(11)

[0041] Among them, Ui is the feature map obtained by convolution operation, f is the activation function, and w i is the weight value, b i ∈R is the bias term, K is the number of convolution kernels, Ui∈R L×D , L is the length of the text sequence, and D is the word embedding dimension value;

[0042] Fusion of multiple feature maps: K feature maps are fused by element-wise addition to obtain a new feature map U∈R L×D , the feature map fusion method is shown in formula (12):

[0043]

[0044] Perform global average pooling on the feature map U to obtain the vector s, which reduces the feature dimension, reduces the noise and irregularity in the feature map, and increases the robustness of the model. The global average pooling calculation method is shown in formula (13):

[0045]

[0046] Among them, s c is the cth element of s, s∈R D×1 , s is a column vector with D elements;

[0047] Input s into a fully connected layer for dimensionality reduction to obtain the compressed representation E v , that is, the text semantic feature vector, E v ∈R d×1 , d<D, the dimension reduction calculation method is shown in formula (14):

[0048] E v =δ(B(w z *s+b z ))#(14)

[0049] Among them, δ is a Relu function, B is a batch normalization function, and w z is the weight value, b z ∈R is the bias term, and the calculation method of d value is shown in formula (15):

[0050]

[0051] Where r is a hyperparameter used to control the scaling ratio, L d Indicates the minimum value of d, which is usually set to 32.

[0052] Preferably, the objective function optimization uses text semantic features and predicted word semantic features to map relationship categories, uses the cross entropy loss function to calculate the difference between the predicted relationship distribution and the true relationship distribution, and trains the text intent recognition model. The objective function optimization input is the text semantic feature E v , the output is the text intent recognition model The objective function optimization comprises the following steps:

[0053] From the text semantic features E v Extract the semantic features of the predicted word E mask , using E mask and E v Predict the category probability. The probability prediction calculation method is shown in formulas (16) and (17):

[0054] p a =σ(W1E mask +b1)#(16)

[0055] p b =σ(W2E v +b2)#(17)

[0056] Among them, σ is the softmax activation function, W1, W2, b1, b2 are the weight matrices of the fully connected layer, and p a is the probability of the predicted value of [MASK] in the prompt template, p b is the probability of the predicted value of the traffic text intent category;

[0057] The text intent recognition model is trained using the joint loss. The joint loss consists of the prediction word loss and the classification loss. The joint loss function is shown in formula (18), the prediction word loss function is calculated as shown in formula (19), and the classification loss function is shown in formula (20):

[0058]

[0059] in, is the joint loss value, ρ s is the weight coefficient of the predicted word loss, is the predicted word loss value, ρ c is the classification loss weight coefficient, is the classification loss value;

[0060]

[0061] Among them, y ia is a symbolic function. If the predicted word [MASK] of the i-th sample is the word represented by a, then y ia =1, otherwise y ia =0; p ia Represents the probability that the predicted word [MASK] of the i-th sample is predicted as word a, SM is the length of the BERT vocabulary, and N is the number of samples;

[0062]

[0063] Among them, y ib is a symbolic function. If the true relationship category of sample i is b, then y ib =1, otherwise y ib =0, p ib represents the probability that sample i belongs to relation category b, and M is the number of relation types;

[0064] In the training phase, the loss is continuously calculated and back-propagated through the above process to update the parameters. The network parameters of the three modules of traffic text representation learning, multi-scale text semantic feature extraction and objective function optimization will be updated, and finally a trained text intent recognition model will be obtained.

[0065] In the testing phase, input the test sample tx i , use prompt templates to structure the call text, and then input it into the model Calculate the probability value of the text representation belonging to each category, and output the intent recognition y corresponding to the maximum probability value.

[0066] In summary, the present application includes at least one of the following beneficial technical effects:

[0067] 1. This application proposes an objective function that integrates prediction word loss and classification loss to fully extract the correlation between intent category and text, so that the model can better understand contextual semantic knowledge. This method uses text semantic features and prediction word features to predict relationship types respectively, and uses cross entropy to calculate the similarity between the probability distribution of prediction results and the probability distribution of true relationships to train the text intent classification model and improve the text classification recall rate. This method can better capture the dependency between words and optimize the model decision boundary, thereby improving the accuracy of relationship classification for new samples. When the weight ratio of prediction word loss and classification loss is 1:1, the model can learn the discriminative features of different intent types;

[0068] 2. This application uses soft and hard mixed prompt knowledge to guide the pre-trained language model to strengthen text representation, and uses a multi-scale convolutional attention network to extract text semantic features and predicted word semantic features from text representation to enhance the model's intent classification performance. The multi-scale text semantic impact analysis experiment shows that convolutional attention networks with different structures can capture text semantic features at different levels. The optimal values ​​of the number of convolutional layers and convolutional kernel size are related to the characteristics of the data set. Shallower convolutional networks are more suitable for capturing shallow semantic information, such as digital information, location information, word order or sentence structure, etc. Therefore, a model with fewer convolutional layers is better for identifying text intent with relatively simple sentences and words.

[0069] 3. This application constructs a hybrid prompt template of soft prompts and hard prompts, and combines it with a pre-trained language model to enhance the representation ability of text. This method retains the clear guidance of hard prompts while using the flexibility of soft prompts to dynamically focus on target words; it can learn highly relevant target intents among multiple intents, especially in identifying sensitive words, customer expectations, sentiment classification and other scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a flowchart of a method for identifying the intent of a conversation text based on knowledge prompts in an embodiment of the present application.

[0071] Figure 2 It is a principle framework diagram of the method for identifying the intent of a conversation text based on knowledge prompts in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The following is combined with Figure 1-2 This application is described in further detail.

[0073] The present application embodiment discloses a method for identifying the intent of a conversation text based on knowledge prompts. Figure 1 , including the following steps:

[0074] S1: Prompt template construction, transforming the relationship classification problem into a relationship word prediction problem, using templates and text context to predict relationship words, fully learning the dependency relationship between words, and optimizing the model decision boundary.

[0075] Specifically, the input of the prompt template is the historical traffic training text set X = {X1, ..., X N} and BERT vocabulary S, the output is a text sequence DT, where N is the number of samples in the training sample set, and the prompt module construction includes the following sub-steps:

[0076] S11: Extract m pseudo-prompt words {u1, ..., u1, ..., u2, ..., u3, ..., u4, ..., u5, ..., u6, ..., u7, ..., u8, ..., u9, ..., u10, ..., u11, ..., u12, ..., u13, ..., u14, ..., u15, ..., u16, ..., u17, ..., u18, ..., u19, ..., u21, m} as a soft prompt template ST, extract x i The co-occurrence word set C = {g1, ..., g C}, calculate the TF-IDF value of each co-occurring word in the sample. The calculation methods of TF, IDF, and TF-IDF are shown in formulas (1), (2), and (3), respectively:

[0077]

[0078] Among them, g i is the i-th co-occurring word, n i It is the word g i In the sample x i The number of occurrences in n k is the number of words in the sample, TF(g i , x i ) is the co-occurring word g i The frequency of words;

[0079]

[0080] Where N is the number of words, g n Represents the word g i The +1 operation is to avoid the denominator being 0. IDF(g i , X) is the co-occurring word g i The inverse document frequency of

[0081] TF-IDF(g i , x i , X) = TF(g i , xi )*IDF(g i ,X)#(3)

[0082] TF-IDF(g i , x i , X) is used to measure the co-occurrence of words in sample x i The importance of .

[0083] S12: Sort the co-occurring words according to the TF-IDF value, and use the sorted co-occurring word set C and placeholders such as [CLS], [SEP], and [MASK] to construct hard prompts. Then, use the hard prompt structured text to obtain the text sequence HT to be processed, and concatenate ST and HT into a text sequence DT to input into the next module. The construction method of the module is shown in formula (4):

[0084] DT={[CLS],g1,…,g c ,ST,[MASK],x i ,[SEP]}#(4)

[0085] Among them, [CLS], [SEP], and [MASK] are placeholder characters, representing classification tokens, sentence separation tokens, and mask tokens, respectively. They are used to capture the information of the entire input sequence, identify the boundaries of sentences, and predict missing words based on the context, thereby helping the model learn richer semantic information; g i is the i-th co-occurring word in the sample, c is the number of co-occurring words, and ST is the soft prompt template.

[0086] S2: Extract text semantic features, perform representation learning on the conversation text, extract features from the preprocessed text data, convert the text into numerical form using word embedding technology, build an encoding vector, use BiLSTM and feedforward neural network to build a prompt encoder, then use BERT encoding, concatenate the two encoding vectors to output the conversation text representation; extract key semantic features in the text by building a multi-scale convolutional attention network.

[0087] Specifically, the input of text semantic feature extraction is the text sequence DT, and the output is the text representation E t , the text semantic feature extraction includes the following steps:

[0088] S21: Use BiLSTM and feedforward neural network to construct a prompt encoder, and use the prompt encoder to encode the soft prompt ST to obtain the soft prompt identifier E S , the input of each word in the soft prompt is taken as a time step, and for each time step t (t = 1, 2, 3...), the forward long short-term memory neural network outputs the hidden state The backward long short-term memory network outputs the hidden state Concatenate the two hidden state vectors to get the feature vector of each character The soft prompt word vector encoding method is shown in formula (5):

[0089]

[0090] The soft prompt feature vector is obtained by merging the forward and reverse end output vectors of the BiLSTM layer. The soft prompt feature vector is shown in formula (6), where l is the length of the soft prompt template:

[0091]

[0092] H s The soft prompt representation E is obtained through a feedforward neural network s , the soft prompt indicates that the calculation is as shown in formula (7):

[0093] E s =tanh(W T H S +b)#(7)

[0094] Among them, W T is the weight, b is the bias term, W T and b are trainable parameter weights of the feedforward neural network, and the activation function is the tanh function.

[0095] S22: Use BERT to encode the HT text sequence to obtain the text sequence representation E ht , input {ht1, ht2, …, ht n}, output The BERT encoding method is shown in formula (8):

[0096]

[0097] in, is the sequence hidden state vector of the BERT embedding layer, and n is the length of the HT text sequence;

[0098] After encoding the characters in HT one by one, concatenate all the character vectors to get the text sequence representation E ht , the splicing algorithm is shown in formula (9):

[0099]

[0100] S23: Set soft prompt to E S and text sequence representation E ht The spliced ​​sequence is represented by E O , the vector concatenation algorithm is shown in formula (10):

[0101] E O =E S ⊕E ht #(10)

[0102] Among them, ⊕ represents the spelling operation, E S is the representation vector of the soft prompt ST, E ht is the representation vector of the text sequence HT;

[0103] S24: Using the Transformer-based BERT model to represent sequences O Perform feature extraction again to obtain the text representation E t .

[0104] In particular, text semantic feature extraction also includes constructing a multi-scale convolutional attention network training. The steps of multi-scale convolutional attention network training include the following steps:

[0105] S25: Use convolution kernels of different scales to extract feature maps of text semantics, and use K filters to filter E t ∈R L×D Convolution operation is performed to obtain K feature maps, and zero padding is used to make the feature maps consistent in scale. The convolution calculation method is shown in formula (11):

[0106] Ui=f(w i *E t +b i ), i∈[1,K]#(11)

[0107] Among them, Ui is the feature map obtained by convolution operation, f is the activation function, and w i is the weight value, b i ∈R is the bias term, K is the number of convolution kernels, Ui∈R L×D , L is the length of the text sequence, and D is the word embedding dimension value;

[0108] S26: Fuse multiple feature maps, and fuse K feature maps by element-wise addition to obtain a new feature map U∈R L×D , the feature map fusion method is shown in formula (12):

[0109]

[0110] S27: Perform global average pooling on the feature map U to obtain vector s, reduce the feature dimension, reduce noise and irregularity in the feature map, and increase the robustness of the model. The global average pooling calculation method is shown in formula (13):

[0111]

[0112] Among them, sc is the cth element of s, s∈R D×1 , s is a column vector with D elements;

[0113] S28: Input s into a fully connected layer for dimensionality reduction to obtain the compressed representation E v , that is, the text semantic feature vector, E v ∈R d×1 , d<D, the dimension reduction calculation method is shown in formula (14):

[0114] E v =δ(B(w z *s+b z ))#(14)

[0115] Among them, δ is a Relu function, B is a batch normalization function, and w z is the weight value, b z ∈R is the bias term, and the calculation method of d value is shown in formula (15):

[0116]

[0117] Where r is a hyperparameter used to control the scaling ratio, L d Indicates the minimum value of d, which is usually set to 32.

[0118] S3: Objective function optimization, using text semantic features and predicted word semantic features to predict relationships, measuring the similarity of probability distribution by calculating the cross entropy between predicted relationships and true relationships, and using this as the objective function to train the text classification model.

[0119] Specifically, the objective function optimization uses the text semantic features and the predicted word semantic features to map the relationship categories, uses the cross entropy loss function to calculate the difference between the predicted relationship distribution and the true relationship distribution, and trains the text intent recognition model. The input of the objective function optimization is the text semantic feature E v , the output is the text intent recognition model The objective function optimization comprises the following steps:

[0120] S31: From text semantic features E v Extract the semantic features of the predicted word E mask , using E mask and E v Predict the category probability. The probability prediction calculation method is shown in formulas (16) and (17):

[0121] p a =σ(W1E mask +b1)#(16)

[0122] pb =σ(W2E v +b2)#(17)

[0123] Among them, σ is the softmax activation function, W1, W2, b1, b2 are the weight matrices of the fully connected layer, and p a is the probability of the predicted value of [MASK] in the prompt template, p b is the probability of the predicted value of the traffic text intent category.

[0124] S32: Use the joint loss to train the text intent recognition model. The joint loss consists of the prediction word loss and the classification loss. The joint loss function is shown in formula (18). The prediction word loss function calculation is shown in formula (19). The classification loss function is shown in formula (20):

[0125]

[0126] in, is the joint loss value, ρ s is the weight coefficient of the predicted word loss, is the predicted word loss value, ρ c is the classification loss weight coefficient, is the classification loss value;

[0127]

[0128] Among them, y ia is a symbolic function. If the predicted word [MASK] of the i-th sample is the word represented by a, then y ia =1, otherwise y ia =0; p ia Represents the probability that the predicted word [MASK] of the i-th sample is predicted as word a, SM is the length of the BERT vocabulary, and N is the number of samples;

[0129]

[0130] Among them, y ib is a symbolic function. If the true relationship category of sample i is b, then y ib =1, otherwise y ib =0, p ib represents the probability that sample i belongs to relation category b, and M is the number of relation types.

[0131] S33: In the training phase, the loss is continuously calculated and back-propagated through the above process to update the parameters. The network parameters of the three modules of traffic text representation learning, multi-scale text semantic feature extraction and objective function optimization will be updated, and finally a trained text intent recognition model will be obtained.

[0132] S34: In the testing phase, input the test sample tx i , use prompt templates to structure the call text, and then input it into the model Calculate the probability value of the text representation belonging to each category, and output the intent recognition y corresponding to the maximum probability value.

[0133] The implementation principle of the embodiment of the present application is as follows: the present application proposes an objective function that integrates the prediction word loss and the classification loss, fully extracts the correlation between the intent category and the text, and facilitates the model to better understand the contextual semantic knowledge. The method uses text semantic features and prediction word features to predict the relationship type respectively, and uses cross entropy to calculate the similarity between the prediction result probability distribution and the true relationship probability distribution to train the text intent classification model and improve the text classification recall rate. This method can better capture the dependency relationship between words and optimize the model decision boundary, thereby improving the relationship classification accuracy of new samples. When the weight ratio of the prediction word loss and the classification loss is 1:1, the model can learn the discriminative features of different intent types; using soft and hard mixed prompt knowledge guidance The pre-trained language model strengthens the text representation. The multi-scale convolutional attention network is used to extract text semantic features and predict word semantic features from the text representation to enhance the model's intent classification performance. The multi-scale text semantic impact analysis experiment shows that convolutional attention networks with different structures can capture text semantic features at different levels. The optimal values ​​of the number of convolutional layers and the size of the convolutional kernel are related to the characteristics of the data set. Shallower convolutional networks are more suitable for capturing shallow semantic information, such as digital information, location information, word order or sentence structure, etc. Therefore, using a model with fewer convolutional layers is better for identifying text intents with relatively simple sentences and words. A hybrid prompt template of soft and hard prompts is constructed, and the pre-trained language model is combined to enhance the text representation ability. While retaining the clear guidance of hard prompts, this method uses the flexibility of soft prompts to dynamically focus on target words; it can learn highly correlated target intents among multiple intents, especially in identifying sensitive words, customer expectations, sentiment classification and other scenarios.

[0134] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for identifying traffic text intention based on knowledge prompts, characterized in that: The steps include: Prompt template construction, transforming the relationship classification problem into a relationship word prediction problem, using templates and text context to predict relationship words, fully learning the dependencies between words, and optimizing the model decision boundary; Extract text semantic features, learn the representation of the conversation text, extract features from the preprocessed text data, convert the text into numerical form using word embedding technology, build a coding vector, build a prompt encoder using BiLSTM and a feedforward neural network, and then use BERT encoding to concatenate two coding vectors to output the conversation text representation; extract key semantic features from the text by building a multi-scale convolutional attention network; Objective function optimization uses text semantic features and predicted word semantic features to predict relationships, measures the similarity of probability distribution by calculating the cross entropy between predicted relationships and true relationships, and uses this as the objective function to train the text classification model.

2. The method for identifying traffic text intention based on knowledge prompts according to claim 1, characterized in that: The input of the prompt template is the historical traffic training text set X = {X1, ..., X N } and BERT vocabulary S, the output is a text sequence DT, where N is the number of samples in the training sample set, and the prompt module construction includes the following sub-steps: Extract m pseudo-cue words {u1, ..., u m } as a soft prompt template ST, extract x i The co-occurrence word set C = {g1, ..., g C }, calculate the TF-IDF value of each co-occurring word in the sample. The calculation methods of TF, IDF, and TF-IDF are shown in formulas (1), (2), and (3), respectively: Among them, g i is the i-th co-occurring word, n i It is the word g i In the sample x i The number of occurrences in n k is the number of words in the sample, TF(g i , x i ) is the co-occurring word g i The frequency of words; Where N is the number of words, g n Represents the word g i The +1 operation is to avoid the denominator being 0. IDF(g i , X) is the co-occurring word g i The inverse document frequency of TF-IDF(g i ,x i ,X)=TF(g i ,x i )*IDF(g i ,X)#(3) TF-IDF(g i , x i , X) is used to measure the co-occurrence of words in sample x i The importance of The co-occurring words are sorted according to the TF-IDF value, and the sorted co-occurring word set C is used to concatenate with placeholders such as [CLS], [SEP], and [MASK] to construct hard prompts. Then, the hard prompt structured text is used to obtain the text sequence HT to be processed, and ST and HT are concatenated into a text sequence DT to be input into the next module. The construction method of the module is shown in formula (4): DT={[CLS],g1,…,g c ,ST,[MASK],x i ,[SEP]}#(4) Among them, [CLS], [SEP], and [MASK] are placeholder characters, representing classification tokens, sentence separation tokens, and mask tokens, respectively. They are used to capture the information of the entire input sequence, identify the boundaries of sentences, and predict missing words based on the context, thereby helping the model learn richer semantic information; g i is the i-th co-occurring word in the sample, c is the number of co-occurring words, and ST is the soft prompt template.

3. The method for identifying traffic text intention based on knowledge prompts according to claim 1, characterized in that: The input of the text semantic feature extraction is the text sequence DT, and the output is the text representation E t , the text semantic feature extraction includes the following steps: A prompt encoder is constructed using BiLSTM and a feedforward neural network, and the prompt encoder is used to encode the soft prompt ST to obtain a soft prompt identifier E S , the input of each word in the soft prompt is taken as a time step, and for each time step t (t = 1, 2, 3...), the forward long short-term memory neural network outputs the hidden state The backward long short-term memory network outputs the hidden state Concatenate the two hidden state vectors to get the feature vector of each character The soft prompt word vector encoding method is shown in formula (5): The soft prompt feature vector is obtained by merging the forward and reverse end output vectors of the BiLSTM layer. The soft prompt feature vector is shown in formula (6), where l is the length of the soft prompt template: H s The soft prompt representation E is obtained through a feedforward neural network S , the soft prompt indicates that the calculation is as shown in formula (7): E S =tanh(W T H S +b)#(7) Among them, W T is the weight, b is the bias term, W T and b are trainable parameter weights of the feedforward neural network, and the activation function is the tanh function; Use BERT to encode the HT text sequence to get the text sequence representation E ht , input {ht1, ht2, …, ht n }, output The BERT encoding method is shown in formula (8): in, is the sequence hidden state vector of the BERT embedding layer, and n is the length of the HT text sequence; After encoding the characters in HT one by one, concatenate all the character vectors to get the text sequence representation E ht , the splicing algorithm is shown in formula (9): The soft prompt indicates E S and text sequence representation E ht The spliced ​​sequence is represented by E O , the vector concatenation algorithm is shown in formula (10): in, Indicates spelling operation, E S is the representation vector of the soft prompt ST, E ht is the representation vector of the text sequence HT; Using the Transformer-based BERT model to represent the sequence E O Perform feature extraction again to obtain the text representation E t .

4. The method for identifying traffic text intention based on knowledge prompts according to claim 3 is characterized in that: The text semantic feature extraction also includes constructing a multi-scale convolutional attention network training.

5. The method for identifying traffic text intention based on knowledge prompts according to claim 4 is characterized in that: The steps of multi-scale convolutional attention network training include the following steps: Convolution kernels of different scales are used to extract the feature map of text semantics, and K filters are used to filter E t ∈R L×D Convolution operation is performed to obtain K feature maps, and zero padding is used to make the feature maps consistent in scale. The convolution calculation method is shown in formula (11): Ui=f(w i *E t +b i ),i∈[1,K]#(11) Among them, Ui is the feature map obtained by convolution operation, f is the activation function, and w i is the weight value, b i ∈R is the bias term, K is the number of convolution kernels, Ui∈R L×D , L is the length of the text sequence, and D is the word embedding dimension value; Fusion of multiple feature maps: K feature maps are fused by element-wise addition to obtain a new feature map U∈R L ×D , the feature map fusion method is shown in formula (12): Perform global average pooling on the feature map U to obtain the vector s, which reduces the feature dimension, reduces the noise and irregularity in the feature map, and increases the robustness of the model. The global average pooling calculation method is shown in formula (13): Among them, s c is the cth element of s, s∈R D×1 , s is a column vector with D elements; Input s into a fully connected layer for dimensionality reduction to obtain the compressed representation E v , that is, the text semantic feature vector, E v ∈R d×1 , d<D, the dimension reduction calculation method is shown in formula (14): E v =δ(B(w z *s+b z ))#(14) Among them, δ is a Relu function, B is a batch normalization function, and w z is the weight value, b z ∈R is the bias term, and the calculation method of d value is shown in formula (15): Where r is a hyperparameter used to control the scaling ratio, L d Indicates the minimum value of d, which is usually set to 32.

6. The method for identifying traffic text intention based on knowledge prompts according to claim 1, characterized in that: The objective function optimization uses text semantic features and predicted word semantic features to map relationship categories, uses the cross entropy loss function to calculate the difference between the predicted relationship distribution and the true relationship distribution, and trains the text intent recognition model. The objective function optimization input is the text semantic feature E v , the output is the text intent recognition model The objective function optimization comprises the following steps: From the text semantic features E v Extract the semantic features of the predicted word E mask , using E mask and E v Predict the category probability. The probability prediction calculation method is shown in formulas (16) and (17): p a =σ(W1E mask +b1)#(16) p b =σ(W2E v +b2)#(17) Among them, σ is the softmax activation function, W1, W2, b1, b2 are the weight matrices of the fully connected layer, and p a is the probability of the predicted value of [MASK] in the prompt template, p b is the probability of the predicted value of the traffic text intent category; The text intent recognition model is trained using the joint loss. The joint loss consists of the prediction word loss and the classification loss. The joint loss function is shown in formula (18), the prediction word loss function is calculated as shown in formula (19), and the classification loss function is shown in formula (20): in, is the joint loss value, ρ s is the weight coefficient of the predicted word loss, is the predicted word loss value, ρ c is the classification loss weight coefficient, is the classification loss value; Among them, y ia is a symbolic function. If the predicted word [MASK] of the i-th sample is the word represented by a, then y ia =1, otherwise y ia =0; p ia Represents the probability that the predicted word [MASK] of the i-th sample is predicted as word a, SM is the length of the BERT vocabulary, and N is the number of samples; Among them, y ib is a symbolic function. If the true relationship category of sample i is b, then y ib =1, otherwise y ib =0, p ib represents the probability that sample i belongs to relation category b, and M is the number of relation types; In the training phase, the loss is continuously calculated and back-propagated through the above process to update the parameters. The network parameters of the three modules of traffic text representation learning, multi-scale text semantic feature extraction and objective function optimization will be updated, and finally a trained text intent recognition model will be obtained. In the testing phase, input the test sample tx i , use prompt templates to structure the call text, and then input it into the model Calculate the probability value of the text representation belonging to each category, and output the intent recognition y corresponding to the maximum probability value.

Images