Intention recognition method, device and related equipment

Abstract

The present disclosure provides an intention recognition method, apparatus and related equipment, the method comprising: obtaining a phoneme sequence corresponding to the audio to be detected, and based on the phoneme sequence, determining a text sequence corresponding to the audio to be detected; based on a pre-trained language model, performing feature extraction and fusion on the text sequence and the phoneme sequence to obtain a sentence feature vector corresponding to the audio to be detected; based on the sentence feature vector, obtaining a prediction result for characterizing the intention of the audio to be detected. The method can effectively improve the accuracy of spoken intention recognition.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent voice technology, and in particular to an intent recognition method, apparatus, and related equipment. Background Art

[0002] The current intelligent voice dialogue system is mainly generated by five parts: speech recognition, spoken language understanding, dialogue management, dialogue generation, and speech synthesis. Spoken language understanding is mainly composed of three parts: domain identification, intent recognition, and slot filling. Intent recognition usually takes the top 1 text output by the speech recognition module for classification prediction. The main means to improve the accuracy of classification prediction are to expand training data, adopt better classification models, and add pre-training. Pre-training is a common means of enhancing text representation. It mainly obtains the corresponding word vector by modeling the relationship between a word in a sentence and its surrounding context. However, when faced with homophones and near-phonetic words with speech recognition errors, the existing intent recognition method will identify the wrong intent, and the accuracy of spoken intent recognition is low. How to improve the accuracy of spoken intent recognition needs to be solved urgently. Summary of the invention

[0003] The present disclosure provides an intention recognition method, apparatus and related equipment, which are used to solve the defect in the prior art that when facing homophones and near-phonetic characters with speech recognition errors, the existing intention recognition method will recognize the wrong intention, so as to improve the accuracy of spoken intention recognition.

[0004] In a first aspect, the present disclosure provides an intent recognition method, comprising: obtaining a phoneme sequence corresponding to audio to be detected, and based on the phoneme sequence, determining a text sequence corresponding to the audio to be detected; based on a pre-trained language model, performing feature extraction and fusion on the text sequence and the phoneme sequence to obtain a sentence feature vector corresponding to the audio to be detected; based on the sentence feature vector, obtaining a prediction result for characterizing the intent of the audio to be detected.

[0005] According to an intention recognition method provided by the present disclosure, obtaining a prediction result for characterizing the intention of the audio to be detected based on the sentence feature vector specifically includes: inputting the sentence feature vector into a fully connected neural network for classification prediction to obtain the prediction result.

[0006] According to an intention recognition method provided by the present disclosure, the feature extraction and fusion of the text sequence and the phoneme sequence to obtain a sentence feature vector corresponding to the audio to be detected includes: inputting the text sequence and the phoneme sequence into the pre-trained language model to obtain a robust word vector representation corresponding to the audio to be detected; based on the robust word vector representation, obtaining a sentence feature vector output by the pre-trained language model and corresponding to the audio to be detected.

[0007] According to an intent recognition method provided by the present disclosure, the pre-trained language model is trained and determined in the following manner, including: inputting a real text sequence and a real phoneme sequence into an initial pre-trained language model, adjusting and determining a first pre-trained language model network parameter; wherein the real text sequence is a text sequence obtained from the Internet; the real phoneme sequence is obtained based on the real text sequence; the first pre-trained language model network parameter is used to characterize the pronunciation relationship between the real text sequence and the real phoneme sequence; inputting a sample text sequence and a corresponding sample phoneme sequence into a pre-trained language model whose network parameters are the first pre-trained language model network parameters, and obtaining a second pre-trained language model network parameter; wherein the sample text sequence and the corresponding sample phoneme sequence are obtained by performing speech recognition on a preset plurality of sample audios; and constructing the pre-trained language model with generalization capability based on the second pre-trained language model network parameters.

[0008] According to an intent recognition method provided by the present disclosure, the inputting of a real text sequence and a real phoneme sequence into an initial pre-trained language model, and adjusting and determining the first pre-trained language model network parameters further include: inputting the real text sequence and the real phoneme sequence into the initial pre-trained language model, masking the real text sequence and the real phoneme sequence, obtaining a first masked text sequence and a first masked phoneme sequence, predicting the masked first masked text sequence and the first masked phoneme sequence through context, and adjusting and determining the first pre-trained language model network parameters; wherein the first masked text sequence is 20% of the text sequences randomly selected from the real text sequence; and the first masked phoneme sequence is 20% of the phoneme sequences randomly selected from the real phoneme sequence.

[0009] According to an intent recognition method provided by the present disclosure, the sample text sequence and the corresponding sample phoneme sequence are input into a pre-trained language model whose network parameters are the first pre-trained language model network parameters, and the second pre-trained language model network parameters are constructed further including: inputting the sample text sequence and the sample phoneme sequence into a pre-trained language model whose network parameters are the first pre-trained language model network parameters, masking the sample text sequence and the sample phoneme sequence to obtain a second masked text sequence and a second masked phoneme sequence, and obtaining the second pre-trained language model network parameters by predicting the masked second masked text sequence and the second masked phoneme sequence through context; wherein the second masked text sequence is 20% of the text sequences randomly selected from the sample text sequence; and the second masked phoneme sequence is 20% of the phoneme sequences randomly selected from the sample phoneme sequence.

[0010] According to an intention recognition method provided by the present disclosure, the real phoneme sequence is obtained according to the real text sequence, specifically comprising: segmenting the real text sequence to obtain the real characters; acquiring the phonemes of the real characters according to a phoneme comparison table; and obtaining the real phoneme sequence based on the phonemes of all the real characters.

[0011] According to an intent recognition method provided by the present disclosure, the network parameters of the second pre-trained language model are adjusted in the following manner, specifically including: determining the real text transcribed from the sample audio listened to manually; based on the cross entropy loss function, calculating the difference between the predicted result representing the intent of the sample audio and the real text; based on the difference, adjusting the network parameters of the first pre-trained language model to obtain the network parameters of the second pre-trained language model.

[0012] In the second aspect, the present disclosure also provides an intention recognition device, including: a speech recognition module, a pre-trained language module and a fully connected neural network classification module. Among them, the speech recognition module is used to obtain a phoneme sequence corresponding to the audio to be detected, and based on the phoneme sequence, determine the text sequence corresponding to the audio to be detected; the pre-trained language module is used to extract and fuse the text sequence and the phoneme sequence based on the pre-trained language model to obtain a sentence feature vector corresponding to the audio to be detected; the fully connected neural network classification module is used to obtain a prediction result for characterizing the intention of the audio to be detected based on the sentence feature vector.

[0013] In a third aspect, the present disclosure further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described intent recognition methods are implemented.

[0014] In a fourth aspect, the present disclosure further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described intent recognition methods.

[0015] The present disclosure provides an intention recognition method, apparatus and related equipment. By obtaining a phoneme sequence corresponding to an audio to be detected, and based on the phoneme sequence, determining a text sequence corresponding to the audio to be detected; based on a pre-trained language model, performing feature extraction and fusion on the text sequence and the phoneme sequence to obtain a sentence feature vector corresponding to the audio to be detected; the pre-trained language model can incorporate the phoneme information corresponding to a character into the pre-training task. Thus, in the process of performing feature extraction and fusion on the text sequence and the phoneme sequence, the information not only has text-level features but also pronunciation features, so that the obtained sentence feature vector has strong anti-noise ability. Based on the sentence feature vector, the prediction result used to characterize the intention of the audio to be detected will be more accurate. For example, when facing similar pronunciation errors in the user's reply such as "yes" and "four", although at the text level, the sentence feature vectors corresponding to the two have very low correlation and are very different, the sentence feature vector obtained by the pre-trained model with fused phoneme information will consider both text-level and pronunciation feature information, that is to say, it can take into account the pronunciation similarity between "yes" and "four", so as to align the two to a certain extent, and then obtain an accurate prediction result used to characterize the intention of the audio to be detected, improving the accuracy of spoken language intention recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present disclosure or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0017] Figure 1 is one of the flow diagrams of the intention recognition method provided by the present disclosure;

[0018] Figure 2 is the step flow diagram of performing feature extraction and fusion on a text sequence and a phoneme sequence based on a pre-trained language model to obtain a sentence feature vector corresponding to an audio to be detected in the intention recognition method provided by the present disclosure;

[0019] Figure 3 is the flow diagram of determining a pre-trained language model through training in the embodiment of the intention recognition method of the present disclosure;

[0020] Figure 4 is the step flow diagram of inputting a sample text sequence and a corresponding sample phoneme sequence into a pre-trained language model with the network parameters of the first pre-trained language model network parameters to obtain the network parameters of the second pre-trained language model in the intention recognition method of the present disclosure;

[0021] Figure 5 is a schematic diagram of the structure of the intention recognition device provided by the present disclosure;

[0022] Figure 6 A schematic diagram of the structure of a pre-trained language module in the intention recognition device provided by the present disclosure;

[0023] Figure 7 A schematic diagram of pre-trained language model training in the intent recognition method provided by the present disclosure;

[0024] Figure 8 A flowchart of a specific embodiment of the method for identifying intentions provided in the present disclosure;

[0025] Fig. 9 It is a schematic diagram of the structure of an electronic device provided by the present disclosure. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present disclosure.

[0027] Combine the following Figure 1 Various embodiments of the intent recognition method of the present disclosure are described.

[0028] Reference Figure 1 , Figure 1 It is one of the flowcharts of the intent recognition method provided by the present disclosure. The intent recognition method provided by the embodiment of the present disclosure includes:

[0029] Step S101, obtaining a phoneme sequence corresponding to the audio to be detected, and determining a text sequence corresponding to the audio to be detected based on the phoneme sequence;

[0030] Step S103: Based on the pre-trained language model, feature extraction and fusion are performed on the text sequence and the phoneme sequence to obtain a sentence feature vector corresponding to the audio to be detected;

[0031] Step S105: Based on the sentence feature vector, obtain a prediction result for characterizing the intention of the audio to be detected.

[0032] The intent recognition method provided by the embodiments of the present disclosure obtains a phoneme sequence corresponding to the audio to be detected, and determines a text sequence corresponding to the audio to be detected based on the phoneme sequence; based on a pre-trained language model, feature extraction and fusion are performed on the text sequence and the phoneme sequence to obtain a sentence feature vector corresponding to the audio to be detected; the pre-trained language model can incorporate the phoneme information corresponding to the characters into the pre-training task. Thus, in the process of feature extraction and fusion of the text sequence and the phoneme sequence, the information has not only the characteristics at the text level but also the pronunciation characteristics, so that the sentence feature vector has strong anti-noise ability. Based on the sentence feature vector, the prediction result used to characterize the intent of the audio to be detected will be more accurate. For example, when facing similar pronunciation errors in the user's reply, such as "yes" and "four", although at the text level, the sentence feature vectors corresponding to the two have very low correlation and are very different, the sentence feature vector obtained by the pre-trained model with fused phoneme information will consider both the text level and the pronunciation characteristics. That is to say, it can take into account the pronunciation similarity between "yes" and "four", so as to align the two to a certain extent, and then obtain an accurate prediction result used to characterize the intent of the audio to be detected, improving the accuracy of spoken language intent recognition.

[0033] The following Figure 1 further describes each step of the illustrated embodiment.

[0034] Step S101: Obtain a phoneme sequence corresponding to the audio to be detected, and determine a text sequence corresponding to the audio to be detected based on the phoneme sequence;

[0035] In this step, a phoneme is the smallest speech unit divided according to the natural attributes of speech. Analyzing according to the pronunciation actions in a syllable, one action constitutes one phoneme. Phonemes are divided into two major categories: vowels and consonants. For example, the Chinese syllable "ā" has only one phoneme, "ài" has two phonemes, "dài" has three phonemes, etc.

[0036] Obtaining the phoneme sequence corresponding to the audio to be detected can be achieved by the acoustic model in the speech recognition module receiving the audio signal for feature extraction and then through conversion. After the acoustic model gives the phoneme sequence, the string sequence with the highest probability is selected from the candidate text sequences as the recognized top1 text, which is used as the text sequence.

[0037] Step S103: Based on a pre-trained language model, perform feature extraction and fusion on the text sequence and the phoneme sequence to obtain a sentence feature vector corresponding to the audio to be detected;

[0038] The pre-trained language model generally uses the BERT model, which is trained using large-scale unlabeled corpus to obtain the semantic representation of text containing rich semantic information. The semantic representation of the text is then fine-tuned in a specific NLP task and finally applied to the NLP task.

[0039] Step S105: Based on the sentence feature vector, obtain a prediction result for characterizing the intention of the audio to be detected.

[0040] In this step, the prediction result of the intention of the audio to be detected may be a classification result, which may be set as required.

[0041] In an optional embodiment, step S105 specifically includes:

[0042] Step S1051: input the sentence feature vector to the fully connected neural network for classification prediction to obtain the prediction result.

[0043] A fully connected neural network (FFNN) feeds information from the front to the back (input and output, respectively). Neural networks are usually described as having multiple layers, where each layer consists of parallel input, hidden or output neurons. A single layer will not have connections, and usually two adjacent layers are fully connected (each neuron in one layer is connected to every neuron in another layer). The simplest practical network has two input neurons and one output neuron, which can be used to model logic gates. A fully connected neural network plays the role of mapping the sentence feature vector to obtain a prediction result that the user can understand.

[0044] People usually train fully connected neural networks via the back-propagation algorithm, giving the network a pair of data sets (input data set + expected output data set), which is called supervised learning, rather than unsupervised learning where you just give it the input and let the network fill in the blanks. The error of back propagation is usually some variation of the difference between the input and the output. Assuming the network has enough hidden neurons, in theory it can always model the relationship between the input and the output. In practice, their use is very limited, but they are often combined with other networks to form new networks.

[0045] In an alternative embodiment, referring to Figure 2 , Figure 2 In the intent recognition method provided in the present disclosure, based on the pre-trained language model, feature extraction and fusion are performed on the text sequence and the phoneme sequence to obtain the sentence feature vector corresponding to the audio to be detected. Step S103 specifically includes:

[0046] Step S2031, inputting a text sequence and a phoneme sequence into a pre-trained language model to obtain a robust word vector representation corresponding to the audio to be detected;

[0047] Robust word vector representation is a stronger and more robust word vector representation. Because there may be errors in the recognized words in speech recognition, but the errors can be traced, that is, the pronunciation is similar. By adding the MLM task of phonemes to the Bert model and assigning phoneme information to the word vector, a more robust word vector representation, that is, a robust word vector representation, can be obtained.

[0048] Step S2032: Based on the robust word vector representation, obtain the sentence feature vector output by the pre-trained language model and corresponding to the audio to be detected.

[0049] The sentence feature vector is the vector corresponding to the [CLS] token output by the Bert model.

[0050] Reference Figure 3 , Figure 3 This is one of the flowcharts of determining a pre-trained language model through training in the embodiment of the intent recognition method disclosed in the present invention, which specifically includes the following steps:

[0051] Step S301: input the real text sequence and the real phoneme sequence into the initial pre-trained language model, and adjust and determine the first pre-trained language model network parameters;

[0052] Step S303: input the sample text sequence and the corresponding sample phoneme sequence into a pre-trained language model whose network parameters are the network parameters of the first pre-trained language model to obtain the network parameters of the second pre-trained language model;

[0053] Step S305: constructing the pre-trained language model with generalization capability based on the second pre-trained language model network parameters;

[0054] Below Figure 3 The various steps of the illustrated embodiment are further described.

[0055] Step S301: input the real text sequence and the real phoneme sequence into the initial pre-trained language model, and adjust and determine the first pre-trained language model network parameters;

[0056] Specifically, because Bert needs a large amount of text to train the initial pre-trained language model, but the collected sample audio does not have that much data, it is necessary to first pre-train by table lookup to obtain the first pre-trained language model network parameters for characterizing the pronunciation relationship between the real text sequence and the real phoneme sequence. It can be understood that the real text sequence is a large amount of text sequence obtained from the Internet, and then the real phoneme sequence is obtained based on the real text sequence.

[0057] Step S303: input the sample text sequence and the corresponding sample phoneme sequence into a pre-trained language model whose network parameters are the network parameters of the first pre-trained language model to obtain the network parameters of the second pre-trained language model;

[0058] In order to adapt to the use in real scenarios, the text obtained by real speech recognition may deviate from the real intended text, so we use the pre-trained language model to adjust the parameters by inputting multiple preset sample audios for speech recognition to obtain the sample text sequence and the corresponding sample phoneme sequence, that is, using noisy training data for parameter adjustment to obtain the second pre-trained language model network parameters, that is, the model network parameters for real use. Figure 7 shown.

[0059] Step S305: constructing the pre-trained language model with generalization capability based on the second pre-trained language model network parameters;

[0060] Based on the trained network parameters, we can obtain the BERT model that fused the phoneme information and can adapt to noisy situations.

[0061] For step S301, in an optional embodiment, it may include:

[0062] Step S3011, inputting the real text sequence and the real phoneme sequence into the initial pre-trained language model, masking the real text sequence and the real phoneme sequence, obtaining a first masked text sequence and a first masked phoneme sequence, predicting the masked first masked text and the first masked phoneme sequence through context, and adjusting and determining the first pre-trained language model network parameters;

[0063] Step S3011 is to modify the representation learning mechanism of the BERT model, namely, MLM (mask language model). The general MLM only needs to mask 15% of the text sequence, but because we have integrated the phoneme information and the correlation has increased, we need to increase the mask ratio to 20%, and the first masked text sequence is the 20% text sequence randomly selected from the real text sequence; the first masked phoneme sequence is the 20% phoneme sequence randomly selected from the real phoneme sequence. In the selected field, in 10% of the cases, the words are randomly replaced, in 10% of the cases, the words are not replaced, and in 80% of the cases, the words are masked, and the first masked text sequence is predicted based on the context, that is, the remaining vocabulary. Similarly, in the selected phoneme segment, in 10% of the cases, the phonemes are randomly replaced, in 10% of the cases, the phonemes are not replaced, and in 80% of the cases, the phonemes are masked, and the first masked phoneme sequence is predicted based on the context, that is, the remaining phonemes.

[0064] For step S303, in an optional embodiment, it may include:

[0065] Step S3031, inputting the sample text sequence and the sample phoneme sequence into a pre-trained language model whose network parameters are the network parameters of the first pre-trained language model, masking the sample text and the sample phoneme sequence to obtain a second masked text and a second masked phoneme sequence, and predicting the masked second masked text and the second masked phoneme sequence through context to obtain the network parameters of the second pre-trained language model;

[0066] For step S3031, similarly, the general MLM only needs to mask 15% of the text sequence, but because we have integrated the phoneme information and the correlation is improved, we need to increase the mask ratio to 20%, and the second masked text sequence is the 20% text sequence randomly selected from the sample text sequence; the second masked phoneme sequence is the 20% phoneme sequence randomly selected from the sample phoneme sequence. In the selected field, in 10% of the cases, the words are randomly replaced, in 10% of the cases, the words are not replaced, and in 80% of the cases, the words are masked, and the masked second masked text sequence is predicted based on the context, that is, the remaining vocabulary. Similarly, in the selected phoneme segment, in 10% of the cases, the phonemes are randomly replaced, in 10% of the cases, the phonemes are not replaced, and in 80% of the cases, the phonemes are masked, and the masked second masked phoneme sequence is predicted based on the context, that is, the remaining phonemes.

[0067] In a preferred embodiment, before step S301, the method further includes:

[0068] Step S300: Obtain a real phoneme sequence according to the real text.

[0069] Step S300 specifically includes:

[0070] Step S3001, segmenting the real text to obtain real characters;

[0071] The first step is word segmentation, because different word combinations have different pronunciations, so it is necessary to first segment the words to determine the pronunciation of the word under this word combination. Chinese word segmentation is generally based on characters.

[0072] Step S3002, obtaining the phonemes of the real word according to the phoneme comparison table;

[0073] Since real text is a massive amount of Internet text and will not go through the speech recognition system, to determine the pronunciation of this word, we generally obtain the phonemes of the word through the phoneme comparison table. When facing a word with multiple sounds, we select the phoneme with the most common pronunciation as its corresponding phoneme.

[0074] Step S3003: Obtain a real phoneme sequence based on the phonemes of all real characters.

[0075] Reference Figure 4 , Figure 4 In the disclosed intention recognition method, a sample text sequence and a corresponding sample phoneme sequence are input into a pre-trained language model whose network parameters are the first pre-trained language model network parameters to obtain the second pre-trained language model network parameters. In a specific embodiment, step S303 specifically includes the following steps:

[0076] Step S401: input the sample text sequence and the corresponding sample phoneme sequence into a pre-trained language model whose network parameters are the network parameters of the first pre-trained language model, and output a sentence feature vector;

[0077] Step S403: obtaining a prediction result for characterizing the intention of the audio sample based on the sentence feature vector;

[0078] Step S405, determining the real text transcribed from the sample audio listened to manually;

[0079] Step S407: Calculate the difference between the prediction result and the real text used to characterize the intention of the audio sample based on the cross entropy loss function;

[0080] Step S409: Based on the difference, adjust the first pre-trained language model network parameters to obtain the second pre-trained language model network parameters.

[0081] In order to obtain the network parameters of the pre-trained language model with more generalization ability, we often need to make multiple adjustments to determine and optimize for specific applications, that is, the Finetune task in the BERT model. The sentence feature vector obtained by the BERT model, like the sentence vector representation obtained by ordinary CNN and LSTM at the end of the model, needs to be mapped to the classification label space through a fully connected neural network. For example, the BERT model obtains a CLS vector of 768 dimensions and needs to map it to 10 classification labels. It needs a fully connected network. The shape of the parameter w is 768*10, and the final 10-dimensional vector logits is obtained. This logits is normalized and scaled by softmax to obtain the probability of each intent. Generally, the label corresponding to the value with the largest probability is taken as the prediction result of the intent of the audio sample. The cross entropy loss function is the loss calculation function, which calculates the difference between the probability of the prediction result of the intent of the audio sample and the true intent 0,1. If the probability of the prediction result of the intent of the audio sample is higher and the intent classification is correct, that is, 1, the loss is smaller. On the contrary, if the prediction is wrong, the loss is greater. The cross entropy loss compares the probability distribution value after the softmax normalized probability scaling and the loss calculation between the labels 0 and 1. The specific calculation formula is:

[0082]

[0083] When the loss value after multiple trainings is less than the preset loss value, or reaches the preset number of trainings, the training ends and the network parameters of the second pre-trained language model are obtained.

[0084] The intention recognition method provided in the present disclosure can be described by taking a specific application as an example to describe its actual application process. Figure 8 , input a piece of speech, perform speech recognition on it, get the TOP1 text and corresponding phonemes, input the pre-trained language model, get robust word vector features, perform sentence embedding on the robust word vector features, that is, perform sentence concatenation according to their position attributes, also known as model transformation, get the sentence feature vector and send it to the fully connected network to get the prediction result that represents the speech intent.

[0085] The intention recognition device provided in an embodiment of the present disclosure is described below. The intention recognition device described below and the intention recognition method described above can be referenced to each other.

[0086] Reference Figure 5 , Figure 5 is a structural diagram of the intention recognition device provided by the present disclosure. The embodiment of the present disclosure also provides an intention recognition device, including: a speech recognition module 50, a pre-trained language module 52 and a fully connected neural network classification module 54. Among them, the speech recognition module 50 is used to obtain a phoneme sequence corresponding to the audio to be detected, and based on the phoneme sequence, determine the text sequence corresponding to the audio to be detected; the pre-trained language module 52 is used to extract and fuse the text sequence and the phoneme sequence based on the pre-trained language model, and obtain the sentence feature vector corresponding to the audio to be detected; the fully connected neural network classification module 54 is used to obtain the prediction result for characterizing the intention of the audio to be detected based on the sentence feature vector.

[0087] The intention recognition device provided by the embodiment of the present disclosure obtains the phoneme sequence corresponding to the audio to be detected through the speech recognition module 50, and determines the text sequence corresponding to the audio to be detected based on the phoneme sequence; then the pre-trained language module 52 is used to extract and fuse the text sequence and the phoneme sequence based on the pre-trained language model to obtain the sentence feature vector corresponding to the audio to be detected; the pre-trained language model in the pre-trained language module 52 can integrate the phoneme information corresponding to the word into the pre-training task, so that in the process of extracting and fusing the text sequence and the phoneme sequence, the information not only has the text level feature but also has the pronunciation feature, so as to have a sentence feature vector with strong anti-noise ability. , the prediction result for characterizing the intention of the audio to be detected obtained by the fully connected neural network classification module 54 will be more accurate. For example, when facing similar pronunciation errors in user replies such as "yes" and "four", although at the text level, the sentence feature vector representations corresponding to the two are very low in correlation and the sentence feature vectors are far apart, the sentence feature vector obtained by the pre-trained model with fused phoneme information will consider both text level and pronunciation feature information, that is, it can take into account the pronunciation similarity between "yes" and "four", so that the two representations can be aligned to a certain extent, thereby obtaining an accurate prediction result for characterizing the intention of the audio to be detected, thereby improving the accuracy of spoken intention recognition.

[0088] In an optional embodiment, the fully connected neural network classification module 54 is further used to: input the sentence feature vector to the fully connected neural network for classification prediction and obtain the prediction result.

[0089] A fully connected neural network (FFNN) feeds information from the front to the back (input and output, respectively). Neural networks are usually described as having multiple layers, where each layer consists of parallel input, hidden or output neurons. A single layer will not have connections, and usually two adjacent layers are fully connected (each neuron in one layer is connected to every neuron in another layer). The simplest practical network has two input neurons and one output neuron, which can be used to model logic gates. A fully connected neural network plays the role of mapping the sentence feature vector to obtain a prediction result that the user can understand.

[0090] People usually train fully connected neural networks via the back-propagation algorithm, giving the network a pair of data sets (input data set + expected output data set), which is called supervised learning, rather than unsupervised learning where you just give it the input and let the network fill in the blanks. The error of back propagation is usually some variation of the difference between the input and the output. Assuming the network has enough hidden neurons, in theory it can always model the relationship between the input and the output. In practice, their use is very limited, but they are often combined with other networks to form new networks.

[0091] Reference Figure 6 , Figure 6 A schematic diagram of the structure of the pre-trained language module 52 in the intention recognition device provided by the present disclosure, wherein the pre-trained language module 52 further includes:

[0092] A robust word vector representation acquisition unit 520 is used to input a text sequence and a phoneme sequence into a pre-trained language model to acquire a robust word vector representation corresponding to the audio to be detected;

[0093] Robust word vector representation is a stronger and more robust word vector representation. Because there may be errors in the recognized words in speech recognition, but the errors can be traced, that is, the pronunciation is similar. By adding the MLM task of phonemes to the Bert model and assigning phoneme information to the word vector, a more robust word vector representation, that is, a robust word vector representation, can be obtained.

[0094] The sentence feature vector acquisition unit 522 is used to acquire the sentence feature vector output by the pre-trained language model and corresponding to the audio to be detected based on the robust word vector representation.

[0095] The sentence feature vector is the vector corresponding to the [CLS] token output by the Bert model.

[0096] In an optional embodiment, the pre-trained language module 52 is further used to train and determine a pre-trained language model, specifically including:

[0097] A modeling unit, used for inputting a real text sequence and a real phoneme sequence into an initial pre-trained language model, and adjusting and determining network parameters of the first pre-trained language model;

[0098] The real text sequence is a text sequence obtained from the Internet; the real phoneme sequence is obtained according to the real text; the first pre-trained language model network parameter is used to characterize the pronunciation relationship between the real text sequence and the real phoneme sequence;

[0099] A network parameter acquisition unit is used to input the sample text sequence and the corresponding sample phoneme sequence into a pre-trained language model whose network parameters are the network parameters of the first pre-trained language model to obtain the network parameters of the second pre-trained language model;

[0100] In order to adapt to use in real scenarios, the text obtained by real speech recognition may deviate from the truly intended text, so we use a pre-trained language model that obtains a sample text sequence and a corresponding sample phoneme sequence by inputting a preset multiple sample audio for speech recognition, and then adjusts the parameters. That is, we use noisy training data for parameter adjustment to obtain the second pre-trained language model network parameters, that is, the model network parameters for actual use.

[0101] A model building unit is used to build the pre-trained language model with generalization capability based on the network parameters of the second pre-trained language model;

[0102] In a specific embodiment, the modeling unit is further configured to:

[0103] Inputting the real text sequence and the real phoneme sequence into the initial pre-trained language model, masking the real text sequence and the real phoneme sequence to obtain a first masked text sequence and a first masked phoneme sequence, predicting the masked first masked text and the first masked phoneme sequence through context, and adjusting and determining the first pre-trained language model network parameters;

[0104] Specifically, because the BERT model requires a large amount of text to train the initial pre-trained language model, but the sample audio collected does not have that much data, it is necessary to first pre-train by table lookup to obtain the first pre-trained language model network parameters to characterize the pronunciation relationship between the real text sequence and the real phoneme sequence. It can be understood that the real text sequence is a large amount of text sequence obtained from the Internet, and then the real phoneme sequence is obtained based on the real text sequence.

[0105] In a specific embodiment, the network parameter acquisition unit is further used to:

[0106] Inputting the sample text sequence and the sample phoneme sequence into a pre-trained language model whose network parameters are the network parameters of the first pre-trained language model, masking the sample text and the sample phoneme sequence to obtain a second masked text and a second masked phoneme sequence, and predicting the masked second masked text and the second masked phoneme sequence through context to obtain the network parameters of the second pre-trained language model;

[0107] Similarly, general MLM only needs to mask 15% of the text sequence, but since we have integrated phoneme information and increased the relevance, we need to increase the masking ratio to 20%, so the second masked text sequence is the 20% text sequence randomly selected from the sample text sequence; the second masked phoneme sequence is the 20% phoneme sequence randomly selected from the sample phoneme sequence. In the selected field, in 10% of cases, the words are randomly replaced, in 10% of cases, the words are not replaced, and in 80% of cases, the words are masked. The masked second masked text sequence is predicted based on the context, i.e., the remaining vocabulary. Similarly, in the selected phoneme segment, in 10% of cases, the phonemes are randomly replaced, in 10% of cases, the phonemes are not replaced, and in 80% of cases, the phonemes are masked. The masked second masked phoneme sequence is predicted based on the context, i.e., the remaining phonemes.

[0108] In an optional embodiment, the modeling unit further includes a real phoneme sequence acquisition unit, which is used to obtain a real phoneme sequence according to the real text, specifically including:

[0109] The word segmentation subunit is used to segment the real text to obtain the real words;

[0110] The reason why we use word segmentation sub-units is that different word combinations have different pronunciations. Therefore, we need to first segment the word to determine the pronunciation of the word under this word combination. Chinese word segmentation is generally based on characters.

[0111] Obtaining a phoneme subunit, for obtaining the phonemes of a real word according to a phoneme comparison table;

[0112] Since real texts are massive Internet texts and will not be passed through a speech recognition system, to determine the pronunciation of each real word in the real text, we generally obtain the phonemes of the word through a phoneme comparison table. When faced with a real word with multiple sounds, we select the phoneme with the most common pronunciation as its corresponding phoneme.

[0113] The phoneme subunits are obtained to obtain a real phoneme sequence based on the phonemes of all real words.

[0114] In a specific embodiment, the network parameter acquisition unit specifically includes:

[0115] The output vector subunit inputs the sample text sequence and the corresponding sample phoneme sequence into a pre-trained language model whose network parameters are the network parameters of the first pre-trained language model, and outputs a sentence feature vector;

[0116] An acquisition result subunit, used for acquiring a prediction result for representing the intent of the audio sample based on the sentence feature vector;

[0117] A text determination subunit, used to determine the real text transcribed from the sample audio manually listened to;

[0118] A calculation subunit, used for calculating the difference between the prediction result for representing the intent of the audio sample and the real text based on a cross entropy loss function;

[0119] The adjusting subunit is used to adjust the first pre-trained language model network parameters based on the difference to obtain the second pre-trained language model network parameters.

[0120] In order to obtain the network parameters of the pre-trained language model with more generalization ability, we often need to make multiple adjustments to determine and optimize for specific applications, that is, the Finetune task in the BERT model. The sentence feature vector obtained by the BERT model, like the sentence vector representation obtained by ordinary CNN and LSTM at the end of the model, needs to be mapped to the classification label space through a fully connected neural network. For example, the BERT model obtains a CLS vector of 768 dimensions and needs to map it to 10 classification labels. It needs a fully connected network. The shape of the parameter w is 768*10, and the final 10-dimensional vector logits is obtained. This logits is normalized and scaled by softmax to obtain the probability of each intent. Generally, the label corresponding to the value with the largest probability is taken as the prediction result of the intent of the audio sample. The cross entropy loss function is the loss calculation function, which calculates the difference between the probability of the prediction result of the intent of the audio sample and the true intent 0,1. If the probability of the prediction result of the intent of the audio sample is higher and the intent classification is correct, that is, 1, the loss is smaller. On the contrary, if the prediction is wrong, the loss is greater. The cross entropy loss compares the probability distribution value after the softmax normalized probability scaling and the loss calculation between the labels 0 and 1. The specific calculation formula is:

[0121]

[0122] When the loss value after multiple trainings is less than the preset loss value, or reaches the preset number of trainings, the training ends and the network parameters of the second pre-trained language model are obtained.

[0123] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the above-mentioned intention recognition method.

[0124] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0125] On the other hand, the present disclosure also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the intention recognition method provided by the above-mentioned methods.

[0126] On the other hand, the present disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the above-mentioned intention recognition methods.

[0127] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for intention recognition, characterized in that: include: Acquire a phoneme sequence corresponding to the audio to be detected, and determine a text sequence corresponding to the audio to be detected based on the phoneme sequence; Based on the pre-trained language model, feature extraction and fusion are performed on the text sequence and the phoneme sequence to obtain a sentence feature vector corresponding to the audio to be detected, wherein the feature extraction and fusion are performed on the text sequence and the phoneme sequence to obtain a sentence feature vector corresponding to the audio to be detected includes: Inputting the text sequence and the phoneme sequence into the pre-trained language model to obtain a robust word vector representation corresponding to the audio to be detected; Based on the robust word vector representation, a sentence feature vector output by the pre-trained language model and corresponding to the audio to be detected is obtained, Wherein, the pre-trained language model has generalization capability; The robust word vector representation is to assign phoneme information to the word vector; Based on the sentence feature vector, a prediction result for characterizing the intention of the audio to be detected is obtained.

2. The method for intention recognition according to claim 1, characterized in that: The step of obtaining a prediction result for characterizing the intention of the audio to be detected based on the sentence feature vector specifically includes: The sentence feature vector is input into a fully connected neural network for classification prediction to obtain the prediction result.

3. The method for identifying intention according to claim 1, characterized in that: The pre-trained language model is trained and determined in the following manner, including: Inputting a real text sequence and a real phoneme sequence into an initial pre-trained language model, adjusting and determining a first pre-trained language model network parameter; wherein the real text sequence is a text sequence obtained from the Internet; the real phoneme sequence is obtained according to the real text sequence; and the first pre-trained language model network parameter is used to characterize the pronunciation relationship between the real text sequence and the real phoneme sequence; Inputting a sample text sequence and a corresponding sample phoneme sequence into a pre-trained language model whose network parameters are the network parameters of the first pre-trained language model to obtain a second pre-trained language model network parameter; wherein the sample text sequence and the corresponding sample phoneme sequence are obtained by performing speech recognition on a plurality of preset sample audios; Based on the second pre-trained language model network parameters, the pre-trained language model with generalization capability is constructed.

4. The method for intention recognition according to claim 3, characterized in that: The step of inputting the real text sequence and the real phoneme sequence into the initial pre-trained language model and adjusting and determining the first pre-trained language model network parameters further includes: Inputting the real text sequence and the real phoneme sequence into the initial pre-trained language model, masking the real text sequence and the real phoneme sequence to obtain a first masked text sequence and a first masked phoneme sequence, predicting the masked first masked text sequence and the first masked phoneme sequence through context, and adjusting and determining the first pre-trained language model network parameters; The first masked text sequence is 20% of the text sequences randomly selected from the real text sequence; the first masked phoneme sequence is 20% of the phoneme sequences randomly selected from the real phoneme sequence.

5. The intention recognition method according to claim 3 or 4, characterized in that: The step of inputting the sample text sequence and the corresponding sample phoneme sequence into the network parameter pre-trained language model with the network parameter of the first pre-trained language model, and constructing the network parameter of the second pre-trained language model further comprises: Inputting the sample text sequence and the sample phoneme sequence into a pre-trained language model whose network parameters are the network parameters of the first pre-trained language model, masking the sample text sequence and the sample phoneme sequence to obtain a second masked text sequence and a second masked phoneme sequence, and predicting the masked second masked text sequence and the second masked phoneme sequence through context to obtain the second pre-trained language model network parameters; The second masked text sequence is 20% of the text sequences randomly selected from the sample text sequence; and the second masked phoneme sequence is 20% of the phoneme sequences randomly selected from the sample phoneme sequence.

6. The method for identifying intention according to claim 3, characterized in that: The real phoneme sequence is obtained according to the real text sequence and specifically comprises: Performing word segmentation on the real text sequence to obtain real words; According to the phoneme comparison table, obtaining the phonemes of the real word; Based on the phonemes of all the real words, the real phoneme sequence is obtained.

7. The method for intention recognition according to claim 5, characterized in that: The network parameters of the second pre-trained language model are adjusted in the following manner, specifically including: Determining the true text of the sample audio transcription by human listening; Based on a cross entropy loss function, calculating the difference between a prediction result representing the intent of the sample audio and the real text; Based on the difference, the first pre-trained language model network parameters are adjusted to obtain the second pre-trained language model network parameters.

8. An intention recognition device, characterized in that: The device is used to implement the intention recognition method described in any one of claims 1 to 7, and the device includes: A speech recognition module, used to obtain a phoneme sequence corresponding to the audio to be detected, and determine a text sequence corresponding to the audio to be detected based on the phoneme sequence; A pre-trained language module, based on a pre-trained language model, is used to extract and fuse features of the text sequence and the phoneme sequence to obtain a sentence feature vector corresponding to the audio to be detected; A fully connected neural network classification module is used to obtain a prediction result for characterizing the intention of the audio to be detected based on the sentence feature vector.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the intention recognition method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the intention recognition method as described in any one of claims 1 to 7 are implemented.

Images