A voice recognition-based dialog interaction method and device

Abstract

The application belongs to the technical field of intelligent interaction, and discloses a dialogue interaction method and device based on voice recognition, which comprises the following steps: acquiring audio data, and pre-processing the audio data to obtain a short-time signal; extracting audio feature information in the short-time signal; inputting the audio feature information into a full-covariance Gaussian mixture model to obtain a real person possibility score; if the real person possibility score is greater than a preset threshold, converting the audio data into text data, and performing intent recognition, entity recognition and emotion recognition on the text data to obtain a recognition result; matching a reply text in a corresponding industry database according to the recognition result; converting the reply text into voice data, and playing the voice data. The application can capture the diversity and complexity of the short-time signal, improve the robustness and accuracy of audio recognition, avoid non-human dialogue with the dialogue engine, and save computing resources.

Description

Technical Field

[0001] This application relates to the field of intelligent interaction technology, and in particular to a dialogue interaction method and device based on speech recognition. Background Technology

[0002] With the development of artificial intelligence technology, the application of intelligent dialogue engines is becoming more and more widespread. Existing dialogue engines usually obtain the user's voice information and then transcribe the speech into text based on end-to-end speech recognition models, speech recognition engines such as SpeechBrain, etc. Then, they use algorithms such as regular expressions, Mel-frequency energy features (MFB), BiLSTM-CRF (bidirectional long short-term memory network and conditional random field), and multi-task learning to identify the user's intent in the input text. Based on the recognition results, they find the corresponding response strategy in the knowledge base, convert the response strategy into audio, and play it to the user.

[0003] However, existing dialogue engines focus on the accuracy of speech conversion and speech-to-text recognition to improve the response effect. But in fact, the user's voice received by the dialogue engine may not be spoken by a real person, but may be spoken by the user's intelligent AI system. This will cause the dialogue engine to have a dialogue with another interactive system, which will not only affect the dialogue engine's construction of the user's personality label and reduce the subsequent dialogue effect, but also waste the dialogue engine's computing resources. Summary of the Invention

[0004] This application provides a speech recognition-based dialogue interaction method and apparatus that can capture the diversity and complexity of short-term signals, improve the robustness and accuracy of audio recognition, avoid dialogue engines interacting with non-humans, and save computing resources.

[0005] In a first aspect, embodiments of this application provide a dialogue interaction method based on speech recognition, comprising:

[0006] Acquire audio data and preprocess it to obtain a short-time signal;

[0007] Extract audio feature information from short-time signals; audio feature information includes signal energy curve, formant curve, short-time zero-crossing rate, frequency domain feature parameters, cepstral feature parameters, and harmonic feature parameters;

[0008] The audio feature information is input into a Gaussian mixture model with full covariance to obtain a real person probability score.

[0009] If the probability score of a real person is greater than a preset threshold, the audio data will be converted into text data.

[0010] Intent recognition, entity recognition, and sentiment recognition are performed on text data to obtain recognition results;

[0011] Based on the recognition results, the corresponding response text is matched with the relevant industry database;

[0012] The dialogue text is converted into audio data, and the audio data is played back.

[0013] Furthermore, the method also includes:

[0014] Extract the target statement from the text data and input it into the autoencoder model to obtain the encoded statement;

[0015] Calculate the reconstruction error of the encoded statement; determine whether the reconstruction error is less than the preset coherence threshold;

[0016] If not, extract each word from the target sentence, and form word vectors from adjacent words; calculate the pinyin similarity between the two words in each word vector;

[0017] Use word vectors whose pinyin similarity is lower than a preset similarity threshold as question vectors;

[0018] Repeat the above steps for each sentence in the text data and record the number of question vectors; divide the number of vectors by the number of word vectors to obtain the verbal error quantitative score.

[0019] The likelihood score of a real person is updated based on the quantitative scoring of verbal slips.

[0020] Furthermore, the method also includes:

[0021] Detect pauses in audio data, and determine the sentence breaks in text data based on the pauses; determine whether the sentence breaks are grammatically correct, and record the number of sentence breaks that are not grammatically correct; divide the number of sentence breaks by the number of pauses to obtain a quantitative score for grammatical issues.

[0022] The likelihood score of a real person is updated based on the quantitative scoring of grammatical issues.

[0023] Furthermore, the method also includes:

[0024] Extract words from text data and their corresponding meanings;

[0025] Obtain user tags and determine if there are any standard words with the same meaning in the user tags;

[0026] If so, determine whether the words in the text data match the standard words;

[0027] Record the number of first words that match the standard word and the number of second words that do not;

[0028] Divide the number of the first word by the number of the second word to obtain the word usage habit score;

[0029] The likelihood of a real person being identified is updated based on their vocabulary usage habits.

[0030] Furthermore, the method also includes:

[0031] Before preprocessing the audio data, determine whether the audio data is noise;

[0032] If the audio data is noise or the timing starts after the audio data has finished playing;

[0033] When the timed duration reaches a preset response threshold, a preset end-of-conversation message is displayed, and the conversation ends. Furthermore, the method also includes:

[0034] After receiving a conversation end command or after ending a conversation, retrieve the historical user input text;

[0035] Extract the words and their meanings from the historical user input text;

[0036] Group words with the same meaning into a word group and calculate the word frequency of each word;

[0037] The word with the highest frequency in each phrase and whose frequency is greater than the preset standard threshold is taken as the standard word; the standard word and its corresponding frequency and meaning are put into user tags.

[0038] Furthermore, the method also includes:

[0039] Before preprocessing the audio data, extract the fundamental frequency and speech rate of the audio data;

[0040] Matching is performed in the user's acoustic feature database based on fundamental frequency and speech rate;

[0041] If a match is found, the audio data is converted into text data; otherwise, the audio data is preprocessed.

[0042] Secondly, embodiments of this application provide a dialogue interaction device based on speech recognition, comprising:

[0043] The preprocessing module is used to acquire audio data and preprocess the audio data to obtain a short-time signal;

[0044] The feature extraction module is used to extract audio feature information from short-time signals; the audio feature information includes signal energy curve, formant curve, short-time zero-crossing rate, frequency domain feature parameters, cepstral feature parameters, and harmonic feature parameters;

[0045] The input module is used to input audio feature information into a fully covariant Gaussian mixture model to obtain a real person probability score.

[0046] The conversion module is used to convert audio data into text data when the real person probability score is greater than a preset threshold.

[0047] The recognition module is used to perform intent recognition, entity recognition, and sentiment recognition on text data to obtain recognition results;

[0048] The matching module is used to match the response text in the corresponding industry database based on the recognition results;

[0049] The playback module is used to convert the dialogue text into audio data and play the audio data.

[0050] Furthermore, the device also includes:

[0051] The statement encoding module is used to obtain the target statement in the text data and input it into the autoencoder model to obtain the encoded statement.

[0052] The reconstruction error module is used to calculate the reconstruction error of the encoded statement and determine whether the reconstruction error is less than the preset coherence threshold.

[0053] The word vector module is used to extract each word in the target sentence and combine adjacent words into a word vector.

[0054] The Pinyin Similarity module is used to calculate the Pinyin similarity between two words in each word vector.

[0055] The question vector module is used to select word vectors with a pinyin similarity lower than a preset similarity threshold as question vectors.

[0056] The question count module is used to repeat the above steps for each statement in the text data and record the number of question vectors.

[0057] The slip-of-speech quantification module is used to divide the number of vectors by the number of word vectors to obtain a slip-of-speech quantification score.

[0058] The slip-of-the-mouth update module is used to update the real person probability score based on the quantitative score of slip-of-the-mouth mistakes.

[0059] Furthermore, the device also includes:

[0060] The pause detection module is used to detect pause positions in audio data and obtain the sentence break positions in text data based on the pause positions.

[0061] The grammar module is used to determine whether the punctuation marks are grammatically correct and to record the number of punctuation marks that are not grammatically correct.

[0062] The grammar quantification module is used to divide the number of sentence breaks by the number of pauses to obtain a quantitative score for grammar issues.

[0063] The grammar update module is used to update the probability score of real people based on the quantitative score of grammar issues.

[0064] Furthermore, the device also includes:

[0065] The first extraction module is used to extract words from text data and the meanings of each word.

[0066] The tag acquisition module is used to acquire user tags and determine whether there are standard words with the same meaning among the user tags.

[0067] The standard judgment module is used to determine whether the words in the text data match the standard words.

[0068] The quantity recording module is used to record the number of first words that match the standard word and the number of second words that do not.

[0069] The word usage quantification module is used to divide the number of the first word by the number of the second word to obtain a word usage habit score.

[0070] The word usage update module is used to update the probability score of real people based on word usage habits.

[0071] In some embodiments, the device further includes:

[0072] The noise detection module is used to determine whether the audio data is noise before preprocessing it.

[0073] The timing module is used to start timing when the audio data is noise or after the voice data has finished playing.

[0074] The conversation end module is used to respond with a preset end voice message and end the conversation when the timed duration reaches a preset response threshold.

[0075] Furthermore, the device also includes:

[0076] The historical text acquisition module is used to acquire historical user input text after receiving a dialogue end command or after the dialogue ends.

[0077] The second extraction module is used to extract the words and their corresponding meanings from the historical user input text.

[0078] The word frequency calculation module is used to group words with the same meaning into a word group and calculate the word frequency of each word.

[0079] The standard determination module is used to identify the word with the highest frequency in each phrase that is greater than a preset standard threshold as the standard word.

[0080] The tag generation module is used to add standard words and their corresponding word frequencies and meanings into user tags.

[0081] Furthermore, the device also includes:

[0082] The acoustic feature extraction module is used to extract the fundamental frequency and speech rate of the audio data before preprocessing.

[0083] The acoustic feature matching module is used to perform matching in the user's acoustic feature database based on fundamental frequency and speech rate; if a match is found, the audio data is converted into text data; otherwise, the audio data is preprocessed.

[0084] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the steps of a speech recognition-based dialogue interaction method as described in any of the above embodiments.

[0085] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of a speech recognition-based dialogue interaction method as described in any of the above embodiments.

[0086] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following:

[0087] This application provides a speech recognition-based dialogue interaction method. Upon receiving user audio data, it first converts it into a short-time signal, extracting audio feature information such as signal energy curves, formant curves, short-time zero-crossing rates, frequency domain feature parameters, cepstral feature parameters, and harmonic feature parameters. A Gaussian mixture model with full covariance is then used to score the likelihood of the audio being spoken by a real person. If the likelihood score is greater than a preset threshold, the audio data is considered to have been spoken by a real person, and text conversion, text recognition, and dialogue operations are performed based on the audio data. The full covariance Gaussian mixture model in this method can model various audio feature information of the audio data. Combining multiple audio feature information can capture the diversity and complexity of short-time signals, improving the robustness and accuracy of audio recognition, avoiding dialogue between the dialogue engine and non-humans, and saving computational resources. Attached Figure Description

[0088] Figure 1 A flowchart illustrating a speech recognition-based dialogue interaction method provided as an exemplary embodiment of this application.

[0089] Figure 2 A schematic diagram of audio data provided for an exemplary embodiment of this application.

[0090] Figure 3 The probability density distribution diagram of the standard GMM model fitted to an exemplary embodiment of this application.

[0091] Figure 4The probability density distribution of the fully covariance GMM model fitted to an exemplary embodiment of this application is shown.

[0092] Figure 5 A flowchart illustrating the steps for calculating a verbal slip quantification score, provided as an exemplary embodiment of this application.

[0093] Figure 6 A flowchart illustrating the steps for calculating a word usage habit score, provided as an exemplary embodiment of this application.

[0094] Figure 7 This is a structural diagram of a speech recognition-based dialogue interaction device provided as an exemplary embodiment of this application. Detailed Implementation

[0095] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0096] Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0097] Please see Figure 1 This application provides a dialogue interaction method based on speech recognition, including:

[0098] Step S11: Acquire audio data and preprocess the audio data to obtain a short-time signal.

[0099] The preprocessing operations for audio data include pre-emphasis, framing, and windowing.

[0100] Specifically, pre-emphasis is used to boost the energy in the high-frequency range, typically employing a first-order high-pass filter, as shown in the formula:

[0101] Y(n) = X(n) - coef*X(n-1)

[0102] Where X(n) is the audio data, and coef is usually taken as 0.97.

[0103] Frame segmentation is the process of cutting audio data into small segments or frames of equal length. The frame length is usually selected from 20ms to 30ms (corresponding to 320-480 sampling points at a sampling rate of 16kHz). There is a certain overlap between frames, usually half the frame length.

[0104] Windowing involves applying a window function, such as a Hamming window, to each frame of signal to reduce the impact of discontinuities at frame boundaries.

[0105] After the above preprocessing operations, multiple frames of short-time signals can be obtained. The following steps are performed on each short-time signal.

[0106] Step S12: Extract audio feature information from the short-time signal; the audio feature information includes signal energy curve, formant curve, short-time zero-crossing rate, frequency domain feature parameters, cepstral feature parameters, and harmonic feature parameters.

[0107] Specifically, the signal energy curve is obtained by calculating the signal energy of each short-time signal frame, which is usually the sum of squares of the signals within the frame, and then arranging and connecting the sums of squares according to the order of the corresponding short-time signals.

[0108] The short-time zero-crossing rate is the number of times a signal crosses zero in each frame, reflecting the non-periodic characteristics of a short-time signal.

[0109] Frequency domain feature parameters are obtained using Short Time Fourier Transform (STFT): a Fast Fourier Transform (FFT) is performed on each frame of short-time signal to obtain the spectrum, and then the frequency domain feature parameters of the spectrum are obtained, such as the spectral centroid. The spectral centroid represents the "center of gravity" of the spectrum and can be calculated using librosa.feature.spectral_centroid.

[0110] The formant curve is obtained by analyzing the spectrum after the above fast Fourier transform. The formant frequency of each short-time signal can be determined, and the formant frequency corresponds to the resonance mode of the audio channel. Similarly, the formant curve is obtained by connecting the short-time signals in sequence.

[0111] The cepstral characteristic parameters are obtained by taking the logarithm of the spectrum and performing an inverse Fourier transform to obtain the cepstral spectrum, and then processing the cepstral spectrum through a discrete cosine transform (DCT) to extract the Mel frequency cepstral coefficients (MFCC).

[0112] Harmonic characteristic parameters are the harmonic components of a short-time signal.

[0113] Step S13: Input the audio feature information into the Gaussian mixture model with full covariance to obtain the probability score of the real person.

[0114] Among them, Gaussian Mixture Models (GMMs) can model audio feature information. By viewing audio data as composed of multiple Gaussian distributions, the distribution of these feature information is described by the mixture of multiple Gaussian distributions.

[0115] The main types of GMM models include the standard GMM model, the diagonal covariance GMM model, the spherical covariance GMM model, and the full covariance GMM model. This application uses the full covariance GMM model.

[0116] During the training process, it is necessary to estimate the parameters of the full covariance GMM, including the mean, variance, and mixing weights of each Gaussian distribution. Specifically, the GMM parameters are estimated by maximizing, which can make the acoustic model more accurately describe the statistical characteristics of speech data. Then, the probability distribution of each audio feature information is calculated using the full covariance GMM model, and the likelihood score of a real person is obtained based on the probability distribution using decision logic such as the Viterbi algorithm.

[0117] The following is the formula expression of the full covariance GMM mathematical model:

[0118]

[0119] where x is a D-dimensional feature vector composed of each audio feature information of the short-time signal of the i-th frame; M represents the number of frames, and w i represents the weight corresponding to M, which is adjusted and determined during the training process; g(x丨u i ∑ i ) represents the Gaussian density of the frame, and the Gaussian density of each frame is a Gaussian function of D audio feature information.

[0120]

[0121] where u i is the mean vector, that is, the mean of the Gaussian distribution; ∑ i is the covariance matrix, which can be full rank; the mixing weights satisfy [[ID=[]26]] All of these are estimated and adjusted during the training process.

[0122] Here, the parameter form of λ is given. For different frames M, there is:

[0123] λ = {w i , u i , ∑ i}

[0124] Please refer to Figure 2 、 Figure 3 and Figure 4 , Figure 2 is a segment of audio data. Inevitably, this segment of audio data will be affected by some noise, but the overall distribution is smooth. Figure 3 is the probability density distribution fitted by the standard GMM model. It can be clearly seen that the standard GMM model will cause serious information loss. And Figure 4 is the probability density distribution fitted by the full covariance GMM model. Compared with the standard GMM model, each Gaussian component of the full covariance GMM model has a different covariance matrix, which enables the model to more flexibly capture the characteristics of the speaker, and the likelihood score of a real person will be more accurate.

[0125] Furthermore, in practical applications, GMM can be combined with Hidden Markov Models (HMM) to form a GMM-HMM model. This combination leverages the clustering analysis and statistical modeling capabilities of GMM, as well as the time series modeling capabilities of HMM, thereby improving the model's recognition accuracy and robustness.

[0126] Step S14: If the probability score of a real person is greater than a preset threshold, the audio data is converted into text data.

[0127] Step S15: Perform intent recognition, entity recognition, and emotion recognition on the text data to obtain the recognition results.

[0128] Step S16: Match the response text in the corresponding industry database based on the recognition results.

[0129] Specifically, the first step is to use natural language processing (NLP) techniques to determine the intent or purpose expressed in the text data. This can be achieved through machine learning or deep learning models. For example, pre-trained language models like BERT can be fine-tuned to adapt to specific business scenarios. Then, entity recognition (NER) is performed to identify specific types of entities from the text data, such as names of people, places, organizations, and dates. This can be achieved using named entity recognition techniques, such as models based on BIO tagging. Entity recognition can be performed using BERT or other deep learning models.

[0130] Next, sentiment analysis is performed to identify the emotional tone of the text, such as positive, negative, or neutral. Sentiment analysis models, such as those based on LSTM and attention mechanisms, can be used to evaluate the sentiment of the text. Feature extraction is a crucial step in the process of intent recognition, entity recognition, and sentiment recognition. Features such as words, phrases, and parts of speech can be extracted from text data, and word embedding techniques such as Word2Vec or GloVe can be used to obtain richer semantic features.

[0131] The industry database consists of various pre-set databases within the dialogue engine, which can be determined based on the entity recognition results.

[0132] Based on the identification results of intent, entity, and emotion, database query or information retrieval techniques are used to match within a defined industry database to find the most suitable response text.

[0133] In practice, semantic analysis of text data can also be performed through word vector analysis, word sense disambiguation, context similarity recognition, intent clarification, etc., but semantic analysis and how to respond are not the focus of this application.

[0134] Step S17: Convert the dialogue text into speech data and play the speech data.

[0135] When converting the dialogue text into speech data, it is necessary to pay attention to the pronunciation of polyphonic characters, determine the pronunciation of polyphonic characters in the dialogue text in combination with the surrounding context, and determine the emotional tone expressed based on the dialogue text.

[0136] The above embodiment provides a speech recognition-based dialogue interaction method. After receiving user audio data, it first converts it into a short-time signal and extracts audio feature information such as signal energy curve, formant curve, short-time zero-crossing rate, frequency domain feature parameters, cepstral feature parameters, and harmonic feature parameters. A fully covariant Gaussian mixture model is then used to score the likelihood of the audio being spoken by a real person. If the likelihood score is greater than a preset threshold, the audio data is considered to have been spoken by a real person, and text conversion, text recognition, and dialogue operations are performed based on the audio data. The fully covariant Gaussian mixture model in this method can model various audio feature information of the audio data. Combining multiple audio feature information can capture the diversity and complexity of short-time signals, improve the robustness and accuracy of audio recognition, avoid dialogue between the dialogue engine and non-humans, and save computational resources.

[0137] In some embodiments, the method further includes:

[0138] Step S01: Before preprocessing the audio data, determine whether the audio data is noise.

[0139] Specifically, audio analysis tools such as AudioAnalyze can be used to determine noise levels; or objective evaluation metrics such as PESQ (Perceptual Evaluation of Speech Quality) can be used to assess audio quality and determine the presence of noise.

[0140] Step S02: If the audio data is noise or after the voice data has finished playing, start timing.

[0141] Step S03: When the timed duration reaches the preset response threshold, reply with the preset end voice and end the conversation.

[0142] The preset response threshold can be set to a noise threshold of 1 minute and a no-response threshold of 3 minutes.

[0143] In practical applications, the number of dialogue interactions that a server can handle is limited. If a user continues to input noise or does not respond for a long time, the dialogue will automatically end; this can avoid unnecessary occupation of computing resources.

[0144] In addition to saving computing resources, setting a preset response threshold to automatically end a conversation can also ensure security. In some cases, a user's silence or the presence of noise may indicate that the user's attention has shifted elsewhere. Ending the conversation can serve as a security measure to prevent unauthorized access or operations.

[0145] In some embodiments, the method further includes:

[0146] Step S21: Before preprocessing the audio data, extract the fundamental frequency and speech rate of the audio data.

[0147] Specifically, the fundamental frequency (FFM) refers to the frequency at which the vocal cords vibrate and produce sound, determining the pitch of the sound. Because everyone's vocal cord structure is different, the fundamental frequency varies from person to person. For example, the FFM range differs between men, women, and people of different ages; even within the same sex, each person's FFM is different.

[0148] Speech rate refers to the speed at which a person speaks, usually measured by the number of words (or syllables) pronounced per unit of time. Speech rate is influenced by a variety of factors, including culture, environment, thinking, and expressive ability. Different people have different speech rates, and even the same person may have different speech rates in different situations.

[0149] The difference between fundamental frequency and speech rate makes these two parameters usable as acoustic features of a biological system for user identification.

[0150] Step S22: Matching is performed in the user's acoustic feature database based on fundamental frequency and speech rate.

[0151] Step S23: If a match is found, the audio data is converted into text data; otherwise, the audio data is preprocessed.

[0152] Among them, the user acoustic feature library is the acoustic features of all users in the historical dialogue collected by the dialogue interaction engine, namely the fundamental frequency and speech rate of all historical users. If there is a match, it means that the user who inputs the audio data at this time is a historical user, who has already undergone real person recognition. This can save the subsequent recognition based on Gaussian mixture model and directly perform text conversion and dialogue.

[0153] The above embodiments construct a user acoustic feature database by recording the acoustic feature information of historical users, which can quickly identify and respond to old users when they use the dialogue engine, eliminating the need for audio recognition again and further saving computing resources.

[0154] Please see Figure 5 In some embodiments, the method further includes:

[0155] Step S31: Obtain the target statement in the text data and input it into the auto-encoding model to obtain the encoded statement.

[0156] Step S32: Calculate the reconstruction error of the encoded statement; determine whether the reconstruction error is less than the preset coherence threshold.

[0157] Among them, the reconstruction error is a key indicator to measure the similarity when the encoded statement is reconstructed back to the original target statement, and can be obtained by calculating the mean square error or mean absolute error between the target statement and the encoded statement.

[0158] Step S33: If not, extract each word in the target statement and form word vectors by combining two adjacent words.

[0159] Specifically, if the reconstruction error is lower than the preset coherence threshold, it means the sentence is coherent, and the user does not have pronunciation errors or repeat speaking situations, and step S15 can be further executed; if the reconstruction error is higher than the preset coherence threshold, it means the sentence is incoherent, and at this time, the specific situation of the sentence incoherence needs to be further detected. This application judges whether the user has self-correction when inputting audio based on whether the sentence is coherent, such as slips of the tongue, repeated reading, etc.

[0160] Step S34: Calculate the pinyin similarity between two words in each word vector.

[0161] Specifically, the pinyin similarity can be obtained by calculating the edit distance between the pinyins of two words.

[0162] "Edit distance" is a method to measure the difference between two sequences, by calculating the minimum number of single-character edits (insertion, deletion or replacement) required to convert one sequence to another. For the pinyins of words in this application, they can be regarded as strings, and the edit distance is used to calculate their similarity.

[0163] Step S35: Take the word vectors with pinyin similarity lower than the preset similarity threshold as problem vectors.

[0164] Specifically, the judgment of the pinyin similarity here is the situation where this application determines whether the user has a slip of the tongue and repeats the correct pronunciation again; for example, assume the two words in the word vector are "捉弄, 作弄", when calculating the pinyin similarity of these two words, it is detected that "zhuo" becomes "zuo", and only the initial consonant "zh" needs to be replaced with "z", and there is only 1 edit operation, that is, the replacement operation, the edit distance, and the pinyin similarity is 1. If the preset similarity threshold is 2, then at this time, it will be judged that this word vector is a problem vector, that is, "捉弄, 作弄" is the user's slip of the tongue situation.

[0165] It can be seen that the value of pinyin similarity is inversely proportional to the degree of similarity. The smaller the pinyin similarity, the smaller the edit distance, and the greater the pinyin similarity between the two words. Therefore, this application regards two words in word vectors below the preset similarity threshold as words with similar pronunciation, which is likely the case where the user made a mistake and then repeated the correct pronunciation.

[0166] Step S36: Repeat the above steps for each statement in the text data and record the number of question vectors.

[0167] Repeat steps S31-S35 above for each statement as the target statement, and summarize the number of problem vectors obtained through filtering.

[0168] Step S37: Divide the number of vectors by the number of word vectors to obtain the verbal error quantification score.

[0169] In practical applications, apart from polyphonic characters and words with neutral tone, artificial intelligence rarely mispronounces words. In particular, current artificial intelligence is becoming more and more familiar with how to determine the pronunciation of polyphonic characters based on context, resulting in fewer and fewer mispronunciations. Moreover, the most crucial point is that even if artificial intelligence mispronounces a word, it will not realize it, that is, it will not self-correct. Therefore, the detection based on self-correction in this application can further improve the accuracy of speech recognition.

[0170] Step S38: Update the real person probability score based on the verbal slip quantification score.

[0171] Specifically, here we add a verbal slip quantification score to the real person probability score, that is, the number of verbal slips will increase the value of the real person probability score; then we determine whether the real person probability score is greater than a preset threshold, and if so, we execute step S15.

[0172] The above embodiment first assesses the coherence of the sentence. When a sentence is found to be disjointed, it uses pinyin similarity to determine whether the disjointedness is due to user self-correction. If so, the likelihood score is updated based on the number of slips of the tongue, and the assessment is then performed again based on the updated likelihood score, further improving the accuracy of human-machine recognition.

[0173] In some embodiments, the method further includes:

[0174] Step S41: Detect pauses in the audio data and obtain the sentence breaks in the text data based on the pauses.

[0175] Step S42: Determine whether the punctuation position conforms to grammar, and record the number of punctuation positions that do not conform to grammar.

[0176] Specifically, syntax analysis techniques in Natural Language Processing (NLP) can be used for syntax judgment, such as Grammarly and LanguageTool. These analysis services can identify which pauses do not conform to grammatical rules.

[0177] Step S43: Divide the number of sentence breaks by the number of pauses to obtain a quantitative score for grammatical problems.

[0178] Step S44: Update the real person probability score based on the quantitative score of grammatical problems.

[0179] Specifically, here the real person probability score is subtracted from the grammar problem quantification score to obtain an updated real person probability score. Then it is determined whether the real person probability score is greater than a preset threshold. If so, step S15 is executed.

[0180] In practical applications, the audio input by users varies in length, as does the converted text data. Therefore, it is necessary to quantify the number of grammatically incorrect punctuation by using the ratio of the number of punctuation breaks to the total number of pauses.

[0181] The above embodiments achieve the detection of the quality of language expression in audio data by judging the number of abnormal pauses in the audio data based on grammar. This is because when artificial intelligence outputs audio, it pays less attention to the content and the required emotion of the audio, but this is often not ignored when a real person speaks. Therefore, the number of grammatical problems can be regarded as inversely proportional to the probability of a real person speaking.

[0182] In some embodiments, the method further includes:

[0183] Step S51: After receiving a dialogue end command or ending the dialogue, obtain the historical user input text.

[0184] The dialogue end command is triggered by the user clicking the end button on the dialogue interaction interface, and the dialogue end is triggered in the above embodiment after detecting continuous noise or after the user does not reply for a timeout.

[0185] The historical user input text is the text data converted from all user audio inputs in the current dialog box that has been responded to. Since the historical user input text is used to create user tags, it is necessary to ensure that the corresponding text is input by a real person. According to the judgment in the above embodiment, only the text data that is judged to be the corresponding text data of a real person will be recognized and responded to. Therefore, this embodiment only uses the text data that has been responded to when creating user tags at the end of the dialogue.

[0186] Step S52: Extract the words and their corresponding meanings from the historical user input text.

[0187] Step S53: Group words with the same meaning into a word group and calculate the word frequency of each word.

[0188] Step S54: Select the word with the highest frequency in each phrase and whose frequency is greater than the preset standard threshold as the standard word.

[0189] For example, "accurate" and "precise" have the same meaning and will be grouped into one word group; "put in" and "put in" have the same meaning and will also be grouped into one word group. In the first word group, the frequency of "accurate" is greater than that of "precise" and also greater than the preset standard threshold, indicating that users habitually use the word "accurate" and regard it as a standard word. In the second word group, although the frequency of "put in" is greater than that of "put in," the frequency of "put in" is less than the preset standard threshold, indicating that neither "put in" nor "put in" appears very often and is insufficient to be judged as a user's habitual word.

[0190] Furthermore, to avoid an excessive number of words meeting the standard word requirements due to an inappropriate preset standard threshold setting, which would put pressure on the storage of user tags, the judgment of the preset standard threshold can be omitted. First, select the word with the highest frequency in each phrase as the representative word of the corresponding meaning, and then sort the representative words and select the top 3 words as the standard words. The top 3 words in the sorting are not fixed values ​​and can be adjusted freely according to the actual situation.

[0191] Step S55: Add the standard words and their corresponding word frequencies and meanings to the user tags.

[0192] Specifically, when adding standard words to user tags, since user tags are updated after each conversation, they may contain previously identified standard words. To ensure that the same meaning corresponds to only one standard word, the meaning of the current standard word to be added needs to be compared with the historical standard words already existing in the user tags. If the same meaning is found, the word with the higher frequency is selected as the standard word.

[0193] The above embodiments have enabled the construction of user tags, laying the foundation for the subsequent determination of users' habitual words.

[0194] Please see Figure 6 In some embodiments, the method further includes:

[0195] Step S61: Extract the words from the text data and the meanings of each word.

[0196] Step S62: Obtain user tags and determine whether there are standard words with the same meaning in the user tags.

[0197] Step S63: If yes, determine whether the words in the text data are consistent with the standard words.

[0198] Step S64: Record the number of first words that match the standard word and the number of second words that do not match.

[0199] Step S65: Divide the number of the first word by the number of the second word to obtain the word usage habit score.

[0200] For example, in the user tags, the recorded standard words are: accurate, aligned, and recognized; in the text data, the words with the same meaning as these standard words are: accurate, aligned, and detected; then only "aligned" has the same meaning as the standard words, while the other two have the same meaning but different words, so the word usage habit score is 1:2, or 0.5 points.

[0201] The above example uses only three words; in actual application, there will be many more words to compare.

[0202] Step S66: Update the probability score of real people based on word usage habits.

[0203] In practical applications, the larger the number of words in the first word, the higher the consistency between the word usage habits of the current audio data and the user's habitual word usage recorded in the user tags, and the greater the likelihood of it being a real person.

[0204] Therefore, here the real person probability score is added to the word usage habit score to obtain an updated real person probability score, and then it is determined whether the real person probability score is greater than the preset threshold. If so, step S15 is executed.

[0205] The above embodiments, by recording users' word usage habits, enable the system to determine whether the word usage of the current audio matches the previously recorded word usage habits after each audio data reception, thereby further improving the accuracy of judging the likelihood of a real person.

[0206] Please see Figure 7 Another embodiment of this application provides a dialogue interaction device based on speech recognition, including:

[0207] The preprocessing module 101 is used to acquire audio data and preprocess the audio data to obtain a short-time signal.

[0208] The feature extraction module 102 is used to extract audio feature information from short-time signals; the audio feature information includes signal energy curve, formant curve, short-time zero-crossing rate, frequency domain feature parameters, cepstral feature parameters, and harmonic feature parameters.

[0209] Input module 103 is used to input audio feature information into a Gaussian mixture model with full covariance to obtain a real person probability score.

[0210] The conversion module 104 is used to convert audio data into text data when the real person probability score is greater than a preset threshold.

[0211] The recognition module 105 is used to perform intent recognition, entity recognition, and emotion recognition on text data to obtain recognition results.

[0212] The matching module 106 is used to match the response text in the corresponding industry database based on the recognition results.

[0213] The playback module 107 is used to convert the dialogue text into voice data and play the voice data.

[0214] Furthermore, the device also includes:

[0215] The statement encoding module is used to obtain the target statement in the text data and input it into the autoencoder model to obtain the encoded statement.

[0216] The reconstruction error module is used to calculate the reconstruction error of the encoded statement and determine whether the reconstruction error is less than the preset coherence threshold.

[0217] The word vector module is used to extract each word in the target sentence and combine adjacent words into a word vector.

[0218] The Pinyin Similarity module is used to calculate the Pinyin similarity between two words in each word vector.

[0219] The question vector module is used to select word vectors with a pinyin similarity lower than a preset similarity threshold as question vectors.

[0220] The question count module is used to repeat the above steps for each statement in the text data and record the number of question vectors.

[0221] The slip-of-speech quantification module is used to divide the number of vectors by the number of word vectors to obtain a slip-of-speech quantification score.

[0222] The slip-of-the-mouth update module is used to update the real person probability score based on the quantitative score of slip-of-the-mouth mistakes.

[0223] Furthermore, the device also includes:

[0224] The pause detection module is used to detect pause positions in audio data and obtain the sentence break positions in text data based on the pause positions.

[0225] The grammar module is used to determine whether the punctuation marks are grammatically correct and to record the number of punctuation marks that are not grammatically correct.

[0226] The grammar quantification module is used to divide the number of sentence breaks by the number of pauses to obtain a quantitative score for grammar issues.

[0227] The grammar update module is used to update the probability score of real people based on the quantitative score of grammar issues.

[0228] Furthermore, the device also includes:

[0229] The first extraction module is used to extract words from text data and the meanings of each word.

[0230] The tag acquisition module is used to acquire user tags and determine whether there are standard words with the same meaning among the user tags.

[0231] The standard judgment module is used to determine whether the words in the text data match the standard words.

[0232] The quantity recording module is used to record the number of first words that match the standard word and the number of second words that do not.

[0233] The word usage quantification module is used to divide the number of the first word by the number of the second word to obtain a word usage habit score.

[0234] The word usage update module is used to update the probability score of real people based on word usage habits.

[0235] In some embodiments, the device further includes:

[0236] The noise detection module is used to determine whether the audio data is noise before preprocessing it.

[0237] The timing module is used to start timing when the audio data is noise or after the voice data has finished playing.

[0238] The conversation end module is used to respond with a preset end voice message and end the conversation when the timed duration reaches a preset response threshold.

[0239] Furthermore, the device also includes:

[0240] The historical text acquisition module is used to acquire historical user input text after receiving a dialogue end command or after the dialogue ends.

[0241] The second extraction module is used to extract the words and their corresponding meanings from the historical user input text.

[0242] The word frequency calculation module is used to group words with the same meaning into a word group and calculate the word frequency of each word.

[0243] The standard determination module is used to identify the word with the highest frequency in each phrase that is greater than a preset standard threshold as the standard word.

[0244] The tag generation module is used to add standard words and their corresponding word frequencies and meanings into user tags.

[0245] Furthermore, the device also includes:

[0246] The acoustic feature extraction module is used to extract the fundamental frequency and speech rate of the audio data before preprocessing.

[0247] The acoustic feature matching module is used to perform matching in the user's acoustic feature database based on fundamental frequency and speech rate; if a match is found, the audio data is converted into text data; otherwise, the audio data is preprocessed.

[0248] The specific limitations of the speech recognition-based dialogue interaction device provided in this embodiment can be found in the embodiment of the speech recognition-based dialogue interaction method described above, and will not be repeated here. Each module in the above-described speech recognition-based dialogue interaction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0249] This application provides a computer device that may include a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it causes the processor to perform steps of a speech recognition-based dialogue interaction method as described in any of the above embodiments.

[0250] The working process, working details, and technical effects of the computer device provided in this embodiment can be found in the embodiment of a dialogue interaction method based on speech recognition described above, and will not be repeated here.

[0251] This application provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of a speech recognition-based dialogue interaction method as described in any of the above embodiments. The computer-readable storage medium refers to a data storage carrier, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The working process, details, and technical effects of the computer-readable storage medium provided in this embodiment can be found in the embodiments of a speech recognition-based dialogue interaction method described above, and will not be repeated here.

[0252] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0253] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0254] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A dialogue interaction method based on speech recognition, characterized in that, include: Acquire audio data and preprocess the audio data to obtain a short-time signal; Extract audio feature information from the short-time signal; the audio feature information includes signal energy curve, formant curve, short-time zero-crossing rate, frequency domain feature parameters, cepstral feature parameters, and harmonic feature parameters; The audio feature information is input into a Gaussian mixture model with full covariance to obtain a real person probability score. If the probability score of the real person is greater than a preset threshold, the audio data is converted into text data; The target statement in the text data is obtained and input into the autoencoder model to obtain the encoded statement; Calculate the reconstruction error of the encoded statement; determine whether the reconstruction error is less than a preset coherence threshold; If not, extract each word from the target sentence and combine adjacent words into a word vector; Calculate the pinyin similarity between two words in each of the aforementioned word vectors; The word vectors whose pinyin similarity is lower than a preset similarity threshold are used as question vectors; Repeat the above steps for each statement in the text data and record the number of vectors in the question vector; Divide the number of vectors by the number of word vectors to obtain the verbal error quantitative score; The probability score of the real person is updated based on the verbal slip quantification score; The text data is subjected to intent recognition, entity recognition, and emotion recognition to obtain the recognition results; Based on the recognition results, match the corresponding response text in the relevant industry database; The dialogue text is converted into speech data, and the speech data is played.

2. The dialogue interaction method based on speech recognition according to claim 1, characterized in that, Also includes: Detect pauses in the audio data, and determine the sentence breaks in the text data based on the pauses. Determine whether the punctuation position conforms to grammar, and record the number of punctuation positions that do not conform to grammar; Divide the number of sentence breaks by the number of pauses to obtain a quantitative score for grammatical problems. The probability score of the real person is updated based on the quantitative score of the grammatical issues.

3. The dialogue interaction method based on speech recognition according to claim 1, characterized in that, Also includes: Extract the words from the text data and their corresponding meanings; Obtain user tags and determine whether there are standard words with the same meaning among the user tags; If so, determine whether the words in the text data are consistent with the standard words; Record the number of first words that match the standard words and the number of second words that do not match them; Divide the number of the first word by the number of the second word to obtain the word usage habit score; The probability score of the real person is updated based on the vocabulary usage habits scored.

4. The dialogue interaction method based on speech recognition according to claim 3, characterized in that, Also includes: Before preprocessing the audio data, determine whether the audio data is noise; If the audio data is noise or if the timing starts after the audio data has finished playing; When the timer reaches the preset response threshold, a preset end voice message will be sent, and the conversation will end.

5. The dialogue interaction method based on speech recognition according to claim 4, characterized in that, Also includes: After receiving a conversation end command or after ending a conversation, retrieve the historical user input text; Extract the words and their meanings from the historical user input text; Group words with the same meaning into a word group and calculate the word frequency of each word; The word with the highest frequency in each phrase and whose frequency is greater than a preset standard threshold is taken as the standard word; The standard words and their corresponding word frequencies and meanings are placed into the user tags.

6. The dialogue interaction method based on speech recognition according to claim 1, characterized in that, Also includes: Before preprocessing the audio data, the fundamental frequency and speech rate of the audio data are extracted; Matching is performed in the user acoustic feature database based on the base frequency and the speech rate; If a match is found, the audio data is converted into text data; otherwise, the audio data is preprocessed.

7. A dialogue interaction device based on speech recognition, characterized in that, include: The preprocessing module is used to acquire audio data and preprocess the audio data to obtain a short-time signal; The feature extraction module is used to extract audio feature information from the short-time signal; the audio feature information includes signal energy curve, formant curve, short-time zero-crossing rate, frequency domain feature parameters, cepstral feature parameters, and harmonic feature parameters; The input module is used to input the audio feature information into a Gaussian mixture model with full covariance to obtain a real person probability score; The conversion module is used to convert the audio data into text data when the real person probability score is greater than a preset threshold. The statement encoding module is used to obtain the target statement in the text data, input it into the autoencoder model, and obtain the encoded statement; The reconstruction error module is used to calculate the reconstruction error of the encoded statement; Determine whether the reconstruction error is less than a preset coherence threshold; The word vector module is used to extract each word in the target sentence and combine adjacent words into a word vector. The Pinyin Similarity module is used to calculate the Pinyin similarity between two words in each word vector; The question vector module is used to use word vectors with pinyin similarity lower than a preset similarity threshold as question vectors; The question count module is used to repeat the above steps for each statement in the text data and record the number of question vectors. The error quantification module is used to divide the number of vectors by the number of word vectors to obtain the error quantification score; The slip-of-the-mouth update module is used to update the real person probability score based on the quantitative score of slip-of-the-mouth mistakes. The recognition module is used to perform intent recognition, entity recognition, and emotion recognition on the text data to obtain recognition results. The matching module is used to match the response text in the corresponding industry database based on the recognition result; The playback module is used to convert the dialogue text into voice data and play the voice data.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the speech recognition-based dialogue interaction method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the speech recognition-based dialogue interaction method as described in any one of claims 1 to 6.

Images