Dialogue interaction method and device based on voice recognition

Abstract

The invention belongs to the technical field of intelligent interaction, and discloses a conversation interaction method and device based on voice recognition, and the method comprises the steps: obtaining audio data, and carrying out the preprocessing of the audio data, and obtaining a short-time signal; extracting audio feature information in the short-time signal; inputting the audio feature information into a full covariance Gaussian mixture model to obtain a real person possibility score; if the real person possibility score is greater than a preset threshold value, converting the audio data into text data, and performing intention recognition, entity recognition and emotion recognition on the text data to obtain a recognition result; matching an answer text in a corresponding industry database according to a recognition result; and converting the answer text into voice data, and playing the voice data. The diversity and complexity of short-time signals can be captured, the robustness and accuracy of audio recognition are improved, dialogues between a dialogue engine and non-people are avoided, and computing resources are saved.

Description

Technical Field

[0001] The present application relates to the field of intelligent interaction technology, and in particular to a conversation interaction method and device based on speech recognition. Background Art

[0002] With the development of artificial intelligence technology, the application of intelligent dialogue engines is becoming more and more extensive. Existing dialogue engines usually transcribe the voice into text based on end-to-end speech recognition models, speech recognition engines such as SpeechBrain, etc. after obtaining the user's voice information, and then use regular expressions, Mel spectrum energy features (MFB), BiLSTM-CRF (bidirectional long short-term memory network and conditional random field), multi-task learning and other algorithms to identify the intention of the user's input text, and find the answer strategy in the corresponding knowledge base based on the recognition results, convert the answer strategy into audio and play it for the user.

[0003] However, existing dialogue engines focus on the accuracy of voice conversion and speech-to-text recognition, as well as improving the effectiveness of conversations. But in fact, the user voice received by the dialogue engine may not be from a real person, but may be from the user's intelligent AI system. This will cause the dialogue engine to have a conversation with another interactive system, which will not only affect the dialogue engine's construction of user personality labels and reduce the effectiveness of subsequent conversations, but also waste the dialogue engine's computing resources. Summary of the invention

[0004] The present application provides a conversation interaction method and device based on speech recognition, which can capture the diversity and complexity of short-term signals, improve the robustness and accuracy of audio recognition, avoid conversations between the conversation engine and non-humans, and save computing resources.

[0005] In a first aspect, an embodiment of the present application provides a conversation interaction method based on speech recognition, comprising:

[0006] Acquire audio data, and pre-process the audio data to obtain a short-time signal;

[0007] Extract audio feature information from short-time signals; the audio feature information includes signal energy curve, resonance peak curve, short-time zero-crossing rate, frequency domain feature parameters, cepstrum feature parameters and harmonic feature parameters;

[0008] The audio feature information is input into the Gaussian mixture model of full covariance to obtain the likelihood score of being a real person;

[0009] If the likelihood score of being a real person is greater than a preset threshold, the audio data is converted into text data;

[0010] Perform intent recognition, entity recognition, and sentiment recognition on text data to obtain recognition results;

[0011] Match the answer text in the corresponding industry database according to the recognition result;

[0012] Convert the conversation text into voice data and play the voice data.

[0013] Furthermore, the method further comprises:

[0014] Obtain the target sentence in the text data and input it into the autoencoder model to obtain the encoded sentence;

[0015] Calculate the reconstruction error of the encoded sentence; determine whether the reconstruction error is less than a preset coherence threshold;

[0016] If not, extract each word in the target sentence, and combine two adjacent words into a word vector; calculate the phonetic similarity of the two words in each word vector;

[0017] The word vectors whose phonetic similarity is lower than the preset similarity threshold are used as question vectors;

[0018] Repeat the above steps for each sentence in the text data, record the number of vectors of the question vector; divide the number of vectors by the number of word vectors to get the quantified score of the slip of the tongue;

[0019] Update the likelihood score of being human based on the quantified score of the slip of the tongue.

[0020] Furthermore, the method further comprises:

[0021] Detect pause positions in the audio data, and obtain sentence segmentation positions of the text data according to the pause positions; determine whether the sentence segmentation positions are grammatical, and record the number of sentence segments that are not grammatical; divide the number of sentence segments by the number of pause positions to obtain a quantitative score of the grammatical problem;

[0022] Update the human likelihood score based on the grammatical question quantitative score.

[0023] Furthermore, the method further comprises:

[0024] Extract words from text data and the meanings of each word;

[0025] Obtain user tags and determine whether there are standard words with the same meaning in the user tags;

[0026] If yes, determine whether the words in the text data are consistent with the standard words;

[0027] The number of first words that were consistent with the standard word and the number of second words that were inconsistent were recorded;

[0028] Divide the number of first words by the number of second words to get the word usage habit score;

[0029] Update the human likelihood score based on the word usage habit score.

[0030] Furthermore, the method further comprises:

[0031] Before preprocessing the audio data, determining whether the audio data is noise;

[0032] If the audio data is noise or after the voice data is played, the timing starts;

[0033] When the timed duration reaches a preset response threshold, a preset ending voice is replied to end the conversation. Further, the method further includes:

[0034] After receiving the conversation end instruction or ending the conversation, obtain the historical user input text;

[0035] Extract the words and meanings of the historical user input text;

[0036] Group words with the same meaning into a phrase and calculate the frequency of each word;

[0037] The word with the largest frequency in each phrase and a frequency greater than a preset standard threshold is taken as the standard word; the standard word and the corresponding frequency and meaning are put into the user tag.

[0038] Furthermore, the method further comprises:

[0039] Before preprocessing the audio data, extract the fundamental frequency and speech rate of the audio data;

[0040] Matching is performed in the user acoustic feature library based on fundamental frequency and speaking rate;

[0041] If there is a match, the audio data is converted into text data; otherwise, the audio data is preprocessed.

[0042] In a second aspect, an embodiment of the present application provides a conversation interaction device based on speech recognition, comprising:

[0043] A preprocessing module is used to obtain audio data and preprocess the audio data to obtain a short-time signal;

[0044] A feature extraction module is used to extract audio feature information from short-time signals; the audio feature information includes signal energy curve, resonance peak curve, short-time zero-crossing rate, frequency domain feature parameters, cepstrum feature parameters and harmonic feature parameters;

[0045] An input module, used to input audio feature information into a Gaussian mixture model of full covariance to obtain a likelihood score of being a real person;

[0046] A conversion module, used for converting audio data into text data when the likelihood score of a real person is greater than a preset threshold;

[0047] The recognition module is used to perform intent recognition, entity recognition, and sentiment recognition on text data to obtain recognition results;

[0048] A matching module is used to match the answer text in the corresponding industry database according to the recognition result;

[0049] The playback module is used to convert the dialogue text into voice data and play the voice data.

[0050] Furthermore, the device also includes:

[0051] The sentence encoding module is used to obtain the target sentence in the text data and input it into the autoencoding model to obtain the encoded sentence.

[0052] The reconstruction error module is used to calculate the reconstruction error of the coding sentence and determine whether the reconstruction error is less than a preset coherence threshold.

[0053] The word vector module is used to extract each word in the target sentence and form a word vector for two adjacent words.

[0054] The phonetic similarity module is used to calculate the phonetic similarity between two words in each word vector.

[0055] The question vector module is used to use the word vectors whose pinyin similarity is lower than a preset similarity threshold as question vectors.

[0056] The question quantity module is used to repeat the above steps for each sentence in the text data and record the vector quantity of the question vector.

[0057] The slip of the tongue quantification module is used to divide the number of vectors by the number of word vectors to obtain a slip of the tongue quantification score.

[0058] The slip of the tongue update module is used to update the likelihood score of the real person according to the quantified score of the slip of the tongue.

[0059] Furthermore, the device also includes:

[0060] The pause detection module is used to detect the pause position in the audio data and obtain the sentence segmentation position of the text data according to the pause position.

[0061] The grammar module is used to determine whether the segmentation position conforms to the grammar and record the number of segmentation positions that do not conform to the grammar;

[0062] The grammar quantification module is used to divide the number of sentence segments by the number of pause positions to obtain a quantitative score of grammar problems.

[0063] The grammar update module is used to update the human likelihood score according to the grammar problem quantitative score.

[0064] Furthermore, the device also includes:

[0065] The first extraction module is used to extract words from text data and the meanings of each word.

[0066] The tag acquisition module is used to obtain user tags and determine whether there are standard words with the same meaning in the user tags.

[0067] The standard judgment module is used to judge whether the words in the text data are consistent with the standard words.

[0068] The quantity recording module is used to record the quantity of the first words that are consistent with the standard words and the quantity of the second words that are inconsistent with the standard words.

[0069] The word usage quantification module is used to divide the number of first words by the number of second words to obtain a word usage habit score.

[0070] The word usage update module is used to update the real person probability score according to the word usage habit score.

[0071] In some embodiments, the apparatus further comprises:

[0072] The noise judgment module is used to judge whether the audio data is noise before preprocessing the audio data.

[0073] The timing module is used to start timing when the audio data is noise or after the voice data is played.

[0074] The conversation ending module is used to reply with a preset ending voice and end the conversation when the timing reaches a preset response threshold.

[0075] Furthermore, the device also includes:

[0076] The historical text acquisition module is used to obtain historical user input text after receiving a dialogue end instruction or ending a dialogue.

[0077] The second extraction module is used to extract words and the meanings of each word in the historical user input text.

[0078] The word frequency calculation module is used to group words with the same meaning into a word group and calculate the word frequency of each word.

[0079] The standard determination module is used to take the word with the largest word frequency in each word group and a word frequency greater than a preset standard threshold as the standard word.

[0080] The tag generation module is used to put standard words and corresponding word frequencies and word meanings into user tags.

[0081] Furthermore, the device also includes:

[0082] The acoustic feature extraction module is used to extract the fundamental frequency and speech rate of the audio data before preprocessing the audio data.

[0083] The acoustic feature matching module is used to match the user's acoustic feature library based on fundamental frequency and speech rate; when there is a match, the audio data is converted into text data; otherwise, the audio data is preprocessed.

[0084] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of a conversation interaction method based on speech recognition as described in any of the above embodiments are performed.

[0085] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a conversation interaction method based on speech recognition as in any of the above embodiments.

[0086] In summary, compared with the prior art, the technical solution provided in the embodiment of the present application has at least the following beneficial effects:

[0087] The embodiment of the present application provides a method for dialogue interaction based on speech recognition. After receiving the user's audio data, it first converts it into a short-time signal, extracts audio feature information such as signal energy curve, resonance peak curve, short-time zero-crossing rate, frequency domain feature parameters, cepstrum feature parameters and harmonic feature parameters, and uses a full covariance Gaussian mixture model to score the likelihood of a real person. When the likelihood score of a real person is greater than a preset threshold, the audio data is considered to be sent by a real person, and text conversion, text recognition and dialogue operations are performed based on the audio data. The full covariance Gaussian mixture model in the above method can model various audio feature information of audio data, and can capture the diversity and complexity of short-time signals by combining multiple audio feature information, improve the robustness and accuracy of audio recognition, avoid dialogue engines from talking to non-humans, and save computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 A flowchart of a conversation interaction method based on speech recognition is provided for an exemplary embodiment of the present application.

[0089] Figure 2 A schematic diagram of audio data provided for an exemplary embodiment of the present application.

[0090] Figure 3 A probability density distribution diagram of a standard GMM model fitting provided for an exemplary embodiment of the present application.

[0091] Figure 4A probability density distribution diagram of a full covariance GMM model fitting provided for an exemplary embodiment of the present application.

[0092] Figure 5 A flowchart of the steps for calculating the quantified score of verbal errors provided by an exemplary embodiment of the present application.

[0093] Figure 6 A flowchart of the steps for calculating word usage habit scores is provided for an exemplary embodiment of the present application.

[0094] Figure 7 A structural diagram of a conversation interaction device based on speech recognition provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0095] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0096] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of this application.

[0097] See also Figure 1 , the embodiment of the present application provides a conversation interaction method based on speech recognition, comprising:

[0098] Step S11, acquiring audio data, and preprocessing the audio data to obtain a short-time signal.

[0099] The operations of preprocessing the audio data include pre-emphasis, framing and windowing.

[0100] Specifically, pre-emphasis is to increase the energy of the high-frequency part, usually using a first-order high-pass filter, the formula is:

[0101] Y(n)=X(n)-coef*X(n-1)

[0102] Among them, X(n) is the audio data, and coef is usually taken as 0.97.

[0103] Framing is the process of cutting audio data into small segments or frames of equal length; the frame length is usually 20ms to 30ms (corresponding to 320-480 sampling points at a sampling rate of 16kHz), and there is a certain overlap between frames, usually half of the frame length.

[0104] Windowing is the process of applying a window function, such as a Hamming window, to each frame of the signal to reduce the impact of discontinuities at the frame boundaries.

[0105] After the above preprocessing operation, multiple frames of short-time signals can be obtained, and the following steps are performed on each short-time signal.

[0106] Step S12, extracting audio feature information from the short-time signal; the audio feature information includes a signal energy curve, a resonance peak curve, a short-time zero-crossing rate, a frequency domain feature parameter, a cepstrum feature parameter, and a harmonic feature parameter.

[0107] Specifically, the signal energy curve is obtained by calculating the signal energy of each frame of short-time signal, usually the sum of squares of the signals in the frame, and then arranging and connecting the sum of squares in the order of the corresponding short-time signals to obtain the signal energy curve.

[0108] The short-time zero-crossing rate is the number of times each frame of the signal crosses zero, which reflects the non-periodic characteristics of the short-time signal;

[0109] The frequency domain feature parameters are obtained by using the short-time Fourier transform (STFT): a fast Fourier transform (FFT) is performed on each frame of the short-time signal to obtain the spectrum, and then the frequency domain feature parameters of the spectrum are obtained, such as the spectrum centroid. The spectrum centroid represents the "center of gravity" of the spectrum and can be calculated by librosa.feature.spectral_centroid.

[0110] The resonance peak curve is obtained by spectrum analysis after the above-mentioned fast Fourier transform, and the resonance peak frequency of each frame of the short-time signal can be determined. The resonance peak frequency corresponds to the resonance mode of the vocal tract. Similarly, the resonance peak curve can be obtained by connecting the short-time signals in sequence.

[0111] The cepstrum feature parameters are obtained by taking the logarithm of the spectrum and then performing an inverse Fourier transform to obtain the cepstrum, which is then processed by discrete cosine transform (DCT) to extract the Mel-frequency cepstrum coefficients (MFCC).

[0112] The harmonic characteristic parameter is the harmonic component of the short-time signal.

[0113] Step S13, input the audio feature information into a Gaussian mixture model of full covariance to obtain a real person possibility score.

[0114] Among them, the Gaussian mixture model (GMM) can model audio feature information by considering the audio data as composed of multiple Gaussian distributions and describing the distribution of these feature information through a mixture of multiple Gaussian distributions.

[0115] The GMM models mainly include the standard GMM model, the diagonal covariance GMM model, the spherical covariance GMM model, the full covariance GMM model, etc. This application adopts the full covariance GMM model.

[0116] During the training process, it is necessary to estimate the parameters of the full covariance GMM, including the mean, variance, and mixing weights of each Gaussian distribution. Specifically, the GMM parameters are estimated by maximizing, which can make the acoustic model more accurately describe the statistical characteristics of speech data. Then, the probability distribution of each audio feature information is calculated using the full covariance GMM model, and the likelihood score of a real person is obtained based on the probability distribution using decision logic such as the Viterbi algorithm.

[0117] The following is the formula expression of the full covariance GMM mathematical model:

[0118]

[0119] where x is a D-dimensional feature vector, which is composed of the audio feature information of the short-time signal of the i-th frame; M represents the number of frames, and w i represents the weight corresponding to M, which is adjusted and determined during the training process; g(x丨u i ∑ i ) represents the Gaussian density of the frame, and the Gaussian density of each frame is a Gaussian function of D audio feature information.

[0120]

[0121] where u i is the mean vector, that is, the mean of the Gaussian distribution; ∑ i is the covariance matrix, which can be full rank; the mixing weights satisfy All of these are estimated and adjusted during the training process.

[0122] Here, the parameter form of λ is given. For different frames M, there is:

[0123] λ = {w i , u i , ∑ i}

[0124] Please refer to Figure 2 、 Figure 3 and Figure 4 , Figure 2 is an audio data segment. Inevitably, this audio data segment will be affected by some noise, but the overall distribution is smooth. Figure 3 is the probability density distribution fitted by the standard GMM model. It can be clearly seen that the standard GMM model will cause serious information loss. And Figure 4 is the probability density distribution fitted by the full covariance GMM model. Compared with the standard GMM model, each Gaussian component of the full covariance GMM model has a different covariance matrix, which enables the model to more flexibly capture the characteristics of the speaker, and the likelihood score of a real person will be more accurate.

[0125] Furthermore, in practical applications, GMM can be combined with Hidden Markov Model (HMM) to form a GMM-HMM model. This combination utilizes the clustering analysis and statistical modeling capabilities of GMM and the time series modeling capabilities of HMM, thereby improving the recognition accuracy and robustness of the model.

[0126] Step S14: if the human possibility score is greater than a preset threshold, the audio data is converted into text data.

[0127] Step S15, performing intent recognition, entity recognition and emotion recognition on the text data to obtain a recognition result.

[0128] Step S16, matching the answer text in the corresponding industry database according to the recognition result.

[0129] Specifically, natural language processing techniques are first needed to determine the intent or purpose expressed in the text data, which can be achieved through machine learning or deep learning models. For example, a pre-trained language model such as BERT can be fine-tuned to suit a specific business scenario; then entity recognition (NER) is performed to identify specific types of entities from the text data, such as names of people, places, organizations, dates, etc. This can be achieved by using named entity recognition techniques, such as models based on BIO tagging. BERT or other deep learning models can be used for entity recognition.

[0130] Next, sentiment analysis is performed to identify the emotional tendencies in the text, such as positive, negative, or neutral. Sentiment analysis models, such as those based on LSTM and Attention mechanisms, can be used to evaluate the sentiment of the text. Feature extraction is a key step in the process of intent recognition, entity recognition, and sentiment recognition. Features such as vocabulary, phrases, and parts of speech can be extracted from text data, or word embedding technologies such as Word2Vec or GloVe can be used to obtain richer semantic features.

[0131] The industry databases are various databases that are pre-set in the dialogue engine and can be determined based on the entity recognition results.

[0132] Based on the recognition results of intent, entity and emotion, database query or information retrieval technology is used to match in the specified industry database to find the most appropriate answer text.

[0133] In the specific implementation process, semantic analysis of text data can also be carried out through word vector analysis, word sense disambiguation, context similarity recognition, intent clarification, etc., but semantic analysis and how to answer are not the focus of this application.

[0134] Step S17, converting the dialogue text into voice data, and playing the voice data.

[0135] When converting the dialogue text into voice data, it is necessary to pay attention to the pronunciation of polyphones, determine the pronunciation of polyphones in the dialogue text based on the context, and determine the emotional tone expressed based on the dialogue text.

[0136] The above embodiment provides a method for dialogue interaction based on speech recognition. After receiving the user's audio data, it first converts it into a short-time signal, extracts audio feature information such as signal energy curve, resonance peak curve, short-time zero-crossing rate, frequency domain feature parameters, cepstrum feature parameters and harmonic feature parameters, and uses a full covariance Gaussian mixture model to score the likelihood of a real person. When the likelihood score of a real person is greater than a preset threshold, the audio data is considered to be sent by a real person, and text conversion, text recognition and dialogue operations are performed based on the audio data. The full covariance Gaussian mixture model in the above method can model various audio feature information of audio data. Combining multiple audio feature information can capture the diversity and complexity of short-time signals, improve the robustness and accuracy of audio recognition, avoid dialogue engines from talking to non-humans, and save computing resources.

[0137] In some embodiments, the method further comprises:

[0138] Step S01, before preprocessing the audio data, determine whether the audio data is noise.

[0139] Specifically, you can use audio analysis tools such as AudioAnalyze to judge the noise; or use objective evaluation indicators such as PESQ (Perceptual Evaluation of Speech Quality) to evaluate the audio quality and determine whether there is noise.

[0140] Step S02: If the audio data is noise or after the voice data is played, start timing.

[0141] Step S03, when the timed duration reaches a preset response threshold, reply with a preset ending voice and end the conversation.

[0142] Among them, the preset response threshold can set the noise threshold to 1 minute and the no response threshold to 3 minutes.

[0143] In actual applications, the number of conversation interactions connected to the server is limited. If a user continues to input noise or does not respond for a long time, the conversation will be automatically ended; this can avoid unnecessary occupation of computing resources.

[0144] In addition to saving computing resources, setting a preset response threshold to automatically end the conversation can also ensure security issues. Because in some cases, user silence or receiving noise may indicate that the user's attention has been diverted elsewhere, ending the conversation can serve as a security measure to prevent unauthorized access or operation.

[0145] In some embodiments, the method further comprises:

[0146] Step S21, before preprocessing the audio data, extract the fundamental frequency and speech rate of the audio data.

[0147] Specifically, fundamental frequency refers to the frequency of the sound produced by the vibration of the vocal cords, which determines the pitch of the sound. Because the structure of each person's vocal cords is different, the fundamental frequency emitted has individual differences. For example, the fundamental frequency range of men, women or people of different ages is different, and even people of the same gender have different fundamental frequencies.

[0148] Speech rate refers to the speed at which a person speaks, usually measured by the number of words (or syllables) pronounced per unit time. Speech rate is affected by many factors, including culture, environment, thinking and expression ability. Different people have different speech rates, and even the same person may have different speech rates in different situations.

[0149] The differences in fundamental frequency and speaking rate make these two parameters useful as biological acoustic features for user identification.

[0150] Step S22: matching in the user acoustic feature library based on fundamental frequency and speech rate.

[0151] Step S23, if there is a match, the audio data is converted into text data; otherwise, the audio data is pre-processed.

[0152] Among them, the user acoustic feature library is the acoustic features of all users in the historical conversations collected by the conversation interaction engine, that is, the fundamental frequency and speaking rate of all historical users. If there is a match, it means that the user who inputs the audio data at this time is an old historical user and is a user who has undergone real-person recognition. The subsequent recognition based on the Gaussian mixture model can be omitted, and text conversion and dialogue can be performed directly.

[0153] The above embodiment records the acoustic feature information of historical users and builds a user acoustic feature library, which can quickly identify and answer old users when they use the dialogue engine, eliminating the process of audio recognition again and further saving computing resources.

[0154] See also Figure 5 In some embodiments, the method further comprises:

[0155] Step S31: Obtain the target statement in the text data and input it into the autoencoder model to obtain the encoded statement.

[0156] Step S32: Calculate the reconstruction error of the encoded statement; determine whether the reconstruction error is less than the preset coherence threshold.

[0157] Among them, the reconstruction error is a key indicator to measure the similarity when the encoded statement is reconstructed back to the original target statement, and it can be obtained by calculating the mean square error or the mean absolute error between the target statement and the encoded statement.

[0158] Step S33: If not, extract each word in the target statement and form word vectors by combining two adjacent words.

[0159] Specifically, if the reconstruction error is lower than the preset coherence threshold, it indicates that the sentence is coherent and the user has not made a pronunciation error or repeated speech. Step S15 can be further executed; if the reconstruction error is higher than the preset coherence threshold, it indicates that the sentence is not coherent. At this time, it is necessary to further detect the specific situation of the incoherent sentence. This application determines whether the user has self-correction, such as slips of the tongue or repeated readings, based on whether the sentence is coherent when inputting audio.

[0160] Step S34: Calculate the pinyin similarity between the two words in each word vector.

[0161] Specifically, the pinyin similarity can be obtained by calculating the edit distance between the pinyins of the two words.

[0162] "Edit distance" is a method to measure the difference between two sequences, by calculating the minimum number of single-character edits (insertion, deletion, or replacement) required to convert one sequence to another. For the pinyins of words in this application, they can be regarded as strings, and the edit distance is used to calculate their similarity.

[0163] Step S35: Take the word vectors with pinyin similarity lower than the preset similarity threshold as problem vectors.

[0164] Specifically, the judgment of the pinyin similarity here is the situation where this application determines whether the user has made a slip of the tongue and repeated the correct pronunciation again; for example, assume that the two words in the word vector are "tease, play a trick on". When calculating the pinyin similarity of these two words, it is detected that "zhuo" becomes "zuo", and only the initial consonant "zh" needs to be replaced with "z". There is only 1 edit operation, that is, the replacement operation, the edit distance, and the pinyin similarity is 1. If the preset similarity threshold is 2, then at this time, it will be determined that this word vector is a problem vector, that is, "tease, play a trick on" is the user's slip of the tongue situation.

[0165] From this, it can be seen that the value of pinyin similarity is inversely proportional to the degree of similarity. The smaller the pinyin similarity, the smaller the edit distance and the greater the pinyin similarity between the two words. Therefore, this application regards two words in the word vector below the preset similarity threshold as words with similar pronunciations, that is, it is very likely that the user repeated the correct pronunciation after a slip of the tongue.

[0166] Step S36, repeat the above steps for each sentence in the text data, and record the vector quantity of the question vector.

[0167] That is, each sentence is taken as the target sentence and the above steps S31-S35 are repeated to summarize the number of question vectors obtained by screening.

[0168] Step S37, divide the number of vectors by the number of word vectors to obtain a quantified score of the slip of the tongue.

[0169] In actual applications, except for polyphones and light-toned words, artificial intelligence basically does not read the wrong words, especially since current artificial intelligence is becoming more and more familiar with how to determine the pronunciation of polyphones based on the context, resulting in fewer and fewer mispronunciations. And the most critical point is that even if artificial intelligence reads the wrong pronunciation, it will not realize it, that is, it will not correct itself, so the self-correction detection based on this application can further improve the accuracy of speech recognition.

[0170] Step S38: updating the human likelihood score according to the quantified score of the slip of the tongue.

[0171] Specifically, here the authenticity probability score is added to the verbal error quantitative score, that is, the number of verbal errors will increase the value of the authenticity probability score; then it is determined whether the authenticity probability score is greater than a preset threshold, if so, step S15 is executed.

[0172] The above embodiment first judges the coherence of the sentence. When detecting incoherence of the sentence, it judges whether the incoherence of the sentence is caused by the user's self-correction through the pinyin similarity. If so, the real person possibility score is updated according to the number of slips of the tongue, and the judgment is made again based on the updated real person possibility score, which further improves the accuracy of human-machine recognition.

[0173] In some embodiments, the method further comprises:

[0174] Step S41, detecting the pause position in the audio data, and obtaining the sentence segmentation position of the text data according to the pause position.

[0175] Step S42, determining whether the segmentation position is grammatically correct, and recording the number of segmentation positions that are not grammatically correct.

[0176] Specifically, grammatical analysis technology in natural language processing (NLP) can be used for grammatical judgment, such as Grammarly, LanguageTool, etc. These analysis services can check which pauses do not comply with grammatical rules.

[0177] Step S43, dividing the number of sentence segments by the number of pause positions to obtain a quantitative score of the grammatical problem.

[0178] Step S44, updating the real person likelihood score according to the grammatical problem quantitative score.

[0179] Specifically, the real person possibility score is subtracted from the grammatical problem quantitative score to obtain an updated real person possibility score, and then it is determined whether the real person possibility score is greater than a preset threshold. If so, step S15 is executed.

[0180] In actual applications, the audio input by users varies in length, and the converted text data also varies in length. Therefore, it is necessary to quantify the number of ungrammatical segments by the ratio of the number of segmentations to the total number of pauses.

[0181] The above embodiment realizes the detection of the quality of language expression of audio data by judging the number of abnormal pauses in the audio data based on grammar. Because artificial intelligence pays less attention to the content and required emotions of the audio when outputting audio, but these are often not ignored when real people speak, so the number of grammatical problems can be regarded as inversely proportional to the possibility of being a real person.

[0182] In some embodiments, the method further comprises:

[0183] Step S51, after receiving a dialogue ending instruction or ending the dialogue, obtaining historical user input text.

[0184] The conversation ending instruction can be triggered by the user clicking the end button on the conversation interaction interface, and the conversation ending is triggered in the above embodiment after continuous noise is detected or the user does not reply after a timeout.

[0185] The historical user input text is the text data converted from all user input audio and replied in the current dialog box. Because the historical user input text is used to create user tags, it is necessary to ensure that the corresponding text is input by a real person. According to the judgment of the above embodiment, only the text data corresponding to the real audio will be recognized and answered. Therefore, this embodiment only uses the text data that has been replied when creating a user tag at the end of the conversation.

[0186] Step S52: extracting the words and the meanings of the words in the historical user input text.

[0187] Step S53, grouping words with the same meaning into a phrase, and calculating the frequency of each word.

[0188] Step S54: taking the word with the largest frequency in each phrase and a frequency greater than a preset standard threshold as a standard word.

[0189] For example, "accurate" and "precise" have the same meaning and will be grouped into one phrase; "put in" and "put in" have the same meaning and will also be grouped into one phrase; in the first phrase, the frequency of "accurate" is greater than that of "precise" and greater than the preset standard threshold, which means that the user is accustomed to using the word "accurate" and regards "accurate" as a standard word. In the second phrase, although the frequency of "put in" is greater than that of "put in", the frequency of "put in" is less than the preset standard threshold, which means that both "put in" and "put in" appear very rarely and are not enough to be judged as the user's habitual words.

[0190] Furthermore, in order to avoid excessive words that meet the standard word requirements due to inappropriate setting of the preset standard threshold, which would put pressure on the storage of user tags, the judgment of the preset standard threshold can be omitted, and the word with the largest frequency in each phrase is first selected as the representative word of the corresponding meaning, and then the representative words are sorted, and the top 3 words are selected as standard words; the top 3 words are not fixed values, and the number selected each time can be freely adjusted according to actual conditions.

[0191] Step S55, putting the standard word and the corresponding word frequency and word meaning into the user tag.

[0192] Specifically, when putting a standard word into a user tag, since the user tag is updated after each conversation, the user tag may store previously determined standard words. At this time, in order to ensure that the same meaning corresponds to only one standard word, it is necessary to compare the meaning of the standard word to be put in with the historical standard words already existing in the user tag. If the same meaning is found, the word with a higher frequency is selected as the standard word.

[0193] The above embodiment realizes the construction of user tags, laying a foundation for the following determination of the user's habitual words.

[0194] See also Figure 6 In some embodiments, the method further comprises:

[0195] Step S61, extracting words from text data and the meanings of each word.

[0196] Step S62, obtaining the user tag, and determining whether there is a standard word with the same meaning in the user tag.

[0197] Step S63: If yes, determine whether the word in the text data is consistent with the standard word.

[0198] Step S64, recording the number of first words that are consistent with the standard words and the number of second words that are inconsistent with the standard words.

[0199] Step S65, dividing the number of the first words by the number of the second words to obtain a word usage habit score.

[0200] For example, in the user tags, the standard words recorded are: precision, alignment, and recognition; the words in the text data that have the same meaning as these standard words are: accuracy, alignment, and detection; then the only word that has the same meaning as the standard word is "alignment", and the remaining two have the same meaning but different words, so the word usage habit score is 1:2, or 0.5 points.

[0201] The above example only uses 3 words. In actual application, there will be more words to be compared.

[0202] Step S66, updating the real person likelihood score according to the word usage habit score.

[0203] In practical applications, the larger the number of first words, the more consistent the word usage habits of the current audio data are with the user's habitual words recorded in the user tag, and the greater the possibility of being a real person.

[0204] Therefore, here the real person possibility score is added to the word usage habit score to obtain an updated real person possibility score, and then it is determined whether the real person possibility score is greater than a preset threshold. If so, step S15 is executed.

[0205] The above embodiment records the user's word usage habits, and thus determines whether the words used in the current audio are consistent with the word usage habits recorded in the past each time audio data is received, thereby further improving the accuracy of the judgment of the possibility of being a real person.

[0206] See also Figure 7 Another embodiment of the present application provides a conversation interaction device based on speech recognition, comprising:

[0207] The preprocessing module 101 is used to obtain audio data and preprocess the audio data to obtain a short-time signal.

[0208] The feature extraction module 102 is used to extract audio feature information from the short-time signal; the audio feature information includes signal energy curve, resonance peak curve, short-time zero-crossing rate, frequency domain feature parameters, cepstrum feature parameters and harmonic feature parameters.

[0209] The input module 103 is used to input the audio feature information into the Gaussian mixture model of the full covariance to obtain a real person possibility score.

[0210] The conversion module 104 is used to convert the audio data into text data when the likelihood score of a real person is greater than a preset threshold.

[0211] The recognition module 105 is used to perform intent recognition, entity recognition and emotion recognition on the text data to obtain recognition results.

[0212] The matching module 106 is used to match the answer text in the corresponding industry database according to the recognition result.

[0213] The playing module 107 is used to convert the dialogue text into voice data and play the voice data.

[0214] Furthermore, the device also includes:

[0215] The sentence encoding module is used to obtain the target sentence in the text data and input it into the autoencoding model to obtain the encoded sentence.

[0216] The reconstruction error module is used to calculate the reconstruction error of the coding sentence and determine whether the reconstruction error is less than a preset coherence threshold.

[0217] The word vector module is used to extract each word in the target sentence and form a word vector for two adjacent words.

[0218] The phonetic similarity module is used to calculate the phonetic similarity between two words in each word vector.

[0219] The question vector module is used to use the word vectors whose pinyin similarity is lower than a preset similarity threshold as question vectors.

[0220] The question quantity module is used to repeat the above steps for each sentence in the text data and record the vector quantity of the question vector.

[0221] The slip of the tongue quantification module is used to divide the number of vectors by the number of word vectors to obtain a slip of the tongue quantification score.

[0222] The slip of the tongue update module is used to update the likelihood score of the real person according to the quantified score of the slip of the tongue.

[0223] Furthermore, the device also includes:

[0224] The pause detection module is used to detect the pause position in the audio data and obtain the sentence segmentation position of the text data according to the pause position.

[0225] The grammar module is used to determine whether the segmentation position conforms to the grammar and record the number of segmentation positions that do not conform to the grammar.

[0226] The grammar quantification module is used to divide the number of sentence segments by the number of pause positions to obtain a quantitative score of grammar problems.

[0227] The grammar update module is used to update the human likelihood score according to the grammar problem quantitative score.

[0228] Furthermore, the device also includes:

[0229] The first extraction module is used to extract words from text data and the meanings of each word.

[0230] The tag acquisition module is used to obtain user tags and determine whether there are standard words with the same meaning in the user tags.

[0231] The standard judgment module is used to judge whether the words in the text data are consistent with the standard words.

[0232] The quantity recording module is used to record the quantity of the first words that are consistent with the standard words and the quantity of the second words that are inconsistent with the standard words.

[0233] The word usage quantification module is used to divide the number of first words by the number of second words to obtain a word usage habit score.

[0234] The word usage update module is used to update the real person probability score according to the word usage habit score.

[0235] In some embodiments, the apparatus further comprises:

[0236] The noise judgment module is used to judge whether the audio data is noise before preprocessing the audio data.

[0237] The timing module is used to start timing when the audio data is noise or after the voice data is played.

[0238] The conversation ending module is used to reply with a preset ending voice and end the conversation when the timing reaches a preset response threshold.

[0239] Furthermore, the device also includes:

[0240] The historical text acquisition module is used to obtain historical user input text after receiving a dialogue end instruction or ending a dialogue.

[0241] The second extraction module is used to extract words and the meanings of each word in the historical user input text.

[0242] The word frequency calculation module is used to group words with the same meaning into a word group and calculate the word frequency of each word.

[0243] The standard determination module is used to take the word with the largest word frequency in each word group and a word frequency greater than a preset standard threshold as the standard word.

[0244] The tag generation module is used to put standard words and corresponding word frequencies and word meanings into user tags.

[0245] Furthermore, the device also includes:

[0246] The acoustic feature extraction module is used to extract the fundamental frequency and speech rate of the audio data before preprocessing the audio data.

[0247] The acoustic feature matching module is used to match the user's acoustic feature library based on fundamental frequency and speech rate; when there is a match, the audio data is converted into text data; otherwise, the audio data is preprocessed.

[0248] For the specific definition of a conversational interaction device based on speech recognition provided in this embodiment, please refer to the embodiment of a conversational interaction method based on speech recognition above, which will not be repeated here. Each module in the above-mentioned conversational interaction device based on speech recognition can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0249] An embodiment of the present application provides a computer device, which may include a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the processor executes the steps of a conversation interaction method based on speech recognition as in any of the above embodiments.

[0250] The working process, working details and technical effects of the computer device provided in this embodiment can be found in the above embodiment of a conversation interaction method based on speech recognition, which will not be described in detail here.

[0251] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of a conversational interaction method based on speech recognition as in any of the above embodiments are implemented. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive and / or a memory stick, etc. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The working process, working details and technical effects of the computer-readable storage medium provided in this embodiment can be found in the above embodiment of a conversational interaction method based on speech recognition, and will not be repeated here.

[0252] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0253] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0254] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A conversation interaction method based on speech recognition, characterized in that: include: Acquiring audio data, and preprocessing the audio data to obtain a short-time signal; Extract audio feature information from the short-time signal; the audio feature information includes a signal energy curve, a resonance peak curve, a short-time zero-crossing rate, a frequency domain feature parameter, a cepstrum feature parameter, and a harmonic feature parameter; Inputting the audio feature information into a Gaussian mixture model of full covariance to obtain a real person possibility score; If the real person possibility score is greater than a preset threshold, converting the audio data into text data; Performing intent recognition, entity recognition, and sentiment recognition on the text data to obtain a recognition result; Matching the answer text in the corresponding industry database according to the recognition result; The answer text is converted into voice data, and the voice data is played.

2. The method for dialogue interaction based on speech recognition according to claim 1, characterized in that: Also includes: Obtaining a target sentence from the text data and inputting it into an autoencoding model to obtain an encoded sentence; Calculating a reconstruction error of the encoded sentence; Determining whether the reconstruction error is less than a preset coherence threshold; If not, extract each word in the target sentence and form a word vector with two adjacent words; Calculate the phonetic similarity of two words in each of the word vectors; The word vector whose pinyin similarity is lower than a preset similarity threshold is used as a question vector; Repeat the above steps for each sentence in the text data, and record the vector quantity of the question vector; Dividing the number of vectors by the number of word vectors to obtain a quantified score of verbal errors; The human likelihood score is updated according to the verbal error quantification score.

3. The method for dialogue interaction based on speech recognition according to claim 1, characterized in that: Also includes: Detecting pause positions in the audio data, and obtaining sentence segmentation positions of the text data according to the pause positions; Determine whether the segmentation position is grammatically correct, and record the number of segmentations at the segmentation position that is not grammatically correct; Divide the number of sentence segments by the number of pause positions to obtain a quantitative score of the grammatical problem; The human likelihood score is updated according to the grammatical problem quantitative score.

4. The method for dialogue interaction based on speech recognition according to claim 1, characterized in that: Also includes: Extracting words from the text data and the meanings of the words; Obtaining a user tag, and determining whether there is a standard word with the same meaning in the user tag; If yes, determining whether the word in the text data is consistent with the standard word; Recording the number of first words that are consistent with the standard word and the number of second words that are inconsistent with the standard word; Divide the number of the first words by the number of the second words to obtain a word usage habit score; The real person likelihood score is updated according to the word usage habit score.

5. The method for dialogue interaction based on speech recognition according to claim 4, characterized in that: Also includes: Before preprocessing the audio data, determining whether the audio data is noise; If the audio data is noise or after the voice data is played, start timing; When the timing reaches the preset response threshold, the preset ending voice is replied and the conversation ends.

6. The method for dialogue interaction based on speech recognition according to claim 5, characterized in that: Also includes: After receiving the conversation end instruction or ending the conversation, obtain the historical user input text; Extracting the words of the historical user input text and the meanings of each word; Group words with the same meaning into a phrase and calculate the frequency of each word; The word with the largest frequency in each phrase and a frequency greater than a preset standard threshold is used as a standard word; The standard word and the corresponding word frequency and word meaning are put into the user tag.

7. The method for dialogue interaction based on speech recognition according to claim 1, characterized in that: Also includes: Before preprocessing the audio data, extracting the fundamental frequency and speech rate of the audio data; Matching in a user acoustic feature library based on the fundamental frequency and the speech rate; If there is a match, the audio data is converted into text data; otherwise, the audio data is preprocessed.

8. A conversational interaction device based on speech recognition, characterized in that: include: A preprocessing module, used to obtain audio data and preprocess the audio data to obtain a short-time signal; A feature extraction module, used to extract audio feature information from the short-time signal; the audio feature information includes a signal energy curve, a resonance peak curve, a short-time zero-crossing rate, a frequency domain feature parameter, a cepstrum feature parameter, and a harmonic feature parameter; An input module, used for inputting the audio feature information into a Gaussian mixture model of full covariance to obtain a likelihood score of a real person; A conversion module, configured to convert the audio data into text data when the likelihood score of a real person is greater than a preset threshold; A recognition module, used to perform intent recognition, entity recognition and emotion recognition on the text data to obtain a recognition result; A matching module, used for matching the answer text in the corresponding industry database according to the recognition result; The playing module is used to convert the dialogue text into voice data and play the voice data.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the conversation interaction method based on speech recognition as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the conversation interaction method based on speech recognition as described in any one of claims 1 to 7 are implemented.

Images