Voice interaction method, voice interaction system and storage medium
By identifying and processing emotional tags of interactive speech and text, and generating responsive speech with emotional richness and naturalness, the problem that existing intelligent dialogue systems cannot effectively identify user emotions and control responsive speech emotions is solved, and the user communication experience is improved.
Patent Information
- Application Number
- CN202411059760.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-05
AI Technical Summary
The existing intelligent dialogue system cannot effectively recognize the user's emotions and cannot control the emotional changes in the response voice, resulting in poor emotional communication with the user.
By receiving interactive voice input by the user, combining interactive voice and interactive text, emotional tags are determined, and the rhythmic characteristics of response text and speech are generated based on the emotional tags, thereby generating emotionally rich and natural response voices.
It improves the emotional classification accuracy of interactive voice and the emotional richness and nature of response voice, and enhances emotional communication between users and the system.
Smart Images

Figure CN118588085B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a voice interaction method, a voice interaction system and a storage medium. Background Art
[0002] In recent years, with the advancement of artificial intelligence technology, the technology of human-computer interaction through intelligent dialogue systems has also developed rapidly. Intelligent dialogue systems can recognize and process the interactive text or interactive voice input by users, generate and output the response text for the interactive text, or output the response voice for the interactive voice.
[0003] However, existing intelligent dialogue systems are usually unable to recognize users' emotions based on interactive text or interactive speech, and are unable to control the degree of emotional change in the generated response speech, which is not conducive to emotional communication with users. Summary of the invention
[0004] In order to solve the above problems, the embodiment of the present application provides a voice interaction method, which can improve the accuracy of emotion classification of interactive speech and improve the emotional richness and naturalness of the generated response speech. Specifically, the embodiment of the present application discloses the following technical solutions:
[0005] A first aspect of an embodiment of the present application provides a voice interaction method, the method comprising: receiving an interactive voice input by a user; determining an emotion label corresponding to the interactive voice based on the interactive voice and an interactive text corresponding to the interactive voice; determining a response text corresponding to the interactive text, and a first prosodic feature and a second prosodic feature corresponding to the response text based on the emotion label; wherein the first prosodic feature is used to characterize the whole sentence prosodic feature of the response text, and the second prosodic feature is used to characterize the local prosodic feature of each character in the response text; and generating and outputting a response voice corresponding to the interactive voice based on the response text, the first prosodic feature and the second prosodic feature.
[0006] In some embodiments, the above-mentioned determining the emotion label corresponding to the above-mentioned interactive speech based on the above-mentioned interactive speech and the interactive text corresponding to the above-mentioned interactive speech includes: determining text emotion features based on the above-mentioned interactive speech and the above-mentioned interactive text, and determining speech emotion features based on the above-mentioned interactive speech; determining the above-mentioned emotion label based on the above-mentioned text emotion features and the above-mentioned speech emotion features.
[0007] In some embodiments, the above-mentioned determination of text sentiment features based on the above-mentioned interactive speech and the above-mentioned interactive text includes: processing the above-mentioned interactive speech based on the first sentiment feature extraction model to obtain whole sentence sentiment features; processing the above-mentioned interactive text based on the second sentiment feature extraction model to obtain character sentiment features; determining the above-mentioned text sentiment features based on the above-mentioned whole sentence sentiment features and the above-mentioned character sentiment features.
[0008] In some embodiments, the above-mentioned determining the speech emotion characteristics based on the above-mentioned interactive speech includes: processing the above-mentioned interactive speech based on a first emotion feature extraction model to obtain implicit speech emotion characteristics; processing the above-mentioned interactive speech based on a third emotion feature extraction model to obtain explicit speech emotion characteristics; determining the above-mentioned speech emotion characteristics based on the above-mentioned implicit speech emotion characteristics and the above-mentioned explicit speech emotion characteristics.
[0009] In some embodiments, the above-mentioned determining the response text corresponding to the above-mentioned interactive text, and the first prosodic feature and the second prosodic feature corresponding to the above-mentioned response text according to the above-mentioned emotion label includes: generating the above-mentioned response text corresponding to the above-mentioned emotion label according to the above-mentioned emotion label; processing the above-mentioned response text and the above-mentioned emotion label based on a first prosodic prediction model and an overall prosodic change parameter to obtain the above-mentioned first prosodic feature; processing the above-mentioned response text and the above-mentioned first prosodic feature based on a second prosodic prediction model and a local prosodic change parameter to obtain the above-mentioned second prosodic feature.
[0010] In some embodiments, the above-mentioned response text and the above-mentioned emotion tag are processed based on the first rhythm prediction model and the overall rhythm change parameters to obtain the above-mentioned first rhythm feature, including: encoding the above-mentioned emotion tag to obtain the whole sentence rhythm feature corresponding to the above-mentioned response text; denoising the above-mentioned whole sentence rhythm feature based on the above-mentioned first rhythm prediction model and the above-mentioned overall rhythm change parameters to obtain the above-mentioned first rhythm feature.
[0011] In some embodiments, the method further includes: performing noise processing on the sample whole sentence rhythmic features based on the first rhythm prediction model and the overall rhythm change parameters to obtain the sample whole sentence rhythmic features after noise addition; performing noise reduction processing based on the overall rhythm change parameters and the sample whole sentence rhythmic features after noise addition to obtain the sample whole sentence rhythmic features after noise reduction; determining the whole sentence rhythmic loss function corresponding to the first rhythm prediction model according to the sample whole sentence rhythmic features and the sample whole sentence rhythmic features after noise reduction; and optimizing the whole sentence rhythmic loss function according to the first optimization parameter of the whole sentence rhythmic loss function.
[0012] In some embodiments, the processing of the response text and the first prosodic feature based on the second prosodic prediction model and the local prosodic change parameters to obtain the second prosodic feature includes: encoding the response text based on an encoder to obtain a response text feature; fusing the first prosodic feature with the response text feature to obtain a fused text feature; wherein the fused text feature is a response text feature containing the first prosodic feature; processing the fused text feature based on the random prosodic predictor in the second prosodic prediction model and the local prosodic change parameters to obtain a random prosodic feature corresponding to the fused text feature; wherein the random prosodic feature includes a random fundamental frequency feature, a random energy feature, and a random duration feature; processing the fused text feature based on the fixed prosodic predictor in the second prosodic prediction model to obtain a fixed prosodic feature corresponding to the fused text feature; wherein the fixed prosodic feature includes a fixed fundamental frequency feature, a fixed energy feature, and a fixed duration feature; and determining the second prosodic feature based on the random prosodic feature, the fixed prosodic feature, and the control coefficient.
[0013] In some embodiments, the above-mentioned fused text features are processed based on the random prosody predictor in the above-mentioned second prosody prediction model and the above-mentioned local prosody change parameters to obtain random prosody features corresponding to the above-mentioned fused text features, including: performing noise processing on the above-mentioned fused text features based on the above-mentioned random prosody predictor and the above-mentioned local prosody change parameters to obtain the noisy fused text features; performing noise reduction processing on the above-mentioned noisy fused text features based on the above-mentioned random prosody predictor and the above-mentioned local prosody change parameters to obtain the above-mentioned random prosody features.
[0014] In some embodiments, the above-mentioned processing of the above-mentioned fused text features based on the random prosody predictor in the above-mentioned second prosody prediction model and the above-mentioned local prosody change parameters to obtain the random prosody features corresponding to the above-mentioned fused text features includes: processing the above-mentioned fused text features based on the random fundamental frequency predictor in the above-mentioned random prosody predictor and the above-mentioned local prosody change parameters to obtain the above-mentioned random fundamental frequency features; processing the above-mentioned fused text features and the above-mentioned random fundamental frequency features based on the random energy predictor in the above-mentioned random prosody predictor and the above-mentioned local prosody change parameters to obtain the above-mentioned random energy features; processing the above-mentioned fused text features, the above-mentioned random fundamental frequency features and the above-mentioned random energy features based on the random duration predictor in the above-mentioned random prosody predictor and the above-mentioned local prosody change parameters to obtain the above-mentioned random duration features; determining the above-mentioned random prosody features based on the above-mentioned random fundamental frequency features, the above-mentioned random energy features and the above-mentioned random duration features.
[0015] In some embodiments, the control coefficient includes a first control coefficient and a second control coefficient, and the second rhythmic feature is determined based on the random rhythmic feature, the fixed rhythmic feature and the control coefficient, including: determining the second rhythmic feature based on the first control coefficient, the second control coefficient, the random rhythmic feature and the fixed rhythmic feature; wherein the first control coefficient is used to determine the weight of the random rhythmic feature, and the second control coefficient is used to determine the weight of the fixed rhythmic feature.
[0016] In some embodiments, the method further includes: obtaining a sample response speech; obtaining sample fundamental frequency information, sample energy information and sample duration information corresponding to the sample response speech, and encoding the sample fundamental frequency information, sample energy information and sample duration information to obtain sample local prosody features; wherein the sample local prosody features include sample fundamental frequency features, sample energy features and sample duration features; according to the sample local prosody features and the random prosody features, determining the random prosody loss function corresponding to the random prosody predictor, and optimizing the random prosody predictor according to the second optimization parameter of the random prosody loss function; wherein the random prosody loss function includes a random fundamental frequency loss function, a random energy loss function and a random duration loss function; according to the sample local prosody features and the fixed prosody features, determining the fixed prosody loss function corresponding to the fixed prosody predictor, and optimizing the fixed prosody predictor according to the third optimization parameter of the fixed prosody loss function; wherein the fixed prosody loss function includes a fixed fundamental frequency loss function, a fixed energy loss function and a fixed duration loss function.
[0017] In some embodiments, the above-mentioned encoder-based encoding processing of the above-mentioned response text to obtain response text features includes: encoding the above-mentioned response text based on the first encoder to obtain character-level features corresponding to the above-mentioned response text; encoding the above-mentioned response text based on the second encoder to obtain phoneme-level features corresponding to the above-mentioned response text; encoding the features obtained by adding the above-mentioned character-level features and the above-mentioned phoneme-level features based on the third encoder to obtain the above-mentioned response text features.
[0018] In some embodiments, the above-mentioned generating and outputting a response speech corresponding to the above-mentioned interactive speech based on the above-mentioned response text, the above-mentioned first prosodic feature and the above-mentioned second prosodic feature includes: determining the speech feature based on the above-mentioned first prosodic feature, the above-mentioned second prosodic feature and the above-mentioned response text feature; generating and outputting the above-mentioned response speech based on the above-mentioned speech feature.
[0019] In some embodiments, after receiving the interactive voice input by the user, the method further includes: performing voice recognition processing on the interactive voice to obtain the interactive text corresponding to the interactive voice; after determining the emotional label corresponding to the interactive voice, the method further includes: processing the interactive text based on a language model to obtain the response text corresponding to the interactive text.
[0020] The second aspect of an embodiment of the present application provides a voice interaction system, which includes: an input module, configured to receive interactive voice input by a user; a sentiment classification module, configured to determine the sentiment label corresponding to the interactive voice based on the interactive voice and the interactive text corresponding to the interactive voice; a rhythm prediction module, configured to determine the response text corresponding to the interactive text, and the first rhythm feature and the second rhythm feature corresponding to the response text based on the sentiment label; wherein the first rhythm feature is used to characterize the whole sentence rhythm feature of the response text, and the second rhythm feature is used to characterize the local rhythm feature of each character in the response text; an output module, configured to generate and output the response voice corresponding to the interactive voice based on the response text, the first rhythm feature and the second rhythm feature.
[0021] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores computer program instructions. When a computer reads the instructions, the voice interaction method described in the first aspect is executed.
[0022] The fourth aspect of an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the voice interaction method described in the first aspect above.
[0023] In the voice interaction method provided in the embodiment of the present application, after receiving the interactive voice input by the user, the emotion label corresponding to the interactive voice can be determined according to the interactive voice and the interactive text corresponding to the interactive voice; and according to the emotion label, the response text corresponding to the interactive text, and the first prosodic feature and the second prosodic feature corresponding to the response text are determined. Among them, the first prosodic feature is used to characterize the whole sentence prosodic feature of the response text, and the second prosodic feature is used to characterize the local prosodic feature of each character in the response text. Afterwards, the response voice corresponding to the interactive voice is generated and output according to the response text, the first prosodic feature and the second prosodic feature.
[0024] By applying the technical solution provided in the present application, the emotion label corresponding to the interactive speech is determined according to the two modes of interactive speech and interactive text, so as to improve the accuracy of emotion classification of the interactive speech input by the user; the whole sentence rhythmic features and local rhythmic features of each character corresponding to the response text are determined according to the emotion label, so as to improve the emotion richness and naturalness of the generated response speech. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0026] Figure 1 A flowchart of a voice interaction method provided in some embodiments of the present application;
[0027] Figure 2 A schematic diagram of a voice interaction system provided in some embodiments of the present application;
[0028] Figure 3 A schematic diagram of a sentiment classification module provided in some embodiments of the present application;
[0029] Figure 4 A schematic diagram of extracting MFCC features provided in some embodiments of the present application;
[0030] Figure 5 A flowchart of another voice interaction method provided for some embodiments of the present application;
[0031] Figure 6 A schematic diagram of a diffusion model provided for some embodiments of the present application;
[0032] Figure 7 A schematic diagram of a second rhythm prediction model provided in some embodiments of the present application;
[0033] Figure 8 A schematic diagram of encoding text information provided in some embodiments of the present application;
[0034] Fig. 9 A schematic diagram of a VITS model provided for some embodiments of the present application;
[0035] Fig.10 A schematic diagram of another voice interaction system provided in some embodiments of the present application;
[0036] Fig.11 A schematic diagram of an electronic device according to some embodiments of the present application. DETAILED DESCRIPTION
[0037] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention and to make the above-mentioned purposes, features and advantages of the embodiments of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention are further described in detail below with reference to the accompanying drawings.
[0038] In existing intelligent dialogue systems, there are usually the following problems:
[0039] 1. Some existing intelligent dialogue systems only support responses in text form, lack effective use of voice, and have poor emotional intervention and comfort functions for users. At the same time, responding to users' interactive text or interactive voice based on knowledge graphs and databases will cause the generated response text to be too rigid, making it difficult to give full play to the advantages of AI's diversified and emotional responses.
[0040] 2. Some existing intelligent dialogue systems only rely on a single modality of information (i.e. text information or voice information) to judge and classify user emotions, resulting in large errors in the classification results.
[0041] 3. When generating response speech, some existing intelligent dialogue systems usually use the VITS model (Conditional Variational Autoencoder with Adversarial Learning for End-to-EndText-to-Speech) to predict the rhythmic features of the response text and generate the response speech based on the response text and rhythmic features. However, the VITS model lacks fine-grained modeling of rhythmic features, and the rhythm of the response speech generated based on the same response text is exactly the same, and the emotional changes of the response speech cannot be controlled, which is not conducive to emotional communication with users.
[0042] Based on the above technical problems, the present application provides a voice interaction method, a voice interaction system and a storage medium, which can perform voice interaction with users; and can improve the accuracy of emotion classification of the interactive voice input by users; and can also control the emotional changes of the output response voice, and improve the emotional richness and naturalness of the response voice.
[0043] The voice interaction method provided in this application is described in detail below in conjunction with the accompanying drawings.
[0044] Figure 1 A flowchart of a voice interaction method provided in some embodiments of the present application, Figure 2 A voice interaction system is provided for some embodiments of the present application. Figure 1 The voice interaction method shown can be Figure 2 The voice interaction system 200 shown is implemented. Figure 1 As shown, the voice interaction method may include steps 110 to 140.
[0045] Step 110: receiving interactive voice input by the user.
[0046] In some embodiments, Figure 2 As shown, the voice interaction system 200 includes an input module 210, a processing module 220, a sentiment classification module 230, a reasoning module 240, a speech synthesis module 250 and an output module 260.
[0047] In some embodiments, the user can input interactive voice to the voice interaction system 200 , and the voice interaction system 200 can obtain the interactive voice through the input module 210 therein and send the interactive voice to the processing module 220 .
[0048] Step 120: Determine the emotion label corresponding to the interactive speech according to the interactive speech and the interactive text corresponding to the interactive speech.
[0049] In some embodiments, the processing module 220 may perform speech recognition processing on the interactive speech to convert the interactive speech into corresponding text information (i.e., interactive text). Exemplarily, the processing module 220 may perform speech recognition processing on the interactive speech based on a speech recognition model, and the type of the speech recognition model is not limited in the embodiments of the present application.
[0050] In some embodiments, after the processing module 220 obtains the interactive text, the emotion classification module 230 obtains the interactive text and the interactive speech, and determines the text emotion features based on the interactive speech and the interactive text, and determines the speech emotion features based on the interactive speech; thereby classifying the user's emotions through the two modalities of text emotion features and speech emotion features, that is, determining the emotion label corresponding to the interactive speech.
[0051] refer to Figure 3 A schematic diagram of a sentiment classification module is shown in FIG. Figure 3As shown, after the sentiment classification module 230 obtains the interactive speech and the interactive text, the sentiment feature extraction model can be used to extract the sentiment feature of the interactive speech, thereby obtaining a 512-dimensional sentiment feature, which is strongly correlated with the sentiment of the interactive text and can be used as the feature of the interactive text in the sentence dimension, that is, the whole sentence sentiment feature of the interactive text; and the sentiment feature extraction model is used to extract the sentiment feature of the interactive text, thereby obtaining the feature of the interactive text in the character dimension, that is, the character sentiment feature of the interactive text. Then, the obtained whole sentence sentiment feature and character sentiment feature of the interactive text are combined to form the features of the text modality, that is, the sentiment feature fusion of the whole sentence sentiment feature and the character sentiment feature is performed in the text embedding layer (text_embedding), thereby obtaining the text sentiment feature.
[0052] Among them, the first emotion feature extraction model can be a Wav2Vec-2.0 model, etc., and the second emotion feature extraction model can be a MegatronBert, etc. The specific types of the first emotion feature extraction model and the second emotion feature extraction model are not limited in the embodiment of the present application.
[0053] At the same time, the emotion classification module 230 can extract implicit speech emotion features from the interactive speech through the first emotion feature extraction model, and extract explicit speech emotion features from the interactive speech through the third emotion feature extraction model. The third emotion feature extraction model can extract Mel-Frequency Cepstral Coefficients (MFCC) of the interactive speech, and use the MFCC features of the interactive speech as explicit speech emotion features of the interactive speech.
[0054] refer to Figure 4 A schematic diagram of extracting MFCC features is shown in FIG. Figure 4 As shown, when extracting the MFCC features of interactive speech, the interactive speech can be pre-emphasized first, that is, the high-frequency part of the interactive speech is enhanced through a high-pass filter, so as to balance the spectrum and improve the signal-to-noise ratio. The formula of the transfer function of the high-pass filter can refer to formula (1):
[0055] (1)
[0056] Among them, t is the time; is the speech sampling value corresponding to time t; is the speech sampling value corresponding to time t-1; is the pre-emphasis result corresponding to time t; is the pre-emphasis coefficient, and its value is usually in the range of [0.9, 1.0].
[0057] Next, after the interactive speech is pre-emphasized, the interactive speech is framed, that is, the interactive speech is divided into multiple short-time window frames. After the interactive speech is framed, each frame of the interactive speech is windowed using a window function to increase the continuity between frames and reduce spectrum leakage. For example, a Hamming window can be used to window each frame of the interactive speech. The formula of the Hamming window function can refer to formula (2):
[0058] (2)
[0059] Where N is the window length, n is the position of each sample point in the window, is the value of the Hamming window function at sample point n.
[0060] After windowing, a discrete Fourier transform (DFT) is performed on each frame of windowed interactive speech to convert the time domain signal into a frequency domain signal. In the frequency domain, the spectral features of the interactive speech are extracted through the Mel filter bank, so that the extracted features are more consistent with the perceptual characteristics of the human auditory system and can provide strong support for emotion classification. After that, the spectral features obtained after passing through the Mel filter bank are logarithmically operated to obtain the spectral features after logarithm. Finally, the spectral features after logarithm are discrete cosine transformed (DCT) to obtain the MFCC features of the interactive speech.
[0061] In some embodiments, after obtaining the implicit speech emotion features and the explicit speech emotion features of the interactive speech, the implicit speech emotion features and the explicit speech emotion features are combined to form the features of the speech modality, that is, the implicit speech emotion features and the explicit speech emotion features are emotionally fused in the speech embedding layer (Speech_embedding) to obtain the speech emotion features.
[0062] Afterwards, the emotion classification module 230 may fuse the obtained text emotion features and speech emotion features through a multilayer perceptron (MLP), and perform emotion classification on the interactive speech according to the fused emotion features, thereby obtaining an emotion label corresponding to the emotion type of the interactive speech. For example, the emotion label may include: positive, negative, neutral, angry, sad, happy, fearful, surprised, disgusted, etc.
[0063] Through the above scheme, by utilizing both interactive text and interactive voice modalities, it is possible to determine the emotional features corresponding to the two modalities respectively, and to combine the interactive information of the two modalities with the corresponding relationship of the latent space to determine the emotional features and perform emotional classification, thereby more comprehensively capturing the emotional features of users and improving the accuracy of emotional classification.
[0064] Step 130, determining the response text corresponding to the interactive text, and the first prosodic feature and the second prosodic feature corresponding to the response text according to the emotion tag.
[0065] In some embodiments, after classifying the emotional features of the interactive speech and interactive text and determining the emotional labels, a response content (i.e., response text) to the interactive text can be generated based on the emotional labels and the interactive text, and a first prosodic feature corresponding to the response text can be determined based on the emotional labels and the response text, and then a second prosodic feature corresponding to the response text can be determined based on the response text and the first prosodic feature.
[0066] Figure 5 A flowchart of another voice interaction method provided in some embodiments of the present application. Figure 5 As shown, the above step 130 may include steps 510 to 530.
[0067] Step 510, based on the emotion tag, generate a response text corresponding to the emotion tag.
[0068] In some embodiments, after classifying the emotional features of the interactive speech and the interactive text and determining the emotional tags, the reasoning module 240 can generate response content (ie, response text) to the interactive text based on the emotional tags and the interactive text.
[0069] Exemplarily, the reasoning module 240 can use a language model to process the interactive text and generate an interactive text that is consistent with the user's emotion corresponding to the emotional label. For example, the language model can be composed of a main model (such as a ChatGLM model) and a fine-tuning model (such as a low-rank adaptation (Low-Rank Adaptation of Large Language Models, LoRA) fine-tuning model). If it is necessary to guide and appease the user's negative emotions, the language model can be trained using psyQA and / or efaqa psychological counseling corpus to generate a LoRA fine-tuning model for psychological aspects. The reasoning module 240 can first load the ChatGLM model, then load the LoRA fine-tuning model, and replace the parameters of the ChatGLM model with the parameters of the LoRA model, so that the response text output by the language model is more in line with the language style of the psychological counselor. Finally, the interactive text is input into the ChatGLM model after parameter replacement, and the ChatGLM model outputs the response text corresponding to the interactive text after reasoning.
[0070] Step 520 , processing the response text and the emotion label based on the first prosody prediction model and the overall prosody change parameter to obtain a first prosody feature.
[0071] In some embodiments, after determining the response text, the speech synthesis module 250 can generate a response speech that matches the user emotion corresponding to the emotion label according to the emotion label and the response text. For example, if the emotion label corresponding to the interactive speech input by the user is negative, that is, the user's emotion is negative, then the generated response speech can be a soothing tone; if the emotion label corresponding to the interactive speech input by the user is positive, that is, the user's emotion is positive, then the generated response speech can be an affirmative and approving tone.
[0072] Exemplarily, when generating a response speech, not only the rhythm of the entire sentence of the response speech (i.e., the rhythm at the paragraph level) can be adjusted, but also the local rhythm of the response speech (i.e., the rhythm of each character in the response speech) can be adjusted, so that the generated response speech can have a unified emotional style as a whole and subtle emotional changes in each part.
[0073] In some embodiments, the speech synthesis module 250 includes a first prosody prediction model, which is a paragraph-level prosody predictor, and can be used to determine the first (i.e., the whole sentence prosody feature at the paragraph level) of the response speech. The paragraph-level prosody predictor can be constructed based on the Denoising Diffusion Probabilistic Models (DDPM). Figure 6 A schematic diagram of a diffusion model is shown in Figure . Figure 6 As shown in the figure, the diffusion model is a generative model based on the forward denoising process and the reverse denoising process. The forward denoising process converts the features into Gradually convert to a white noise The reverse denoising process converts this white noise into Gradually transform into features The following describes in detail the training process of the first prosody prediction model and the process of determining the first prosody feature based on the first prosody prediction model.
[0074] In some embodiments, the process of training the first prosody prediction model includes: performing noise processing on the prosody features of the sample whole sentence based on the first prosody prediction model and the overall prosody change parameters to obtain the prosody features of the sample whole sentence after noise; performing noise reduction processing on the prosody features of the sample whole sentence based on the overall prosody change parameters and the prosody features of the sample whole sentence after noise to obtain the prosody features of the sample whole sentence after noise reduction; determining the prosody loss function of the whole sentence corresponding to the first prosody prediction model according to the prosody features of the sample whole sentence and the prosody features of the sample whole sentence after noise reduction; and optimizing the prosody loss function of the whole sentence according to the first optimization parameter of the prosody loss function of the whole sentence.
[0075] In some examples, during the training of the paragraph-level prosody predictor, when the prosody features of the sample whole sentence are subjected to noise processing, the prosody features of the sample whole sentence can be gradually noised according to the overall prosody change parameter (i.e., the time step, or the number of diffusion times), that is, the prosody features of the sample whole sentence are subjected to noise processing. After T time steps of forward diffusion, the rhythmic features of the whole sentence of the sample after adding noise are obtained. The time step can be set according to the actual situation. The implementation of the forward noise addition process can refer to formula (3):
[0076] (3)
[0077] in, is the variance parameter of the noise corresponding to the tth time step, It can increase with the increase of time step T; is the identity matrix.
[0078] After the noise is added, the rhythmic features of the whole sentence of the noised sample are analyzed. The reverse denoising process is used to denoise the rhythmic features of the whole sentence of the sample after adding noise. Sampling by time step, through the neural network Gradually reduce noise and obtain the rhythmic features of the whole sentence of the sample after noise reduction The implementation of the reverse noise reduction process can refer to formula (4):
[0079] (4)
[0080] in, is the noise factor, It is a neural network used to predict the noise corresponding to time step t; the neural network can be based on the U-shaped convolutional neural network (U-Net), while adding residual connection and skip connection mechanisms, and the rhythmic features of the whole sentence of the sample are analyzed through a multi-layer convolutional network. It performs multiple downsampling and then multiple upsampling to restore it, and adds a transformer layer in the middle of up and down sampling for self-supervision to improve the expression ability.
[0081] After that, according to the rhythmic features of the whole sentence And the rhythmic features of the whole sentence after denoising , determine the whole sentence prosody loss function corresponding to the first prosody prediction model , and according to the whole sentence rhythm loss function The first optimization parameter of optimizes the whole sentence rhythm loss function. Neural Network The noise corresponding to each time step is predicted through training data learning, and the first optimization parameter is optimized so that the predicted noise is as close as possible to the actual added noise. Refer to formula (5):
[0082] (5)
[0083] in, is the estimation of the prosodic features of the whole sentence of the sample corresponding to time step t, Estimate the prosodic features of the whole sentence of the sample corresponding to the previous time step, is the first optimization parameter.
[0084] In some embodiments, the response text and the emotion tag are processed based on the first rhythm prediction model and the overall rhythm change parameters to obtain the first rhythm feature, including: encoding the emotion tag to obtain the whole sentence rhythm feature corresponding to the response text; and denoising the whole sentence rhythm feature based on the first rhythm prediction model and the overall rhythm change parameters to obtain the first rhythm feature.
[0085] In the process of reasoning based on the paragraph-level prosody predictor, the sentiment label obtained by the sentiment classification module 230 is first encoded to obtain the whole sentence prosody feature corresponding to the response text; then, the whole sentence prosody feature is denoised based on the paragraph-level prosody predictor and the overall prosody change parameter to obtain the predicted value output by the paragraph-level prosody predictor (i.e., the first prosody feature).
[0086] By increasing or decreasing the time step of the chapter-level prosody predictor, the emotional expression effect of the first prosody feature can be increased or decreased accordingly. Increasing the time step can make the chapter-level prosody predictor have a stronger ability to reduce the noise of the input whole sentence prosody features, and the output first prosody features have higher accuracy and more significant effects on emotional expression. Therefore, the time step in the chapter-level prosody predictor can be adjusted according to the user's emotional changes. For example, a larger time step is used in the stage where the user's emotions are more negative; a smaller time step is used in the stage where the user's negative emotions are weakened, so as to adjust the first prosody feature in line with the user's emotional changes to improve the user experience.
[0087] Step 530: Process the response text and the first prosodic feature based on the second prosodic prediction model and the local prosodic change parameter to obtain a second prosodic feature.
[0088] In some embodiments, the speech synthesis module 250 further includes a second prosody prediction model, which includes a fundamental frequency predictor, an energy predictor, and a duration predictor, and can model fine-grained emotions (i.e., character-level prosody), thereby obtaining a second prosody feature that has a richer and more accurate expression of emotions, and the second prosody feature is used to characterize the local prosody feature of each character in the response text. The second prosody prediction model can also be based on the following example: Figure 6 The diffusion model shown is constructed. The training process of the second prosody prediction model and the process of determining the second prosody feature based on the second prosody prediction model are described in detail below.
[0089] refer to Figure 7 A schematic diagram of a second rhythm prediction model is shown in FIG. Figure 7 As shown, the process of training the second prosody prediction model and obtaining the second prosody feature based on the second prosody prediction model can be performed simultaneously, that is, the training process and the reasoning process of the second prosody prediction model can be performed simultaneously.
[0090] In some embodiments, the response text and the first prosody feature are processed based on the second prosody prediction model and the local prosody change parameters to obtain the second prosody feature, including: encoding the response text based on the encoder to obtain the response text feature; fusing the first prosody feature and the response text feature to obtain a fused text feature; processing the fused text feature based on the random prosody predictor and the local prosody change parameters in the second prosody prediction model to obtain a random prosody feature corresponding to the fused text feature; processing the fused text feature based on the fixed prosody predictor in the second prosody prediction model to obtain a fixed prosody feature corresponding to the fused text feature; and determining the second prosody feature based on the random prosody feature, the fixed prosody feature and the control coefficient.
[0091] Among them, the fused text feature is the answer text feature containing the first prosody feature; the second prosody prediction model includes a random prosody predictor and a fixed prosody predictor; the random prosody predictor includes a random fundamental frequency predictor, a random energy predictor and a random duration predictor; the fixed prosody predictor includes a fixed fundamental frequency predictor, a fixed energy predictor and a fixed duration predictor; the random prosody features include random fundamental frequency features, random energy features and random duration features; the fixed prosody features include fixed fundamental frequency features, fixed energy features and fixed duration features.
[0092] refer to Figure 8 A schematic diagram of encoding text information is shown in FIG. Figure 8As shown, the response text can be encoded based on the first encoder (such as the BERT pre-trained model, etc.) to obtain the character-level features corresponding to the response text. The complex relationship between words and words, sentences and sentences in the response text can be obtained through the pre-trained BERT language model, and the character-level features corresponding to the response text can be generated. The character-level features can reflect complex rhythmic information, thereby effectively optimizing the rhythmic effect of the synthesized response speech. In addition, the response text can be encoded based on the second encoder (such as an acoustic model based on deep learning, a speech recognition model, etc.) to obtain the phoneme-level features corresponding to the response text. The character-level features are then mapped through an embedding layer (Embedding) and converted to the same dimension as the phoneme-level features through a convolutional network; finally, the features obtained by adding the two are encoded based on the third encoder (i.e., a text encoder) to obtain the response text features. It should be noted that the specific types of the first encoder, the second encoder, and the third encoder are not limited in the embodiment of the present application.
[0093] Next, the first prosodic feature and the answer text feature are fused to obtain a fused text feature, that is, the fused text feature is the answer text feature containing the first prosodic feature. Then, the fused text feature is processed based on the random fundamental frequency predictor and the local prosodic change parameter (i.e., the time step, or the number of diffusion times) to obtain the random fundamental frequency feature corresponding to the fused text feature; then, the fused text feature predicted by the fundamental frequency is processed based on the random energy predictor and the local prosodic change parameter to obtain the random energy feature corresponding to the fused text feature; then, the fused text feature predicted by the fundamental frequency and energy is processed based on the random duration predictor and the local prosodic change parameter to obtain the random duration feature corresponding to the fused text feature.
[0094] Among them, the fused text features are processed based on the random fundamental frequency predictor, random energy predictor and random duration predictor to obtain the random fundamental frequency features, random energy features and random duration features respectively. The principles of the process are the same, that is, the fused text features are firstly subjected to noise processing according to the local rhythm change parameters to obtain the fused text features after noise addition; then, the fused text features after noise addition are subjected to noise reduction processing according to the local rhythm change parameters to obtain the fused text features containing the random rhythm features. The specific process and related formulas can be referred to above Figure 6 The corresponding embodiments will not be described in detail here.
[0095] In some embodiments, the process of training the second prosody prediction model includes: obtaining sample response speech; obtaining sample fundamental frequency information, sample energy information and sample duration information corresponding to the sample response speech, and encoding the sample fundamental frequency information, sample energy information and sample duration information to obtain sample local prosody features; determining the random prosody loss function corresponding to the random prosody predictor based on the sample local prosody features and the random prosody features, and optimizing the random prosody predictor based on the second optimization parameter of the random prosody loss function; determining the fixed prosody loss function corresponding to the fixed prosody predictor based on the sample local prosody features and the fixed prosody features, and optimizing the fixed prosody predictor based on the third optimization parameter of the fixed prosody loss function.
[0096] Among them, the local rhythmic features of the sample include sample fundamental frequency features, sample energy features and sample duration features; the random rhythmic loss function includes random fundamental frequency loss function, random energy loss function and random duration loss function; the fixed rhythmic loss function includes fixed fundamental frequency loss function, fixed energy loss function and fixed duration loss function.
[0097] Exemplarily, at least one sample response speech can be obtained from a sample speech database, and the fundamental frequency of the sample response speech can be extracted using the pysptk.sptk.rapt package, and then the sample fundamental frequency information at the character level can be obtained by clipping, transforming and interpolating the sample response speech. Also, the sample response speech can be subjected to a short-time Fourier Transform (STFT) operation using the librosa package to obtain amplitude and phase information, and the amplitude information can be squared and then squared for each column and then squared to obtain character-level sample energy information. Also, the sample response speech can be forcibly aligned with the sample response text corresponding to the sample response speech through a monotonic alignment search (MAS), thereby obtaining the sample duration information corresponding to the sample response speech. Then, the obtained sample fundamental frequency information, sample energy information and sample duration information are encoded to obtain sample fundamental frequency features, sample energy features and sample duration features.
[0098] Exemplarily, the random prosody loss function corresponding to the random prosody predictor can be determined according to the sample local prosody features and the random prosody features output by each random prosody predictor, and the random prosody predictor can be optimized according to the second optimization parameter of the random prosody loss function. For example, the random fundamental frequency loss function corresponding to the random fundamental frequency predictor can be determined according to the random fundamental frequency features output by the random fundamental frequency predictor and the sample fundamental frequency features; and the random energy loss function corresponding to the random energy predictor can be determined according to the random energy features output by the random energy predictor and the sample energy features; and the random duration loss function corresponding to the random duration predictor can be determined according to the random duration features output by the random duration predictor and the sample duration features.
[0099] Among them, the determination of the random fundamental frequency loss function, random energy loss function or random duration loss function can refer to Figure 6 The corresponding formula (5) in the embodiment is used to fusion text features Noise addition and noise reduction are performed to obtain the predicted random rhythmic features Then, according to the local rhythmic features of the sample and predicted random rhythmic features , determine the random rhythm loss function , and according to the second optimization parameter For random rhythm loss function Optimize. Random rhythm loss function The formula for determining can be referred to as formula (6):
[0100] (6)
[0101] In some embodiments, in addition to determining the random prosodic features, the fixed prosodic features corresponding to the fused text features may also be determined based on the fixed prosodic predictor. Among them, the fixed fundamental frequency predictor / fixed energy predictor / fixed duration predictor may be a predictor including two layers of long short-term memory (LSTM) and one fully connected layer, which can make the output of the predictor have a relatively stable structure. After the fused text features are input into the fixed fundamental frequency predictor / fixed energy predictor / fixed duration predictor, the fixed fundamental frequency features, fixed energy features and fixed duration features corresponding to the fused text features may be obtained respectively. Among them, the formula for processing the fused text features based on the fixed fundamental frequency predictor / fixed energy predictor / fixed duration predictor to obtain the fixed fundamental frequency features / fixed energy features / fixed duration features corresponding to the fused text features may refer to formula (7):
[0102] (7)
[0103] in, To integrate text features, Indicates the fusion of text features through convolutional neural networks (CNN) Extract fixed rhythmic features. is the diagonal covariance matrix, is the variance constant. (Right now ) is a Gaussian distribution, and the predicted value can be sampled from the generated Gaussian distribution , or directly use as a predicted value.
[0104] Furthermore, based on the fixed prosody features predicted by the fixed prosody predictor and the local prosody features of the sample, a fixed prosody loss function corresponding to the fixed prosody predictor can be determined, and the fixed prosody predictor can be optimized according to the third optimization parameter in the fixed prosody loss function. The formula for determining the fixed prosody loss function can refer to formula (8):
[0105] (8)
[0106] in, is a fixed rhythm loss function; is the number of predictions; is the local rhythmic feature of the i-th sample; is the fixed rhythmic feature of the ith prediction, as in formula (7) .
[0107] Exemplarily, the fixed prosody predictor can be optimized using algorithms such as gradient descent. When optimizing the fixed prosody predictor using the gradient descent algorithm, the formula For the third optimization parameter Optimize. is the learning rate, is a fixed rhythm loss function For the third optimization parameter gradient.
[0108] After determining the above random rhythm loss function and the above fixed rhythm loss function, the loss function corresponding to the second rhythm prediction model can be determined: It can be understood that optimizing the random prosody loss function and the fixed prosody loss function is to optimize the second prosody prediction model.
[0109] In some embodiments, the control coefficient includes a first control coefficient and a second control coefficient; determining the second rhythmic feature according to the random rhythmic feature, the fixed rhythmic feature and the control coefficient includes: determining the second rhythmic feature according to the first control coefficient, the second control coefficient, the random rhythmic feature and the fixed rhythmic feature; wherein the first control coefficient is used to determine the weight of the random rhythmic feature, and the second control coefficient is used to determine the weight of the fixed rhythmic feature.
[0110] After determining the random prosodic features and the fixed prosodic features, the weights of the random prosodic features and the fixed prosodic features can be adjusted respectively by using the first control coefficient corresponding to the random prosodic features and the second control coefficient corresponding to the fixed prosodic features, thereby adjusting the prosodic diversity and prosodic stability respectively, and determining the second prosodic features corresponding to the answering text. For example, the first control coefficient and the second control coefficient can be adjusted to appropriate values, so that the local prosody of the answering speech generated based on the random prosodic features and the fixed prosodic features has both good variability and good stability.
[0111] Through the above scheme, the present application utilizes the sampling generation characteristics within the distribution of the diffusion model itself, which can make the predicted random rhythmic features have both the overall characteristics within the distribution and the one-to-many randomness, so that different rhythms can be determined for the same response text. At the same time, controlling the time step corresponding to the noise reduction process can correspondingly control the degree of rhythmic change, thereby improving the naturalness and controllability of the rhythm. In addition, while predicting the duration characteristics of the response text features, the present application also adds the prediction of the fundamental frequency and energy characteristics of the response text features, so that the character-level rhythmic features of the response text can be jointly determined based on the fundamental frequency, energy and duration characteristics, which can improve the emotional richness and naturalness of the character-level rhythmic features.
[0112] Step 140: Generate and output a response speech corresponding to the interactive speech according to the response text, the first prosodic feature and the second prosodic feature.
[0113] In some embodiments, the speech synthesis module 250 also includes a speech synthesis model. After determining the first prosodic feature and the second prosodic feature, the response text feature (i.e., the target text feature) containing the first prosodic feature and the second prosodic feature can be input into the speech synthesis model. The speech synthesis model can determine the corresponding speech feature according to the target text feature, and generate the response speech corresponding to the interactive speech according to the speech feature. Finally, the output module 260 outputs the response speech, thereby realizing the speech interaction with the user.
[0114] For example, the speech synthesis model may be a variational inference with adversarial learning for end-to-end text-to-speech (VITS) model. Fig. 9 A schematic diagram of a VITS model is shown in FIG. Fig. 9 As shown, the target text features determined based on the second prosody prediction model can be input into the flow-based models in the VITS model, so as to obtain the response speech features corresponding to the target text features based on the flow model; then, the response speech features are processed based on the decoder in the VITS model to convert the response speech features into corresponding audio waveforms, so as to obtain the response speech.
[0115] In some embodiments, continue to refer to Fig. 9 , training the VITS model can improve the quality and naturalness of the generated response speech. Please refer to the following example for the training process of the VITS model:
[0116] 1. Audio Reconstruction Loss
[0117] The audio reconstruction loss is used to measure the difference between the Mel spectrum generated by the VITS model and the true Mel spectrum. For example, the linear spectrum corresponding to the training sample speech can be extracted first, and the posterior encoder (PosteriorEncoder) in the VITS model can obtain the Mel spectrum corresponding to the training sample speech based on the linear spectrum corresponding to the training sample speech. (The mel spectrum is the training sample speech The real Mel spectrum of Output implicit features , and implicit features The corresponding mean and variance, and according to the implicit features The corresponding mean and variance generate the posterior distribution ; Finally, the implicit features are transformed into Decoded into the estimated answering voice , and extract its Mel spectrum . Thus, the loss function corresponding to audio reconstruction is obtained Refer to formula (9):
[0118] (9)
[0119] 2. KL Divergence Loss
[0120] KL discreteness loss is used to measure the posterior distribution and the conditional prior distribution The difference between the two. The loss function corresponding to KL divergence Refer to formula (10):
[0121] (10)
[0122] Among them, the conditional prior distribution is the target text feature output by the second prosody prediction model; after the target text feature is improved in expressiveness by the flow model, it is combined with the above implicit feature output by the posterior encoder Perform alignment processing to obtain the alignment matrix .
[0123] 3. Adversarial Training Loss
[0124] Adversarial training loss is achieved by introducing a discriminator into the VITS model. To distinguish between real audio and synthesized audio generated by the decoder, thereby improving the quality of the generated response speech. The adversarial training loss consists of the following three parts:
[0125] a. Discriminator Loss
[0126] The discriminator loss is used to measure the discriminator For training sample speech and synthesized audio The discriminator loss corresponds to the loss function Refer to formula (11):
[0127] (11)
[0128] b. Generator Loss
[0129] The generator loss is used to measure the synthesized audio Discriminator Identified as training sample speech The loss function corresponding to the generator loss is Refer to formula (12):
[0130] (12)
[0131] c. Feature Matching Loss
[0132] Feature matching loss is used to measure the synthetic audio And training sample speech In the discriminator The difference between different levels of features. The loss function corresponding to the feature matching loss is Refer to formula (13):
[0133] (13)
[0134] in, is the number of layers of the discriminator network, is the discriminator The characteristics of the layer, For the The number of features of the layer.
[0135] Through the above scheme, the audio reconstruction quality, the alignment of the potential representation and the authenticity of the generated audio in the VITS model are comprehensively considered. The model is optimized through audio reconstruction loss, KL discreteness loss and adversarial training loss. By optimizing these loss functions, the VITS model can generate high-quality and natural response speech.
[0136] By applying the technical solution of the present application, the emotion classification module in the voice interaction system can perform emotion classification on the interactive speech input by the user based on multi-modality (text mode and voice mode), so that the emotion classification result can be more accurate. In addition, the speech synthesis module in the voice interaction system can determine the paragraph-level rhythmic features (i.e., the whole sentence rhythmic features) and character-level rhythmic features corresponding to the response text based on the response text corresponding to the interactive speech, so that the generated response speech can have a unified emotional style as a whole, and can also have subtle emotional changes locally for different sentence meanings.
[0137] Fig.10 A schematic diagram of another voice interaction system provided in some embodiments of the present application. Fig.10 As shown, the voice interaction system 1000 includes an input module 1001, a sentiment classification module 1002, a rhythm prediction module 1003 and an output module 1004.
[0138] The input module 1001 is configured to receive interactive voice input by the user.
[0139] The emotion classification module 1002 is configured to determine the emotion label corresponding to the interactive speech according to the interactive speech and the interactive text corresponding to the interactive speech.
[0140] The rhythm prediction module 1003 is configured to determine the response text corresponding to the interactive text, and the first rhythm feature and the second rhythm feature corresponding to the response text according to the emotion label; wherein the first rhythm feature is used to characterize the rhythm feature of the entire sentence of the response text, and the second rhythm feature is used to characterize the local rhythm feature of each character in the response text.
[0141] The output module 1004 is configured to generate and output a response speech corresponding to the interactive speech according to the response text, the first prosodic feature and the second prosodic feature.
[0142] In some embodiments, the emotion classification module 1002 is specifically configured to: determine text emotion features according to interactive speech and interactive text, and determine speech emotion features according to interactive speech; determine emotion labels according to text emotion features and speech emotion features.
[0143] In some embodiments, the sentiment classification module 1002 is specifically configured to: process the interactive speech based on the first sentiment feature extraction model to obtain the sentiment features of the whole sentence; process the interactive text based on the second sentiment feature extraction model to obtain the character sentiment features; and determine the text sentiment features based on the sentiment features of the whole sentence and the character sentiment features.
[0144] In some embodiments, the emotion classification module 1002 is specifically configured to: process the interactive speech based on the first emotion feature extraction model to obtain implicit speech emotion features; process the interactive speech based on the third emotion feature extraction model to obtain explicit speech emotion features; determine the speech emotion features based on the implicit speech emotion features and the explicit speech emotion features.
[0145] In some embodiments, the rhythm prediction module 1003 is specifically configured to: generate a response text corresponding to the emotion tag based on the emotion tag; process the response text and the emotion tag based on a first rhythm prediction model and overall rhythm change parameters to obtain a first rhythm feature; process the response text and the first rhythm feature based on a second rhythm prediction model and local rhythm change parameters to obtain a second rhythm feature.
[0146] In some embodiments, the rhythm prediction module 1003 is specifically configured to: encode the emotion label to obtain the whole sentence rhythm feature corresponding to the response text; perform noise reduction on the whole sentence rhythm feature based on the first rhythm prediction model and the overall rhythm change parameter to obtain the first rhythm feature.
[0147] like Fig.10 As shown, the voice interaction system 1000 also includes a first training module 1005 .
[0148] In some embodiments, the first training module 1005 is configured to: perform noise processing on the sample whole sentence rhythmic features based on the first rhythm prediction model and the overall rhythm change parameters to obtain the sample whole sentence rhythmic features after noise addition; perform noise reduction processing on the sample whole sentence rhythmic features based on the overall rhythm change parameters and the noised sample whole sentence rhythmic features to obtain the sample whole sentence rhythmic features after noise reduction; determine the whole sentence rhythmic loss function corresponding to the first rhythm prediction model according to the sample whole sentence rhythmic features and the sample whole sentence rhythmic features after noise reduction; and optimize the whole sentence rhythmic loss function according to the first optimization parameter of the whole sentence rhythmic loss function.
[0149] In some embodiments, the prosody prediction module 1003 is specifically configured as follows: encoding the response text based on the encoder to obtain the response text feature; fusing the first prosody feature and the response text feature to obtain the fused text feature; wherein the fused text feature is the response text feature containing the first prosody feature; processing the fused text feature based on the random prosody predictor and the local prosody change parameter in the second prosody prediction model to obtain the random prosody feature corresponding to the fused text feature; wherein the random prosody feature includes the random fundamental frequency feature, the random energy feature and the random duration feature; processing the fused text feature based on the fixed prosody predictor in the second prosody prediction model to obtain the fixed prosody feature corresponding to the fused text feature; wherein the fixed prosody feature includes the fixed fundamental frequency feature, the fixed energy feature and the fixed duration feature; determining the second prosody feature based on the random prosody feature, the fixed prosody feature and the control coefficient.
[0150] In some embodiments, the prosody prediction module 1003 is specifically configured to: perform noise processing on the fused text features based on the random prosody predictor and the local prosody change parameters to obtain the noisy fused text features; perform noise reduction processing on the noisy fused text features based on the random prosody predictor and the local prosody change parameters to obtain the random prosody features.
[0151] In some embodiments, the rhythm prediction module 1003 is specifically configured as follows: based on the random fundamental frequency predictor and local rhythm change parameters in the random rhythm predictor, the fused text features are processed to obtain random fundamental frequency features; based on the random energy predictor and local rhythm change parameters in the random rhythm predictor, the fused text features and the random fundamental frequency features are processed to obtain random energy features; based on the random duration predictor and local rhythm change parameters in the random rhythm predictor, the fused text features, the random fundamental frequency features and the random energy features are processed to obtain random duration features; based on the random duration predictor and local rhythm change parameters in the random rhythm predictor, the random rhythm features are determined.
[0152] In some embodiments, the control coefficient includes a first control coefficient and a second control coefficient, and the rhythm prediction module 1003 is specifically configured to: determine the second rhythm feature according to the first control coefficient, the second control coefficient, the random rhythm feature and the fixed rhythm feature; wherein the first control coefficient is used to determine the weight of the random rhythm feature, and the second control coefficient is used to determine the weight of the fixed rhythm feature.
[0153] like Fig.10 As shown, the voice interaction system 1000 also includes a second training module 1006 .
[0154] In some embodiments, the second training module 1006 is configured to: obtain a sample response speech;
[0155] The sample fundamental frequency information, sample energy information and sample duration information corresponding to the sample response speech are obtained, and the sample fundamental frequency information, sample energy information and sample duration information are encoded to obtain the sample local prosody features; wherein the sample local prosody features include sample fundamental frequency features, sample energy features and sample duration features; according to the sample local prosody features and the random prosody features, the random prosody loss function corresponding to the random prosody predictor is determined, and the random prosody predictor is optimized according to the second optimization parameter of the random prosody loss function; wherein the random prosody loss function includes the random fundamental frequency loss function, the random energy loss function and the random duration loss function; according to the sample local prosody features and the fixed prosody features, the fixed prosody loss function corresponding to the fixed prosody predictor is determined, and the fixed prosody predictor is optimized according to the third optimization parameter of the fixed prosody loss function; wherein the fixed prosody loss function includes the fixed fundamental frequency loss function, the fixed energy loss function and the fixed duration loss function.
[0156] In some embodiments, the prosody prediction module 1003 is specifically configured as follows: based on the first encoder, encode the response text to obtain character-level features corresponding to the response text; based on the second encoder, encode the response text to obtain phoneme-level features corresponding to the response text; based on the third encoder, encode the features obtained by adding character-level features and phoneme-level features to obtain response text features.
[0157] In some embodiments, the output module 1004 is specifically configured to: determine the speech feature according to the first prosodic feature, the second prosodic feature and the response text feature; and generate and output the response speech according to the speech feature.
[0158] like Fig.10 As shown, the voice interaction system 1000 also includes a processing module 1007 .
[0159] In some embodiments, the processing module 1007 is configured to: perform speech recognition processing on the interactive speech to obtain interactive text corresponding to the interactive speech; and process the interactive text based on a language model to obtain a response text corresponding to the interactive text.
[0160] Fig.11 A schematic diagram of an electronic device provided for some embodiments of the present application. In some embodiments, the electronic device includes one or more processors and a memory. The memory is configured to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the voice interaction method in the above-mentioned embodiment.
[0161] like Fig.11 As shown, the electronic device 1100 includes: a processor (processor) 1101 and a memory 1102. Exemplarily, the electronic device 1100 may also include: a communication interface (Communications Interface) 1103 and a communication bus 1104.
[0162] The processor 1101, the memory 1102 and the communication interface 1103 communicate with each other via the communication bus 1104. The communication interface 1103 is used to communicate with other devices such as client or other server network elements.
[0163] In some embodiments, the processor 1101 is used to execute the program 1105, and specifically can execute the relevant steps in the above-mentioned voice interaction method embodiment. Specifically, the program 1105 can include program code, and the program code includes computer executable instructions.
[0164] Exemplarily, the processor 1101 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement some embodiments of the present application. The electronic device 1100 may include one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.
[0165] In some embodiments, the memory 1102 is used to store the program 1105. The memory 1102 may include a high-speed RAM memory, and may also include a non-volatile memory (NVM), such as at least one disk memory.
[0166] Program 1105 can be specifically called by processor 1101 to enable electronic device 1100 to perform voice interaction method operations.
[0167] Some embodiments of the present application provide a computer-readable storage medium, which stores at least one executable instruction. When the executable instruction is executed on the electronic device 1100, the electronic device 1100 executes the voice interaction method in the above embodiment.
[0168] The executable instructions can be specifically used to enable the electronic device 1100 to perform voice interaction method operations.
[0169] For example, the computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.
[0170] The beneficial effects that can be achieved by the computer-readable storage medium provided in some embodiments of the present application can be referred to the beneficial effects in the corresponding voice interaction method provided above, and will not be repeated here.
[0171] It should be noted that, in the application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0172] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0173] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.
[0174] For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by an instruction execution system, apparatus, or device or in conjunction with such instruction execution system, apparatus, or device.
[0175] More specific examples (a non-exhaustive list) of computer readable media include the following: an electrical connection having one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM).
[0176] In addition, the computer readable medium can even be a paper or other suitable medium on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, and then editing, interpreting or processing in other suitable ways as necessary, and then storing it in the computer memory. It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof.
[0177] In the above embodiments, multiple steps or methods may be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it may be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0178] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.
Claims
1. A voice interaction method, It is characterized in that include: Receive interactive voice input by the user; Determining an emotion label corresponding to the interactive speech according to the interactive speech and the interactive text corresponding to the interactive speech; According to the emotion tag, generating a response text corresponding to the emotion tag; Processing the response text and the emotion tag based on the first prosody prediction model and the overall prosody change parameter to obtain a first prosody feature; wherein the first prosody feature is used to characterize the prosody feature of the entire sentence of the response text; Encoding the response text based on the encoder to obtain response text features; The first prosodic feature and the response text feature are fused to obtain a fused text feature; wherein the fused text feature is the response text feature including the first prosodic feature; Processing the fused text feature based on a random prosody predictor and a local prosody change parameter in a second prosody prediction model to obtain a random prosody feature corresponding to the fused text feature; Processing the fused text feature based on the fixed prosody predictor in the second prosody prediction model to obtain a fixed prosody feature corresponding to the fused text feature; Determining a second prosodic feature according to the random prosodic feature, the fixed prosodic feature and the control coefficient; wherein the second prosodic feature is used to characterize a local prosodic feature of each character in the response text; A response speech corresponding to the interactive speech is generated and output according to the response text, the first prosodic feature and the second prosodic feature.
2. The method according to claim 1, It is characterized in that The determining, according to the interactive speech and the interactive text corresponding to the interactive speech, the emotion label corresponding to the interactive speech includes: Determining text emotion features according to the interactive speech and the interactive text, and determining speech emotion features according to the interactive speech; The emotion label is determined according to the text emotion feature and the speech emotion feature.
3. The method according to claim 2, It is characterized in that The determining of text emotion features according to the interactive speech and the interactive text includes: Processing the interactive speech based on a first emotion feature extraction model to obtain an entire sentence emotion feature; Processing the interactive text based on a second sentiment feature extraction model to obtain character sentiment features; The text sentiment feature is determined according to the whole sentence sentiment feature and the character sentiment feature.
4. The method according to claim 2, It is characterized in that The step of determining the speech emotion feature according to the interactive speech comprises: Processing the interactive speech based on a first emotion feature extraction model to obtain an implicit speech emotion feature; Processing the interactive speech based on a third emotion feature extraction model to obtain an explicit speech emotion feature; The speech emotion feature is determined according to the implicit speech emotion feature and the explicit speech emotion feature.
5. The method according to claim 1, It is characterized in that The step of processing the response text and the emotion label based on the first prosody prediction model and the overall prosody change parameter to obtain a first prosody feature includes: Encoding the emotion tag to obtain the whole sentence rhythm feature corresponding to the response text; The whole sentence prosodic feature is subjected to noise reduction processing based on the first prosodic prediction model and the overall prosodic change parameter to obtain the first prosodic feature.
6. The method according to claim 5, It is characterized in that The method further comprises: Performing noise processing on the prosodic features of the whole sentence of the sample based on the first prosodic prediction model and the overall prosodic change parameter to obtain the prosodic features of the whole sentence of the sample after noise addition; Performing noise reduction processing based on the overall rhythm change parameter and the rhythmic features of the sample whole sentence after noise addition to obtain the rhythmic features of the sample whole sentence after noise reduction; Determine a whole sentence prosody loss function corresponding to the first prosody prediction model according to the sample whole sentence prosody feature and the sample whole sentence prosody feature after denoising; The whole sentence prosody loss function is optimized according to a first optimization parameter of the whole sentence prosody loss function.
7. The method according to claim 1, It is characterized in that The random rhythm features include random fundamental frequency features, random energy features and random duration features; the fixed rhythm features include fixed fundamental frequency features, fixed energy features and fixed duration features.
8. The method according to claim 1, It is characterized in that The step of processing the fused text feature based on the random prosody predictor and the local prosody change parameter in the second prosody prediction model to obtain the random prosody feature corresponding to the fused text feature includes: Performing noise processing on the fused text feature based on the random prosody predictor and the local prosody change parameter to obtain a noisy fused text feature; The random prosody predictor and the local prosody change parameter are used to perform denoising on the noisy fused text feature to obtain the random prosody feature.
9. The method according to claim 7, It is characterized in that The step of processing the fused text feature based on the random prosody predictor and the local prosody change parameter in the second prosody prediction model to obtain the random prosody feature corresponding to the fused text feature includes: Processing the fused text feature based on the random fundamental frequency predictor in the random prosody predictor and the local prosody change parameter to obtain the random fundamental frequency feature; Processing the fused text feature and the random fundamental frequency feature based on the random energy predictor in the random prosody predictor and the local prosody change parameter to obtain the random energy feature; The fused text feature, the random fundamental frequency feature and the random energy feature are processed based on the random duration predictor in the random prosody predictor and the local prosody change parameter to obtain the random duration feature; The random rhythm feature is determined based on the random fundamental frequency feature, the random energy feature and the random duration feature.
10. The method according to any one of claims 1 to 9, It is characterized in that The control coefficient includes a first control coefficient and a second control coefficient, and determining the second prosodic feature according to the random prosodic feature, the fixed prosodic feature and the control coefficient includes: The second prosodic feature is determined according to the first control coefficient, the second control coefficient, the random prosodic feature and the fixed prosodic feature; wherein the first control coefficient is used to determine the weight of the random prosodic feature, and the second control coefficient is used to determine the weight of the fixed prosodic feature.
11. The method according to any one of claims 1 to 9, It is characterized in that The method further comprises: Get sample answering voice; Acquire sample fundamental frequency information, sample energy information and sample duration information corresponding to the sample response speech, and encode the sample fundamental frequency information, the sample energy information and the sample duration information to obtain sample local prosodic features; wherein the sample local prosodic features include sample fundamental frequency features, sample energy features and sample duration features; Determine a random prosody loss function corresponding to the random prosody predictor according to the sample local prosody feature and the random prosody feature, and optimize the random prosody predictor according to a second optimization parameter of the random prosody loss function; wherein the random prosody loss function includes a random fundamental frequency loss function, a random energy loss function and a random duration loss function; According to the local prosody features of the samples and the fixed prosody features, a fixed prosody loss function corresponding to the fixed prosody predictor is determined, and the fixed prosody predictor is optimized according to a third optimization parameter of the fixed prosody loss function; wherein the fixed prosody loss function includes a fixed fundamental frequency loss function, a fixed energy loss function and a fixed duration loss function.
12. The method according to any one of claims 1 to 9, It is characterized in that The encoding process of the response text based on the encoder to obtain the response text features includes: Encoding the response text based on the first encoder to obtain character-level features corresponding to the response text; Encoding the response text based on a second encoder to obtain phoneme-level features corresponding to the response text; The feature obtained by adding the character-level feature and the phoneme-level feature is encoded based on the third encoder to obtain the response text feature.
13. The method according to any one of claims 1 to 9, It is characterized in that The step of generating and outputting a response speech corresponding to the interactive speech according to the response text, the first prosodic feature, and the second prosodic feature includes: determining a speech feature according to the first prosodic feature, the second prosodic feature and the answer text feature; The response speech is generated and outputted according to the speech feature.
14. The method according to any one of claims 1 to 9, It is characterized in that After receiving the interactive voice input by the user, the method further includes: Performing speech recognition processing on the interactive speech to obtain the interactive text corresponding to the interactive speech; After determining the emotion label corresponding to the interactive speech, the method further includes: The interactive text is processed based on a language model to obtain the response text corresponding to the interactive text.
15. A voice interaction system, It is characterized in that include: An input module, configured to receive interactive speech input by a user; The emotion classification module is configured to determine the emotion label corresponding to the interactive speech according to the interactive speech and the interactive text corresponding to the interactive speech; The prosody prediction module is configured to generate a response text corresponding to the emotion tag according to the emotion tag; process the response text and the emotion tag based on a first prosody prediction model and an overall prosody change parameter to obtain a first prosody feature; wherein the first prosody feature is used to characterize the prosody feature of the entire sentence of the response text; encode the response text based on an encoder to obtain a response text feature; fuse the first prosody feature and the response text feature to obtain a fused text feature; wherein the fused text feature is a response text feature containing the first prosody feature; process the fused text feature based on a random prosody predictor and a local prosody change parameter in a second prosody prediction model to obtain a random prosody feature corresponding to the fused text feature; process the fused text feature based on a fixed prosody predictor in the second prosody prediction model to obtain a fixed prosody feature corresponding to the fused text feature; determine a second prosody feature based on the random prosody feature, the fixed prosody feature and a control coefficient; wherein the second prosody feature is used to characterize the local prosody feature of each character in the response text; The output module is configured to generate and output a response speech corresponding to the interactive speech according to the response text, the first prosodic feature and the second prosodic feature.
16. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the voice interaction method according to any one of claims 1-14 is implemented.
17. A computer program product, It is characterized in that The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the voice interaction method according to any one of claims 1 to 14.
