A multi-emotion speech synthesis method based on digital human

Abstract

The present invention discloses a multi-emotion speech synthesis method based on digital human, comprising the following steps: obtaining audio corpus under multiple emotions; extracting text information and phoneme timestamps in the audio corpus to construct a first training data set; training a pre-constructed phoneme timestamp prediction model through the first training data set; predicting the phoneme stamps and pronunciation frame lengths of each pronunciation audio in the text information through the trained phoneme timestamp prediction model to generate a phoneme sequence; inputting the phoneme sequence into a pre-trained acoustic model to obtain synthesized audio; the present invention can avoid the artificial addition of rhythm tags and improve the model training efficiency; and achieve a more realistic audio synthesis effect by learning the time domain and frequency domain features of the audio.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual digital human, and more specifically to a multi-emotion speech synthesis method based on digital human. Background Art

[0002] At present, although traditional speech synthesis can eliminate mechanical sounds and make it difficult for people to distinguish whether it is a real person's voice or a synthesized voice, the same text only has a single emotion. For speech synthesis of different emotions, different models are often used to train different audio corpora to obtain audio with corresponding emotions. This not only requires a large amount of audio corpora, but also the switching between different emotions will show a mechanical feeling, which is difficult to satisfy in the highly anthropomorphic voice interaction of digital humans. In terms of multi-emotion speech synthesis, although there is no hard indicator such as bit error rate, it also faces similar problems as speech tasks such as speech recognition. Although speech synthesis can synthesize the corresponding audio for the text that needs to be synthesized speech, its emotional richness is still not as high as that of humans. This is particularly important in the voice interaction of digital humans.

[0003] Ordinary deep learning models, such as tacotron, are based on statistical models to achieve speech synthesis. Often, a model can only train a speech synthesis system for one emotion. To achieve multi-emotional expression, it is necessary to switch back and forth between different emotion models. This not only requires a large amount of corpus for training, but also switching back and forth between different speech models will cause a serious mechanical feeling. In addition, the sounds synthesized by different emotional speech models cannot continuously model emotions, that is, the emotions expressed are either this or that, and there is no soft emotional expression in the middle. Directly throwing different emotional audio corpora with emotional labels into a model for training will cause the synthesized speech to not conform to the rhythmic characteristics of people's normal expression of emotions, that is, it will not catch the words that should be expressed with emphasis, and thus it will sound very strange.

[0004] Adding prosodic labels to the training corpus can alleviate the above problems, but labeling the corpus is a tedious process that requires a lot of manual review to achieve a good result. In addition, in actual voice interaction, the text to be synthesized also needs to be prosodicly annotated, which makes the system complicated, and incorrect annotations are likely to affect the final synthesis effect.

[0005] Therefore, how to provide a multi-emotion speech synthesis method based on digital human is a problem that technicians in this field need to solve urgently. Summary of the invention

[0006] In view of this, the present invention provides a multi-emotion speech synthesis method based on digital humans, which can avoid the artificial addition of prosody labels and improve the efficiency of model training; and achieve a more realistic audio synthesis effect by learning the time domain and frequency domain features of audio.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A multi-emotional speech synthesis method based on digital human, comprising the following steps:

[0009] Obtain audio corpus with multiple emotions;

[0010] Extracting text information and phoneme timestamp labels from the audio corpus to construct a training data set;

[0011] Performing supervised training on a pre-built timestamp prediction model using the training data set;

[0012] The text information to be synthesized is predicted by using the trained timestamp prediction model and the preset emotion vector to obtain a phoneme timestamp;

[0013] The phoneme timestamp is input into the trained acoustic model to obtain synthesized audio.

[0014] Furthermore, in the audio corpus under the multiple emotions, the emotions include one or more of neutral, happy, sad, angry, surprised and fear.

[0015] Furthermore, extracting text information and phoneme timestamp labels from the audio corpus and constructing a first training data set comprises:

[0016] The audio corpus is edited into multiple audios,

[0017] Performing speech recognition on the audio through an ASR speech recognition model to obtain text information; converting the text information into text pinyin, and decomposing it into multiple phonemes according to a phoneme dictionary to form a phoneme sequence;

[0018] Performing phoneme alignment on the phoneme sequence to obtain a phoneme timestamp label for each audio;

[0019] The training data set is generated by inserting a pause frame length into the phoneme sequence and concatenating the phoneme timestamp labels corresponding to the audio.

[0020] Furthermore, the phoneme timestamp prediction model is a Bilstm model, and the loss function of the Bilstm model adopts a mean square error loss function.

[0021] Further, the acoustic model includes a synthesizer and a vocoder;

[0022] Generating a model embedding vector according to the phoneme timestamp;

[0023] Fitting the model embedding vector through the synthesizer to obtain mel spectrum features;

[0024] The mel spectrum feature is input into the vocoder to obtain synthesized audio.

[0025] Further, a model embedding vector is generated according to the phoneme timestamp, the steps are:

[0026] According to the phoneme timestamp, the pronunciation frame length difference between each phoneme and its adjacent preceding and following pinyins is calculated respectively, and the pronunciation frame length difference is spliced ​​into the phoneme timestamp label to generate a model embedding vector.

[0027] Furthermore, the vocoder is a pre-trained hifi-gan model.

[0028] Furthermore, the mel spectrum feature has 80 channels, hop_size is 256, and win_size is 1024.

[0029] Furthermore, after extracting the text information of the audio information, regularization processing is performed to convert the special characters that need to be converted into Chinese into Chinese to obtain the regularized standard text. The beneficial effects of the present invention are:

[0030] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a multi-emotion speech synthesis method based on digital humans, which adopts emotion vectors combined with timestamp information, and uses deep learning technology to learn speech emotion characteristics, thereby reducing the errors that may be caused by the rhythm labeling step, making the model pronunciation more natural and the emotions more realistic, while eliminating the rhythm prediction step, speeding up the process and improving real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0032] Figure 1 The accompanying drawing is a schematic diagram of a multi-emotion speech synthesis method based on digital human provided by the present invention;

[0033] Figure 2 The accompanying drawing is a schematic diagram of a method for extracting phoneme timestamp labels in the present invention;

[0034] Figure 3The accompanying drawing is a schematic diagram of a method for constructing a training data set in the present invention;

[0035] Figure 4 The accompanying drawing is a schematic diagram of the acoustic model training method in the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] like Figure 1 The embodiment of the present invention discloses a multi-emotion speech synthesis method based on a digital human, comprising the following steps:

[0038] S1: Obtain audio corpus with multiple emotions;

[0039] In one embodiment, in the audio corpus with multiple emotions, the emotions may include one or more of neutral, happy, sad, angry, surprised and fear. In addition, the audio corpus has a consistent volume and a consistent tone.

[0040] The specific steps are:

[0041] Professional announcers record multi-emotional audio materials in professional recording studios. Announcers need to record audio materials for six emotions, including neutral, happy, sad, angry, surprised, and fear. Each emotion should record about 10,000 audios. Announcers should record audio materials for various emotions separately, and try to keep the same volume, and the speaking speed and tone of the same emotion should be kept consistent.

[0042] To ensure recording efficiency, the recording is not paused during a recording. The pauses or non-script sounds of the announcer during the recording process need to be edited out later. After the audios recorded in different batches are edited, the volume of each audio needs to be normalized to ensure the quality of the training audio.

[0043] S2: Extract text information and phoneme timestamps from audio corpus to build training dataset;

[0044] In one embodiment, the text information is converted into text pinyin and decomposed into multiple phonemes according to a phoneme dictionary to form a phoneme sequence;

[0045] Perform phoneme alignment on the phoneme sequence to obtain the phoneme timestamp label of each audio;

[0046] Insert pause frame lengths into the phoneme sequence and concatenate the phoneme timestamp labels corresponding to the audio to generate a training data set. The pause frame length can be obtained by factoring the timestamp label. The phoneme timestamp contains the phoneme and the corresponding duration, one is the phoneme and the other is the time. The concatenation here means copying the phoneme into a multiple of its time frame length. For example, for the phoneme abc, the time frame lengths of the three phonemes are 123 respectively, so the phoneme copy frame length is abbccc.

[0047] The specific steps are:

[0048] like Figure 2 , the announcer records the audio and processes the audio clips to obtain training audio corpora one by one. In order to prevent the announcer from making mistakes such as wrong words, missing words, and extra words when recording the audio, each audio corpus is first processed for speech recognition to obtain the text content of each audio. After manual verification, the correct audio and corresponding text information are obtained. After obtaining the correct text information, the text content is converted into pinyin, and the pinyin and pinyin phoneme dictionary are sorted out. The audio files and the corresponding pinyin files are sorted out, and the acoustic model is used to align the phonemes through the phoneme dictionary, such as the MFA tool. After phoneme alignment, the phoneme timestamp information of each audio can be obtained. The text, phoneme timestamp, audio file and other information of each audio corpus are sorted out as training data text for the next step of training the model.

[0049] S3: Performing supervised training on a pre-built phoneme timestamp prediction model using the first training data set;

[0050] like Figure 3 , based on the text information and phoneme timestamp information in the first training data set, a phoneme timestamp prediction model is trained.

[0051] The purpose of predicting timestamp information is to predict the phoneme timestamp information corresponding to the audio sound after inputting the text during speech synthesis, so that when performing audio synthesis, we know which phoneme should be pronounced at a certain moment. The unit of timestamp is represented by frames, and the time for a phoneme to be pronounced can take up several frames. The input of the phoneme timestamp prediction model only contains text information, but there will be intermittent pauses in the actual audio pronunciation. In order to be able to represent this intermittent pause in the output prediction structure, an sp symbol is added after each text pinyin to indicate the pause frame length. If there is no pause frame length at this sp, the corresponding label is 0. The prediction model structure adopts the Bilstm model, and the loss function adopts the mean square error loss function. After inputting the text pinyin, it is converted into a phoneme sequence containing sp by querying the phoneme dictionary of the pinyin. The phoneme sequence is spliced ​​with the corresponding emotional label vector after embedding, and then input into the Bilstm network. The corresponding label is the frame length of each phoneme pronunciation. Through training, the model converges.

[0052] S4: predicting the text information to be synthesized by using the trained timestamp prediction model and the preset emotion vector to obtain a phoneme timestamp;

[0053] S5: Input the phoneme timestamp into the trained acoustic model to obtain synthesized audio.

[0054] In one embodiment, after training the phoneme timestamp prediction model, the pronunciation content within each frame length of the duet pronunciation audio can be obtained by inputting text.

[0055] The next step is to generate audio content through the acoustic model.

[0056] The input is a phoneme sequence. Each phoneme unit consists of phonemes of its corresponding frame length. The phoneme sequence is first concatenated with the emotion vector after embedding.

[0057] In order to capture the acoustic characteristics of the emotions in multi-emotional audio, the acoustic model needs to find the phonemes where the stress is located.

[0058] Different audio emotion features are reflected in the time domain and frequency domain. Here we have obtained the phoneme timestamp information, so we can use the frame length information of the phoneme to let the model learn the time domain information of the phoneme where the stress is located.

[0059] The current frame length of each character and the frame length difference relative to the previous character and the frame length difference of the next character are concatenated into the embedding vector of the previous phoneme sequence, and then input into the acoustic model, so that pronunciation audio with different emotions can be learned.

[0060] like Figure 4The acoustic model mainly consists of two parts, the synthesizer and the vocoder. Here, the mel spectrum feature extracted from the audio prediction is used as the output label of the synthesizer. The loss function is calculated based on the mel spectrum and output label output by the synthesizer, and the model training is performed to determine the convergence of the synthesizer; the mel channel is 80, the hop_size is 256, and the win_size is 1024.

[0061] The vocoder uses the HiFi-GAN model, and the vocoder uses a pre-trained model, that is, it has been pre-trained on large-scale data in advance.

[0062] Train the synthesizer model by inputting the embedding vector of the phoneme sequence to fit the corresponding audio mel spectrum features. Finally, input the generated mel spectrum features into the previously pre-trained hifi-gan model to generate audio, thereby testing the model effect.

[0063] In this embodiment, when the trained model is actually used, it is only necessary to obtain the text and the emotion to be expressed, and there is no need to input a prosody label for the text.

[0064] After entering the text and sentiment labels, first use regular expressions to convert special characters that need to be converted into Chinese, such as Arabic numerals, telephone numbers, unit names, and operation symbols.

[0065] The processed sentences are converted into corresponding phoneme labels through the phoneme dictionary, and the sp symbol is inserted into the phoneme sequence, and then the emotion vector is spliced ​​and input into the timestamp prediction model. After obtaining the timestamp of each phoneme, the timestamp information is calculated and combined with the previously input emotion phoneme vector to obtain the expanded emotion phoneme vector with the phoneme timestamp information, which is input into the synthesizer model to obtain the output mel spectrum feature. Then it is input into the hifi-gan model to generate the audio file corresponding to the text and complete the multi-emotion speech synthesis.

[0066] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0067] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-emotional speech synthesis method based on digital human, characterized in that: The following steps are involved: Obtain audio corpus with multiple emotions; Extracting text information and phoneme timestamp labels from the audio corpus to construct a training data set; Performing supervised training on a pre-built timestamp prediction model using the training data set; The text information to be synthesized is predicted by using the trained timestamp prediction model and the preset emotion vector to obtain a phoneme timestamp; The phoneme timestamp is input into the trained acoustic model to obtain synthesized audio.

2. The multi-emotional speech synthesis method based on digital human according to claim 1, characterized in that: In the audio corpus under the multiple emotions, the emotions include one or more of neutral, happy, sad, angry, surprised and fearful.

3. The multi-emotion speech synthesis method based on digital human according to claim 1, characterized in that: The steps of extracting text information and phoneme timestamp labels from the audio corpus and constructing a training data set include: The audio corpus is edited into multiple audios. Performing speech recognition on the audio through an ASR speech recognition model to obtain text information; converting the text information into text pinyin, and decomposing it into multiple phonemes according to a phoneme dictionary to form a phoneme sequence; Performing phoneme alignment on the phoneme sequence to obtain a phoneme timestamp label for each audio; The training data set is generated by inserting a pause frame length into the phoneme sequence and concatenating the phoneme timestamp labels corresponding to the audio.

4. The multi-emotional speech synthesis method based on digital human according to claim 1, characterized in that: The timestamp prediction model is a Bilstm model, and the loss function of the Bilstm model adopts a mean square error loss function.

5. The multi-emotion speech synthesis method based on digital human according to claim 1, characterized in that: The acoustic model includes a synthesizer and a vocoder; Generating a model embedding vector according to the phoneme timestamp; Fitting the model embedding vector through the synthesizer to obtain mel spectrum features; The mel spectrum feature is input into the vocoder to obtain synthesized audio.

6. The multi-emotion speech synthesis method based on digital human according to claim 5, characterized in that: Generate a model embedding vector according to the phoneme timestamp, the steps are: According to the phoneme timestamp, the pronunciation frame length difference between each phoneme and its adjacent preceding and following pinyins is calculated respectively, and the pronunciation frame length difference is spliced ​​into the phoneme timestamp label to generate a model embedding vector.

7. The multi-emotion speech synthesis method based on digital human according to claim 5, characterized in that: The vocoder is a pre-trained hifi-gan model.

8. The multi-emotion speech synthesis method based on digital human according to claim 5, characterized in that: The mel spectrum feature has 80 channels, hop_size is 256, and win_size is 1024.

9. The multi-emotion speech synthesis method based on digital human according to claim 1, characterized in that: After extracting the text information of the audio information, regularization processing is performed to convert special characters that need to be converted into Chinese into Chinese to obtain regularized standard text.

Images