A short video blogger stylized speech synthesis method based on attention variational autoencoder

Abstract

This paper presents a stylized speech synthesis method for short video bloggers based on attention-based variational autoencoders, belonging to the technical field of variational autoencoder network models. This stylized speech synthesis method divides the short video blogger speech synthesis process into two parts: the first part constructs a stylized speech synthesis dataset using short video blogger videos, and the second part synthesizes stylized speech using an attention-based variational autoencoder. This method incorporates a novel iAFF attention mechanism module to establish a system based on... β - The supervised attention mechanism network of the VAE method can directly synthesize stylized blogger speech through label attributes.

Description

Technical Field

[0001] This invention relates to the field of variational autoencoder model technology, and more specifically, to a method for stylized speech synthesis for short video bloggers based on attention-based variational autoencoders. Background Technology

[0002] Speech synthesis technology is the technology that converts text into speech that humans can understand. It uses a combination of computer algorithms and acoustic models to transform input text into natural and fluent speech output. With the continuous improvement of computer performance and speech synthesis algorithms, speech synthesis systems have become more complex and produce higher quality outputs. Today, speech synthesis technology is widely used in various fields, such as voice broadcasting and virtual blogger voice synthesis, bringing fun and convenience to our daily lives.

[0003] Traditional speech generation methods typically consist of two modules: a front-end and a back-end. The front-end module analyzes the input text and extracts the linguistic information needed by the back-end module, including text regularization, part-of-speech prediction, polyphonic character disambiguation, and prosodic prediction. The back-end generates speech waveforms using specific methods based on the analysis results from the front-end.

[0004] Currently, speech synthesis typically uses an end-to-end approach, where inputting text or phonetic characters allows the system to directly generate the corresponding audio waveform. However, if it is desired to synthesize multiple styles of voice, voice samples from different speakers are required for synthesis; it is not possible to directly synthesize multiple styles of voice. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this patent proposes a stylized speech synthesis method for short video bloggers based on an attention-based variational autoencoder. This method can generate a variety of stylized voices. It can synthesize multiple stylized voices based on provided character styles, without requiring specific speaker voice samples. Furthermore, the method incorporates Big Five personality traits and emotional style tags, enabling the synthesis of voices with diverse personalities and emotions.

[0006] This invention relates to a stylized speech synthesis method for short video bloggers based on an attention-based variational autoencoder, comprising two parts. The first part involves constructing a stylized speech synthesis dataset using short video blogger videos. The second part involves generating stylized speech from short video bloggers using an attention-based variational autoencoder. The Tacotron2 generative model is then used to extract speech from the video blogger dataset, and an attention-based variational autoencoder is used to combine the voice with the attributes of the short video blogger to generate stylized speaker embeddings. Finally, the Tacotron2 model is used to synthesize speech based on these speech embeddings. Stylized voices are synthesized using blogger-related attribute features.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a method for stylized speech synthesis for short video bloggers based on attention variational autoencoders, comprising the following steps:

[0008] S1. Construct a face-voice dataset, download the first impression dataset, and extract face images and corresponding voice sounds from the first impression video dataset;

[0009] S2. Obtain the contour detection image of the key facial features. Input the face dataset image in step S1. Use the DeepFace multiple face attribute feature extraction model to extract the emotional and personality attribute features of the face from the face dataset image in step S1. Use X-Vector to extract the voice attribute features from the voice dataset.

[0010] S3. Construct a supervised attention mechanism network based on the β-VAE method, add an iAFF module between the encoder and decoder, and combine the attention module to establish a supervised attention mechanism network based on the β-VAE method;

[0011] The speaker embedding features extracted by X-Vector are used to generate latent variables z through the encoder. The speaker attribute features extracted from the image are combined with the latent variables z, and then passed through the iAFF attention module before being sent to the decoder for decoding. After decoding by the decoder, the speaker embedding is generated.

[0012] Add a condition to the β-VAE to synthesize stylized speech based on the attributes of the input. The output is based on the input. In the training set, there are data pairs (x, y), where y is the input, i.e., the condition, and x is the expected output.

[0013] The model loss function is as follows:

[0014] L cβvae =-β·D KL (q(z|x,y)||p(z|y))+E q(z|x，y) (log(p(x|z,y)))

[0015] Among them, D KL It is the KL divergence, which makes the latent variables generated by the encoder conform to the standard normal distribution as much as possible. p represents the true vector, q represents the result vector, z represents the latent vector, and E q(z|x，y) Representing the reconstruction loss, it is the MSE loss between the vector decoded by the decoder and the input vector, reflecting the difference between the VAE-generated result and the input. The corresponding goal is to make the VAE-generated result as similar to the input as possible.

[0016] S4. Using the facial attribute features and corresponding voice features obtained in step S2 as the training dataset, train the supervised attention mechanism network based on the β-VAE method constructed in step S3 using Adam.

[0017] S5. Input attribute feature values. Based on the facial attribute features and corresponding voice features obtained in step S2, input them into the supervised attention mechanism network based on the β-VAE method trained in step S4 to generate speech embedding.

[0018] S6. The stylized sound obtained in step S5 is embedded into the multi-speaker Tacotron2 model for speech synthesis to synthesize the sound.

[0019] Step S2, obtaining facial attribute features and corresponding voice features, includes the following steps:

[0020] S2.1 Extracting sound features through X-Vector: X-Vector accepts input of arbitrary length and transforms it into a feature representation of fixed length;

[0021] S2.2 Feature Extraction Based on Face Shape: The MTCNN face detection model is used to extract face markers to calculate face shape. The MTCNN network structure is a three-tiered cascade network consisting of P-Net, R-Net, and O-Net layers. In the MTCNN three-layer network structure, the network output includes whether it is a face, bounding box regression values, and face feature point coordinates. The final loss function of the model can be specifically expressed as:

[0022] L mtcnn =α det L det +α box L box +α landmarks L landmarks

[0023] Where, α det L det The face classification loss uses cross-entropy as the loss function, α. box L box The bounding box regression loss uses Euclidean distance as the loss function, α. landmarks L landmarks The loss is based on facial feature point coordinates, using Euclidean distance as the loss function.

[0024] S2.3 Extracts features based on emotion, age, and gender, and uses the DeepFace framework to extract multiple attribute feature values;

[0025] S2.4 Based on the weight feature extraction, the BMI weight prediction model is used to predict the weight attribute features of a person from the image;

[0026] S2.5 extracts features based on the Big Five personality traits, using a dataset of short video bloggers that capture first impressions. This dataset contains annotations of the Big Five personality traits, and the attribute features of the Big Five personality traits are used directly.

[0027] 3. The method for stylized speech synthesis of short video bloggers based on attention variational autoencoder according to claim 1, characterized in that, in step S6, the speaker stylized features obtained in step S5 are embedded using the Tacotron2 model for speech synthesis, and the generated speaker feature embeddings are synthesized into the sound we hear through the Tacotron model, and different styles of speech are synthesized according to attribute labels.

[0028] Compared with existing technologies, this application has the following advantages: The short video blogger stylized speech synthesis method based on attention variational autoencoder can learn more comprehensive speech feature information by incorporating a new iAFF attention mechanism module. This method establishes a supervised attention mechanism network based on the β-VAE method, combining character attribute features with the speaker's voice, and can synthesize different styles of speaker voices based on character attribute labels. Attached Figure Description

[0029] Figure 1 This is a flowchart of a short video blogger stylized speech synthesis method based on attention variational autoencoders.

[0030] Figure 2 It is a network model structure for a short video blogger-style speech synthesis method based on attention variational autoencoders.

[0031] Figure 3 This is a screenshot of the generated stylized speech synthesis method for short video bloggers based on attention variational autoencoders.

[0032] Figure 4 This is a diagram of the synthesized sound effect. Detailed Implementation

[0033] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments: These embodiments are used as examples to further illustrate this application. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention.

[0034] Figure 1 A flowchart of a short video blogger stylized speech synthesis method based on attention variational autoencoders is shown. This method specifically includes the following steps:

[0035] (1) Construct a face-voice dataset, download the first impression dataset, and extract face images and corresponding voice sounds from the first impression video dataset;

[0036] (2) Obtain the contour detection image of the key facial features, input the face dataset image in step (1), use DeepFace and other face attribute feature extraction models to extract the emotional, personality and other attribute features of the face from the face dataset image in step (1), and use X-Vector to extract the sound attribute features from the sound dataset.

[0037] (3) Construct a supervised attention mechanism network based on the β-VAE method, add an iAFF module between the encoder and decoder, and combine the attention module to establish a supervised attention mechanism network based on the β-VAE method;

[0038] (4) Using the facial attribute features and corresponding voice features obtained in step (2) as the training dataset, Adam is used to train the supervised attention mechanism network based on the β-VAE method constructed in step (3);

[0039] (5) Input attribute feature values, obtain the face attribute features and corresponding voice features according to step (2), and input them into the supervised attention mechanism network based on the β-VAE method trained in step (4) to generate speech embedding;

[0040] (6) The stylized sound obtained in step (5) is embedded into the multi-speaker Tacotron2 model for speech synthesis.

[0041] Step (2) to obtain facial attribute features and corresponding voice features includes the following steps:

[0042] (2.1) Sound features are extracted through X-Vector. Thanks to the StatisticsPooling layer in its network, X-Vector can accept input of any length and transform it into a fixed-length feature representation.

[0043] (2.2) Based on facial feature extraction, the MTCNN face detection model is used to extract facial landmarks to calculate the face shape. The MTCNN network structure is a three-tiered network, which can be divided into three layers: P-Net, R-Net, and O-Net. This model adopts the idea of ​​candidate boxes plus classifier, which can simultaneously balance speed and accuracy to achieve fast and efficient face detection. In the P, R, and O networks of MTCNN, the network output forms are basically the same, which are whether it is a face, the bounding box regression value, and the coordinates of facial feature points (in P-Net and R-Net, the coordinates of facial feature points have no practical meaning). The final loss function of the model can be specifically expressed as:

[0044] L mtcnn =α det L det +α box L box +α landmarks L landmarks

[0045] Where, α det L det The face classification loss uses cross-entropy as the loss function, α. box L box The bounding box regression loss uses Euclidean distance as the loss function, α. landmarks L landmarks The loss is based on facial feature point coordinates, using Euclidean distance as the loss function.

[0046] (2.3) Based on emotion, age, and gender features, multiple attribute feature values ​​are extracted using the DeepFace framework. DeepFace is a lightweight face recognition and facial attribute analysis (age, gender, emotion) framework. It is a hybrid face recognition framework that incorporates state-of-the-art models: VGG-Face, Google FaceNet, OpenFace, DeepID, ArcFace, and Dlib, achieving an accuracy of 97.53% on face recognition tasks.

[0047] (2.4) Based on the weight extraction features, the BMI weight prediction model is used to predict the weight attribute features of people from the images.

[0048] (2.5) Based on the Big Five personality traits, the first impression short video blogger dataset was used. This dataset contains the annotation of the Big Five personality traits, and the attribute features of the Big Five personality traits can be used directly.

[0049] In step (3), a supervised attention mechanism network based on the β-VAE method is constructed. The generator includes an encoder and a decoder. An iAFF module is added between the encoder and the decoder. The speaker embedding features extracted by X-Vector are used to generate latent variables z through the encoder. The speaker attribute features extracted from the image are combined with the latent variables z, and then passed through the iAFF attention module before being sent to the decoder for decoding. After decoding by the decoder, the speaker embedding is generated. A condition is added to the β-VAE, which can synthesize stylized speech based on the input attribute conditions. The output is based on the input. In the training set, there are data pairs (x, y), where y is the input, which is the condition, and x is the output we expect.

[0050] The model loss function is as follows:

[0051] L cβvae =-β·D KL (q(z|x,y)||p(z|y))+E q(z|x，y) (log(p(x|z,y)))

[0052] Among them, D KL It is the KL divergence that makes the latent variables generated by the encoder conform to the standard normal distribution as closely as possible, E q(z|x，y) Representing the reconstruction loss, it is the MSE loss between the vector decoded by the decoder and the input vector, reflecting the difference between the result generated by the VAE and the input. The corresponding goal is to make the result generated by the VAE as similar to the input as possible.

[0053] Step (6) uses the Tacotron2 model to embed the speaker stylized features obtained in step (5) for speech synthesis. The generated speaker feature embedding is synthesized by the Tacotron model into the sound we hear. Different styles of speech can be synthesized according to the attribute labels.

[0054] Figure 2 This paper presents a stylized speech synthesis model structure for short video bloggers based on an attention-based variational autoencoder. As shown in the figure, this method designs a network structure for stylized speech synthesis for short video bloggers. Based on β-VAE, a supervised attention mechanism network based on the β-VAE method is constructed. An iAFF module is added between the encoder and decoder, and combined with the attention module, a supervised attention mechanism network based on the β-VAE method is established.

[0055] Using a set of voice features and corresponding personality attribute features as input, the speaker's voice embedding is obtained through a network model structure based on attention variational autoencoder for stylized speech synthesis of short video bloggers. Then, the stylized voice can be synthesized by passing it through the Tacotron model. Figure 3 The lower layer shows a smaller fluctuation in the sound signal graph representing the average of the Big Five personality traits, while the upper layer shows a larger fluctuation in the sound signal graph representing increased extroversion and openness. It can be seen that increasing the openness and extroversion of the Big Five personality traits increases the number of vibrations, pitch, amplitude, and loudness, proving the feasibility and effectiveness of this method. In daily life, there are also differences between male and female voices; the fundamental frequency of female voices is generally higher than that of male voices. Figure 4 The left side shows a synthesized male voice, and the right side shows a synthesized female voice. As can be seen from the image, the fundamental frequency of the female voice is higher than that of the male voice.

Claims

1. A short video blogger stylized speech synthesis method based on attention variational autoencoder, characterized in that, Includes the following steps: S1. Construct a face-voice dataset, download the first impression dataset, and extract face images and corresponding voice sounds from the first impression video dataset; S2. Obtain the contour detection image of the key facial features. Input the face dataset image in step S1. Use the DeepFace multiple facial attribute feature extraction model to extract the emotion and Big Five personality attribute features of the face from the face dataset image in step S1. Use X-Vector to extract the voice attribute features from the voice dataset. S3. Construct a supervised attention mechanism network based on the β-VAE method, add an iAFF module between the encoder and decoder, and combine the attention module to establish a supervised attention mechanism network based on the β-VAE method; The speaker embedding features extracted by X-Vector are used to generate latent variables z through the encoder. The speaker attribute features extracted from the image are combined with the latent variables z, and then passed through the iAFF attention module before being sent to the decoder for decoding. After decoding by the decoder, the speaker embedding is generated. Add a condition to the β-VAE to synthesize stylized speech based on the attributes of the input. The output is based on the input. In the training set, there are data pairs (x, y), where y is the input, i.e., the condition, and x is the expected output. The model loss function is as follows: L cβvae −β·D KL (q(z|x,y)||p(z|y))+E q(z|x,y) (log(p(x|z,y))) Among them, D KL It is the KL divergence, which makes the latent variables generated by the encoder conform to the standard normal distribution as much as possible. p represents the true vector, q represents the result vector, z represents the latent vector, and E q(z|x,y) Representing the reconstruction loss, it is the MSE loss between the vector decoded by the decoder and the input vector, reflecting the difference between the VAE-generated result and the input. The corresponding goal is to make the VAE-generated result as similar to the input as possible. S4. Using the facial attribute features and corresponding voice features obtained in step S2 as the training dataset, train the supervised attention mechanism network based on the β-VAE method constructed in step S3 using Adam. S5. Input attribute feature values. Based on the facial attribute features and corresponding voice features obtained in step S2, input them into the supervised attention mechanism network based on the β-VAE method trained in step S4 to generate speech embedding. S6. The stylized sound obtained in step S5 is embedded into the multi-speaker Tacotron2 model for speech synthesis to synthesize the sound.

2. The method for stylized speech synthesis of short video bloggers based on attention variational autoencoders according to claim 1, characterized in that, Step S2, obtaining facial attribute features and corresponding voice features, includes the following steps: S2.1 Extracting sound features through X-Vector: X-Vector accepts input of arbitrary length and transforms it into a feature representation of fixed length; S2.2 Feature Extraction Based on Face Shape: The MTCNN face detection model is used to extract face markers to calculate face shape. The MTCNN network structure is a three-tiered cascade network consisting of P-Net, R-Net, and O-Net layers. In the MTCNN three-layer network structure, the network output includes whether it is a face, bounding box regression values, and face feature point coordinates. The final loss function of the model can be specifically expressed as: L mtcnn =a det L det +a box L box +a landmarks L lamdmarks Where, α det L det The face classification loss uses cross-entropy as the loss function, α. box L box The bounding box regression loss uses Euclidean distance as the loss function, α. landmarks L landmarks The loss is based on facial feature point coordinates, using Euclidean distance as the loss function. S2.3 Extracts features based on emotion, age, and gender, and uses the DeepFace framework to extract multiple attribute feature values; S2.4 Based on the weight feature extraction, the BMI weight prediction model is used to predict the weight attribute features of a person from the image; S2.5 extracts features based on the Big Five personality traits, using a dataset of short video bloggers that capture first impressions. This dataset contains annotations of the Big Five personality traits, and the attribute features of the Big Five personality traits are used directly.

3. The short video blogger stylized speech synthesis method based on attention variational autoencoder according to claim 1, characterized in that, Step S6 uses the Tacotron2 model to embed the speaker stylized features obtained in step S5 for speech synthesis. The generated speaker feature embeddings are synthesized by the Tacotron model to form the sound we hear. Different styles of speech are synthesized according to the attribute labels.

Images