A deep learning-based speech synthesis method for intelligent robots

Abstract

The present invention relates to the technical field of intelligent speech synthesis, and discloses a method for intelligent robot speech synthesis based on deep learning, comprising: step 1, text processing: receiving input text, and performing word segmentation, part-of-speech tagging, and phoneme tagging on the text to obtain a text embedding sequence for subsequent sentiment analysis steps; step 2, sentiment analysis: based on the generated text embedding sequence, using a BERT-based sentiment classification model to perform sentiment analysis on the text to obtain a text sentiment embedding for multimodal sentiment fusion; step 3, receiving visual input, extracting visual sentiment embedding through a convolutional neural network, and inputting the visual sentiment embedding into the multimodal sentiment fusion. The deep learning model can learn speech data, extract natural phonemes, intonation, and rhythm features in the speech, generate a natural and fluent speech synthesis effect, and can be close to human speech in terms of speech speed, intonation, and sound quality, thereby enhancing the user's auditory experience.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent speech synthesis, and in particular to a method for intelligent robot speech synthesis based on deep learning. Background Art

[0002] With the development of artificial intelligence and voice interaction technology, intelligent speech synthesis has become an important part of human-computer interaction and has been widely used in intelligent customer service, virtual assistants, navigation systems, entertainment equipment and other fields. However, traditional speech synthesis methods, such as splicing-based waveform synthesis and parametric speech synthesis, have limitations in naturalness, emotional expression and personalized customization, and cannot fully meet users' needs for high-quality voice interaction.

[0003] The main problems faced by traditional speech synthesis technology include:

[0004] Traditional speech synthesis technology is limited by the size of the corpus and the splicing technology. The synthesized speech lacks natural coherence and fluency, and the intonation is stiff, which can easily make people feel "machine-like".

[0005] Voice interaction requires information transfer and the ability to convey emotions through speech to achieve a richer user experience. However, traditional synthesis methods lack the ability to model emotional features and are unable to generate emotionally charged speech, which impacts the user experience.

[0006] Different users and application scenarios have different preferences and requirements for voice timbre, intonation, and other aspects. Traditional speech synthesis technology cannot flexibly generate personalized voices and cannot meet users' diverse needs for synthesized speech.

[0007] Traditional speech synthesis systems rely on corpora of specific languages ​​and dialects and require specialized training for different languages ​​or dialects. Due to language differences and the diversity of speech features, traditional methods face technical difficulties when dealing with the synthesis of multiple languages ​​and dialects. The synthesis effect is poor, and there is a lack of versatility and adaptability, which limits the scope of application of the system.

[0008] In recent years, advances in deep learning technology have brought new solutions to the field of speech synthesis. Deep learning can automatically extract speech features such as phonemes, intonation, and rhythm by training neural networks with data, and has achieved significant improvements in speech synthesis effects. Speech synthesis methods based on deep learning can generate natural and fluent speech, support emotional expression, have personalized customization capabilities, and can adapt to the needs of multiple languages ​​and dialects. However, existing deep learning speech synthesis technology has room for optimization in terms of model complexity and computing resource requirements, and different application scenarios have different requirements for speech synthesis quality, and further innovation and improvement are still needed.

[0009] Therefore, those skilled in the art provide an intelligent robot speech synthesis method based on deep learning to solve the above-mentioned problems. Summary of the Invention

[0010] In view of the shortcomings of the existing technology, the present invention provides an intelligent robot speech synthesis method based on deep learning to solve the problems raised in the above background technology.

[0011] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for intelligent robot speech synthesis based on deep learning, comprising:

[0012] Step 1: Text processing: Receive input text and perform word segmentation, part-of-speech tagging, and phoneme tagging on the text to obtain a text embedding sequence for subsequent sentiment analysis steps.

[0013] Step 2: Sentiment analysis: Based on the generated text embedding sequence, use the BERT-based sentiment classification model to perform sentiment analysis on the text to obtain text sentiment embedding for multimodal sentiment fusion;

[0014] Step 3: Visual emotion information extraction: Receive visual input, extract visual emotion embedding through convolutional neural network, and input the visual emotion embedding into multimodal emotion fusion;

[0015] Step 4: Multimodal emotion fusion: Based on the obtained text emotion embedding and visual emotion embedding, the Transformer model is used to fuse the text emotion embedding and the visual emotion embedding to obtain a multimodal emotion embedding, which is used as the input for emotion control.

[0016] Step 5, emotion control interface: Provide a user interface to allow users to adjust the weight coefficients in the multimodal emotion embedding to generate the final emotion embedding;

[0017] Step 6: The speech synthesis model generates a spectrogram: Based on the obtained text embedding sequence and the final emotion embedding, the text embedding sequence and the final emotion embedding are input into the VI TS variational inference model to generate a spectrogram;

[0018] Step 7: The vocoder generates an audio waveform: The generated spectrogram is input into the HiFi-GAN vocoder, and the audio waveform is generated by the GAN to obtain the audio output of speech synthesis;

[0019] Step 8: Data enhancement and model optimization: During the training process, self-supervised learning and semi-supervised learning are used for data enhancement and model optimization, including:

[0020] Extract unlabeled data features by comparing predictive coding;

[0021] Semi-supervised training using the mean teacher model framework;

[0022] Generate diverse audio training data through data augmentation methods to optimize the model's generation performance in different scenarios.

[0023] Preferably, in step 2, the text sentiment embedding generation method includes:

[0024] Use the BERT-based sentiment classifier to perform sentiment analysis on the text and generate sentiment category C and sentiment intensity S;

[0025] Among them, the sentiment category C represents the sentiment label of the text, through the formula:

[0026] P(C|T)=softmax(W c BERT(T)+b c ),

[0027] Among them, W c represents the weight matrix of the classification layer, b c represents the bias term, T represents the input text, and BERT(T) represents the output after the input text T passes through the BERT model.

[0028] P(C|T) represents the probability of category C given the text T;

[0029] Emotional intensity S represents the intensity of emotional expression, through the formula:

[0030] S=W s BERT(T)+b s ,

[0031] Among them, W s represents the weight matrix for intensity prediction, b s Represents the bias term, and BERT(T) represents the output after the input text T passes through the BERT model.

[0032] Preferably, in step 3, the visual emotion embedding generation method includes:

[0033] Receive visual information I and extract visual emotion embedding E through the convolutional neural network model I ;

[0034] Among them, visual emotion embedding E I By formula:

[0035] E I =FC(CNN(I)),

[0036] Among them, CNN represents convolutional neural network, which is used to extract visual features, and FC represents fully connected layer.

[0037] Preferably, in step 4, the multimodal emotion fusion method includes:

[0038] Using Transformer model to fuse text sentiment embedding E T and visual emotion embedding E I , generate multimodal sentiment embedding E multi ;

[0039] Multimodal sentiment embedding E multi By formula:

[0040] E multi =Transformer([E T ;E I ]),

[0041] Among them, E T It means that the text and visual sentiment embeddings are concatenated and input into the Transformer.

[0042] Preferably, in step 5, the emotion control interface allows the user to adjust the emotion weight α to generate the final emotion embedding E final , E final Expressed as:

[0043] E final =E multi α+E T (1-α),

[0044] Among them, E multi represents multimodal sentiment embedding, E T represents text sentiment embedding, and α is the weight factor controlling text and visual sentiment.

[0045] Preferably, in step 6, the speech synthesis model generates a spectrogram M containing emotion control based on a variational inference model VITS, including:

[0046] Combine text embedding H and sentiment embedding E final Fusion generates sentiment semantic vector H.;

[0047] The spectrogram M is generated by variational inference, and the objective function is defined as:

[0048] logp(M|H.)≈E q(z|M) [logp(M|z,H.)]-D KL (q(z|M)||p(z|H.)),

[0049] Where z is the latent variable, q(z|M) represents the posterior distribution of the spectrogram M, p(z|H.) represents the prior distribution, and D KL is the KL divergence, Eq(z|M) represents the expectation about the distribution q(z|M), logp(M|H.) represents the logarithmic probability of observing M given H.,

[0050] logp(M|z,H.) represents the log probability of generating M given the latent variable z and the condition H.

[0051] Preferably, noise is added during the generation process, and a regularization term R(H′) is defined to make the model output more stable, wherein the regularization term R(H′) is expressed as:

[0052] R(H′)=λ∑ i ||h′ i || 2 ,

[0053] Among them, λ is the regularization coefficient, h′ i are the elements in the sentiment semantic vector.

[0054] Preferably, in step 7, the vocoder converts the generated spectrogram M into an audio waveform based on a HiFi-generative adversarial network, and the vocoder includes:

[0055] The adversarial training between the generator G and the discriminator D is controlled by the generative adversarial network loss, which is defined as:

[0056] L GAN =E M [logD(M)]+E G(M) [log(1-D(G(M)))],

[0057] Among them, D(M) represents the discriminant output of the real spectrogram M, G(M) represents the spectrogram generated by the generator, and L GAN is the adversarial loss of the generated adversarial network, E M [logD(M)] represents the expectation of the real data M, E G(M) [log(1-D(G(M)))] represents the expectation of generating the data G(M).

[0058] Preferably, in step 7, the feature matching loss L FM To further improve the audio fidelity, the feature matching loss is defined as:

[0059]

[0060] Among them, D (l) Represents the features of the discriminator layer l, N l Indicates the dimension of the l-th layer feature, L FM is the feature matching loss, L is the number of feature layers in the discriminator, and D(l) (M) is the feature representation of the discriminator D on the real data M at the lth layer, D (l) (G(M)) is the feature representation of the discriminator D on the generated data G(M) at the lth layer.

[0061] Preferably, in step 8, the self-supervised data enhancement uses contrastive predictive coding to extract unlabeled data features, and enhances the generalization ability of the model through contrastive learning. The contrastive predictive coding loss is defined as:

[0062]

[0063] Among them, L CPC To compare the loss function of predictive coding, T is the total number of time steps, K is the number of future time steps to be predicted, and x t+k is the feature vector of time step t+k, x j is the feature vector of negative samples, To calculate the context vector x at time step t t Its feature vector x at the future time step t+k t+k The inner product of is the normalization term.

[0064] The present invention provides a method for intelligent robot speech synthesis based on deep learning. It has the following beneficial effects:

[0065] 1. The present invention can learn voice data through a deep learning model, extract the natural phonemes, intonation and rhythm features in the voice, and generate a natural and smooth speech synthesis effect. It can be close to human speech in terms of speaking speed, intonation and sound quality, thereby enhancing the user's auditory experience.

[0066] 2. The present invention utilizes the adaptability of deep learning models to generate personalized speech synthesis effects according to different application scenarios or user needs. For example, by adjusting model parameters or input conditions to generate speech with specific emotions, gender or timbre, the preferences of different user groups can be met, giving the intelligent robot a unique personality.

[0067] 3. The present invention utilizes the deep learning model's ability to model emotional features, allowing the robot to incorporate emotional colors into the speech synthesis process, making the synthesized speech vivid and achieving a cheerful tone or a calm intonation. This can be achieved by adjusting the model's input parameters, thereby enhancing the emotional interaction effect between the robot and the user.

[0068] 4. The present invention supports multi-language and multi-dialect speech synthesis. Thanks to the deep learning model's training ability on multi-language data, it can achieve natural and smooth synthesis effects in Mandarin, English and dialects, which is convenient for application in intelligent robots in different language environments, expanding its scope of use and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 Flowchart of the present invention. DETAILED DESCRIPTION

[0070] To help those skilled in the art understand the present invention, the following will provide a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only partial embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0071] The present invention is described in detail below with reference to the accompanying drawings:

[0072] Example:

[0073] Please see the attached Figure 1 , an embodiment of the present invention provides an intelligent robot speech synthesis method based on deep learning, comprising:

[0074] Step 1: Text processing: Receive input text and perform word segmentation, part-of-speech tagging, and phoneme tagging on the text to obtain a text embedding sequence for subsequent sentiment analysis steps.

[0075] Step 2: Sentiment analysis: Based on the generated text embedding sequence, use the BERT-based sentiment classification model to perform sentiment analysis on the text to obtain text sentiment embedding for multimodal sentiment fusion;

[0076] Step 3: Visual emotion information extraction: Receive visual input, extract visual emotion embedding through convolutional neural network, and input the visual emotion embedding into multimodal emotion fusion;

[0077] Step 4: Multimodal emotion fusion: Based on the obtained text emotion embedding and visual emotion embedding, the Transformer model is used to fuse the text emotion embedding and the visual emotion embedding to obtain a multimodal emotion embedding, which is used as the input for emotion control.

[0078] Step 5, emotion control interface: Provide a user interface to allow users to adjust the weight coefficients in the multimodal emotion embedding to generate the final emotion embedding;

[0079] Step 6: The speech synthesis model generates a spectrogram: Based on the obtained text embedding sequence and the final emotion embedding, the text embedding sequence and the final emotion embedding are input into the VI TS variational inference model to generate a spectrogram;

[0080] Step 7: The vocoder generates an audio waveform: The generated spectrogram is input into the HiFi-GAN vocoder, and the audio waveform is generated by the GAN to obtain the audio output of speech synthesis;

[0081] Step 8: Data enhancement and model optimization: During the training process, self-supervised learning and semi-supervised learning are used for data enhancement and model optimization, including:

[0082] Extract unlabeled data features by comparing predictive coding;

[0083] Semi-supervised training using the mean teacher model framework;

[0084] Generate diverse audio training data through data augmentation methods to optimize the model's generation performance in different scenarios.

[0085] Through multimodal emotion fusion and emotion control interfaces, synthesized speech can reflect a variety of emotional characteristics. BERT is used for text sentiment analysis, and convolutional neural networks are used to extract visual emotion information. This emotion information is integrated through the Transformer model, making the generated speech natural and fluent, capable of expressing rich emotional colors, making human-computer interaction more friendly and emotionally resonant.

[0086] The emotion control interface provides users with the function of adjusting emotion weights, allowing users to customize the emotional expression of speech in different situations and needs. Personalized emotion control improves the flexibility of speech synthesis, meets users' personalized needs for speech output, can adapt to diverse usage scenarios, and enhance the user experience.

[0087] The VI TS variational inference model is used to generate spectrograms, which are then passed through the HiFi-GAN vocoder to generate audio waveforms. This ensures that the generated speech sound is realistic, detailed, and clear. The VI TS model generates detailed spectral information, while the HiFi-GAN generates high-quality audio waveforms through generative adversarial network learning, reducing noise and distortion. The result is high-quality, low-noise synthesized speech.

[0088] During the training process, the present invention applies self-supervised and semi-supervised learning, extracting features through comparative predictive coding using unlabeled data. At the same time, a mean teacher model is used for semi-supervised training. Data augmentation and model optimization strategies effectively improve the generalization ability of the model in different situations, enabling it to generate high-quality speech output in a variety of application environments, reducing dependence on labeled data.

[0089] Through multimodal emotion fusion and data enhancement, this method is highly adaptable to multilingual, multi-dialect, and multi-modal emotion information, supporting cross-language and cross-cultural speech synthesis needs and expanding the applicability of intelligent robots in different language environments and application fields.

[0090] The introduction of self-supervised and semi-supervised learning techniques reduces dependence on labeled data. At the same time, by generating diversified training data through data augmentation, the training efficiency of the model is further improved, which can reduce the cost of obtaining labeled data and accelerate the model optimization process.

[0091] In summary, the method of the present invention has significantly improved the naturalness of speech, emotional expression, personalization, sound quality, adaptability and training efficiency. It can provide high-quality speech synthesis output for intelligent robots and meet users' needs for high-quality voice interaction in different situations.

[0092] In step 2, the text sentiment embedding generation method includes:

[0093] Use the BERT-based sentiment classifier to perform sentiment analysis on the text and generate sentiment category C and sentiment intensity S;

[0094] Among them, the sentiment category C represents the sentiment label of the text, through the formula:

[0095] P(C|T)=softmax(W c BERT(T)+b c ),

[0096] Among them, W c represents the weight matrix of the classification layer, b c represents the bias term, T represents the input text, and BERT(T) represents the output after the input text T passes through the BERT model.

[0097] P(C|T) represents the probability of category C given the text T;

[0098] Emotional intensity S represents the intensity of emotional expression, through the formula:

[0099] S=W s BERT(T)+b s ,

[0100] Among them, W s represents the weight matrix for intensity prediction, b s Represents the bias term, and BERT(T) represents the output after the input text T passes through the BERT model.

[0101] The BERT-based sentiment classifier can identify emotions in text and generate sentiment categories to clarify the type of emotion expressed in the text. This classification method can effectively identify emotions in the text and provide emotion labels for speech synthesis, helping to generate emotion-matched speech output, making the synthesized speech fit the text content and emotional needs.

[0102] The introduction of emotional intensity can refine the expression of emotions. By numerically quantifying the strength of emotions, it can add a sense of emotional depth to speech synthesis. The control of emotional intensity enables speech synthesis to express specific emotions and flexibly adjust between different intensities of emotions, making the generated speech realistic and expressive, and enriching the emotional depth of the speech output.

[0103] The dual control of emotion category and emotion intensity makes the synthesis system emotionally flexible and adaptable to different scenarios. For example, in customer service applications, a gentle tone and low emotion intensity can be determined, while in entertainment or gaming scenarios, strong emotion expression can be used. This tailors the needs of specific application scenarios and improves the effectiveness of human-computer interaction.

[0104] Through emotional labeling and intensity control, users can obtain natural and delicate emotional feedback during the interaction, making the robot-generated voice output more humane and interactive. Especially when the emotional intensity is controllable, users will get a real auditory experience, significantly improving user satisfaction and usage experience.

[0105] In summary, the text emotion embedding generation method brings delicate and flexible emotional expression to intelligent speech synthesis through dual control of emotion category and intensity, significantly improving the emotional authenticity and personalized adaptability of speech output, and can meet diverse user needs and application scenarios.

[0106] In step 3, the visual emotion embedding generation method includes:

[0107] Receive visual information I and extract visual emotion embedding E through the convolutional neural network model I ;

[0108] Among them, visual emotion embedding E I By formula:

[0109] E I =FC(CNN(I)),

[0110] Among them, CNN represents convolutional neural network, which is used to extract visual features, and FC represents fully connected layer.

[0111] By extracting visual emotion embedding, the system can combine the emotional features of text and visual information to achieve multimodal emotion fusion. Multimodal processing can fully understand and express emotions, enabling intelligent robots to make accurate and context-appropriate emotional expressions based on visual information in speech synthesis, providing a richer interactive experience.

[0112] Visual emotion embedding can capture subtle differences in emotional expression in images or videos, such as facial expressions, posture, and environmental emotions. It extracts visual features through a convolutional neural network and converts them into emotion embeddings through fully connected layers. This preserves the visual details and makes speech synthesis realistic and contextual.

[0113] Convolutional neural networks excel at extracting visual features and can identify complex visual emotional information. Through the visual emotion embedding extracted by convolutional neural networks, the system can effectively distinguish different visual emotions and generate matching voice emotions, enabling speech synthesis systems to recognize and express emotions, providing support for applications in different emotional scenarios.

[0114] Visual information can convey instantaneous emotional changes. The introduction of visual emotion embedding enables the system to dynamically adapt to the user's emotional changes. For example, by detecting changes in the user's facial expression through visual signals, the system can adjust the emotional expression of the voice in real time, making human-computer interaction more flexible and natural, and able to respond to user emotional changes in real time, improving the interactive experience.

[0115] Visual emotion embedding can incorporate the user's external emotional feedback into the speech synthesis process, making the generated speech personalized and interactive. For example, after seeing the user's facial expressions and body movements, the system can adjust the tone and emotional expression of the speech to produce speech output consistent with the user's emotions, enhancing the user's sense of participation.

[0116] In summary, the present invention realizes the fusion of multimodal emotional information through the generation method of visual emotion embedding, enhances the realism and richness of emotional expression, enables intelligent robots to present delicate, dynamic and personalized emotional effects in speech synthesis, and provides users with an immersive and interactive voice interaction experience.

[0117] In step 4, the multimodal emotion fusion method includes:

[0118] Using Transformer model to fuse text sentiment embedding E T and visual emotion embedding E I , generate multimodal sentiment embedding E multi ;

[0119] Multimodal sentiment embedding E multi By formula: E multi =Transformer([E T;E I ]),

[0120] Among them, E T It means that the text and visual sentiment embeddings are concatenated and input into the Transformer.

[0121] By using the Transformer model to fuse textual and visual sentiment embeddings, we can effectively integrate information from different modalities. The Transformer model excels at processing the interrelationships between multimodal data. Therefore, when fusing textual and visual sentiment embeddings, we can capture the deep connections between them, making the generated multimodal sentiment embedding more complete and expressive, and providing more accurate emotional information for subsequent speech synthesis.

[0122] Embedding the emotional features of text and visuals into a multimodal emotional space enables the system to express richer and more authentic emotions in speech synthesis. Through multimodal fusion, the speech synthesis system can simultaneously consider the emotional features of text and visual information, and the generated speech is more natural and coherent, further enhancing the user's auditory experience and interactive immersion.

[0123] The Transformer model's feature extraction and relationship modeling capabilities can fully utilize its self-attention mechanism when fusing text and visual emotional information, weighting important emotional information in different modalities, thereby improving the accuracy of emotion recognition and making the emotions of speech synthesis more in line with actual situations.

[0124] In summary, the use of the Transformer model for multimodal emotion fusion enables the intelligent robot speech synthesis system of the present invention to achieve efficient and accurate multimodal emotion integration. The generated speech synthesis is more natural, rich and in line with situational requirements in emotional expression. At the same time, it improves the real-time response capability and flexibility of the system, providing users with an immersive and personalized voice interaction experience.

[0125] In step 5, the emotion control interface allows users to adjust the emotion weight α to generate the final emotion embedding E final , E final Expressed as:

[0126] E final =E multi α+E T (1-α),

[0127] Among them, E multi represents multimodal sentiment embedding, E T represents text sentiment embedding, and α is the weight factor controlling text and visual sentiment.

[0128] Through the emotion control interface, users can flexibly adjust the weight of text and visual emotions according to actual needs and application scenarios, and personalize the emotional expression of speech, making the generated speech more in line with their personalized needs and improving user satisfaction.

[0129] The weight adjustment function of the emotion control interface enables the system to adapt to different scenarios. In some applications, such as education or entertainment, users may want visual emotions to dominate, while in applications such as news broadcasts or smart assistants, they may want to rely more on textual emotions.

[0130] In different situations, the emotional information conveyed by text and visual emotions is different. By adjusting the emotional weight α, users can control the fusion ratio of multimodal emotions, so that the final emotional embedding more accurately reflects the emotional characteristics of the text or vision, and generates emotional speech that meets actual needs, making emotional expression more accurate and delicate.

[0131] In summary, the emotion control interface provides users with the ability to autonomously regulate emotional output by adjusting the emotion weight α, bringing the advantages of personalized, diversified, accurate and dynamically adaptive emotional expression, significantly enhancing the adaptability of the speech synthesis system, making the generated speech more emotionally layered and personalized, and meeting users' needs for high-quality emotional voice interaction.

[0132] In step 6, the speech synthesis model generates a spectrogram M containing emotion control based on the variational inference model VITS, including:

[0133] Combine text embedding H and sentiment embedding E final Fusion generates sentiment semantic vector H.;

[0134] The spectrogram M is generated by variational inference, and the objective function is defined as:

[0135] logp(M|H.)≈E q(z|M) [logp(M|z,H.)]-D KL (q(z|M)||p(z|H.)),

[0136] Where z is the latent variable, q(z|M) represents the posterior distribution of the spectrogram M, p(z|H.) represents the prior distribution, and D KL is the KL divergence, E q(z|M) represents the expectation about the distribution q(z|M), logp(M|H.) represents the logarithmic probability of observing M given H.,

[0137] logp(M|z,H.) represents the logarithmic probability of generating M given the latent variable z and the condition H.;

[0138] Noise is added during the generation process, and the regularization term R(H′) is defined to make the model output more stable. The regularization term R(H′) is expressed as:

[0139] R(H′)=λ∑ i ||h′ i || 2 ,

[0140] Among them, λ is the regularization coefficient, h′ i are the elements in the sentiment semantic vector.

[0141] By fusing text embedding with emotion embedding to generate an emotion semantic vector, this method accurately reflects text content and emotion information in spectrogram generation. The emotion control method can reflect more delicate emotion levels and intensity in the generated speech, making the generated speech more natural and rich in emotion expression, meeting the needs of emotional speech in different scenarios.

[0142] A variational inference model is used to generate spectrograms. The model is optimized by comparing the posterior distribution with the prior distribution, making the generated spectrograms more realistic. Variational inference effectively improves the model's accuracy and consistency in generating spectrograms, making the generated audio close to natural human pronunciation in terms of sound quality, intonation, and emotional expression, significantly improving the quality of speech synthesis.

[0143] Adding noise and defining regularization terms during the generation process make the model output more stable. The regularization terms constrain the elements in the emotional semantic vector, effectively avoiding overfitting or fluctuation problems in the model during the generation process. This allows speech synthesis to maintain consistent sound quality and emotional expression in different situations, ensuring a more reliable user experience.

[0144] In summary, by generating a spectrogram containing emotion control through variational inference based on VITS, the present invention achieves a speech synthesis effect that is rich in emotion, high in sound quality, strong in stability, and has diversity. It has significantly improved the delicacy of emotional expression, generation quality, and model adaptability, providing users with a natural, reliable, and diverse emotional speech synthesis experience.

[0145] In step 7, the vocoder converts the generated spectrogram M into an audio waveform based on the HiFi-Generative Adversarial Network. The vocoder includes:

[0146] The adversarial training between the generator G and the discriminator D is controlled by the generative adversarial network loss, which is defined as:

[0147] L GAN =E M [logD(M)]+E G(M) [log(1-D(G(M)))],

[0148] Among them, D(M) represents the discriminant output of the real spectrogram M, G(M) represents the spectrogram generated by the generator, and L GAN is the adversarial loss of the generated adversarial network, E M [logD(M)] represents the expectation of the real data M, E G(M) [log(1-D(G(M)))] represents the expectation of generating data G(M);

[0149] In step 7, the feature matching loss L FM To further improve the audio fidelity, the feature matching loss is defined as:

[0150]

[0151] Among them, D (l) Represents the features of the discriminator layer l, N l Indicates the dimension of the l-th layer feature, L FM is the feature matching loss, L is the number of feature layers in the discriminator, and D (l) (M) is the feature representation of the discriminator D on the real data M at the lth layer, D (l) (G(M)) is the feature representation of the discriminator D on the generated data G(M) at the lth layer.

[0152] Through adversarial training of the generator and discriminator in a generative adversarial network, the generated audio waveform is made closer to real audio. The generative adversarial loss establishes an adversarial mechanism between real and generated audio, prompting the generator to continuously improve the naturalness and clarity of the audio, making the speech synthesis output closer to the effect of human speech and reducing the harshness of machine sound.

[0153] The introduction of feature matching loss effectively improves the fidelity of generated audio. By matching the feature representations of different feature layers of the discriminator, the generator can learn the characteristic structure of real audio at multiple levels, making the generated audio waveform richer and more realistic in detail, preserving the high-frequency information in the audio, and making the sound quality clearer and more delicate.

[0154] The Generative Adversarial Network loss suppresses distortion and artifacts in the generated audio through adversarial training. The discriminator continuously distinguishes the differences between the generated audio and the real audio, allowing the generator to adjust the generation process, reduce distortion in the audio, and generate purer and more natural speech, providing a better listening experience.

[0155] The feature matching loss provides additional training signals during the optimization process, allowing the generator to approach the features of real audio at multiple levels. The feature matching loss encourages the generator to match the features of real audio in the feature space. The constraint mechanism makes the training process more stable, avoids problems such as mode collapse in adversarial training, and improves the training efficiency and stability of the model.

[0156] In summary, through the generative adversarial loss and feature matching loss based on HiFi-GAN, the speech synthesis system of the present invention achieves high-fidelity, natural, stable and delicate audio generation effects. The vocoder method has significantly improved fidelity, clarity, stability and adaptability, providing users with a better speech synthesis experience.

[0157] In step 8, self-supervised data enhancement uses contrastive predictive coding to extract unlabeled data features and enhance the generalization ability of the model through contrastive learning. The contrastive predictive coding loss is defined as:

[0158]

[0159] Among them, L CPC To compare the loss function of predictive coding, T is the total number of time steps, K is the number of future time steps to be predicted, and x t+k is the feature vector of time step t+k, x j is the feature vector of negative samples, To calculate the context vector x at time step t t Its feature vector x at the future time step t+k t+k The inner product of is the normalization term.

[0160] Contrastive predictive coding uses contrastive learning to enable the model to learn the correlation between time steps without relying on labeled data. Self-supervised learning can improve the model's ability to understand unseen data, making the model more generalizable across different scenarios, adapting to diverse speech synthesis tasks, and reducing the risk of overfitting.

[0161] Comparative predictive coding can extract useful features from unlabeled data, thereby expanding the scope of model training data. Unlabeled data is generally easy to obtain and inexpensive. By extracting features through comparative predictive coding, we reduce dependence on labeled data, further improve model performance, and provide a rich training resource for speech synthesis.

[0162] The contrasting predictive coding loss function predicts the features of future time steps based on the current time step t. The similarity between the current context vector and future features is calculated through the inner product, which helps the model capture long-distance dependencies in the time series, making the generated speech more natural and fluent in temporal coherence, thereby improving the overall auditory quality of the speech.

[0163] In summary, using contrastive predictive coding as a means of self-supervised data enhancement can make full use of unlabeled data, significantly improve the model's generalization ability, detail restoration and noise resistance, and provide the speech synthesis system with richer, more natural and reliable speech output effects, with significant advantages in data utilization efficiency, training cost and generation quality.

[0164] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for intelligent robot speech synthesis based on deep learning, characterized in that: include: Step 1: Text processing: Receive input text and perform word segmentation, part-of-speech tagging, and phoneme tagging on the text to obtain a text embedding sequence for subsequent sentiment analysis steps. Step 2: Sentiment analysis: Based on the generated text embedding sequence, use the BERT-based sentiment classification model to perform sentiment analysis on the text to obtain text sentiment embedding for multimodal sentiment fusion; Step 3: Visual emotion information extraction: Receive visual input, extract visual emotion embedding through convolutional neural network, and input the visual emotion embedding into multimodal emotion fusion; Step 4: Multimodal emotion fusion: Based on the obtained text emotion embedding and visual emotion embedding, the Transformer model is used to fuse the text emotion embedding and the visual emotion embedding to obtain a multimodal emotion embedding, which is used as the input for emotion control. Step 5, emotion control interface: Provide a user interface to allow users to adjust the weight coefficients in the multimodal emotion embedding to generate the final emotion embedding; Step 6: The speech synthesis model generates a spectrogram: Based on the obtained text embedding sequence and the final emotion embedding, the text embedding sequence and the final emotion embedding are input into the VITS variational inference model to generate a spectrogram M. The speech synthesis model is based on the variational inference model VITS, generating a spectrogram M containing emotion control, including: Combine text embedding H and sentiment embedding E final Fusion generates sentiment semantic vector H.; The spectrogram M is generated by variational inference, and the objective function is defined as: logp(M|H.)≈E q(z|M )[logp(M|z,H.)]-D KL (q(z|M)||p(z|H.)), Where z is the latent variable, q(z|M) represents the posterior distribution of the spectrogram M, p(z|H.) represents the prior distribution, and D KL is the KL divergence, E q(z|M) represents the expectation about the distribution q(z|M), logp(M|H.) represents the logarithmic probability of observing M given H., logp(M|z,H.) represents the logarithmic probability of generating M given the latent variable z and the condition H.; Noise is added during the generation process, and the regularization term R(H′) is defined to make the model output more stable. The regularization term R(H′) is expressed as: R(H′)=λ∑ i ||h′ i || 2 , Among them, λ is the regularization coefficient, h′ i are the elements in the sentiment semantic vector; Step 7: The vocoder generates an audio waveform: The generated spectrogram is input into the HiFi-GAN vocoder, and the audio waveform is generated by the GAN to obtain the audio output of speech synthesis; The vocoder is based on HiFi-Generative Adversarial Network, which converts the generated spectrogram M into an audio waveform. The vocoder includes: The adversarial training between the generator G and the discriminator D is controlled by the generative adversarial network loss, which is defined as: L GAN =E M [logD(M)]+E G(M )[log(1-D(G(M)))], Among them, D(M) represents the discriminant output of the real spectrogram M, G(M) represents the spectrogram generated by the generator, and L GAN is the adversarial loss of the generated adversarial network, E M [logD(M)] represents the expectation of the real data M, E G(M) [log(1-D(G(M)))] represents the expectation of generating data G(M); Through feature matching loss L FM To further improve the audio fidelity, the feature matching loss is defined as: Among them, D (l) Represents the features of the discriminator layer l, N l Indicates the dimension of the l-th layer feature, L FM is the feature matching loss, L is the number of feature layers in the discriminator, and D (l) (M) is the feature representation of the discriminator D on the real data M at the lth layer, D (l) (G(M)) is the feature representation of the discriminator D on the generated data G(M) at the lth layer; Step 8: Data enhancement and model optimization: During the training process, self-supervised learning and semi-supervised learning are used for data enhancement and model optimization, including: Extract unlabeled data features by comparing predictive coding; Semi-supervised training using the mean teacher model framework; Generate diverse audio training data through data augmentation methods to optimize the model's performance in different scenarios; Self-supervised data augmentation uses contrastive predictive coding to extract unlabeled data features and enhances the generalization ability of the model through contrastive learning. The contrastive predictive coding loss is defined as: Among them, L CPC To compare the loss function of predictive coding, T is the total number of time steps, K is the number of future time steps to be predicted, and x t+k is the feature vector of time step t+k, x j is the feature vector of negative samples, To calculate the context vector x at time step t t Its feature vector x at the future time step t+k t+k The inner product of is the normalization term.

2. The method for intelligent robot speech synthesis based on deep learning according to claim 1, characterized in that: In step 2, the text sentiment embedding generation method includes: Use the BERT-based sentiment classifier to perform sentiment analysis on the text and generate sentiment category C and sentiment intensity S; Among them, the sentiment category C represents the sentiment label of the text, through the formula: P(C|T)=softmax(W c ·BERT(T)+b c ), Among them, W c represents the weight matrix of the classification layer, b c represents the bias term, T represents the input text, and BERT(T) represents the output after the input text T passes through the BERT model. P(C|T) represents the probability of category C given the text T; Emotional intensity S represents the intensity of emotional expression, through the formula: S=W s ·BERT(T)+b s , Among them, W s represents the weight matrix for intensity prediction, b s Represents the bias term, and BERT(T) represents the output after the input text T passes through the BERT model.

3. The method for intelligent robot speech synthesis based on deep learning according to claim 1, characterized in that: In step 3, the visual emotion embedding generation method includes: Receive visual information I and extract visual emotion embedding E through the convolutional neural network model I ; Among them, visual emotion embedding E I By formula: E I =FC(CNN(I)), Among them, CNN represents convolutional neural network, which is used to extract visual features, and FC represents fully connected layer.

4. The method for intelligent robot speech synthesis based on deep learning according to claim 1, characterized in that: In step 4, the multimodal emotion fusion method includes: Using Transformer model to fuse text sentiment embedding E T and visual emotion embedding E I , generate multimodal sentiment embedding E multi ; Multimodal sentiment embedding E multi By formula: E multi =Transformer([E T ;E I ]), Among them, E T It means that the text and visual sentiment embeddings are concatenated and input into the Transformer.

5. The method for intelligent robot speech synthesis based on deep learning according to claim 1, characterized in that: In step 5, the emotion control interface allows the user to adjust the emotion weight α to generate the final emotion embedding E final , E final Expressed as: E final =E multi α+E T (1-α), Among them, E multi represents multimodal sentiment embedding, E T represents text sentiment embedding, and α is the weight factor controlling text and visual sentiment.

Images