Intelligent robot speech synthesis method based on deep learning

Abstract

The invention relates to the technical field of intelligent speech synthesis, and discloses an intelligent robot speech synthesis method based on deep learning, which comprises the following steps: 1, text processing: receiving an input text, and carrying out word segmentation, part-of-speech tagging and phoneme tagging on the text to obtain a text embedding sequence for a subsequent sentiment analysis step; step 2, sentiment analysis: according to the generated text embedding sequence, performing sentiment analysis on the text by using a BERT-based sentiment classification model to obtain text sentiment embedding for multi-modal sentiment fusion; and step 3, receiving visual input, extracting visual emotion embedding through the convolutional neural network, and inputting the visual emotion embedding into multi-modal emotion fusion. Through the deep learning model, voice data can be learned, natural phonemes, intonation and rhythm characteristics in voice can be extracted, a natural and smooth voice synthesis effect is generated, the voice speed, the intonation and the tone quality are close to those of human voice, and the auditory experience of a user is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent speech synthesis, and in particular to an intelligent robot speech synthesis method based on deep learning. Background Art

[0002] With the development of artificial intelligence and voice interaction technology, intelligent speech synthesis has become an important part of human-computer interaction and has been widely used in intelligent customer service, virtual assistants, navigation systems, entertainment equipment and other fields. However, traditional speech synthesis methods, such as splicing-based waveform synthesis and parameterized speech synthesis, have limitations in naturalness, emotional expression and personalized customization, and cannot fully meet users' needs for high-quality voice interaction.

[0003] The main problems faced by traditional speech synthesis technology include:

[0004] Traditional speech synthesis technology is limited by the size of the corpus and the splicing technology. The synthesized speech lacks natural coherence and fluency, and the intonation is stiff, which makes people feel that it is "machine-like".

[0005] Voice interaction requires information transfer and the expression of emotions through voice to achieve a richer user experience. However, traditional synthesis methods lack the ability to model emotional features and cannot generate emotionally charged voices, which affects the user experience.

[0006] Different users and application scenarios have different preferences and requirements for voice timbre, intonation, etc. Traditional speech synthesis technology cannot flexibly generate personalized speech and cannot meet users' diverse needs for synthesized speech.

[0007] Traditional speech synthesis systems use corpora of specific languages ​​and dialects and require special training for different languages ​​or dialects. Due to language differences and the diversity of speech features, traditional methods have technical difficulties in dealing with the synthesis of multiple languages ​​and multiple dialects, and the synthesis effect is poor. They also lack versatility and adaptability, which limits the scope of application of the system.

[0008] In recent years, advances in deep learning technology have brought new solutions to the field of speech synthesis. Deep learning can automatically extract speech features such as phonemes, intonation, and rhythm by training neural networks through data, and has achieved significant improvements in speech synthesis effects. Speech synthesis methods based on deep learning can generate natural and fluent speech, support emotional expression, have personalized customization capabilities, and can adapt to the needs of multiple languages ​​and dialects. However, the existing deep learning speech synthesis technology has room for optimization in terms of model complexity and computing resource requirements, and different application scenarios have different requirements for speech synthesis quality, which still requires further innovation and improvement.

[0009] Therefore, those skilled in the art provide an intelligent robot speech synthesis method based on deep learning to solve the above-mentioned problems. Summary of the invention

[0010] In view of the deficiencies in the prior art, the present invention provides an intelligent robot speech synthesis method based on deep learning to solve the problems raised in the above background technology.

[0011] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for intelligent robot speech synthesis based on deep learning, comprising:

[0012] Step 1: Text processing: Receive input text, perform word segmentation, part-of-speech tagging and phoneme tagging on the text, and obtain a text embedding sequence for subsequent sentiment analysis steps;

[0013] Step 2: Sentiment analysis: Based on the generated text embedding sequence, use the BERT-based sentiment classification model to perform sentiment analysis on the text to obtain text sentiment embedding for multimodal sentiment fusion;

[0014] Step 3: Extract visual emotion information: Receive visual input, extract visual emotion embedding through convolutional neural network, and input the visual emotion embedding into multimodal emotion fusion;

[0015] Step 4: Multimodal emotion fusion: Based on the obtained text emotion embedding and the obtained visual emotion embedding, the Transformer model is used to fuse the text emotion embedding and the visual emotion embedding to obtain a multimodal emotion embedding, and the multimodal emotion embedding is used as the input of emotion control.

[0016] Step 5, emotion control interface: Provide a user interface to allow users to adjust the weight coefficients in the multimodal emotion embedding to generate the final emotion embedding;

[0017] Step 6: The speech synthesis model generates a spectrogram: Based on the obtained text embedding sequence and the final emotion embedding, the text embedding sequence and the final emotion embedding are input into the VI TS variational inference model to generate a spectrogram;

[0018] Step 7: The vocoder generates an audio waveform: the generated spectrogram is input into the HiFi-GAN vocoder, and the audio waveform is generated by the GAN to obtain the audio output of speech synthesis;

[0019] Step 8: Data enhancement and model optimization: During the training process, self-supervised learning and semi-supervised learning are used for data enhancement and model optimization, including:

[0020] Extract unlabeled data features by comparing predictive coding;

[0021] Semi-supervised training using the mean teacher model framework;

[0022] Generate diverse audio training data through data augmentation methods to optimize the model's generation performance in different scenarios.

[0023] Preferably, in step 2, the text sentiment embedding generation method includes:

[0024] Use the BERT-based sentiment classifier to perform sentiment analysis on the text and generate sentiment category C and sentiment intensity S;

[0025] Among them, the sentiment category C represents the sentiment label of the text, through the formula:

[0026] P(C|T)=softmax(W c BERT(T)+b c ),

[0027] Among them, W c represents the weight matrix of the classification layer, b c represents the bias term, T represents the input text, and BERT(T) represents the output after the input text T passes through the BERT model.

[0028] P(C|T) represents the probability of category C given the text T;

[0029] The emotional intensity S represents the intensity of emotional expression, through the formula:

[0030] S=W s BERT(T)+b s ,

[0031] Among them, W s represents the weight matrix for intensity prediction, b s represents the bias term, and BERT(T) represents the output after the input text T passes through the BERT model.

[0032] Preferably, in step 3, the visual emotion embedding generation method comprises:

[0033] Receive visual information I and extract visual emotion embedding E through the convolutional neural network model I ;

[0034] Among them, visual emotion embedding E I By formula:

[0035] E I =FC(CNN(I)),

[0036] Among them, CNN represents convolutional neural network, which is used to extract visual features, and FC represents fully connected layer.

[0037] Preferably, in step 4, the multimodal emotion fusion method includes:

[0038] Using Transformer model to fuse text sentiment embedding E T and visual emotion embedding E I , generate multimodal sentiment embedding E multi ;

[0039] Multimodal sentiment embedding multi By formula:

[0040] E multi = Transformer([E T ; E I ]),

[0041] Among them, E T It means that the text and visual sentiment embeddings are concatenated and input into the Transformer.

[0042] Preferably, in step 5, the emotion control interface allows the user to adjust the emotion weight α to generate the final emotion embedding E final , E final It is expressed as:

[0043] E final =E multi α+E T (1-α),

[0044] Among them, E multi represents multimodal sentiment embedding, E T represents text sentiment embedding, and α is the weight factor controlling text and visual sentiment.

[0045] Preferably, in step 6, the speech synthesis model generates a spectrogram M containing emotion control based on a variational inference model VITS, including:

[0046] Embedding text H and sentiment E final Fusion generates sentiment semantic vector H.;

[0047] The spectrogram M is generated by variational inference, and the objective function is defined as:

[0048] logp(M|H.)≈E q(z|M) [logp(M|z,H.)]-D KL (q(z|M)||p(z|H.)),

[0049] Among them, z is the latent variable, q(z|M) represents the posterior distribution of the spectrogram M, p(z|H.) represents the prior distribution, and D KL is the KL divergence, Eq(z|M) represents the expectation of the distribution q(z|M), logp(M|H.) represents the logarithmic probability of observing M given H.,

[0050] logp(M|z,H.) represents the log probability of generating M given the latent variable z and condition H.

[0051] Preferably, noise is added in the generation process, and a regularization term R(H′) is defined to make the model output more stable, wherein the regularization term R(H′) is expressed as:

[0052] R(H′)=λ∑ i ||h′ i || 2 ,

[0053] Among them, λ is the regularization coefficient, h′ i are the elements in the sentiment semantic vector.

[0054] Preferably, in step 7, the vocoder converts the generated spectrogram M into an audio waveform based on the HiFi-generative adversarial network, and the vocoder comprises:

[0055] The adversarial training between the generator G and the discriminator D is controlled by the generative adversarial network loss, which is defined as:

[0056] L GAN =E M [logD(M)]+E G(M) [log(1-D(G(M)))],

[0057] Among them, D(M) represents the discriminant output of the real spectrogram M, G(M) represents the spectrogram generated by the generator, and L GAN To generate the adversarial loss of the adversarial network, E M [logD(M)] represents the expectation of the real data M, E G(M) [log(1-D(G(M)))] represents the expectation of generating the data G(M).

[0058] Preferably, in step 7, the feature matching loss L FM To further improve the audio fidelity, the feature matching loss is defined as:

[0059]

[0060] Among them, D (l) represents the features of the discriminator layer l, N l Indicates the dimension of the l-th layer feature, L FM is the feature matching loss, L is the number of feature layers in the discriminator, and D(l) (M) is the feature representation of the discriminator D for the real data M at the lth layer, D (l) (G(M)) is the feature representation of the discriminator D on the generated data G(M) at the lth layer.

[0061] Preferably, in step 8, the self-supervised data enhancement uses contrastive predictive coding to extract unlabeled data features, and enhances the generalization ability of the model through contrastive learning. The contrastive predictive coding loss is defined as:

[0062]

[0063] Among them, L CPC To compare the loss function of predictive coding, T is the total number of time steps, K is the number of future time steps to be predicted, and x t+k is the feature vector at time step t+k, x j is the feature vector of negative samples, To calculate the context vector x at time step t t Its feature vector x at the future time step t+k t+k The inner product of is the normalization term.

[0064] The present invention provides a method for intelligent robot speech synthesis based on deep learning. It has the following beneficial effects:

[0065] 1. The present invention can learn voice data through a deep learning model, extract natural phonemes, intonation and rhythm features in the voice, generate a natural and smooth speech synthesis effect, and can be close to human voice in terms of speech speed, intonation and sound quality, thereby enhancing the user's auditory experience.

[0066] 2. The present invention utilizes the adaptability of deep learning models to generate personalized speech synthesis effects according to different application scenarios or user needs. For example, by adjusting model parameters or input conditions to generate speech with specific emotions, gender or timbre, the preferences of different user groups can be met, giving the intelligent robot a unique personality.

[0067] 3. The present invention utilizes the deep learning model's ability to model emotional features, allowing the robot to incorporate emotional colors into the speech synthesis process, making the synthesized speech vivid and achieving a cheerful tone or a calm intonation. This can be achieved by adjusting the model's input parameters, thereby enhancing the emotional interaction effect between the robot and the user.

[0068] 4. The present invention supports multi-language and multi-dialect speech synthesis. Thanks to the training ability of the deep learning model on multi-language data, it can achieve natural and smooth synthesis effects in Mandarin, English and dialects, which is convenient for application in intelligent robots in different language environments, expanding its scope of use and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0070] In order to make the technical personnel in the technical field understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in combination with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a partial embodiment of the present invention, not a complete embodiment. Based on the embodiment of the present invention, other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.

[0071] The present invention is described in detail below in conjunction with the accompanying drawings:

[0072] Example:

[0073] Please refer to the attached Figure 1 , an embodiment of the present invention provides an intelligent robot speech synthesis method based on deep learning, comprising:

[0074] Step 1: Text processing: Receive input text, perform word segmentation, part-of-speech tagging and phoneme tagging on the text, and obtain a text embedding sequence for subsequent sentiment analysis steps;

[0075] Step 2: Sentiment analysis: Based on the generated text embedding sequence, use the BERT-based sentiment classification model to perform sentiment analysis on the text to obtain text sentiment embedding for multimodal sentiment fusion;

[0076] Step 3: Extract visual emotion information: Receive visual input, extract visual emotion embedding through convolutional neural network, and input the visual emotion embedding into multimodal emotion fusion;

[0077] Step 4: Multimodal emotion fusion: Based on the obtained text emotion embedding and the obtained visual emotion embedding, the Transformer model is used to fuse the text emotion embedding and the visual emotion embedding to obtain a multimodal emotion embedding, and the multimodal emotion embedding is used as the input of emotion control;

[0078] Step 5, emotion control interface: Provide a user interface to allow users to adjust the weight coefficients in the multimodal emotion embedding to generate the final emotion embedding;

[0079] Step 6: The speech synthesis model generates a spectrogram: Based on the obtained text embedding sequence and the final emotion embedding, the text embedding sequence and the final emotion embedding are input into the VI TS variational inference model to generate a spectrogram;

[0080] Step 7: The vocoder generates an audio waveform: the generated spectrogram is input into the HiFi-GAN vocoder, and the audio waveform is generated by the GAN to obtain the audio output of speech synthesis;

[0081] Step 8: Data enhancement and model optimization: During the training process, self-supervised learning and semi-supervised learning are used for data enhancement and model optimization, including:

[0082] Extract unlabeled data features by comparing predictive coding;

[0083] Semi-supervised training using the mean teacher model framework;

[0084] Generate diverse audio training data through data augmentation methods to optimize the model's generation performance in different scenarios.

[0085] Through multimodal emotion fusion and emotion control interface, the synthesized speech can reflect a variety of emotional characteristics. BERT is used for text emotion analysis, and convolutional neural network is used to extract visual emotion information. The emotion information is fused through the Transformer model, so that the generated speech is natural and fluent, and can express rich emotional colors, making human-computer interaction more friendly and emotionally resonant.

[0086] The emotion control interface provides users with the function of adjusting emotion weights, allowing users to customize the emotional expression of voice in different situations and needs. Personalized emotion control improves the flexibility of speech synthesis, meets users' personalized needs for voice output, can adapt to diversified usage scenarios, and improves users' experience.

[0087] The VI TS variational inference model is used to generate the spectrogram, and then the H i F i-GAN vocoder is used to generate the audio waveform, so that the generated speech sound quality is realistic, delicate and clear. The VI TS model can generate detailed spectral information, and H i F i-GAN generates high-quality audio waveforms through the learning of generative adversarial networks, reducing noise and distortion, so that the final synthesized speech has the characteristics of high sound quality and low noise;

[0088] During the training process, the present invention applies self-supervised and semi-supervised learning, uses unlabeled data to extract features through comparative predictive coding, and uses the mean teacher model for semi-supervised training. The data enhancement and model optimization strategies effectively improve the generalization ability of the model in different situations, so that it can generate high-quality speech output in a variety of application environments and reduce dependence on labeled data.

[0089] Through multimodal emotion fusion and data enhancement, this method is highly adaptable to multi-language, multi-dialect and different modal emotion information, can support cross-language and cross-cultural speech synthesis needs, and expand the applicability of intelligent robots in different language environments and application fields;

[0090] The introduction of self-supervised and semi-supervised learning techniques reduces the dependence on labeled data. At the same time, by generating diversified training data through data enhancement, the training efficiency of the model is further improved, which can reduce the cost of obtaining labeled data and accelerate the model optimization process.

[0091] In summary, the method of the present invention has significant improvements in the naturalness of speech, emotional expression, personalization, sound quality, adaptability and training efficiency, and can provide high-quality speech synthesis output for intelligent robots to meet users' needs for high-quality voice interaction in different situations.

[0092] In step 2, the text sentiment embedding generation method includes:

[0093] Use the BERT-based sentiment classifier to perform sentiment analysis on the text and generate sentiment category C and sentiment intensity S;

[0094] Among them, the sentiment category C represents the sentiment label of the text, through the formula:

[0095] P(C|T)=softmax(W c BERT(T)+b c ),

[0096] Among them, W c represents the weight matrix of the classification layer, b c represents the bias term, T represents the input text, and BERT(T) represents the output after the input text T passes through the BERT model.

[0097] P(C|T) represents the probability of category C given the text T;

[0098] The emotional intensity S represents the intensity of emotional expression, through the formula:

[0099] S=W s BERT(T)+b s ,

[0100] Among them, W s represents the weight matrix for intensity prediction, b s represents the bias term, and BERT(T) represents the output after the input text T passes through the BERT model.

[0101] The BERT-based sentiment classifier can identify sentiment in text and generate sentiment categories to clarify the type of sentiment expressed in the text. The classification method can effectively identify the sentiment in the text, provide sentiment labels for speech synthesis, and help generate sentiment-matching speech output, so that the synthesized speech fits the text content and emotional needs.

[0102] The introduction of emotion intensity can refine the expression of emotion. By numerically quantifying the intensity of emotion, the emotion level can be increased for speech synthesis. The control of emotion intensity enables speech synthesis to express specific emotion and can flexibly adjust between different intensities of the emotion, making the generated speech realistic and expressive, and enriching the emotion level of speech output.

[0103] The dual control of emotion category and emotion intensity makes the synthesis system emotionally flexible and can adapt to different scene requirements. For example, in customer service applications, a gentle tone and low emotion intensity can be determined, while in entertainment or game scenarios, strong emotion expression can be used to meet the needs of specific application scenarios and improve the effect of human-computer interaction.

[0104] Through emotion labeling and intensity control, users can obtain natural and delicate emotional feedback during the interaction, making the robot-generated voice output more humane and interactive. Especially when the emotion intensity is controllable, users will get a real auditory experience, significantly improving user satisfaction and usage experience.

[0105] In summary, the text emotion embedding generation method brings delicate and flexible emotional expression to intelligent speech synthesis through dual control of emotion category and intensity, significantly improves the emotional authenticity and personalized adaptability of speech output, and can meet diverse user needs and application scenarios.

[0106] In step 3, the visual emotion embedding generation method includes:

[0107] Receive visual information I and extract visual emotion embedding E through the convolutional neural network model I ;

[0108] Among them, visual emotion embedding E I By formula:

[0109] E I =FC(CNN(I)),

[0110] Among them, CNN represents convolutional neural network, which is used to extract visual features, and FC represents fully connected layer.

[0111] By extracting visual emotion embedding, the system can combine the emotional features of text and visual information to achieve multimodal emotion fusion. Multimodal processing can fully understand and express emotions, so that intelligent robots can make accurate and scene-appropriate emotional expressions based on visual information in speech synthesis, providing a richer interactive experience.

[0112] Visual emotion embedding can capture the subtle differences in emotional expressions in images or videos, such as facial expressions, postures, and environmental emotions. It extracts visual features through convolutional neural networks and converts them into emotion embeddings through fully connected layers, retaining the visual details and making speech synthesis realistic and situational.

[0113] Convolutional neural networks excel in extracting visual features and can identify complex visual emotional information. Through the visual emotion embedding extracted by convolutional neural networks, the system can effectively distinguish different visual emotions and generate matching voice emotions, enabling the speech synthesis system to have the ability to recognize and express emotions, providing support for applications in different emotional scenarios.

[0114] Visual information can convey instantaneous emotional changes. The introduction of visual emotion embedding enables the system to dynamically adapt to the user's emotional changes. For example, by detecting changes in the user's facial expression through visual signals, the system can adjust the emotional expression of the voice in real time, making human-computer interaction more flexible and natural, and can respond to the user's emotional changes in real time, improving the interactive experience;

[0115] Visual emotion embedding can incorporate the user's external emotional feedback into the speech synthesis process, making the generated speech personalized and interactive. For example, after seeing the user's facial expressions and body movements, the system can adjust the voice tone and emotional expression to produce speech output consistent with the user's emotions, thereby enhancing the user's sense of participation.

[0116] In summary, the present invention realizes the fusion of multimodal emotional information through the generation method of visual emotion embedding, enhances the realism and richness of emotional expression, enables the intelligent robot to present delicate, dynamic and personalized emotional effects in speech synthesis, and provides users with an immersive and interactive voice interaction experience.

[0117] In step 4, the multimodal emotion fusion method includes:

[0118] Using Transformer model to fuse text sentiment embedding E T and visual emotion embedding E I , generate multimodal sentiment embedding E multi ;

[0119] Multimodal sentiment embedding multi By formula: E multi = Transformer([E T; E I ]),

[0120] Among them, E T It means that the text and visual sentiment embeddings are concatenated and input into the Transformer.

[0121] By using the Transformer model to fuse text emotion embedding and visual emotion embedding, information from different modalities can be effectively integrated. The Transformer model is good at processing the relationship between multimodal data. Therefore, when fusing text and visual emotion embedding, it can capture the deep relationship between them, making the generated multimodal emotion embedding more complete and expressive, and providing more accurate emotion information for subsequent speech synthesis;

[0122] The emotional features of text and vision are embedded in the multimodal emotional space, so that the system can reflect richer and more realistic emotional expressions in speech synthesis. Through multimodal fusion, the speech synthesis system can consider the emotional features of text and visual information at the same time, and the generated speech is more natural and coherent in emotion, further enhancing the user's auditory experience and interactive immersion.

[0123] The Transformer model's feature extraction and relationship modeling capabilities can make full use of its self-attention mechanism when fusing text and visual sentiment information, weighting important sentiment information in different modalities to improve the accuracy of sentiment recognition and make the sentiment of speech synthesis more in line with the actual situation.

[0124] In summary, the Transformer model is used for multimodal emotion fusion, so that the intelligent robot speech synthesis system of the present invention can achieve efficient and accurate multimodal emotion integration, and the generated speech synthesis is more natural, rich and in line with situational requirements in emotional expression. At the same time, the real-time response capability and flexibility of the system are improved, providing users with an immersive and personalized voice interaction experience.

[0125] In step 5, the emotion control interface allows users to adjust the emotion weight α to generate the final emotion embedding E final , E final It is expressed as:

[0126] E final =E multi α+E T (1-α),

[0127] Among them, E multi represents multimodal sentiment embedding, E T represents text sentiment embedding, and α is the weight factor controlling text and visual sentiment.

[0128] Through the emotion control interface, users can flexibly adjust the weights of text and visual emotions according to actual needs and application scenarios, and control the emotional expression of speech in a personalized way, so that the generated speech is more in line with their personalized needs and improve user satisfaction.

[0129] The weight adjustment function of the emotion control interface enables the system to adapt to different scene requirements. In some applications, such as education or entertainment, users may want visual emotions to dominate, while in applications such as news broadcasting or smart assistants, they may want to rely more on text emotions.

[0130] In different situations, the emotional information conveyed by text and visual emotions is different. By adjusting the emotional weight α, users can control the fusion ratio of multimodal emotions, so that the final emotional embedding can more accurately reflect the emotional characteristics of the text or vision, and generate emotional speech that meets actual needs, making emotional expression more accurate and delicate.

[0131] In summary, the emotion control interface provides users with the ability to autonomously regulate their emotion output by adjusting the emotion weight α, bringing the advantages of personalized, diversified, precise and dynamically adaptive emotion expression, significantly enhancing the adaptability of the speech synthesis system, making the generated speech more emotional and personalized, and meeting users' needs for high-quality emotional speech interaction.

[0132] In step 6, the speech synthesis model generates a spectrogram M containing emotion control based on the variational inference model VITS, including:

[0133] Embedding text H and sentiment E final Fusion generates sentiment semantic vector H.;

[0134] The spectrogram M is generated by variational inference, and the objective function is defined as:

[0135] logp(M|H.)≈E q(z|M) [logp(M|z,H.)]-D KL (q(z|M)||p(z|H.)),

[0136] Among them, z is the latent variable, q(z|M) represents the posterior distribution of the spectrogram M, p(z|H.) represents the prior distribution, and D KL is the KL divergence, E q(z|M) represents the expectation of the distribution q(z|M), logp(M|H.) represents the logarithmic probability of observing M given H.,

[0137] logp(M|z, H.) represents the logarithmic probability of generating M given the latent variable z and condition H.;

[0138] Noise is added during the generation process, and the regularization term R(H′) is defined to make the model output more stable. The regularization term R(H′) is expressed as:

[0139] R(H′)=λ∑ i ||h′ i || 2 ,

[0140] Among them, λ is the regularization coefficient, h′ i are the elements in the sentiment semantic vector.

[0141] By fusing text embedding with emotion embedding to generate an emotion semantic vector, this method can accurately reflect the text content and emotion information in the spectrogram generation. The emotion control method can reflect more delicate emotion levels and emotion intensity in the generated speech, making the generated speech more natural and rich in emotion expression, meeting the needs of emotional speech in different scenarios.

[0142] The variational inference model is used to generate the spectrogram. The model is optimized by comparing the posterior distribution and the prior distribution, making the generated spectrogram more realistic. Variational inference effectively improves the model's generation accuracy and consistency of the spectrogram, making the generated audio close to natural human pronunciation in terms of sound quality, intonation, and emotional expression, significantly improving the quality of speech synthesis.

[0143] Adding noise and defining regularization terms during the generation process makes the model output more stable. The regularization terms constrain the elements in the emotional semantic vector, effectively avoiding overfitting or fluctuation problems in the model during the generation process, so that speech synthesis can maintain consistent sound quality and emotional expression in different situations, ensuring a more reliable user experience.

[0144] In summary, by generating a spectrogram containing emotion control based on VITS variational inference, the present invention achieves a speech synthesis effect with rich emotions, high sound quality, strong stability and diversity, and has significant improvements in the delicacy of emotional expression, generation quality and model adaptability, providing users with a natural, reliable and diverse emotional speech synthesis experience.

[0145] In step 7, the vocoder converts the generated spectrogram M into an audio waveform based on the HiFi-Generative Adversarial Network. The vocoder includes:

[0146] The adversarial training between the generator G and the discriminator D is controlled by the generative adversarial network loss, which is defined as:

[0147] L GAN =E M [logD(M)]+E G(M) [log(1-D(G(M)))],

[0148] Among them, D(M) represents the discriminant output of the real spectrogram M, G(M) represents the spectrogram generated by the generator, and L GAN To generate the adversarial loss of the adversarial network, E M [logD(M)] represents the expectation of the real data M, E G(M) [log(1-D(G(M)))] represents the expectation of generating data G(M);

[0149] In step 7, the feature matching loss L FM To further improve the audio fidelity, the feature matching loss is defined as:

[0150]

[0151] Among them, D (l) represents the features of the discriminator layer l, N l Indicates the dimension of the l-th layer feature, L FM is the feature matching loss, L is the number of feature layers in the discriminator, and D (l) (M) is the feature representation of the discriminator D for the real data M at the lth layer, D (l) (G(M)) is the feature representation of the discriminator D on the generated data G(M) at the lth layer.

[0152] Through adversarial training of the generator and discriminator in the generative adversarial network, the generated audio waveform is closer to the real audio. The generative adversarial loss establishes an adversarial mechanism between the real audio and the generated audio, prompting the generator to continuously improve the naturalness and clarity of the audio, making the output of speech synthesis closer to the effect of human speech and reducing the stiffness of machine sound.

[0153] The introduction of feature matching loss effectively improves the fidelity of generated audio. By matching the feature representations of different feature layers of the discriminator, the generator can learn the feature structure of real audio at multiple levels, making the generated audio waveform richer and more realistic in details, retaining the high-frequency information in the audio, and making the sound quality clearer and more delicate.

[0154] The generative adversarial network loss suppresses distortion and artifacts in the generated audio through adversarial training. The discriminator continuously distinguishes the difference between the generated audio and the real audio, allowing the generator to adjust the generation process, reduce the distortion in the audio, and generate purer and more natural speech, providing a better listening experience.

[0155] The feature matching loss provides additional training signals in the optimization process, allowing the generator to approach the features of real audio at multiple levels. The feature matching loss encourages the generator to match the features of real audio in the feature space. The constraint mechanism makes the training process more stable, avoids problems such as mode collapse in adversarial training, and improves the training efficiency and stability of the model.

[0156] In summary, through the generative adversarial loss and feature matching loss based on HiFi-GAN, the speech synthesis system of the present invention achieves high-fidelity, natural, stable and delicate audio generation effects. The vocoder method has significant improvements in fidelity, clarity, stability and adaptability, providing users with a better speech synthesis experience.

[0157] In step 8, self-supervised data enhancement uses contrastive predictive coding to extract unlabeled data features and enhances the generalization ability of the model through contrastive learning. The contrastive predictive coding loss is defined as:

[0158]

[0159] Among them, L CPC To compare the loss function of predictive coding, T is the total number of time steps, K is the number of future time steps to be predicted, and x t+k is the feature vector at time step t+k, x j is the feature vector of negative samples, To calculate the context vector x at time step t t Its feature vector x at the future time step t+k t+k The inner product of is the normalization term.

[0160] Contrastive predictive coding uses contrastive learning to allow the model to learn the correlation between time steps without relying on labeled data. Self-supervised learning can improve the model's ability to understand unseen data, making the model more generalizable in different scenarios, adapting to a variety of speech synthesis tasks, and reducing the risk of overfitting.

[0161] The comparative predictive coding method can extract useful features from unlabeled data and expand the training data range of the model. Unlabeled data is usually easy to obtain and low-cost. By extracting features through comparative predictive coding, the dependence on labeled data is reduced, the performance of the model is further improved, and rich training resources are provided for speech synthesis.

[0162] The contrast prediction coding loss function predicts the features of future time steps based on the current time step t, and calculates the similarity between the current context vector and future features through the inner product, which helps the model capture long-distance dependencies in the time series, making the generated speech more natural and fluent in time coherence, thereby improving the overall auditory quality of the speech.

[0163] In summary, by using contrastive predictive coding as a means of self-supervised data enhancement and making full use of unlabeled data, the model's generalization ability, detail restoration and noise resistance can be significantly improved, providing the speech synthesis system with richer, more natural and reliable speech output effects, with significant advantages in data utilization efficiency, training cost and generation quality.

[0164] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for intelligent robot speech synthesis based on deep learning, characterized in that: include: Step 1: Text processing: Receive input text, perform word segmentation, part-of-speech tagging and phoneme tagging on the text, and obtain a text embedding sequence for subsequent sentiment analysis steps; Step 2: Sentiment analysis: Based on the generated text embedding sequence, use the BERT-based sentiment classification model to perform sentiment analysis on the text to obtain text sentiment embedding for multimodal sentiment fusion; Step 3: Extract visual emotion information: Receive visual input, extract visual emotion embedding through convolutional neural network, and input the visual emotion embedding into multimodal emotion fusion; Step 4: Multimodal emotion fusion: Based on the obtained text emotion embedding and the obtained visual emotion embedding, the Transformer model is used to fuse the text emotion embedding and the visual emotion embedding to obtain a multimodal emotion embedding, and the multimodal emotion embedding is used as the input of emotion control; Step 5, emotion control interface: Provide a user interface to allow users to adjust the weight coefficients in the multimodal emotion embedding to generate the final emotion embedding; Step 6: The speech synthesis model generates a spectrogram: Based on the obtained text embedding sequence and the final emotion embedding, the text embedding sequence and the final emotion embedding are input into the VITS variational inference model to generate a spectrogram; Step 7: The vocoder generates an audio waveform: the generated spectrogram is input into the HiFi-GAN vocoder, and the audio waveform is generated by the GAN to obtain the audio output of speech synthesis; Step 8: Data enhancement and model optimization: During the training process, self-supervised learning and semi-supervised learning are used for data enhancement and model optimization, including: Extract unlabeled data features by comparing predictive coding; Semi-supervised training using the mean teacher model framework; Generate diverse audio training data through data augmentation methods to optimize the model's generation performance in different scenarios.

2. The method for intelligent robot speech synthesis based on deep learning according to claim 1, characterized in that: In step 2, the text sentiment embedding generation method includes: Use the BERT-based sentiment classifier to perform sentiment analysis on the text and generate sentiment category C and sentiment intensity S; Among them, the sentiment category C represents the sentiment label of the text, through the formula: P(C|T)=softmax(W c ·BERT(T)+b c ), Among them, W c represents the weight matrix of the classification layer, b c represents the bias term, T represents the input text, and BERT(T) represents the output after the input text T passes through the BERT model. P(C|T) represents the probability of category C given the text T; The emotional intensity S represents the intensity of emotional expression, through the formula: S=W s ·BERT(T)+b s , Among them, W s represents the weight matrix for intensity prediction, b s represents the bias term, and BERT(T) represents the output after the input text T passes through the BERT model.

3. The method for intelligent robot speech synthesis based on deep learning according to claim 1, characterized in that: In step 3, the visual emotion embedding generation method includes: Receive visual information I and extract visual emotion embedding E through the convolutional neural network model I ; Among them, visual emotion embedding E I By formula: E I =FC(CNN(I)), Among them, CNN represents convolutional neural network, which is used to extract visual features, and FC represents fully connected layer.

4. The method for intelligent robot speech synthesis based on deep learning according to claim 1, characterized in that: In step 4, the multimodal emotion fusion method includes: Using Transformer model to fuse text sentiment embedding E T and visual emotion embedding E I , generate multimodal sentiment embedding E multi ; Multimodal sentiment embedding multi By formula: E multi = Transformer([E T ; E I ]), Among them, E T It means that the text and visual sentiment embeddings are concatenated and input into the Transformer.

5. The method for intelligent robot speech synthesis based on deep learning according to claim 1, characterized in that: In step 5, the emotion control interface allows the user to adjust the emotion weight α to generate the final emotion embedding E final , E final Expressed as: E final =E multi α+E T (1-α), Among them, E multi represents multimodal sentiment embedding, E T represents text sentiment embedding, and α is the weight factor controlling text and visual sentiment.

6. The method for intelligent robot speech synthesis based on deep learning according to claim 1, characterized in that: In step 6, the speech synthesis model generates a spectrogram M containing emotion control based on the variational inference model VITS, including: Embedding text H and sentiment E final Fusion generates sentiment semantic vector H.; The spectrogram M is generated by variational inference, and the objective function is defined as: logp(M|H.)≈E q(z|M) [logp(M|z,H.)]-D KL (q(z|M)||p(z|H.)), Among them, z is the latent variable, q(z|M) represents the posterior distribution of the spectrogram M, p(z|H.) represents the prior distribution, and D KL is the KL divergence, E q(z|M) represents the expectation of the distribution q(z|M), logp(M|H.) represents the logarithmic probability of observing M given H., logp(M|z,H.) represents the log probability of generating M given the latent variable z and condition H.

7. The method for intelligent robot speech synthesis based on deep learning according to claim 6, characterized in that: Noise is added in the generation process, and a regularization term R(H′) is defined to make the model output more stable, where the regularization term R(H′) is expressed as: R(H′)=λ∑ i ||h′ i || 2 , Among them, λ is the regularization coefficient, h′ i are the elements in the sentiment semantic vector.

8. The method for intelligent robot speech synthesis based on deep learning according to claim 1, characterized in that: In step 7, the vocoder converts the generated spectrogram M into an audio waveform based on the HiFi-generative adversarial network, and the vocoder includes: The adversarial training between the generator G and the discriminator D is controlled by the generative adversarial network loss, which is defined as: L GAN =E M [logD(M)]+E G(M) [log(1-D(G(M)))], Among them, D(M) represents the discriminant output of the real spectrogram M, G(M) represents the spectrogram generated by the generator, and L GAN To generate the adversarial loss of the adversarial network, E M [logD(M)] represents the expectation of the real data M, E G(M) [log(1-D(G(M)))] represents the expectation of generating the data G(M).

9. The method for intelligent robot speech synthesis based on deep learning according to claim 1, characterized in that: In step 7, the feature matching loss L FM To further improve the audio fidelity, the feature matching loss is defined as: Among them, D (l) represents the features of the discriminator layer l, N l Indicates the dimension of the l-th layer feature, L FM is the feature matching loss, L is the number of feature layers in the discriminator, and D (l) (M) is the feature representation of the discriminator D for the real data M at the lth layer, D (l) (G(M)) is the feature representation of the discriminator D on the generated data G(M) at the lth layer.

10. The method for intelligent robot speech synthesis based on deep learning according to claim 1, characterized in that: In step 8, the self-supervised data enhancement uses contrastive predictive coding to extract unlabeled data features, and enhances the generalization ability of the model through contrastive learning. The contrastive predictive coding loss is defined as: Among them, L CPC To compare the loss function of predictive coding, T is the total number of time steps, K is the number of future time steps to be predicted, and x t+k is the feature vector of time step t+k, x j is the feature vector of negative samples, To calculate the context vector x at time step t t Its feature vector x at the future time step t+k t+k The inner product of is the normalization term.

Images