Speech synthesis method and device based on hierarchical emotion distribution, equipment and medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a speech synthesis method based on hierarchical emotion distribution, which comprises the following steps: acquiring audio data and text transcription, segmenting the audio into phoneme, word and sentence level speech units, extracting acoustic features such as pitch, speech rate and rhythm, generating an emotion distribution vector by using a pre-training model, processing the text transcription to generate text embedding information, combining the emotion distribution vector, the text embedding information and the acoustic features, generating prosodic features through a variable converter, generating a mel-frequency spectrum graph based on the prosodic features by using a decoder, and converting the mel-frequency spectrum graph into a time-domain audio signal by using a vocoder and then outputting the time-domain audio signal. The audio data is hierarchically segmented into phoneme, word and sentence levels, the acoustic features, the emotion distribution vector and the text embedding information are combined, the prosodic features are generated by using the variable converter, and therefore the fine control of emotional expression is realized, and the naturalness and emotional expressiveness of speech synthesis are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a speech synthesis method and device based on hierarchical emotion distribution, equipment and storage medium. BACKGROUND

[0002] With the rapid development of financial services, user interaction experience is increasingly important in financial services. Intelligent customer service, virtual assistants and other intelligent interaction systems have been widely used in financial institutions to handle customer inquiries, business transactions and other related services. However, most of the current intelligent voice systems cannot express emotions naturally and accurately during the interaction process, resulting in a lack of affinity and naturalness in the interaction between customers and the system. As an important means of enhancing the expressiveness of intelligent interaction systems, emotion speech synthesis technology can make virtual assistants and intelligent customer service more humanized and enhance user experience by giving voice emotional color.

[0003] In the prior art, the adjustment of voice emotion often relies on pre-set emotion models, and the emotion intensity cannot be accurately adjusted according to actual needs. The customer needs in the financial field are complex and diverse, and the emotional expression needs to be highly accurate. However, the existing technology cannot achieve fine-grained control of emotions, especially in the financial business interaction, the emotional needs are difficult to respond effectively.

[0004] Emotional speech synthesis involves the adjustment of pitch, speech rate, rhythm and other prosodic features. However, the existing financial intelligent customer service system adjusts these prosodic features in a rough way, lacks pertinence and systematicness, and is difficult to effectively convey the emotional changes in the voice through the adjustment of prosodic features. This results in the lack of naturalness of the voice generated by the system, and cannot achieve the expected interaction effect.

[0005] The existing technology usually models the voice emotion based on a single level, which cannot accurately capture the emotional changes on different speech units (such as phonemes, words, sentences, etc.). This modeling method is particularly inadequate in the financial scenario, because when customers consult different types of financial problems, they may need to express different emotions at the word and sentence level, and the existing technology cannot support such multi-level emotional expression.

[0006] Therefore, the existing technology in the field of emotional speech synthesis, especially in the application scenario of the financial industry, has the problems of insufficient emotion control accuracy, lack of systematic adjustment of prosodic features, and weak hierarchical emotion modeling capability. These deficiencies affect the user experience of intelligent customer service and virtual assistants, and limit their deep application in financial services. SUMMARY

[0007] The main purpose of the present application is to provide a speech synthesis method and device based on hierarchical emotion distribution, equipment and storage medium, aiming at solving the technical problems that the prior art lacks fine control and hierarchical modeling of speech emotion, and it is difficult to accurately express emotional changes on different speech units.

[0008] To achieve the above-mentioned purpose, the present application provides a speech synthesis method based on hierarchical emotion distribution, comprising:

[0009] obtaining audio data and corresponding text transcription;

[0010] segmenting the audio data into phoneme-level speech units, word-level speech units and sentence-level speech units;

[0011] extracting the acoustic features of each speech unit, including pitch, speech rate and rhythm;

[0012] using a pre-trained model to analyze the emotional intensity of each speech unit to generate an emotion distribution vector;

[0013] processing the text transcription to obtain text embedding information;

[0014] combining the emotion distribution vector, the text embedding information and the acoustic features, and inputting a variable converter in a speech synthesis framework to generate prosodic features;

[0015] generating a mel-spectrogram based on the prosodic features through a decoder;

[0016] converting the mel-spectrogram into a time-domain audio signal through a vocoder, and outputting the time-domain audio signal as target emotional speech.

[0017] Further, to achieve the above-mentioned purpose, the present application provides a speech synthesis device based on hierarchical emotion distribution, comprising:

[0018] a data acquisition module for acquiring audio data and corresponding text transcription;

[0019] a speech segmentation module for segmenting the audio data into phoneme-level speech units, word-level speech units and sentence-level speech units;

[0020] an acoustic feature extraction module for extracting the acoustic features of each speech unit, including pitch, speech rate and rhythm;

[0021] an emotion analysis module for using a pre-trained model to analyze the emotional intensity of each speech unit to generate an emotion distribution vector;

[0022] a text embedding module for processing the text transcription to obtain text embedding information;

[0023] a feature fusion and variable converter module for combining the emotion distribution vector, the text embedding information and the acoustic features, and inputting a variable converter in a speech synthesis framework to generate prosodic features;

[0024] a decoder module for generating a mel-spectrogram based on the prosodic features by a decoder;

[0025] a vocoder module for converting the mel-spectrogram into a time-domain audio signal by a vocoder, and outputting the time-domain audio signal as target emotion speech.

[0026] Further, to achieve the above object, the present application also provides a computer device, which comprises a memory, a processor and a hierarchical emotion distribution based speech synthesis program stored in the memory and executable on the processor, and the hierarchical emotion distribution based speech synthesis program implements the steps of the hierarchical emotion distribution based speech synthesis method when executed by the processor.

[0027] Further, to achieve the above object, the present application also provides a computer readable storage medium, which stores a hierarchical emotion distribution based speech synthesis program, and the hierarchical emotion distribution based speech synthesis program implements the steps of the hierarchical emotion distribution based speech synthesis method when executed by a processor.

[0028] Beneficial effects: The present application relates to the field of artificial intelligence, and discloses a hierarchical emotion distribution based speech synthesis method, which acquires audio data and text transcription, segments the audio into phoneme, word and sentence level speech units, extracts acoustic features such as pitch, speech rate and rhythm, generates an emotion distribution vector using a pre-trained model, processes the text transcription to generate text embedding information, combines the emotion distribution vector, the text embedding information and the acoustic features, generates prosodic features through a variable converter, generates a mel-spectrogram based on the prosodic features by a decoder, and converts the mel-spectrogram into a time-domain audio signal by a vocoder and outputs it. The present application segments the audio data into phoneme, word and sentence levels hierarchically, combines acoustic features, emotion distribution vectors and text embedding information, and generates prosodic features through a variable converter, thereby achieving fine control of emotional expression. The decoder generates a mel-spectrogram based on these prosodic features, and finally the mel-spectrogram is converted into a time-domain audio signal by a vocoder to generate natural and expressive emotion speech, effectively solving the problems of inaccurate emotional expression and insufficient hierarchical modeling in the prior art, and significantly improving the naturalness and emotional expressiveness of speech synthesis. BRIEF DESCRIPTION OF DRAWINGS

[0029] The application will be further described below in conjunction with the accompanying drawings and embodiments, wherein:

[0030] Figure 1 An application environment diagram of the speech synthesis method based on hierarchical emotion distribution in an embodiment of the application;

[0031] Figure 2 A flow diagram of the speech synthesis method based on hierarchical emotion distribution in an embodiment of the application;

[0032] Figure 3 A function module diagram of the speech synthesis device based on hierarchical emotion distribution in a preferred embodiment of the application;

[0033] Figure 4 A structure diagram of a computer device in an embodiment of the application;

[0034] Figure 5 Another structure diagram of a computer device in an embodiment of the application. DETAILED DESCRIPTION

[0035] It should be understood that the specific embodiments described herein are merely intended to explain the application, and are not intended to limit the application.

[0036] The speech synthesis method based on hierarchical emotion distribution provided by the embodiments of the application can be applied in an application environment as shown in Figure 1 , wherein a user end communicates with a service end through a network. The service end can obtain audio data and text transcription through the user end, segment the audio into phoneme, word and sentence level speech units, extract acoustic features such as pitch, speech rate and rhythm, generate an emotion distribution vector using a pre-trained model, process the text transcription to generate text embedding information, combine the emotion distribution vector, the text embedding information and the acoustic features, generate prosody features through a variable converter, generate a mel-spectrogram based on the prosody features by a decoder, and convert the mel-spectrogram into a time-domain audio signal by a vocoder and output. The application segments the audio data into phoneme, word and sentence levels hierarchically, combines acoustic features, emotion distribution vectors and text embedding information, and generates prosody features using a variable converter, thereby realizing fine control of emotional expression. The decoder generates a mel-spectrogram based on these prosody features, and finally converts the mel-spectrogram into a time-domain audio signal through a vocoder to generate natural and expressive emotional speech, effectively solving the problems of inaccurate emotional expression and insufficient hierarchical modeling in the prior art, and significantly improving the naturalness and emotional expressiveness of speech synthesis. The user end can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The service end can be implemented by an independent server or a server cluster composed of multiple servers. The application will be described in detail below through specific embodiments.

[0037] Referring to Figure 2 , Figure 2 A flowchart of an embodiment of the speech synthesis method based on hierarchical emotion distribution provided by the present application is shown. It should be noted that although a logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that shown.

[0038] As Figure 2 shown, the speech synthesis method based on hierarchical emotion distribution provided by the present application includes the following steps:

[0039] S10, obtaining audio data and corresponding text transcription;

[0040] In this embodiment, the audio data is usually user input or existing audio extracted from a database. These audio data can be natural language recordings, and the speech file can be input generated by a speech recognition system, or it can be a pre-recorded audio file.

[0041] The text transcription is the textual information corresponding to the audio, which is usually extracted from the audio by speech recognition technology (such as an ASR system) or provided by the user. For example, in the financial field, the audio of the conversation between the customer and the intelligent customer service and its corresponding text content will become the core input in this technical step.

[0042] Real-time audio data can be obtained through a voice collection device (such as a microphone), or an already recorded audio file can be extracted from a database or storage by calling an external interface. In actual application, customer service telephone of a financial institution, voice chat record or interactive recording of an automatic voice assistant can all be used as the source of audio data. Text transcription can be generated by speech recognition technology. In specific implementation, automatic speech recognition (ASR) technology is used to extract the text content from the audio data. The ASR system analyzes the acoustic features of the audio and converts them into the corresponding text content. In the financial scenario, ASR can be used to automatically transcribe the customer's oral consultation record into text.

[0043] In one specific embodiment, a hybrid processing architecture is introduced, which preliminarily identifies locally on the device, and when the device computing power is insufficient, it uploads the audio to the cloud for higher-precision identification. For example, when a customer uses a mobile APP of a bank, the system can first process simple voice input locally, and if it involves complex content, it will send the audio to the cloud for further processing.

[0044] By obtaining audio data and corresponding text transcription, the basic input for the subsequent speech synthesis process is provided, which can accurately reflect the user's voice content and the corresponding text information.

[0045] S20, segmenting the audio data into phoneme-level speech units, word-level speech units, and sentence-level speech units;

[0046] In this embodiment, the audio data is segmented into phoneme-level, word-level, and sentence-level speech units. Different levels of speech units represent different levels of speech, as follows:

[0047] A phoneme is the smallest unit of speech. Phoneme segmentation is the process of breaking down audio data into the most basic speech sound segments. This step is usually achieved through forced alignment tools such as Montreal ForcedAligner.

[0048] Word-level segmentation divides speech streams into lexical levels. This step usually relies on phoneme-level alignment information and combines word boundaries in text transcripts to map speech to each specific word.

[0049] Sentence-level segmentation processes larger ranges of speech by analyzing sentence boundaries in text transcripts, such as punctuation marks, to divide the start and end positions of sentences. This segmentation method is more suitable for processing long speech data with coherent semantics.

[0050] Using tools such as Montreal ForcedAligner or other alignment tools, the audio data is forced to align with the text transcript, generating a mapping of phonemes and speech. This tool identifies the start and end times of phonemes in audio by recognizing phonemes in audio, and divides the audio data into small phoneme segments. The system first identifies each phoneme through an acoustic model, and maps these phonemes to audio segments, generating timestamps so that the audio file is accurately segmented into phoneme units.

[0051] Based on the phoneme segmentation results, the phoneme information is mapped to the corresponding words. Using the word boundary information in the text transcript, the system can identify the start and end positions of each word and divide the audio segments into corresponding word units. By matching the start and end times of word boundaries and phonemes, the system can accurately locate the corresponding segments of each word in the audio, thereby performing segmentation.

[0052] Based on the punctuation marks and grammatical structures in the text transcript, the sentence boundaries are identified, and the audio data is divided according to the sentence structure. For example, by identifying punctuation marks such as commas and periods, the start and end points of each sentence in the audio are determined. Text processing tools will perform syntactic analysis based on punctuation marks, syntax analysis, etc., to divide the text into sentence units and further segment the corresponding speech data.

[0053] For example, when a customer asks "I want to check my account balance" through voice, the system first divides the customer's voice data into phonemes, words, and sentence-level voice units. Specifically, phoneme-level division can accurately identify each pronunciation detail, word-level division is used to identify the user's query keywords such as "account balance", and sentence-level division is used to identify the entire query intent. This hierarchical processing enables the system to accurately process and sentiment recognize each unit.

[0054] By performing multi-level segmentation on audio data, the system can accurately process voice content at the phoneme, word, and sentence levels. This segmentation method lays the foundation for subsequent sentiment analysis and speech synthesis, significantly improving the accuracy and naturalness of speech synthesis.

[0055] S30, extracting acoustic features of each voice unit, the acoustic features including pitch, speech rate and rhythm;

[0056] In this embodiment, acoustic features are extracted for each segmented voice unit (such as phonemes, words, sentences), which directly affect the emotional expression and naturalness of speech, mainly including:

[0057] Pitch: Pitch refers to the frequency characteristics of speech, which reflects the frequency of vocal cord vibration. High pitch is usually associated with high-pitched or excited emotions, while low pitch is associated with calm or sad emotions. Changes in pitch are an important part of prosody.

[0058] Speech Rate: Speech rate refers to the speed of pronunciation of voice units, which has an important impact on the rhythm of speech. Faster speech rate usually conveys tense or excited emotions, while slower speech rate conveys calm or heavy emotions.

[0059] Rhythm: Rhythm refers to the distribution pattern of stress and duration of voice units. Different rhythm patterns can convey different emotions and semantic relationships.

[0060] The system detects pitch changes in speech through acoustic analysis tools such as Praat and OpenSMILE. The process of extracting pitch usually relies on Fast Fourier Transform (FFT) or autocorrelation methods to identify fluctuations in the fundamental frequency of the speech signal. For each segmented voice unit, the system extracts the fundamental frequency curve (F0) to generate pitch features. For example, when a user asks a financial question, the system can identify changes in customer emotions such as anxiety or dissatisfaction, and reflect them through pitch changes.

[0061] The speech rate feature is obtained by calculating the duration of each phoneme or word in the speech unit. Short duration of speech units indicates fast speech rate, while long duration indicates slow speech rate. By timestamping the start and end of each phoneme, the system can calculate the duration of each word or sentence and determine its speech rate feature. For example, when the customer's emotion is urgent, the system can detect a faster speech rate, which is often associated with anxiety or urgency.

[0062] Rhythm is mainly extracted by analyzing the stress and length patterns in the speech. This step can use tools based on prosodic analysis to detect the stress position and length distribution of the speech, and identify the prosodic structure of the sentence. The system analyzes the length of each speech unit and detects the stress distribution to generate the rhythm pattern of the speech. In financial interactions, changes in the rhythm of the customer's speech can reflect their emotional fluctuations, helping the system respond better.

[0063] Example: In a financial intelligent customer service system, when a customer asks "Why was 200 yuan deducted from my account?" through a voice assistant, the system will process the customer's speech and extract its pitch, speech rate and rhythm features. By analyzing the acceleration of the speech rate and the change of the pitch, the system can detect that the customer may be in an anxious state. Then, when generating a reply, the system can respond with a more soothing and friendly tone of voice: "We have already queried the relevant information for you and will explain it to you in detail later, please be patient." This emotional speech synthesis method can effectively alleviate the customer's anxiety and improve the customer's satisfaction with financial services.

[0064] By extracting acoustic features such as pitch, speech rate and rhythm, we can capture the emotional information hidden in the speech and make the speech output more natural and expressive through subsequent emotional synthesis process. In the financial field, this fine processing of acoustic features can help the intelligent customer service system understand the emotional state of the customer, so as to make more considerate and efficient response, significantly improving the user experience.

[0065] S40, using a pre-trained model to analyze the emotional intensity of each speech unit and generating an emotional distribution vector;

[0066] In this embodiment, a pre-trained model is used to analyze the emotional intensity of each segmented speech unit and generate an emotional distribution vector. Each speech unit (phoneme, word, sentence) is subjected to emotional recognition by the pre-trained model, outputting a multi-dimensional vector representing the emotional features of the unit. The model is trained on a large amount of emotional data and can extract emotional features from the input speech unit. The pre-trained model may be a model based on deep neural networks (such as BERT, GPT, etc.), which can extract emotional features from acoustic features based on audio.

[0067] The system analyzes the emotional intensity of the speech unit based on acoustic features such as pitch, speech rate, rhythm, etc. For example, higher pitch and faster speech rate might correspond to stronger emotions like anger or excitement, while lower pitch and slower speech rate might correspond to calm or sad emotions. The pre-trained model can match these acoustic features with pre-defined emotion categories.

[0068] The emotion distribution vector is a multi-dimensional vector, with each dimension representing a pre-defined emotion category (e.g. happy, angry, sad, etc.). After the emotional intensity analysis, the pre-trained model outputs the emotion distribution vector for the speech unit, representing the intensity of each emotion.

[0069] The system first extracts the acoustic features (pitch, speech rate, rhythm, etc.) of the speech unit and inputs these features into the pre-trained model. The pre-trained model might be trained on emotion-labeled datasets (e.g. IEMOCAP, MELD). The pre-trained model uses these acoustic features to infer the emotional intensity of the speech unit. The model matches these features with its internal emotion classification system, generating an emotional intensity score. The pre-trained model outputs an emotion distribution vector, with each dimension of the vector representing an emotion type. This vector represents the distribution of the speech unit across multiple emotion dimensions. For example, a speech unit might contain both "happy" and "anxious" components, but with different weights, and the final emotion vector can represent the relative intensity of these two emotions.

[0070] Example illustration: In the scenario of an intelligent voice assistant in a financial service, a customer might make some consultation or query request through the voice assistant, such as "I want to check my recent credit card transaction record". In this scenario, the system first processes the voice data through the previous steps to get the segmented speech units. Then, the system uses the pre-trained model to analyze the emotional intensity of each speech unit and generates an emotion distribution vector.

[0071] Specifically, the pre-trained model analyzes the acoustic features (pitch, speech rate, rhythm, etc.) in the speech, for example, when the customer says "credit card transaction", the speech rate might be slightly faster and the pitch might be slightly higher, reflecting the customer's emotional tendency to quickly obtain information. Through these features, the pre-trained model can judge the customer's emotional state, which might contain anxious and anxious components. Then, the model generates an emotion distribution vector, with each dimension of the vector possibly representing different emotional intensity, such as high intensity of anxiety and low intensity of happiness.

[0072] By using a pre-trained model for sentiment intensity analysis, an accurate sentiment distribution vector can be generated for each speech unit, allowing the system to capture subtle emotional changes in each speech unit. Combined with the sentiment distribution vector, the system can more accurately express different emotional states when generating speech, improving the expressiveness and naturalness of speech synthesis.

[0073] S50, processing the text transcription to obtain text embedding information;

[0074] In this embodiment, text embedding information for speech synthesis is generated by processing text transcription data. Text embedding is a process of converting text information into vector representation, which compresses semantic, lexical and structural information in text into numerical vectors for model processing. In this technical solution, text embedding information is used to generate natural speech in conjunction with sentiment distribution vector and acoustic features.

[0075] The data of the text transcription is converted from letter form to phoneme form. Phonemes are the basic units of speech, and this step is achieved through a language model or conversion tool. Letter-to-phoneme conversion allows the system to closely interface text data with speech units.

[0076] The converted phoneme sequence is encoded into a vector representation by a text encoder. Common text encoders include pre-trained models such as BERT, GPT, etc., which can capture semantic information in text and represent this information as low-dimensional embedding vectors.

[0077] Additional linguistic features such as part of speech, syntactic structure and semantic information are extracted from the text. These linguistic features provide more semantic and contextual information for text embedding, allowing the model to better understand the meaning of the text and generate more contextually appropriate speech in conjunction with emotional information.

[0078] By fusing the above processing results, the text information is embedded into a low-dimensional vector space to form the final text embedding vector. This embedding vector contains the semantic, phonetic and linguistic features of the text, serving as the basic input for subsequent speech synthesis and emotional expression.

[0079] The text data is converted into phoneme sequences using a letter-to-phoneme conversion model, such as a G2P model, to ensure that the pronunciation information of the text is consistent with the speech processing. The phoneme sequences of the text are encoded using a language model such as BERT to generate a vectorized representation of the text. This encoder not only captures the semantic information of the words but also models the meaning of the words in context. The syntactic, part-of-speech, and semantic information of the text is extracted using natural language processing (NLP) tools. These features are appended to the text embedding vectors to enhance the speech synthesis model's understanding of the text. All the processed information is fused to generate the final text embedding vectors, providing a complete semantic input for the subsequent speech synthesis.

[0080] By processing the text transcription data to generate text embedding information, the system can effectively embed the pronunciation information, semantics, and contextual information of the text into low-dimensional vectors, providing a more comprehensive input for the speech synthesis model.

[0081] S60, the emotion distribution vector, the text embedding information, and the acoustic features are combined and input into a variable converter in the speech synthesis framework to generate prosodic features.

[0082] In this embodiment, the emotion distribution vector, text embedding information, and acoustic features are combined, and then the final prosodic features are generated through a variable converter. These prosodic features are used for speech synthesis to control the pitch, rhythm, duration, and other aspects of the speech, resulting in natural and emotionally expressive speech.

[0083] The emotion distribution vector represents the emotional characteristics of each speech unit. Different dimensions of each vector correspond to different emotion categories, indicating the intensity of the speech unit in different emotions. This vector is generated through a pre-trained model.

[0084] The text embedding information represents the semantic and contextual information of the text content corresponding to the speech unit. These information is generated through a text encoder, which closely integrates the semantic information of the text with the speech synthesis process, ensuring that the generated speech is consistent with the meaning of the text.

[0085] Acoustic features include pitch, speech rate, rhythm, and other aspects, which are used to control the physical characteristics of the speech. These features provide a basis for the subsequent generation of prosodic features, ensuring that the emotional expression in the speech synthesis process is natural and smooth.

[0086] The variable converter is a key module that combines the emotion distribution vector, text embedding information, and acoustic features. It converts these inputs into prosodic features such as pitch, duration, and energy through a deep learning model or other mathematical transformations, which are used for subsequent speech generation.

[0087] The system first combines the emotion distribution vector, text embedding information, and acoustic features to construct a joint representation. The key step is to effectively combine the features from different information sources through multi-modal fusion techniques, so that the generated prosody features can express emotion, text meaning, and speech characteristics at the same time.

[0088] The joint representation is input into a variable converter, which can be a complex neural network (such as a multi-layer perceptron or convolutional neural network). The task of the variable converter is to generate prosodic information such as pitch, duration, and volume based on the input multi-modal features. The converter is trained to capture the complex relationships between text, emotion, and acoustic features.

[0089] The output of the variable converter is a prosody feature vector, which represents the pitch variation, speech rate adjustment, and energy distribution of the speech. Through these prosodic features, the system can generate more natural and emotionally rich speech.

[0090] By multi-modal fusion of emotion distribution vector, text embedding information, and acoustic features, and using the variable converter to generate prosodic features, the system can achieve more delicate speech synthesis.

[0091] S70, generating a mel-spectrogram based on the prosodic features through a decoder;

[0092] In this embodiment, the generated prosodic features are converted into a mel-spectrogram through a decoder. The mel-spectrogram is a time and frequency distribution representation of the speech signal, and is a key intermediate representation form in the speech synthesis process. It is used to represent the frequency components and time domain information of the speech signal, and is generated from the prosodic features through the decoder to provide a basis for subsequent speech synthesis.

[0093] The input of the decoder is the prosodic features (including pitch, duration, energy, etc.) generated from the variable converter, which are used to control the specific form of the synthesized speech. The decoder maps these features to the frequency domain to generate a mel-spectrogram.

[0094] The mel-spectrogram is a frequency spectrum graph that has been transformed by the mel transform, and the frequency scale has been adjusted according to the perception of the human auditory system, making it more suitable for speech synthesis. It captures the timing changes of different frequency components in the speech signal, allowing the speech synthesis system to better simulate real speech.

[0095] The decoder can be implemented based on different models, such as long short-term memory network (LSTM), variational autoencoder (VAE), etc. The role of the decoder is to convert the prosodic features into frequency features in the time domain, and the generated mel-spectrogram is used for subsequent vocoder to generate time-domain audio signals.

[0096] The decoder receives prosodic features (pitch, duration, energy, etc.) generated by the variable converter, which control the detailed performance in speech synthesis. The decoder maps the prosodic features to the frequency domain through a deep learning model or other mathematical transformations, generating a mel-spectrogram. This process involves converting time-series data into frequency-domain representations and adjusting the intensity of different frequency components based on prosodic features. The mel-spectrogram is compressed by mel filters to make the frequency components more consistent with human auditory perception, resulting in a frequency-time distribution graph that provides input for the subsequent vocoder step.

[0097] By converting prosodic features into mel-spectrograms through the decoder, the system can accurately capture frequency changes and time-domain information in speech, ensuring that the generated speech synthesis is natural and expressive. Through precise control of prosodic features, parameters such as pitch, speech rate, and volume in the speech synthesis process can be finely adjusted to generate speech that is more consistent with human auditory perception.

[0098] S80, converting the mel-spectrogram into a time-domain audio signal through the vocoder, and outputting the time-domain audio signal as target emotional speech.

[0099] In this embodiment, the mel-spectrogram generated by the decoder is converted into a time-domain audio signal, i.e., the final speech signal. The vocoder is a key component of the speech synthesis process, which converts information in the frequency domain (mel-spectrogram) into actual audio signals, generating audible speech output.

[0100] The task of the vocoder (Vocoder) is to convert the frequency-domain representation in the mel-spectrogram into a time-domain audio signal. Common vocoders include WaveNet, Griffin-Lim, MelGAN, etc., which can generate high-quality speech output based on the spectrogram.

[0101] The input to the vocoder is the previously generated mel-spectrogram, which represents the frequency distribution of speech. This spectrogram contains information such as pitch, volume, and duration, but it is a frequency-domain representation, not a direct audio signal.

[0102] The vocoder converts the mel-spectrogram in the frequency domain into a time-domain audio signal. The audio signal generated by the vocoder is a speech that can be directly played and can reflect the prosodic features and emotional information input previously.

[0103] The final time-domain audio signal contains all the information of emotional speech, including prosodic features such as pitch, rhythm, and duration. These emotional speech can be directly output by intelligent customer service or virtual assistants, realizing natural speech interaction with emotional expression.

[0104] The system first inputs the Mel-spectrogram into a vocoder, which converts the frequency information into corresponding audio signals. For example, using a WaveNet vocoder, the model generates time-domain signals recursively. WaveNet can capture complex patterns and details in speech and generate very natural speech output. When using fast vocoders such as MelGAN, the vocoder generates time-domain audio signals based on the Mel-spectrogram. MelGAN uses a generative adversarial network approach to speed up the generation process, ensuring efficient processing while generating high-quality audio signals. The time-domain audio signals output by the vocoder are the final speech results, containing the details of prosodic features and emotional expressions, forming natural and fluent speech. The generated time-domain audio signals are output as target emotional speech, and the system can directly play them to the user. These speeches express the emotional features required by the user, making the speech synthesis system have a more natural human-computer interaction effect.

[0105] Example: In a financial service scenario, a customer may make a request through an intelligent voice assistant, such as "Please help me query my transaction records last month." After generating prosodic features and converting them into a Mel-spectrogram, the system inputs the spectrogram into a MelGAN vocoder, which converts it into a time-domain audio signal. The generated emotional speech may contain a gentle tone and a slow pace to calm the customer's emotions. The system replies through voice: "Your transaction records have been queried, and the total amount is 2000 yuan." The audio is generated by MelGAN and has natural sound quality and emotional expression, achieving high-quality user interaction.

[0106] Converting the Mel-spectrogram into a time-domain audio signal through a vocoder can accurately convert frequency domain information into high-quality speech output. Using advanced vocoders such as WaveNet or MelGAN can generate more natural and emotional speech. In the financial scenario, the speech system can interact with the user with very high fidelity and emotional expression, significantly improving the user experience.

[0107] The application relates to the technical field of artificial intelligence, and discloses a speech synthesis method based on hierarchical emotion distribution, which comprises the following steps: acquiring audio data and text transcription, segmenting the audio into phoneme, word and sentence level speech units, extracting acoustic features such as pitch, speech rate and rhythm, generating an emotion distribution vector by using a pre-training model, processing the text transcription to generate text embedding information, combining the emotion distribution vector, the text embedding information and the acoustic features, generating prosodic features through a variable converter, generating a mel-spectrogram based on the prosodic features by using a decoder, and converting the mel-spectrogram into a time-domain audio signal by using a vocoder and outputting the time-domain audio signal. The audio data is hierarchically segmented into phoneme, word and sentence levels, the acoustic features, the emotion distribution vector and the text embedding information are combined, and the prosodic features are generated by using the variable converter, so that fine control of emotional expression is realized. The decoder generates the mel-spectrogram based on the prosodic features, and finally the mel-spectrogram is converted into the time-domain audio signal by using the vocoder, so that natural and expressive emotional speech is generated, the problems of inaccurate emotional expression and insufficient hierarchical modeling in the prior art are effectively solved, and the naturalness and emotional expressiveness of the speech synthesis are significantly improved.

[0108] In one embodiment, S20 comprises:

[0109] S201, aligning the audio data with the text transcription by using a forced alignment tool;

[0110] S202, segmenting the audio data into phoneme level speech units according to the alignment result;

[0111] S203, segmenting the audio data into word level speech units according to the word boundaries in the text transcription;

[0112] S204, segmenting the audio data into sentence level speech units according to the sentence boundaries in the text transcription.

[0113] In the embodiment, the audio data is hierarchically segmented, so that subsequent processing can perform speech analysis and synthesis at different granularities. The segmentation process depends on the alignment result of the audio data and the text transcription, and is divided according to the boundaries of phonemes, words and sentences.

[0114] The forced alignment tool (such as Montreal ForcedAligner) is used to synchronize and align the audio data with the text transcription. The forced alignment is used to accurately determine the time stamp of each speech unit (phoneme, word, sentence) in the audio. The tool calculates the start and end time of each unit based on the sound features in the audio and the text content by using an acoustic model and a language model.

[0115] According to the alignment results, the audio data is segmented into phoneme-level speech units. Phonemes are the smallest units of speech that reflect specific pronunciations in speech. After segmentation, each phoneme unit has precise start and end times for subsequent feature extraction and sentiment analysis.

[0116] Based on the word boundaries in the text transcript, the audio is further segmented into word-level speech units. Word-level segmentation combines multiple phonemes into a word, better expressing semantics. This segmentation process is achieved by matching word boundaries in the text transcript with phoneme start and end times.

[0117] Finally, according to the sentence boundaries in the text (such as punctuation or syntax), the audio data is segmented into sentence-level units. This segmentation method is suitable for processing longer speech segments, helping the system understand complete sentence expressions and providing higher-level semantic information for speech synthesis processes.

[0118] Using tools such as Montreal ForcedAligner, audio data is aligned with text transcripts. This tool generates timestamps based on audio acoustic features and phoneme information in the text, ensuring that each phoneme corresponds to its position in the audio. Based on the alignment results, the audio file is divided into individual phonemes. This step ensures that the start and end times of each pronunciation segment are accurately recorded and provides a basis for subsequent acoustic feature extraction. Based on the word boundaries in the text transcript, the system aggregates phoneme units into word-level units. By detecting the start and end points of words in the text, the system can generate corresponding audio segments. Finally, based on the punctuation or syntax structure in the text, the system divides the audio into sentence units. By aligning the sentence structure in the text, better sentiment analysis and speech synthesis can be performed.

[0119] Example explanation: In financial services, customers initiate requests through voice assistants, such as "Please help me query last month's transaction records." The system first aligns the customer's voice with the text transcript using forced alignment tools to determine the timestamp of each phoneme. Then, the system segments the audio into phoneme-level units to ensure that the start and end times of each pronunciation are accurately marked. Then, by analyzing word boundaries, the system combines phonemes into word units to identify key information such as "query" and "transaction records." Finally, according to the sentence structure, the system identifies complete sentences and performs corresponding query operations. Through this hierarchical processing method, the system can more accurately and quickly respond to customer requests and provide natural voice feedback when generating voice replies, improving customer experience.

[0120] By segmenting the audio data into phoneme, word, and sentence-level units, the system can perform fine-grained processing of the speech, ensuring the naturalness and accuracy of the speech synthesis. In financial services, this segmentation approach can help intelligent customer service systems accurately analyze user speech inputs and express personalized emotions based on each level of speech unit, providing more personalized voice services to customers.

[0121] In one embodiment, S30 comprises:

[0122] S301, for each phoneme-level speech unit, determining a pitch contour to extract pitch acoustic features;

[0123] S302, for each word-level speech unit, measuring the duration to extract the speech rate acoustic features;

[0124] S303, for each sentence-level speech unit, obtaining an energy contour to extract the rhythm acoustic features;

[0125] S304, extracting additional acoustic features including formant and harmonic-to-noise ratio from the phoneme-level speech units and the word-level speech units;

[0126] S305, associating the extracted acoustic features with the corresponding speech units.

[0127] In this embodiment, the system extracts multiple acoustic features from phoneme, word, and sentence-level speech units for subsequent emotional speech synthesis. The main features include pitch, speech rate, and rhythm, combined with other acoustic features such as formant and harmonic-to-noise ratio, to generate more accurate speech performance through multi-level analysis.

[0128] For each phoneme-level speech unit, the system extracts pitch information through the **pitch contour**. Pitch is a frequency feature in speech that reflects the speed of vocal cord vibration and is one of the important features for conveying speech emotion. The pitch contour can capture the subtle pitch changes at the phoneme level, helping the system accurately express the emotional intensity in speech.

[0129] For each word-level speech unit, the system extracts speech rate features by measuring the duration. The speed of speech can convey the emotional state of the speaker, with faster speech often indicating anxiety or excitement, and slower speech often indicating calmness or seriousness.

[0130] For each sentence-level speech unit, the system extracts rhythm features by obtaining an energy contour. The energy contour represents the strength variation of the speech and is usually used to express the prosodic structure of the sentence. By analyzing the energy changes at the sentence level, the system can generate more natural speech expressions.

[0131] The Fast Fourier Transform (FFT) or autocorrelation method is used to extract the fundamental frequency contour (F0) of each phoneme, capturing the pitch variation. The fundamental frequency extraction tool can use software such as Praat or OpenSMILE. According to the start and end time of the phoneme, the system calculates the duration of each speech unit at the word level. The measurement of duration can capture the speech rate variation, helping the system to identify the emotional state. Using the energy contour analysis tool, the system generates an energy contour by calculating the intensity variation of the speech waveform. These energy variations can capture the rhythm characteristics of the speech, enabling the system to express the prosody at the sentence level.

[0132] Formants are peaks in the speech spectrum that represent the resonance frequencies of the speech vocal tract, mainly used to distinguish different vowels and consonants. Phonemes are the smallest units of pronunciation in speech, and extracting formants at the phoneme level can capture the pronunciation characteristics of speech units. Linear Predictive Coding (LPC) or Fourier transform is used to analyze the frequency of speech units at the phoneme level. The frequencies of formants are closely related to the shape and size of the vocal tract, and the changes in these frequencies can distinguish the pronunciation positions of vowels and consonants. For example, the formant distribution of vowels / a / and / i / is different, and through these features, the system can recognize and simulate complex speech characteristics.

[0133] The harmonic-to-noise ratio reflects the proportion of harmonic components and noise components in speech. Harmonic components mainly come from vocal cord vibration, while noise is the irregular part of speech, such as airflow noise or hoarse components. The higher the harmonic-to-noise ratio, the clearer and more stable the speech. At the word level, extracting the harmonic-to-noise ratio can measure the intelligibility of speech. The proportion of harmonic components and noise components is usually calculated by autocorrelation function or other periodicity analysis methods. A higher harmonic-to-noise ratio indicates that the speech has a stronger harmonic component, which is suitable for speech synthesis and recognition to control and optimize intelligibility.

[0134] Extracting formants at the phoneme level helps to capture subtle pronunciation characteristics, ensuring that the pronunciation details of the original speech can be accurately reproduced during speech synthesis. Extracting the harmonic-to-noise ratio at the word level helps to evaluate the intelligibility and stability of speech, especially in long words or sentences, maintaining the clarity and naturalness of speech is particularly important.

[0135] This embodiment can capture the emotional changes in speech by accurately extracting acoustic features such as pitch, speech rate and rhythm, and combine additional acoustic features such as formants and harmonic-to-noise ratio to enhance the naturalness and emotional expression of speech synthesis.

[0136] In one embodiment, the above S40 comprises:

[0137] S401, obtaining an emotion recognition model pre-trained based on an emotion dataset;

[0138] S402, input the acoustic features of each speech unit into the emotion recognition model to determine the emotion intensity score of each speech unit in the predefined emotion category;

[0139] S403, based on the emotion intensity score of each speech unit, form an emotion distribution vector representing the emotion state, each dimension of the emotion distribution vector representing a predefined emotion category.

[0140] In this embodiment, by using a pre-trained emotion recognition model to analyze the acoustic features of speech units, the emotion intensity of speech is determined, and an emotion distribution vector is generated. The emotion distribution vector is used to represent the emotion state of each speech unit, helping the system to express emotion during speech synthesis.

[0141] The pre-trained model is a deep learning model trained on a dataset with emotion labels. This model can recognize different emotion categories (such as happiness, anger, sadness, etc.). Common datasets include IEMOCAP, MELD, etc., which guide the model to learn emotion patterns by labeling emotion categories.

[0142] The system inputs the acoustic features of each speech unit (such as pitch, speech rate, rhythm, formant, etc.) into the pre-trained model. Acoustic features are the basic information for emotion analysis, which can reflect the emotional expression in speech, such as higher pitch usually reflects excitement or anger, while lower pitch may reflect calm or sad emotions.

[0143] The pre-trained model calculates the emotion intensity score of each speech unit in different emotion categories based on the input acoustic features. These scores reflect the performance of speech in different emotion dimensions. For example, a certain speech unit may have an intensity score of 0.8 in the "anger" category, and 0.2 in the "joy" category.

[0144] Based on the emotion intensity score of each speech unit, the system generates an emotion distribution vector. Each dimension of the vector corresponds to a predefined emotion category, representing the intensity of the speech unit in that emotion category. For example, a three-dimensional emotion distribution vector may represent "anger", "sadness" and "joy" three categories, and the value of each dimension corresponds to the intensity of the emotion.

[0145] The system uses a sentiment recognition model pre-trained based on a sentiment annotation dataset. The model can be LSTM, Transformer or CNN, and is trained through supervised learning of a sentiment dataset. The acoustic features of each speech unit extracted earlier are input into the sentiment recognition model. The model will determine the sentiment state expressed by the speech unit according to these acoustic features. The model calculates the sentiment intensity scores of each speech unit in different sentiment categories. These scores change with changes in the pitch, rhythm, etc. of the speech, and the model can identify subtle differences in sentiment. Finally, the system generates a sentiment distribution vector based on the sentiment intensity scores. This vector will be used in the subsequent speech synthesis stage to ensure that the generated speech expresses the corresponding sentiment.

[0146] The present embodiment uses a pre-trained sentiment recognition model to analyze the sentiment intensity of the acoustic features of each speech unit, and the system can generate a sentiment distribution vector representing the sentiment state of each speech unit. These vectors provide accurate sentiment input for subsequent speech synthesis, ensuring that the generated speech can convey accurate sentiment information.

[0147] In one embodiment, S50 comprises:

[0148] S501, converting the text transcription into a phoneme sequence using a grapheme-to-phoneme (G2P) conversion model;

[0149] S502, encoding the phoneme sequence through a text encoder to generate text embedding information;

[0150] S503, extracting linguistic features including part-of-speech, syntax and semantics from the text transcription;

[0151] S504, fusing the linguistic features into the text embedding information and embedding the context information of the text transcription in the fused text embedding information to generate enhanced text embedding information.

[0152] In the present embodiment, the text transcription is processed to generate embedding information that can reflect the semantics and context of the text. This embedding information can provide a semantic basis for subsequent speech synthesis.

[0153] The system first uses a grapheme-to-phoneme (G2P) conversion model to convert the text transcription into a phoneme sequence. Phonemes are the basic units of speech, and converting graphemes to phonemes helps to subsequently process the correspondence between speech and text. The G2P model maps the graphemes in the text transcription to the corresponding phonemes through mapping rules or learning models.

[0154] The text encoder (such as BERT or GPT) encodes the phoneme sequence to generate preliminary text embedding information. The text encoder captures the semantics, context, and word-to-word relationships in the text through a deep learning model. This embedding information is a low-dimensional vector representation that can reflect the core meaning of the text.

[0155] The system extracts linguistic features such as part-of-speech, syntactic structure, and semantic information from the text transcription. Part-of-speech (such as verbs, nouns, etc.) and syntactic structure (such as subject-verb-object relationships) can help the system better understand the semantic levels and structure of the text. Semantic information includes the specific meaning of words, contextual associations, and more.

[0156] The extracted linguistic features are fused with the text embedding information to enhance the expressiveness of the embedding information. The fusion process includes embedding part-of-speech, syntax, and other features into the text embedding vector, allowing it to contain more semantic information. In addition, the system also embeds the context information of the text to ensure that the generated embedding information not only reflects the meaning of individual words, but also captures the semantic dependency between contexts.

[0157] The text transcription is converted into phonemes using a G2P model (such as CMU Pronouncing Dictionary or Phonetisaurus). The system converts the sequence of letters into a sequence of phonemes based on the pronunciation rules of each letter, providing pronunciation information for subsequent speech synthesis. The converted phoneme sequence is input into the text encoder (such as BERT or GPT) to generate the embedding information of the text. The text encoder captures semantic and contextual information through multiple layers of networks, generating a vectorized representation that reflects the linguistic features of the text. Natural language processing (NLP) tools are used to analyze the text transcription, extracting part-of-speech, syntactic structure, and semantic information. These linguistic features can supplement the embedding information generated by the text encoder, helping the system better understand the deep meaning of the text. The extracted part-of-speech, syntax, and semantic features are fused with the text embedding information, and contextual information is embedded during the fusion process. This contextual information allows the embedding information to reflect the coherence and semantic dependency of the text, resulting in more natural speech.

[0158] This embodiment generates text embedding information by processing text transcription, allowing the system to combine the phonetic features, semantic information, and contextual dependencies of the text together, providing a rich semantic foundation for subsequent speech synthesis. Compared to directly using letters or words, the combination of phoneme sequences and linguistic features makes the generated speech more natural, coherent, and accurately conveys the semantics of the text.

[0159] In one embodiment, the above S60 includes:

[0160] S601, align the emotion distribution vector with the corresponding text embedding information at the phoneme level, word level, and sentence level;

[0161] S602, fuse the aligned emotion distribution vector, text embedding information, and the acoustic features to form a combined feature representation;

[0162] S603, apply an attention mechanism to the combined feature representation to enhance the association between the emotion distribution vector and the text embedding information;

[0163] S604, input the attention mechanism-enhanced combined feature representation into a variable converter in the speech synthesis framework to generate prosodic features.

[0164] In this embodiment, the emotion distribution vector, text embedding information, and acoustic features are combined, and then the final prosodic features are generated through a variable converter. These prosodic features control the key features of the synthesized speech, such as pitch, rhythm, and duration, making the generated speech natural and emotionally expressive.

[0165] The system aligns the emotion distribution vector with the text embedding information at different levels (phoneme, word, sentence). The purpose of alignment is to closely combine the emotion and the semantics of the text to ensure that each level of speech unit can accurately express emotion and semantics. For example, the system aligns subtle pitch changes at the phoneme level, important emotional features at the word level, and overall semantic and emotional expression at the sentence level.

[0166] The aligned emotion distribution vector, text embedding information, and acoustic features (pitch, speech rate, rhythm, etc.) are fused. The purpose of fusion is to unify these different sources of information into a combined feature representation. This combined feature representation contains the emotion, semantics, and physical features in the speech and is the basis for generating prosodic features later.

[0167] The attention mechanism is applied to the combined feature representation to enhance the association between the emotion distribution vector and the text embedding information. The attention mechanism can automatically assign weights to each input element (such as emotional information in the emotion distribution vector and semantic information in the text embedding), allowing the system to focus on the most important information for speech synthesis. This makes the synthesized speech not only convey semantics but also naturally express emotions.

[0168] Finally, the attention mechanism-enhanced combined feature representation is input into the variable converter in the speech synthesis framework. The variable converter is a deep learning model or transformation module that is responsible for converting the combined feature representation into prosodic features (such as pitch, duration, rhythm, etc.). These prosodic features are directly used to generate the final speech output, ensuring the naturalness and consistency of emotional expression of the speech.

[0169] The system matches the emotion distribution vector with the text embedding information at phoneme, word, and sentence levels through an alignment algorithm. Through detailed alignment, it ensures the combination of emotion information of different granularities with text semantics. The emotion distribution vector, text embedding information, and acoustic features are fused in multiple modalities to form a combined feature representation. This fusion can be achieved through simple concatenation, weighting, or a multi-modal fusion layer of a deep learning model. The attention mechanism dynamically adjusts the weights to allocate appropriate attention to the emotion distribution and text information. Its output is an enhanced combined feature representation, enabling the system to generate emotions and prosody in speech more accurately. The combined feature representation enhanced by the attention mechanism is input into the variable converter to generate the final prosody features. These prosody features control the basic sound parameters of the speech, such as pitch, duration, and volume, ensuring that the generated speech is natural and emotionally rich.

[0170] This embodiment enables the system to generate more natural prosody features by combining the emotion distribution vector, text embedding information, and acoustic features. The application of the attention mechanism further enhances the accuracy and naturalness of emotion expression.

[0171] In one embodiment, in the above S80, converting the mel-spectrogram into a time-domain audio signal through a vocoder includes:

[0172] S801, analyzing the emotional features of the target emotional speech based on the prosody features;

[0173] S802, determining a vocoder model based on the emotional features of the target emotional speech, and adjusting the sampling rate and frequency resolution of the vocoder model;

[0174] S803, configuring the generation step of the vocoder model to optimize the speed and quality of speech generation;

[0175] S804, converting the mel-spectrogram into a preliminary time-domain audio signal using the configured vocoder model;

[0176] S805, post-processing the preliminary time-domain audio signal to generate a final time-domain audio signal, the post-processing including dynamic range compression, frequency equalization, and gain control.

[0177] In this embodiment, the mel-spectrogram is converted into a final time-domain audio signal through a vocoder to generate the target emotional speech. The task of the vocoder is to convert the mel-spectrogram in the frequency domain into a playable time-domain audio signal, combining the emotional features of the speech to ensure that the generated speech is natural and expressive.

[0178] The system first analyzes the emotional features of the target emotional speech based on the generated prosodic features (pitch, rhythm, duration, etc.). Emotional features include the overall emotional expression of the speech, such as calm, anxious, excited, etc., which determines the prosody, intensity, and form of expression of the speech.

[0179] Based on the emotional features, the system selects the appropriate vocoder model. Commonly used vocoder models include WaveNet, MelGAN, etc. The selection and parameters (such as sampling rate, frequency resolution) of the vocoder model need to be adjusted according to the characteristics of the target speech to ensure the quality of the audio and the accuracy of the emotional expression.

[0180] The system configures the generation step of the vocoder to optimize the speed and quality of speech generation. The step is the time interval setting when the vocoder generates audio signals, which directly affects the fluency and computational efficiency of the generated audio.

[0181] Using the configured vocoder model, the mel-spectrogram is converted into a preliminary time-domain audio signal. This process converts frequency domain information (mel-spectrogram) into actual playable speech signals, but the preliminary signal may need further processing to improve quality.

[0182] Post-processing is performed on the preliminary time-domain audio signal to generate the final time-domain audio signal. Post-processing steps include dynamic range compression (adjusting the volume difference of the audio to improve the listening experience), frequency equalization (adjusting the frequency response of the audio to enhance the clarity of the audio), and gain control (optimizing the volume level). These processing steps ensure that the final generated audio signal has high-quality sound performance.

[0183] The system extracts the emotional features of the speech from the prosodic features and uses these features to adjust the output of the speech synthesis. For example, when expressing anger, the system may choose features with faster speech speed and higher pitch; while expressing calm, the system will choose lower pitch and slower speech speed. The system determines to use vocoder models such as WaveNet or MelGAN based on emotional features. Then, adjust the sampling rate and frequency resolution to optimize the quality of the output audio. For example, a higher sampling rate can generate more detailed speech details, but will increase the computational overhead. Configure the generation step of the vocoder to make the speech generation both fast and of high quality. The configuration of the step needs to balance the speed of generation on the premise of ensuring the quality of the speech to meet the needs of real-time or near-real-time speech synthesis. Using the configured vocoder, the mel-spectrogram is converted into a preliminary time-domain audio signal. At this time, the speech can already be played, but further optimization of the sound quality is needed. Through dynamic range compression, reduce the part of the speech with too high volume, enhance the details. Use frequency equalization to ensure that the frequency distribution in the audio is reasonable and enhance the clarity. Finally, through gain control, optimize the overall volume of the audio to ensure that the speech is clear and natural when played.

[0184] The embodiment converts the mel-spectrogram into a time-domain audio signal through a vocoder, and the system can generate natural speech conforming to prosody features and emotional features. The post-processing step ensures that the finally generated speech reaches a high level in terms of quality and intelligibility.

[0185] In an embodiment, a speech synthesis device based on hierarchical emotional distribution is provided, which corresponds to the speech synthesis method based on hierarchical emotional distribution in the above embodiment. Referring to Figure 3 , Figure 3 A functional module schematic diagram of a preferred embodiment of the speech synthesis device based on hierarchical emotional distribution of the present application. Data acquisition module 10, speech segmentation module 20, acoustic feature extraction module 30, emotion analysis module 40, text embedding module 50, feature fusion and variable converter module 60, decoder module 70 and vocoder module 80. The detailed description of each functional module is as follows:

[0186] The data acquisition module 10 is used to acquire audio data and corresponding text transcription;

[0187] The speech segmentation module 20 is used to segment the audio data into phoneme-level speech units, word-level speech units and sentence-level speech units;

[0188] The acoustic feature extraction module 30 is used to extract acoustic features of each speech unit, including pitch, speech rate and rhythm;

[0189] The emotion analysis module 40 is used to analyze the emotional intensity of each speech unit using a pre-trained model to generate an emotional distribution vector;

[0190] The text embedding module 50 is used to process the text transcription to obtain text embedding information;

[0191] The feature fusion and variable converter module 60 is used to combine the emotional distribution vector, the text embedding information and the acoustic features, and input them into a variable converter in a speech synthesis framework to generate prosody features;

[0192] The decoder module 70 is used to generate a mel-spectrogram based on the prosody features through a decoder;

[0193] The vocoder module 80 is used to convert the mel-spectrogram into a time-domain audio signal through a vocoder, and the time-domain audio signal is taken as target emotional speech and output.

[0194] In an embodiment, the speech segmentation module 20 is specifically used for:

[0195] aligning the audio data with the text transcription using a forced alignment tool;

[0196] segmenting the audio data into phoneme-level speech units according to the alignment result;

[0197] segmenting the audio data into word-level speech units according to word boundaries in the text transcription;

[0198] segmenting the audio data into sentence-level speech units according to sentence boundaries in the text transcription.

[0199] In an embodiment, the acoustic feature extraction module 30 is specifically configured to:

[0200] determining a pitch contour for each phoneme-level speech unit to extract a pitch acoustic feature;

[0201] measuring a duration for each word-level speech unit to extract a speaking rate acoustic feature;

[0202] obtaining an energy contour for each sentence-level speech unit to extract a rhythm acoustic feature;

[0203] extracting additional acoustic features including formant and harmonic-to-noise ratio from the phoneme-level speech units and the word-level speech units;

[0204] associating the extracted acoustic features with corresponding speech units.

[0205] In an embodiment, the sentiment analysis module 40 is specifically configured to:

[0206] obtaining a sentiment recognition model pre-trained based on a sentiment dataset;

[0207] inputting the acoustic features of each speech unit into the sentiment recognition model to determine a sentiment intensity score of the each speech unit on a predefined sentiment category;

[0208] forming a sentiment distribution vector representing a sentiment state based on the sentiment intensity score of each speech unit, each dimension of the sentiment distribution vector representing a predefined sentiment category.

[0209] In an embodiment, the text embedding module 50 is specifically configured to:

[0210] converting the text transcription into a phoneme sequence using a letter-to-phoneme conversion model;

[0211] encoding the phoneme sequence through a text encoder to generate text embedding information;

[0212] extracting linguistic features including part-of-speech, syntax, and semantics from the text transcription;

[0213] The linguistic features are fused into the text embedding information, and context information of the text transcription is embedded in the fused text embedding information to generate enhanced text embedding information.

[0214] In an embodiment, the feature fusion and variable converter module 60 is specifically configured to:

[0215] The sentiment distribution vector is aligned with the corresponding text embedding information at the phoneme level, the word level and the sentence level;

[0216] The aligned sentiment distribution vector, text embedding information and acoustic features are fused to form a combined feature representation;

[0217] An attention mechanism is applied to the combined feature representation to enhance the association between the sentiment distribution vector and the text embedding information;

[0218] The combined feature representation enhanced by the attention mechanism is input into a variable converter in a speech synthesis framework to generate prosody features.

[0219] In an embodiment, the vocoder module 80 is specifically configured to:

[0220] The sentiment features of the target emotional speech are analyzed based on the prosody features;

[0221] Based on the sentiment features of the target emotional speech, a vocoder model is determined, and the sampling rate and frequency resolution of the vocoder model are adjusted;

[0222] The generation step of the vocoder model is configured to optimize the speed and quality of speech generation;

[0223] The mel-spectrogram is converted into a preliminary time-domain audio signal using the configured vocoder model;

[0224] The preliminary time-domain audio signal is post-processed to generate a final time-domain audio signal, and the post-processing includes dynamic range compression, frequency equalization and gain control.

[0225] In one embodiment, a computer device is provided, which can be a server, and its internal structure diagram can be as shown in Figure 4As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media, internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external user terminal through the network connection. The computer program is executed by the processor to realize the functions or steps of the server side of the hierarchical emotion distribution based speech synthesis method.

[0226] In one embodiment, a computer device is provided, which can be a user terminal, and its internal structure diagram can be as shown in the figure. Figure 5 As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media, internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external server through the network connection. The computer program is executed by the processor to realize the functions or steps of the user side of the hierarchical emotion distribution based speech synthesis method

[0227] In one embodiment, a computer device is provided, including a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to realize the following steps:

[0228] Obtain audio data and corresponding text transcription;

[0229] Segment the audio data into phoneme-level speech units, word-level speech units and sentence-level speech units;

[0230] Extract acoustic features of each speech unit, including pitch, speech rate and rhythm;

[0231] Analyze the emotional intensity of each speech unit using a pre-trained model to generate an emotion distribution vector;

[0232] Process the text transcription to obtain text embedding information;

[0233] Combine the emotion distribution vector, the text embedding information and the acoustic features, and input the variable converter in the speech synthesis framework to generate prosodic features;

[0234] generating a mel-spectrogram based on the prosody features by a decoder;

[0235] converting the mel-spectrogram into a time-domain audio signal by a vocoder, outputting the time-domain audio signal as target emotional speech.

[0236] In one embodiment, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement the following steps:

[0237] obtaining audio data and corresponding text transcription;

[0238] segmenting the audio data into phoneme-level speech units, word-level speech units and sentence-level speech units;

[0239] extracting acoustic features of each speech unit, the acoustic features including pitch, speech rate and rhythm;

[0240] analyzing the emotional intensity of each speech unit using a pre-trained model to generate an emotional distribution vector;

[0241] processing the text transcription to obtain text embedding information;

[0242] combining the emotional distribution vector, the text embedding information and the acoustic features, and inputting a variable converter in a speech synthesis framework to generate prosody features;

[0243] generating a mel-spectrogram based on the prosody features by a decoder;

[0244] converting the mel-spectrogram into a time-domain audio signal by a vocoder, outputting the time-domain audio signal as target emotional speech.

[0245] It should be noted that the functions or steps that the computer readable storage medium or the computer device can implement described above can be referred to the related descriptions of the server side and the user side in the foregoing method embodiments, and to avoid repetition, they will not be described here.

[0246] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0247] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0248] It should be noted that if non-company software tools or components appear in the embodiments of the present application, they are only used for example introduction and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A speech synthesis method based on hierarchical emotion distribution, characterized in that, Includes the following steps: Acquire audio data and corresponding text transcription; The audio data is segmented into phoneme-level speech units, word-level speech units, and sentence-level speech units; The acoustic features of each speech unit are extracted, including pitch, speech rate, and rhythm; Obtain an emotion recognition model pre-trained based on an emotion dataset; The acoustic features of each speech unit are input into the emotion recognition model to determine the emotion intensity score of each speech unit in a predefined emotion category; Based on the emotional intensity score of each speech unit, an emotional distribution vector representing the emotional state is formed, where each dimension of the emotional distribution vector represents a predefined emotional category; The text transcription is processed to obtain text embedding information; The emotion distribution vector, the text embedding information, and the acoustic features are combined and input into the variable converter in the speech synthesis framework to generate prosodic features; A Mel spectrogram is generated based on the prosodic features using a decoder; Based on the prosodic features, the emotional features of the target emotional speech are analyzed; Based on the emotional features of the target emotional speech, a vocoder model is determined, and the sampling rate and frequency resolution of the vocoder model are adjusted. Configure the generation step size of the vocoder model to optimize speech generation speed and quality; Using the configured vocoder model, the Mel spectrogram is converted into a preliminary time-domain audio signal; The initial time-domain audio signal is post-processed to generate the final time-domain audio signal and output. The post-processing includes dynamic range compression, frequency equalization, and gain control.

2. The speech synthesis method based on hierarchical emotion distribution as described in claim 1, characterized in that, The audio data is segmented into phoneme-level speech units, word-level speech units, and sentence-level speech units, including: The audio data was aligned with the text transcription using a forced alignment tool. Based on the alignment results, the audio data is segmented into phoneme-level speech units; Based on word boundaries in the text transcription, the audio data is segmented into word-level speech units; The audio data is segmented into sentence-level speech units based on sentence boundaries in the text transcription.

3. The speech synthesis method based on hierarchical emotion distribution as described in claim 1, characterized in that, The acoustic features of each speech unit are extracted, including pitch, speech rate, and rhythm, including: For each phoneme-level speech unit, the fundamental frequency profile is determined to extract pitch acoustic features; For each word-level speech unit, the duration is measured to extract speech rate acoustic features; For each sentence-level speech unit, energy profiles are obtained to extract rhythmic acoustic features; Additional acoustic features, including formants and harmonic-to-noise ratio, are extracted from phoneme-level and word-level speech units. The extracted acoustic features are associated with the corresponding speech units.

4. The speech synthesis method based on hierarchical emotion distribution as described in claim 1, characterized in that, Processing the text transcription to obtain text embedding information includes: The text was transcribed into a phoneme sequence using a letter-to-phoneme conversion model. The phoneme sequence is encoded using a text encoder to generate embedded text information. Linguistic features, including part-of-speech, syntax, and semantics, are extracted from the text transcription. The linguistic features are fused into the text embedding information, and the contextual information of the text transcription is embedded into the fused text embedding information to generate enhanced text embedding information.

5. The speech synthesis method based on hierarchical emotion distribution as described in claim 1, characterized in that, The emotion distribution vector, the text embedding information, and the acoustic features are combined and input into a variable converter in the speech synthesis framework to generate prosodic features, including: The sentiment distribution vector is aligned with the corresponding text embedding information at the phoneme, word, and sentence levels. The aligned sentiment distribution vector, text embedding information, and acoustic features are fused to form a combined feature representation; An attention mechanism is applied to the combined feature representation to enhance the correlation between the sentiment distribution vector and the text embedding information; The combined feature representation enhanced by the attention mechanism is input into the variable converter in the speech synthesis framework to generate prosodic features.

6. A speech synthesis device based on hierarchical emotion distribution, characterized in that, The speech synthesis device based on hierarchical emotion distribution includes: The data acquisition module is used to acquire audio data and corresponding text transcriptions; The speech segmentation module is used to segment the audio data into phoneme-level speech units, word-level speech units, and sentence-level speech units. An acoustic feature extraction module is used to extract the acoustic features of each speech unit, including pitch, speech rate, and rhythm. The sentiment analysis module is used to obtain a sentiment recognition model pre-trained based on a sentiment dataset; input the acoustic features of each speech unit into the sentiment recognition model to determine the sentiment intensity score of each speech unit on a predefined sentiment category; and form a sentiment distribution vector representing the sentiment state based on the sentiment intensity score of each speech unit, wherein each dimension of the sentiment distribution vector represents a predefined sentiment category. The text embedding module is used to process the text transcription to obtain text embedding information; The feature fusion and variable converter module is used to combine the emotion distribution vector, the text embedding information and the acoustic features, and input them into the variable converter in the speech synthesis framework to generate prosodic features; Decoder module, used to generate Mel spectrogram based on the prosodic features using a decoder; A vocoder module is used to analyze the emotional features of the target emotional speech based on the prosodic features; determine a vocoder model based on the emotional features of the target emotional speech, and adjust the sampling rate and frequency resolution of the vocoder model; configure the generation step size of the vocoder model to optimize the speech generation speed and quality; use the configured vocoder model to convert the Mel spectrogram into a preliminary time-domain audio signal; perform post-processing on the preliminary time-domain audio signal to generate and output the final time-domain audio signal, wherein the post-processing includes dynamic range compression, frequency equalization, and gain control.

7. A computer device, characterized in that, The computer device includes a memory, a processor, and a speech synthesis program based on hierarchical emotion distribution stored in the memory and running on the processor, wherein the speech synthesis program based on hierarchical emotion distribution, when executed by the processor, implements the steps of the speech synthesis method based on hierarchical emotion distribution as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a speech synthesis program based on hierarchical emotion distribution, which, when executed by a processor, implements the steps of the speech synthesis method based on hierarchical emotion distribution as described in any one of claims 1-5.

Images