Speech synthesis method and device based on hierarchical emotion distribution, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a speech synthesis method based on hierarchical emotional distribution, which comprises the following steps: acquiring audio data and text transcription, segmenting audio into phoneme, word and sentence-level speech units, extracting acoustic characteristics such as pitch, speed and rhythm, generating an emotional distribution vector by using a pre-training model; the method comprises the following steps: processing text transcription to generate text embedded information, combining an emotion distribution vector, the text embedded information and acoustic features, generating rhythm features through a variable converter, generating a Mel-frequency spectrogram by a decoder based on the rhythm features, and converting the Mel-frequency spectrogram into a time domain audio signal by a vocoder and outputting the time domain audio signal. According to the method, the audio data are hierarchically segmented into phonemes, words and sentences, the acoustic features, the emotion distribution vectors and the text embedding information are combined, and the rhythm features are generated through the variable converter, so that fine control over emotion expression is achieved, and the naturalness and emotion expressive force of speech synthesis are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, device, equipment and storage medium based on hierarchical emotion distribution. Background Art

[0002] With the rapid development of financial services, the importance of user interaction experience in financial services is increasing. Intelligent interactive systems such as intelligent customer service and virtual assistants have been widely used in financial institutions to handle customer consultation, business processing and other related services. However, most intelligent voice systems are currently unable to express emotions naturally and accurately during the interaction process, resulting in a lack of affinity and naturalness in the interaction between customers and the system. As an important means to enhance the expressiveness of intelligent interactive systems, emotional speech synthesis technology can make virtual assistants and intelligent customer service more humane and enhance user experience by giving voice emotional colors.

[0003] In the existing technology, the adjustment of voice emotion often relies on a preset emotion model, and it is impossible to accurately adjust the emotion intensity according to actual needs. Customer needs in the financial field are complex and diverse, and emotional expression requires high precision. However, the existing technology cannot achieve fine-grained control of emotions, especially in financial business interactions, where emotional needs are difficult to respond effectively.

[0004] Emotional speech synthesis involves the adjustment of multiple prosodic features such as pitch, speech speed, and rhythm. However, the existing financial intelligent customer service systems mostly adjust these prosodic features in a rough manner, lacking pertinence and systematization, and it is difficult to effectively convey the emotional changes in speech through the adjustment of prosodic features. This results in the lack of naturalness in the speech generated by the system and fails to achieve the expected interactive effect.

[0005] The existing technology for modeling speech emotions is usually based on a single-level processing, which cannot accurately capture emotional changes in different speech units (such as phonemes, words, sentences, etc.). This modeling method is particularly insufficient in financial scenarios, because when customers consult different types of financial issues, they may need to express different emotions at the vocabulary and sentence levels, and existing technology is difficult to support this multi-level emotional expression.

[0006] Therefore, in the field of emotional speech synthesis, especially in the application scenarios of the financial industry, the existing technologies have problems such as insufficient emotion control accuracy, lack of systematic adjustment of rhythmic features, and weak hierarchical emotion modeling capabilities. These deficiencies affect the user experience of intelligent customer service and virtual assistants, and limit their in-depth application in financial services. Summary of the invention

[0007] The main purpose of the present invention is to provide a speech synthesis method, device, equipment and storage medium based on hierarchical emotion distribution, aiming to solve the technical problem that the existing technology lacks fine control and hierarchical modeling of speech emotions and is difficult to accurately express emotional changes in different speech units.

[0008] To achieve the above object, the present invention provides a speech synthesis method based on hierarchical emotion distribution, comprising:

[0009] Get audio data and corresponding text transcription;

[0010] Segmenting the audio data into phoneme-level speech units, word-level speech units, and sentence-level speech units;

[0011] Extracting acoustic features of each speech unit, wherein the acoustic features include pitch, speech rate and rhythm;

[0012] Use the pre-trained model to analyze the emotional intensity of each speech unit and generate an emotional distribution vector;

[0013] processing the text transcription to obtain text embedding information;

[0014] The sentiment distribution vector, the text embedding information and the acoustic feature are combined and input into a variable converter in a speech synthesis framework to generate a prosodic feature;

[0015] Generate a mel-spectrogram based on the prosodic features by a decoder;

[0016] The mel-spectrogram is converted into a time-domain audio signal by a vocoder, and the time-domain audio signal is output as the target emotional speech.

[0017] Furthermore, to achieve the above object, the present invention provides a speech synthesis device based on hierarchical emotion distribution, comprising:

[0018] A data acquisition module, used to acquire audio data and corresponding text transcription;

[0019] A speech segmentation module, used for segmenting the audio data into phoneme-level speech units, word-level speech units and sentence-level speech units;

[0020] An acoustic feature extraction module, used to extract the acoustic features of each speech unit, wherein the acoustic features include pitch, speech rate and rhythm;

[0021] The sentiment analysis module is used to analyze the sentiment intensity of each speech unit using a pre-trained model and generate a sentiment distribution vector;

[0022] a text embedding module for processing the text transcription to obtain text embedding information;

[0023] A feature fusion and variable converter module, used for combining the sentiment distribution vector, the text embedding information and the acoustic feature, and inputting the result into a variable converter in a speech synthesis framework to generate a prosodic feature;

[0024] A decoder module, configured to generate a mel-spectrogram based on the prosodic features through a decoder;

[0025] The vocoder module is used to convert the mel-spectrogram into a time-domain audio signal through a vocoder, and output the time-domain audio signal as the target emotional speech.

[0026] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a speech synthesis program based on hierarchical emotion distribution stored in the memory and executable on the processor, wherein the speech synthesis program based on hierarchical emotion distribution, when executed by the processor, implements the steps of the speech synthesis method based on hierarchical emotion distribution as described above.

[0027] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a speech synthesis program based on hierarchical emotion distribution is stored, and when the speech synthesis program based on hierarchical emotion distribution is executed by a processor, the steps of the speech synthesis method based on hierarchical emotion distribution as described above are implemented.

[0028] Beneficial effects: The present invention relates to the field of artificial intelligence technology, and discloses a speech synthesis method based on hierarchical emotion distribution, which obtains audio data and text transcription, divides the audio into speech units at the phoneme, word and sentence levels, extracts acoustic features such as pitch, speech rate and rhythm, generates emotion distribution vectors using a pre-trained model, processes text transcription to generate text embedding information, combines the emotion distribution vector, text embedding information and acoustic features, generates prosodic features through a variable converter, generates a mel spectrogram based on the prosodic features by a decoder, and converts the mel spectrogram into a time-domain audio signal and outputs it. The present invention hierarchically divides audio data into phoneme, word and sentence levels, and combines acoustic features, emotion distribution vectors and text embedding information, and generates prosodic features using a variable converter, thereby achieving fine control of emotional expression. The decoder generates a mel spectrogram based on these prosodic features, and finally converts the mel spectrogram into a time-domain audio signal through a vocoder, generating natural and expressive emotional speech, effectively solving the problems of inaccurate emotional expression and insufficient hierarchical modeling in the prior art, and significantly improving the naturalness and emotional expressiveness of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0030] Figure 1 A schematic diagram of an application environment of a speech synthesis method based on hierarchical emotion distribution in an embodiment of the present invention;

[0031] Figure 2 It is a flow chart of an embodiment of a speech synthesis method based on hierarchical emotion distribution of the present invention;

[0032] Figure 3 It is a functional module diagram of a preferred embodiment of a speech synthesis device based on hierarchical emotion distribution of the present invention;

[0033] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0034] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0035] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0036] The speech synthesis method based on hierarchical emotion distribution provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the user end communicates with the server end through a network. The server end can obtain audio data and text transcription through the user end, segment the audio into speech units at the phoneme, word and sentence levels, extract acoustic features such as pitch, speech rate and rhythm, generate an emotion distribution vector using a pre-trained model, process the text transcription to generate text embedding information, combine the emotion distribution vector, text embedding information and acoustic features, generate prosodic features through a variable converter, the decoder generates a Mel spectrogram based on the prosodic features, and the vocoder converts the Mel spectrogram into a time-domain audio signal and outputs it. The present invention hierarchically segments audio data into phoneme, word and sentence levels, and combines acoustic features, emotion distribution vectors and text embedding information, and generates prosodic features using a variable converter, thereby achieving fine control of emotional expression. The decoder generates a Mel spectrogram based on these prosodic features, and finally converts the Mel spectrogram into a time-domain audio signal through a vocoder to generate a natural and expressive emotional speech, which effectively solves the problems of imprecise emotional expression and insufficient hierarchical modeling in the prior art, and significantly improves the naturalness and emotional expressiveness of speech synthesis. The user end may be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices. The server end may be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0037] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a speech synthesis method based on hierarchical emotion distribution provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0038] like Figure 2 As shown, the speech synthesis method based on hierarchical emotion distribution proposed by the present invention comprises the following steps:

[0039] S10, obtaining audio data and corresponding text transcription;

[0040] In this embodiment, the audio data is usually an existing audio input by a user or extracted from a database. The audio data can be a recording of a natural language, and the voice file can be an input generated by a voice recognition system, or it can be a pre-recorded audio file.

[0041] Transcription is the text information corresponding to the audio, which is usually extracted from the audio through speech recognition technology (such as ASR system) or provided by the user. For example, in the financial field, the audio of the conversation between the customer and the intelligent customer service and its corresponding text content will become the core input in this technical step.

[0042] Real-time audio data can be obtained through voice acquisition devices (such as microphones), or recorded audio files can be extracted from databases or storage by calling external interfaces. In practical applications, customer service telephones of financial institutions, voice chat records, or interactive recordings of automatic voice assistants can all be used as sources of audio data. Text transcription can be generated through speech recognition technology. In specific implementations, automatic speech recognition (ASR) technology is used to extract text content from audio data. The ASR system analyzes the acoustic features of the audio and converts it into corresponding text content. In financial scenarios, ASR can be used to automatically transcribe customers' verbal consultation records into text.

[0043] In one specific implementation, a hybrid processing architecture is introduced, where initial recognition is performed locally on the device, and when the device's computing power is insufficient, higher-precision recognition is performed by uploading the audio to the cloud. For example, when a customer uses a bank's mobile app, the system can first process simple voice input locally, and if it involves complex content, the audio is sent to the cloud for further processing.

[0044] By obtaining audio data and the corresponding text transcription, it provides basic input for the subsequent speech synthesis process, which can accurately reflect the user's voice content and corresponding text information.

[0045] S20, dividing the audio data into phoneme-level speech units, word-level speech units, and sentence-level speech units;

[0046] In this embodiment, the audio data is segmented and refined into speech units at the phoneme level, word level, and sentence level. The speech units at different levels represent different levels of speech, as follows:

[0047] Phonemes are the smallest unit of speech. Phoneme segmentation is to break down audio data into the most basic speech sound segments. This step is usually achieved through forced alignment tools (such as Montreal ForcedAligner).

[0048] Word-level segmentation divides the speech stream into lexical levels. This step usually relies on phoneme-level alignment information and combines word boundaries in the text transcription to map the speech to each specific word.

[0049] Sentence-level segmentation processes a larger range of speech, dividing the start and end positions of sentences by analyzing sentence boundaries (such as punctuation marks) in text transcription. This segmentation method is more suitable for processing long speech data with coherent meaning.

[0050] Use tools such as Montreal ForcedAligner or other alignment tools to force align audio data with text transcriptions to generate a mapping of phonemes to speech. This tool segments audio data into small phoneme segments by identifying the start and end times of phonemes in the audio. The system first identifies each phoneme through an acoustic model, and then matches these phonemes to audio segments and generates timestamps, so that the audio file is accurately segmented into phoneme units.

[0051] Based on the results of phoneme segmentation, the phoneme information is mapped to the corresponding word. Using the word boundary information in the text transcription, the system can identify the start and end positions of each word and divide the audio segment into corresponding word units. By matching the word boundaries and the start and end times of the phonemes, the system can accurately locate the corresponding segment of each word in the audio and perform segmentation.

[0052] Identify sentence boundaries based on punctuation and grammatical structures in text transcriptions, and divide audio data into sentence structures. For example, by identifying punctuation marks such as commas and periods, determine the start and end points of each sentence in the audio. Text processing tools perform syntactic analysis on the text based on punctuation, grammatical analysis, etc., divide the text into sentence units, and then segment the corresponding voice data.

[0053] For example, when a customer asks "I want to check my account balance" by voice, the system will first segment the customer's voice data into speech units at the phoneme, word, and sentence levels. Specifically, the phoneme-level segmentation can accurately identify every pronunciation detail, the word level is used to identify the user's query keywords such as "account balance", and the sentence level is used to identify the entire query intent. This hierarchical processing enables the system to accurately process and recognize emotions for each unit.

[0054] By performing multi-level segmentation on audio data, the system can accurately process speech content at the phoneme, word, and sentence levels. This segmentation method lays the foundation for subsequent sentiment analysis and speech synthesis, and can significantly improve the accuracy and naturalness of speech synthesis.

[0055] S30, extracting acoustic features of each speech unit, wherein the acoustic features include pitch, speech rate and rhythm;

[0056] In this embodiment, acoustic features are extracted from each segmented speech unit (such as phonemes, words, sentences). These features directly affect the emotional expression and naturalness of the speech, and mainly include:

[0057] Pitch: Pitch is the frequency characteristic of speech, which reflects the frequency of the speaker's vocal cords vibrating. High pitch is usually associated with high emotion or excitement, while low pitch is associated with calm or sad emotions. Pitch changes are an important part of rhythm.

[0058] Speech Rate: Speech rate refers to the speed of pronunciation of speech units, which has an important impact on the rhythm of speech. Faster speech rates usually convey nervous or excited emotions, while slower speech rates convey calm or heavy emotions.

[0059] Rhythm: Rhythm refers to the distribution pattern of stress and length of speech units. Different rhythm patterns can convey different emotions and semantic relationships.

[0060] The system detects pitch changes in speech through acoustic analysis tools (such as Praat, OpenSMILE). The process of pitch extraction usually relies on fast Fourier transform (FFT) or autocorrelation methods to identify fluctuations in the fundamental frequency in the speech signal. For each segmented speech unit, the system extracts the fundamental frequency curve (F0) to generate pitch features. For example, when a user asks a financial question, the system can identify changes in the customer's emotions, such as anxiety or dissatisfaction, and reflect them through pitch changes.

[0061] The speed characteristics are obtained by calculating the duration of each phoneme or word in the speech unit. A speech unit with a short duration indicates a fast speech rate, while a long duration indicates a slow speech rate. By marking the start and end of each phoneme with a timestamp, the system can calculate the duration of each word or sentence and determine its speed characteristics. For example, when a customer is in a hurry, the system can detect a faster speech rate, which is often associated with anxiety or urgency.

[0062] Rhythm is mainly extracted by analyzing the stress and length patterns in speech. This step can use tools based on prosody analysis to detect the stress position and length distribution of speech and identify the rhythmic structure of sentences. The system analyzes the length of each speech unit and detects the stress distribution to generate the rhythmic pattern of speech. In financial interactions, the rhythmic changes in customers' voices can reflect their emotional fluctuations, thereby helping the system to respond better.

[0063] Example description: In the financial intelligent customer service system, when a customer asks through a voice assistant, "Why was 200 yuan deducted from my account?", the system will process the customer's voice and extract its pitch, speaking speed and rhythm features. By analyzing the increase in speaking speed and the change in pitch, the system can detect that the customer may be in an anxious state. Then, when generating a reply, the system can respond in a more soothing and friendly tone by adjusting the speaking speed and lowering the pitch: "We have found the relevant information for you and will explain it to you in detail later. Please be patient." This emotional speech synthesis method can effectively alleviate customers' anxiety and improve their satisfaction with financial services.

[0064] By extracting acoustic features such as pitch, speaking speed and rhythm, we can capture the emotional information hidden in the voice, and through the subsequent emotional synthesis process, make the voice output more natural and expressive. In the financial field, the refined processing of this acoustic feature can help the intelligent customer service system understand the customer's emotional state, so as to make a more considerate and efficient response, significantly improving the user experience.

[0065] S40, analyzing the emotion intensity of each speech unit using the pre-trained model to generate an emotion distribution vector;

[0066] In this embodiment, the emotion intensity of each segmented speech unit is analyzed by a pre-training model, and an emotion distribution vector is generated. Each speech unit (phoneme, word, sentence) is emotion-recognized by a pre-training model, and a multi-dimensional vector is output to represent the emotion characteristics of the unit. The model is trained with a large amount of emotion data, and can extract its emotion characteristics based on the input speech unit. The pre-training model may be a model based on a deep neural network (such as BERT, GPT, etc.), which can extract emotion characteristics through acoustic features based on audio.

[0067] The system analyzes the emotional intensity of speech units based on acoustic features such as pitch, speaking rate, rhythm, etc. For example, higher pitch and faster speaking rate may correspond to stronger emotions such as anger or excitement, while lower pitch and slower speaking rate may correspond to calm or sad emotions. The pre-trained model is able to match these acoustic features with pre-defined emotion categories.

[0068] The emotion distribution vector is a multi-dimensional vector, each dimension of which represents a predefined emotion category (such as joy, anger, sadness, happiness, etc.). After analyzing the emotion intensity, the pre-trained model will output the emotion distribution vector of the speech unit to represent the intensity of each emotion.

[0069] The system first extracts the acoustic features of the speech unit (pitch, speaking rate, rhythm, etc.) and inputs these features into the pre-trained model. The pre-trained model may be trained using emotion-annotated datasets (such as IEMOCAP, MELD). The pre-trained model uses these acoustic features to infer the emotional intensity of the speech unit. The model matches these features with its internal emotion classification system to generate an emotional intensity score. The pre-trained model outputs an emotional distribution vector, where each dimension in the vector represents an emotional type. The vector represents the distribution of the speech unit across multiple emotional dimensions. For example, a speech unit may contain both "happy" and "anxious" components, but with different weights. The resulting emotional vector can represent the relative intensity of these two emotions.

[0070] Example description: In the intelligent voice assistant scenario in financial services, customers may make some consultation or query requests through the voice assistant, such as "I want to check my recent credit card transaction records." In this scenario, the system first processes the voice data through the previous steps to obtain segmented voice units. Then, the system uses the pre-trained model to analyze the emotional intensity of each voice unit and generate an emotional distribution vector.

[0071] Specifically, the pre-trained model analyzes the acoustic features of speech (pitch, speaking speed, rhythm, etc.). For example, when a customer says "credit card transaction", the speed of speech may be slightly faster and the pitch may also increase, which reflects the customer's emotional tendency to obtain information quickly. Through these features, the pre-trained model can judge the customer's emotional state, which may contain anxious and impatient components. Then, the model generates an emotional distribution vector, and each dimension of the vector may represent different emotional intensities, such as anxiety for high intensity and happiness for low intensity.

[0072] By using the pre-trained model to analyze the intensity of emotion, it is possible to generate accurate emotion distribution vectors on different speech units, allowing the system to capture the subtle emotional changes of each speech unit. Combined with the emotion distribution vector, the system can more accurately express different emotional states when generating speech, improving the expressiveness and naturalness of speech synthesis.

[0073] S50, processing the text transcription to obtain text embedding information;

[0074] In this embodiment, text embedding information for speech synthesis is generated by processing text transcription data. Text embedding is a process of converting text information into vector representation, which compresses the semantic, lexical and structural information in the text into numerical vectors for easy model processing. In this technical solution, text embedding information is used to generate natural speech in conjunction with sentiment distribution vectors and acoustic features.

[0075] Convert the text transcribed data from letter form to phoneme form. Phonemes are the basic units of speech, and this step is achieved through language models or conversion tools. The letter-to-phoneme conversion enables the system to closely connect text data with speech units.

[0076] The converted phoneme sequence is encoded into a vector representation through a text encoder. Common text encoders include pre-trained models such as BERT and GPT, which can capture semantic information in text and represent this information as a low-dimensional embedding vector.

[0077] Extract additional linguistic features from the text, such as part of speech, syntactic structure, and semantic information. These linguistic features provide more semantic and contextual information for text embedding, allowing the model to better understand the meaning of the text and generate more contextual speech in combination with sentiment information.

[0078] By integrating the above processing results, the text information is embedded into a low-dimensional vector space to form the final text embedding vector. This embedding vector contains the semantics, phonemes, and linguistic features of the text, and serves as the basic input for subsequent speech synthesis and emotional expression.

[0079] Use letter-to-phoneme conversion models (such as the G2P model) to convert text data into phoneme sequences to ensure that the pronunciation information of the text is consistent with the speech processing. Encode the phoneme sequence of the text through language models such as BERT to generate a vectorized representation of the text. This encoder can not only capture the semantic information of words, but also carefully model the meaning of words in context. Extract the grammatical, part-of-speech, and semantic information of the text through natural language processing (NLP) tools. Attach these features to the text embedding vector to enhance the speech synthesis model's ability to understand the text. Fuse all the processed information to generate the final text embedding vector, providing complete text semantic input for subsequent speech synthesis.

[0080] By processing text transcription data to generate text embedding information, the system can effectively embed the pronunciation information, semantics, and contextual information of the text into a low-dimensional vector, providing richer input for the speech synthesis model.

[0081] S60, combining the emotion distribution vector, the text embedding information and the acoustic feature, and inputting the result into a variable converter in a speech synthesis framework to generate a prosodic feature;

[0082] In this embodiment, the sentiment distribution vector, text embedding information and acoustic features are combined, and then the final prosodic features are generated through a variable transformer. These prosodic features are used for speech synthesis to control the pitch, rhythm, duration, etc. of the speech, thereby producing natural and emotional speech expressions.

[0083] The sentiment distribution vector represents the sentiment characteristics of each speech unit. The different dimensions of each vector correspond to different sentiment categories, indicating the intensity of the speech unit in different sentiments. This vector is generated by the previous pre-training model.

[0084] Text embedding information represents the semantic and contextual information of the text content corresponding to the speech unit. This information is generated by a text encoder, which closely combines the semantic information of the text with the speech synthesis process to ensure that the generated speech is consistent with the meaning of the text.

[0085] Acoustic features include pitch, speaking speed, rhythm, etc., which are used to control the physical characteristics of speech. These features provide the basis for the subsequent generation of prosodic features, ensuring that the emotional expression in the speech synthesis process is natural and smooth.

[0086] The variable transformer is a key module that combines sentiment distribution vectors, text embedding information, and acoustic features. It transforms these inputs into prosodic features such as pitch, duration, energy, etc. through deep learning models or other mathematical transformations for subsequent speech generation.

[0087] The system first merges the sentiment distribution vector, text embedding information and acoustic features to construct a joint representation. The key to this step is to effectively combine the features of different information sources through multimodal fusion technology, so that the generated prosodic features can simultaneously express sentiment, text meaning and speech features.

[0088] The joint representation is fed into a variational transformer, which can be a complex neural network (such as a multi-layer perceptron or a convolutional neural network) whose task is to generate prosodic information such as pitch, duration, and volume based on the multimodal features of the input. The transformer is trained to capture the complex relationship between textual, sentiment, and acoustic features.

[0089] The output of the variable transformer is a prosodic feature vector that represents the pitch variation, speech rate adjustment, and energy distribution of the speech. Through these prosodic features, the system can generate more natural and emotional speech.

[0090] By multimodally fusing sentiment distribution vectors, text embedding information, and acoustic features, and using variable transformers to generate prosodic features, the system can achieve more sophisticated speech synthesis.

[0091] S70, generating a mel-spectrogram based on the prosody feature through a decoder;

[0092] In this embodiment, the generated prosodic features are converted into a Mel-spectrogram through a decoder. The Mel-spectrogram is a representation of the time and frequency distribution of a speech signal and is a key intermediate representation in the speech synthesis process. It is used to characterize the frequency components and time domain information of a speech signal and is generated from the prosodic features through a decoder to provide a basis for subsequent speech synthesis.

[0093] The input to the decoder is the prosodic features (including pitch, duration, energy, etc.) generated from the variable transformer, which are used to control the specific presentation of the synthesized speech. The decoder maps these features to the frequency domain to generate a mel-spectrogram.

[0094] Mel-spectrogram is a Mel-transformed spectrogram, where the frequency scale is readjusted according to the perception of the human auditory system, making it more suitable for speech synthesis. It captures the temporal changes of different frequency components in the speech signal, allowing the speech synthesis system to better simulate real speech.

[0095] The decoder can be implemented based on different models, such as long short-term memory network (LSTM), variational autoencoder (VAE), etc. The role of the decoder is to convert the rhythmic features into frequency features in the time domain, and the generated Mel spectrum is used for the subsequent vocoder to generate time-domain audio signals.

[0096] The decoder receives the prosodic features (pitch, duration, energy, etc.) generated by the variable transformer, which control the detailed performance in speech synthesis. The decoder maps the prosodic features to the frequency domain through a deep learning model or other mathematical transformation to generate a Mel-spectrogram. This process involves converting time series data into a frequency domain representation and adjusting the intensity of different frequency components according to the prosodic features. The Mel-spectrogram compresses the spectrum through a Mel filter to make the frequency components more consistent with the perception of the human ear, and finally obtains a frequency-time distribution diagram to provide input for the subsequent vocoder step.

[0097] By converting prosodic features into mel-spectrograms through the decoder, the system can accurately capture the frequency changes and time domain information in the speech, ensuring that the generated speech synthesis effect is natural and expressive. Through the precise control of prosodic features, parameters such as pitch, speech rate, and volume in the speech synthesis process can be finely adjusted to generate speech that is more in line with human auditory perception.

[0098] S80, converting the Mel-spectrogram into a time-domain audio signal through a vocoder, and outputting the time-domain audio signal as the target emotional speech.

[0099] In this embodiment, the mel-spectrogram generated by the decoder is converted into a time-domain audio signal, that is, the final speech signal. The vocoder is a key component of the speech synthesis process, which converts the information in the frequency domain (mel-spectrogram) into an actual audio signal and generates an audible speech output.

[0100] The task of the vocoder is to convert the frequency domain representation in the Mel-spectrogram into a time domain audio signal. Common vocoders include WaveNet, Griffin-Lim, MelGAN, etc., which can generate high-quality speech output based on the spectrogram.

[0101] The input to the vocoder is the previously generated Mel spectrogram, which represents the frequency distribution of the speech. The spectrogram contains information such as pitch, volume, and duration, but it is a frequency domain representation, not a direct audio signal.

[0102] The vocoder converts the mel-spectrogram in the frequency domain into an audio signal in the time domain. The audio signal generated by the vocoder is a speech that can be played directly and can reflect the prosodic features and emotional information of the previous input.

[0103] The final time-domain audio signal contains all the information of emotional speech, including rhythmic features such as pitch, rhythm, duration, etc. These emotional speech can be directly output by intelligent customer service or virtual assistants to achieve natural voice interaction with emotional expression.

[0104] The system first inputs the Mel spectrogram into the vocoder, which converts the frequency information and generates the corresponding audio signal. For example, with the WaveNet vocoder, the model generates a time domain signal through recursion. WaveNet is able to capture complex patterns and details in speech and generate very natural speech output. When using a fast vocoder such as MelGAN, the vocoder generates a time domain audio signal based on the Mel spectrogram. MelGAN uses a generative adversarial network to accelerate the generation process, ensuring that high-quality audio signals are generated while processing efficiently. The time domain audio signal output by the vocoder is the final speech result, which contains the details of the prosodic features and emotional expressions, forming a natural and fluent speech. The generated time domain audio signal is output as the target emotional speech, and the system can play it directly to the user. These voices express the emotional characteristics required by the user, making the speech synthesis system have a more natural human-computer interaction effect.

[0105] Example description: In a financial service scenario, a customer may make a request through an intelligent voice assistant, such as "Please help me check last month's transaction records." After the system generates rhythmic features and converts them into Mel spectrograms, the spectrograms are input into the MelGAN vocoder, which converts them into time-domain audio signals. The generated emotional speech may contain a gentle tone and a soothing speaking speed to appease the customer's emotions. The system replies through voice: "Your transaction records have been queried, and the total amount is 2,000 yuan." The audio is generated by MelGAN, with natural sound quality and emotional expression, achieving high-quality user interaction.

[0106] By converting the Mel-spectrogram into a time-domain audio signal through a vocoder, the frequency domain information can be accurately converted into high-quality speech output. Using advanced vocoders such as WaveNet or MelGAN can generate more natural and emotional speech. In financial scenarios, the voice system can interact with users with extremely high fidelity and emotional expression, significantly improving the user experience.

[0107] The present invention relates to the field of artificial intelligence technology, and discloses a speech synthesis method based on hierarchical emotion distribution, which obtains audio data and text transcription, divides the audio into speech units at the phoneme, word and sentence levels, extracts acoustic features such as pitch, speech rate and rhythm, generates an emotion distribution vector using a pre-trained model, processes the text transcription to generate text embedding information, combines the emotion distribution vector, text embedding information and acoustic features, generates prosodic features through a variable converter, generates a mel spectrogram based on the prosodic features by a decoder, and converts the mel spectrogram into a time-domain audio signal and outputs it. The present invention hierarchically divides audio data into phoneme, word and sentence levels, and combines acoustic features, emotion distribution vectors and text embedding information, and generates prosodic features using a variable converter, thereby achieving fine control of emotional expression. The decoder generates a mel spectrogram based on these prosodic features, and finally converts the mel spectrogram into a time-domain audio signal through a vocoder, generating a natural and expressive emotional speech, effectively solving the problems of inaccurate emotional expression and insufficient hierarchical modeling in the prior art, and significantly improving the naturalness and emotional expressiveness of speech synthesis.

[0108] In one embodiment, the above S20 includes:

[0109] S201, aligning the audio data with the text transcription using a forced alignment tool;

[0110] S202, dividing the audio data into phoneme-level speech units according to the alignment result;

[0111] S203, dividing the audio data into word-level speech units according to word boundaries in the text transcription;

[0112] S204: Segment the audio data into sentence-level speech units according to sentence boundaries in the text transcription.

[0113] In this embodiment, the audio data is segmented hierarchically so that subsequent processing can perform speech analysis and synthesis at different granularities. The segmentation process relies on the alignment of the audio data with the text transcription and is divided according to the boundaries of phonemes, words and sentences.

[0114] Forced alignment tools, such as Montreal ForcedAligner, are used to align audio data with text transcriptions. Forced alignment is used to accurately determine the timestamps of each speech unit (phoneme, word, sentence) in the audio. The tool uses acoustic and language models to calculate the start and end times of each unit based on the sound features and text content in the audio.

[0115] Based on the alignment results, the audio data is segmented into phoneme-level speech units. Phoneme is the smallest unit of speech, which reflects the specific pronunciation in speech. After segmentation, each phoneme unit has a precise start and end time for subsequent feature extraction and sentiment analysis.

[0116] The audio is further segmented into word-level speech units based on word boundaries in the text transcription. Word-level segmentation can combine multiple phonemes into one word to better express semantics. This segmentation process is achieved by matching word boundaries and phoneme start and end times in the text transcription.

[0117] Finally, the audio data is segmented into sentence-level units based on sentence boundaries in the text (such as punctuation or grammatical structures). This segmentation method is suitable for processing longer speech segments, helping the system understand complete sentence expressions and providing higher-level semantic information for the speech synthesis process.

[0118] Use tools such as Montreal ForcedAligner to align audio data with text transcriptions. This tool generates timestamps based on the acoustic features of the audio and the phoneme information in the text, ensuring that each phoneme corresponds to its position in the audio. Based on the alignment results, the audio file is divided into individual phonemes. This step ensures that the start and end times of each pronunciation segment are accurately recorded and provides a basis for subsequent acoustic feature extraction. Based on the word boundaries in the text transcription, the system aggregates phoneme units into word-level units. By detecting the start and end points of words in the text, the system can generate corresponding audio segments. Finally, based on the punctuation or syntactic structure in the text, the system divides the audio into sentence units. By aligning the sentence structure in the text, sentiment analysis and speech synthesis can be better performed.

[0119] Example description: In financial services, customers initiate requests through voice assistants, such as "Please help me check last month's transaction records." The system first aligns the customer's voice with the text transcription through a forced alignment tool to determine the timestamp of each phoneme. Next, the system divides the audio into phoneme-level units to ensure that the start and end times of each pronunciation are accurately marked. Then, by analyzing word boundaries, the system combines phonemes into word units and identifies key information such as "query" and "transaction records." Finally, based on the sentence structure, the system recognizes complete sentences and performs corresponding query operations. Through this layered processing approach, the system can respond to customer requests more accurately and quickly, and provide natural voice feedback when generating voice responses, thereby improving customer experience.

[0120] By segmenting the audio data into units at the phoneme, word, and sentence levels, the system can process the speech in a fine-grained manner to ensure the naturalness and accuracy of speech synthesis. In financial services, this segmentation method can help the intelligent customer service system accurately analyze the user's voice input and perform personalized emotional expression based on each level of speech unit, providing customers with more humane voice services.

[0121] In one embodiment, the above S30 includes:

[0122] S301, for each phoneme-level speech unit, determining a fundamental frequency contour to extract a pitch acoustic feature;

[0123] S302, for each word-level speech unit, measuring the duration to extract speech rate acoustic features;

[0124] S303, for each sentence-level speech unit, obtaining an energy profile to extract rhythmic acoustic features;

[0125] S304, extracting additional acoustic features including formant and harmonic-to-noise ratio from the phoneme-level speech units and the word-level speech units;

[0126] S305: Associating the extracted acoustic features with corresponding speech units.

[0127] In this embodiment, the system extracts multiple acoustic features from speech units at the phoneme, word, and sentence levels for subsequent emotional speech synthesis. The main features include pitch, speech rate, and rhythm, and are combined with other acoustic features such as formant and harmonic-to-noise ratio to generate more accurate speech representation through multi-level analysis.

[0128] For each phoneme-level speech unit, the system extracts pitch information through the pitch contour. Pitch is a frequency feature in speech that reflects the speed of vocal cord vibration and is one of the important features for conveying speech emotions. The pitch contour can capture tiny pitch changes at the phoneme level, helping the system to accurately express the emotional intensity in speech.

[0129] For each word-level speech unit, the system extracts the speech rate feature by measuring the duration. The speed of speech can convey the speaker's emotional state. A faster speech rate often indicates anxiety or excitement, while a slower speech rate often indicates calmness or seriousness.

[0130] For each sentence-level speech unit, the system extracts rhythmic features by obtaining energy contours. Energy contours represent the changes in the intensity of speech and are usually used to express the rhythmic structure of sentences. By analyzing the energy changes at the sentence level, the system can generate more natural speech expressions.

[0131] The fundamental frequency contour (F0) of each phoneme is extracted using the Fast Fourier Transform (FFT) or autocorrelation method to capture pitch variations. Fundamental frequency extraction tools can use software such as Praat or OpenSMILE. Based on the start and end times of the phonemes, the system calculates the duration of each word-level speech unit. The duration measurement can capture changes in speech rate and help the system identify emotional states. Using the energy contour analysis tool, the system generates energy contours by calculating the intensity changes of the speech waveform. These energy changes can capture the rhythmic characteristics of speech, allowing the system to express sentence-level prosody.

[0132] Resonance peaks are peaks in the speech spectrum that represent the resonance frequency of the speech vocal tract, and are mainly used to distinguish different vowels and consonants. Phonemes are the smallest pronunciation units of speech. Extracting resonance peaks at the phoneme level can capture the pronunciation characteristics of speech units. Use linear predictive coding (LPC) or Fourier transform to perform frequency analysis on speech units at the phoneme level. The frequency of resonance peaks is closely related to the shape and size of the vocal tract, and the changes in these frequencies can distinguish the pronunciation positions of vowels and consonants. For example, the vowels / a / and / i / have different resonance peak distributions. Through these features, the system can recognize and simulate complex speech features.

[0133] The harmonic noise ratio reflects the ratio of harmonic components to noise components in speech. Harmonic components mainly come from vocal cord vibration, while noise is the irregular part of speech, such as airflow noise or hoarseness. The higher the harmonic noise ratio, the clearer and more stable the speech. At the word level, extracting the harmonic noise ratio can measure the clarity of speech. The ratio of harmonic components to noise components is usually calculated by autocorrelation function or other periodic analysis methods. A higher harmonic noise ratio indicates that the speech has a stronger harmonic component, which is suitable for controlling and optimizing clarity during speech synthesis and recognition.

[0134] Extracting formants at the phoneme level helps capture subtle pronunciation features and ensures that the pronunciation details of the original speech can be accurately reproduced during speech synthesis. Extracting the harmonic-to-noise ratio at the word level helps evaluate the clarity and stability of speech, especially in long words or sentences, where maintaining clarity and naturalness of speech is particularly important.

[0135] This embodiment can capture the emotional changes in speech by accurately extracting acoustic features such as pitch, speaking rate and rhythm, and combine additional acoustic features such as resonance peak and harmonic-noise ratio to enhance the naturalness and emotional expression of speech synthesis.

[0136] In one embodiment, the above S40 includes:

[0137] S401, obtaining an emotion recognition model pre-trained based on an emotion dataset;

[0138] S402, inputting the acoustic features of each speech unit into the emotion recognition model to determine the emotion intensity score of each speech unit in a predefined emotion category;

[0139] S403, based on the emotion intensity score of each speech unit, forming an emotion distribution vector representing the emotion state, wherein each dimension of the emotion distribution vector represents a predefined emotion category.

[0140] In this embodiment, the acoustic features of the speech unit are analyzed by using a pre-trained emotion recognition model to determine the emotion intensity of the speech and generate an emotion distribution vector. The emotion distribution vector is used to represent the emotional state of each speech unit, helping the system to express emotions during speech synthesis.

[0141] The pre-trained model is a deep learning model trained on a dataset with emotion annotations. The model is able to recognize different emotion categories (such as happiness, anger, sadness, etc.). Commonly used datasets include IEMOCAP, MELD, etc. These datasets guide the model to learn emotion patterns by annotating emotion categories.

[0142] The system inputs the acoustic features of each speech unit (such as pitch, speech rate, rhythm, formant, etc.) into the pre-trained model. Acoustic features are the basic information for sentiment analysis and can reflect the emotional expression in speech. For example, higher pitches usually reflect excitement or anger, while lower pitches may reflect calm or sad emotions.

[0143] The pre-trained model calculates the emotional intensity scores of each speech unit in different emotional categories based on the input acoustic features. These scores reflect the performance of speech in different emotional dimensions. For example, the intensity score of a speech unit in the "anger" category may be 0.8, while it may be 0.2 in the "joy" category.

[0144] Based on the emotion intensity score of each speech unit, the system generates an emotion distribution vector. Each dimension of the vector corresponds to a predefined emotion category and represents the intensity of the speech unit in the emotion category. For example, a three-dimensional emotion distribution vector may represent the three categories of "anger", "sadness" and "joy", and the value of each dimension corresponds to the intensity of the emotion.

[0145] The system uses a pre-trained emotion recognition model based on an emotion annotation dataset. The model can be LSTM, Transformer, or CNN, and is trained with supervised learning on an emotion dataset. The acoustic features of each speech unit extracted previously are input into the emotion recognition model. The model uses these acoustic features to determine the emotional state expressed by the speech unit. The model calculates the emotion intensity score for each speech unit in different emotion categories. These scores change with changes in the pitch, rhythm, etc. of the speech, and the model is able to recognize small differences in emotion. Finally, the system generates an emotion distribution vector based on the emotion intensity score. This vector will be used in the subsequent speech synthesis stage to ensure that the generated speech expresses the corresponding emotion.

[0146] In this embodiment, by using the pre-trained emotion recognition model to analyze the acoustic features of each speech unit for emotion intensity, the system can generate an emotion distribution vector representing the emotional state of each speech unit. These vectors provide accurate emotion input for subsequent speech synthesis, ensuring that the generated speech can convey accurate emotion information.

[0147] In one embodiment, the above S50 includes:

[0148] S501, converting the text transcription into a phoneme sequence using a letter-to-phoneme conversion model;

[0149] S502, encoding the phoneme sequence through a text encoder to generate text embedding information;

[0150] S503, extracting linguistic features including part of speech, syntax and semantics from the text transcription;

[0151] S504, integrating the linguistic features into the text embedding information, and embedding the context information of the text transcription into the integrated text embedding information to generate enhanced text embedding information.

[0152] In this embodiment, the text transcription is processed to generate embedded information that can reflect the text semantics and context. The embedded information can provide a semantic basis for subsequent speech synthesis.

[0153] The system first converts the text transcription into a phoneme sequence using a letter-to-phoneme (G2P) conversion model. Phonemes are the basic units of speech, and converting letters into phonemes helps with the subsequent processing of the correspondence between speech and text. The G2P model maps letters in the text transcription to corresponding phonemes through mapping rules or learning models.

[0154] The text encoder (such as BERT or GPT, etc.) encodes the phoneme sequence to generate preliminary text embedding information. The text encoder captures the semantics, context, and relationship between words in the text through a deep learning model. The embedded information is a low-dimensional vector representation that can reflect the core meaning of the text.

[0155] The system extracts linguistic features such as part of speech, syntactic structure, and semantic information from the text transcription. Part of speech (such as verbs, nouns, etc.) and syntactic structure (such as subject-verb-object relationship) can help the system better understand the semantic hierarchy and structure of the text. Semantic information includes the specific meaning of the word, contextual association, etc.

[0156] The extracted linguistic features are fused with the text embedding information to enhance the expressiveness of the embedded information. The fusion process includes embedding features such as part of speech and syntax into the text embedding vector so that it contains more semantic information. In addition, the system also embeds the contextual information of the text to ensure that the generated embedding information can not only reflect the meaning of individual words, but also capture the semantic dependencies between contexts.

[0157] Use a G2P model (such as CMU Pronouncing Dictionary or Phonetisaurus) to convert letters in the text transcription into phonemes. The system converts the letter sequence into a phoneme sequence based on the pronunciation rules of each letter to provide pronunciation information for subsequent speech synthesis. The converted phoneme sequence is input into a text encoder (such as BERT or GPT) to generate text embedding information. The text encoder captures semantic and contextual information through a multi-layer network and generates a vectorized representation that reflects the language features in the text. Use natural language processing (NLP) tools to analyze the text transcription and extract part of speech, syntactic structure, and semantic information. These linguistic features can complement the embedding information generated by the text encoder and help the system better understand the deeper meaning of the text. The extracted part of speech, syntactic, and semantic features are fused with the text embedding information, and contextual information is embedded in the fusion process. This contextual information enables the embedding information to reflect the coherence and semantic dependencies in the text, thereby generating more natural speech.

[0158] This embodiment generates text embedding information by processing text transcription. The system can combine the text's phonetic features, semantic information, and contextual dependencies to provide a rich semantic basis for subsequent speech synthesis. Compared to using letters or words directly, the combination of phoneme sequences and linguistic features makes the generated speech more natural and coherent, and can accurately convey the semantics in the text.

[0159] In one embodiment, the above S60 includes:

[0160] S601, aligning the sentiment distribution vector with corresponding text embedding information at the phoneme level, word level and sentence level;

[0161] S602, fusing the aligned sentiment distribution vector, text embedding information and the acoustic feature to form a combined feature representation;

[0162] S603, applying an attention mechanism to the combined feature representation to enhance the association between the sentiment distribution vector and the text embedding information;

[0163] S604, input the combined feature representation enhanced by the attention mechanism into the variable converter in the speech synthesis framework to generate prosodic features.

[0164] In this embodiment, the sentiment distribution vector, text embedding information and acoustic features are combined, and then the final prosodic features are generated through a variable transformer. These prosodic features control the key features of the synthesized speech, such as pitch, rhythm, and duration, so that the generated speech expression is natural and emotional.

[0165] The system aligns the sentiment distribution vector with the text embedding information at different levels (phonemes, words, sentences). The purpose of alignment is to closely combine sentiment and text semantics to ensure that speech units at each level can accurately express sentiment and semantics. For example, the system aligns subtle pitch changes at the phoneme level, important sentiment features at the word level, and overall semantic and sentiment expressions at the sentence level.

[0166] The aligned sentiment distribution vector, text embedding information and acoustic features (pitch, speaking speed, rhythm, etc.) are fused. The purpose of fusion is to unify the information from these different sources into a combined feature representation. This combined feature representation contains the sentiment, semantic and physical features in the speech, and is the basis for the subsequent generation of prosodic features.

[0167] The attention mechanism is applied to the combined feature representation to enhance the association between the sentiment distribution vector and the text embedding information. The attention mechanism can automatically assign weights to each input element (such as sentiment information in the sentiment distribution vector and semantic information in the text embedding), allowing the system to focus on the information that is most important for speech synthesis. This allows the synthesized speech to not only convey semantics, but also express emotions naturally.

[0168] Finally, the combined feature representation enhanced by the attention mechanism is input into the variational transformer in the speech synthesis framework. The variational transformer is a deep learning model or transformation module that is responsible for converting the combined feature representation into prosodic features (such as pitch, duration, rhythm, etc.). These prosodic features are directly used to generate the final speech output, ensuring the naturalness of the speech and the consistency of emotional expression.

[0169] The system uses an alignment algorithm to match the sentiment distribution vector with the text embedding information at the phoneme, word, and sentence levels. Through refined alignment, sentiment information of different granularities is combined with text semantics. The sentiment distribution vector, text embedding information, and acoustic features are multimodally fused to form a combined feature representation. This fusion can be achieved through simple concatenation, weighting, or multimodal fusion layers of deep learning models. The attention mechanism allocates appropriate attention to the sentiment distribution and text information by dynamically adjusting weights. Its output is an enhanced combined feature representation, which enables the system to generate emotions and rhythm in speech more accurately. The combined feature representation enhanced by the attention mechanism is input into the variable transformer to generate the final prosodic features. These prosodic features control the basic sound parameters of speech, such as pitch, duration, and volume, ensuring that the generated speech is natural and emotional.

[0170] In this embodiment, by combining the sentiment distribution vector, text embedding information and acoustic features, the system can generate more natural rhythmic features. The application of the attention mechanism further enhances the accuracy and naturalness of sentiment expression.

[0171] In one embodiment, in the above S80, converting the Mel-spectrogram into a time-domain audio signal by a vocoder includes:

[0172] S801, analyzing the emotional features of the target emotional speech based on the prosodic features;

[0173] S802, determining a vocoder model based on the emotional characteristics of the target emotional speech, and adjusting the sampling rate and frequency resolution of the vocoder model;

[0174] S803, configuring the generation step size of the vocoder model to optimize speech generation speed and quality;

[0175] S804, using the configured vocoder model, converting the Mel-spectrogram into a preliminary time-domain audio signal;

[0176] S805, post-processing the preliminary time-domain audio signal to generate a final time-domain audio signal, wherein the post-processing includes dynamic range compression, frequency equalization and gain control.

[0177] In this embodiment, the mel-spectrogram is converted into a final time-domain audio signal by a vocoder to generate the target emotional speech. The task of the vocoder is to convert the mel-spectrogram in the frequency domain into a playable time-domain audio signal, combined with the emotional characteristics of the speech, to ensure that the generated speech is natural and expressive.

[0178] The system first analyzes the emotional characteristics of the target emotional speech based on the generated prosodic features (pitch, rhythm, duration, etc.). Emotional characteristics include the overall emotional expression of the speech, such as calmness, anxiety, excitement, etc., which determine the rhythm, strength and expression of the speech.

[0179] Based on the emotional features, the system selects a suitable vocoder model. Commonly used vocoder models include WaveNet, MelGAN, etc. The selection and parameters of the vocoder model (such as sampling rate, frequency resolution) need to be adjusted according to the characteristics of the target speech to ensure the accuracy of audio quality and emotional expression.

[0180] The system configures the generation step size of the vocoder to optimize the speed and quality of speech generation. The step size is the time interval setting when the vocoder generates an audio signal, which directly affects the smoothness and computational efficiency of the generated audio.

[0181] Using the configured vocoder model, the mel-spectrogram is converted into a preliminary time-domain audio signal. This process converts the frequency domain information (mel-spectrogram) into an actual playable speech signal, but the preliminary signal may need further processing to improve quality.

[0182] The preliminary time-domain audio signal is post-processed to generate the final time-domain audio signal. The post-processing steps include dynamic range compression (adjusting the volume difference of the audio to improve the listening experience), frequency equalization (adjusting the frequency response of the audio to enhance the clarity of the audio) and gain control (optimizing the volume level). These processing steps ensure that the final generated audio signal has high-quality sound performance.

[0183] The system extracts emotional features of speech from prosodic features and uses these features to adjust the output of speech synthesis. For example, when expressing anger, the system may choose features with faster speech rate and higher pitch; when expressing calmness, the system will choose lower pitch and slower speech rate. Based on the emotional features, the system determines to use a vocoder model such as WaveNet or MelGAN. Then, the sampling rate and frequency resolution are adjusted to optimize the output audio quality. For example, a higher sampling rate can generate finer speech details, but it increases the computational overhead. The generation step size of the vocoder is configured to make the speech generation fast and high-quality. The configuration of the step size needs to balance the generation speed while ensuring the speech quality to meet the needs of real-time or near-real-time speech synthesis. Using the configured vocoder, the Mel spectrogram is converted into a preliminary time-domain audio signal. At this point, the speech can be played, but the sound quality needs to be further optimized. Through dynamic range compression, the parts of the speech with excessive volume are reduced and the details are enhanced. Use frequency equalization to ensure that the frequency distribution in the audio is reasonable and enhance the clarity. Finally, through gain control, the overall volume of the audio is optimized to ensure that the speech is clear and natural when played.

[0184] In this embodiment, the mel-spectrogram is converted into a time-domain audio signal by a vocoder, and the system can generate natural speech that conforms to the prosodic and emotional characteristics. The post-processing step ensures that the final generated speech reaches a high level in quality and clarity.

[0185] In one embodiment, a speech synthesis device based on hierarchical emotion distribution is provided, and the speech synthesis device based on hierarchical emotion distribution corresponds one-to-one to the speech synthesis method based on hierarchical emotion distribution in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of a speech synthesis device based on hierarchical emotion distribution of the present invention. Data acquisition module 10, speech segmentation module 20, acoustic feature extraction module 30, emotion analysis module 40, text embedding module 50, feature fusion and variable converter module 60, decoder module 70 and vocoder module 80. The functional modules are described in detail as follows:

[0186] A data acquisition module 10, for acquiring audio data and corresponding text transcription;

[0187] A speech segmentation module 20, used for segmenting the audio data into phoneme-level speech units, word-level speech units and sentence-level speech units;

[0188] An acoustic feature extraction module 30, for extracting acoustic features of each speech unit, wherein the acoustic features include pitch, speech rate and rhythm;

[0189] The sentiment analysis module 40 is used to analyze the sentiment intensity of each speech unit using a pre-trained model and generate a sentiment distribution vector;

[0190] a text embedding module 50 for processing the text transcription to obtain text embedding information;

[0191] A feature fusion and variable converter module 60 is used to combine the sentiment distribution vector, the text embedding information and the acoustic feature, and input them into a variable converter in a speech synthesis framework to generate a prosodic feature;

[0192] A decoder module 70, configured to generate a mel-spectrogram based on the prosodic features through a decoder;

[0193] The vocoder module 80 is used to convert the mel-spectrogram into a time-domain audio signal through a vocoder, and output the time-domain audio signal as the target emotional speech.

[0194] In one embodiment, the speech segmentation module 20 is specifically used for:

[0195] aligning the audio data with the text transcription using a forced alignment tool;

[0196] According to the alignment result, the audio data is segmented into phoneme-level speech units;

[0197] Segmenting the audio data into word-level speech units based on word boundaries in the text transcription;

[0198] The audio data is segmented into sentence-level speech units according to sentence boundaries in the text transcription.

[0199] In one embodiment, the acoustic feature extraction module 30 is specifically used for:

[0200] For each phoneme-level speech unit, the fundamental frequency contour is determined to extract the pitch acoustic features;

[0201] For each word-level speech unit, the duration is measured to extract the acoustic features of speech rate;

[0202] For each sentence-level speech unit, the energy profile is obtained to extract rhythmic acoustic features;

[0203] Extract additional acoustic features including formant and harmonic-to-noise ratio from phoneme-level speech units and word-level speech units;

[0204] The extracted acoustic features are associated with the corresponding speech units.

[0205] In one embodiment, the sentiment analysis module 40 is specifically used to:

[0206] Obtain an emotion recognition model pre-trained based on an emotion dataset;

[0207] Inputting the acoustic features of each speech unit into the emotion recognition model to determine the emotion intensity score of each speech unit on a predefined emotion category;

[0208] Based on the emotion intensity score of each speech unit, an emotion distribution vector representing the emotion state is formed, and each dimension of the emotion distribution vector represents a predefined emotion category.

[0209] In one embodiment, the text embedding module 50 is specifically used for:

[0210] converting the text transcription into a phoneme sequence using a grapheme-to-phoneme conversion model;

[0211] Encoding the phoneme sequence through a text encoder to generate text embedding information;

[0212] extracting linguistic features including part of speech, syntax and semantics from the text transcription;

[0213] The linguistic features are fused into the text embedding information, and context information of the text transcription is embedded into the fused text embedding information to generate enhanced text embedding information.

[0214] In one embodiment, the feature fusion and variable converter module 60 is specifically used to:

[0215] Aligning the sentiment distribution vector with the corresponding text embedding information at the phoneme level, word level and sentence level;

[0216] fusing the aligned sentiment distribution vector, text embedding information and the acoustic features to form a combined feature representation;

[0217] Applying an attention mechanism to the combined feature representation to enhance the association between the sentiment distribution vector and the text embedding information;

[0218] The combined feature representation enhanced by the attention mechanism is input into the variable transformer in the speech synthesis framework to generate prosodic features.

[0219] In one embodiment, the vocoder module 80 is specifically configured to:

[0220] Analyzing the emotional features of the target emotional speech based on the prosodic features;

[0221] Determining a vocoder model based on the emotional characteristics of the target emotional speech, and adjusting the sampling rate and frequency resolution of the vocoder model;

[0222] configuring a generation step size of the vocoder model to optimize speech generation speed and quality;

[0223] Using the configured vocoder model, convert the Mel-spectrogram into a preliminary time-domain audio signal;

[0224] The preliminary time-domain audio signal is post-processed to generate a final time-domain audio signal, wherein the post-processing includes dynamic range compression, frequency equalization and gain control.

[0225] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a speech synthesis method based on hierarchical emotion distribution.

[0226] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the functions or steps on the user side of a speech synthesis method based on hierarchical emotion distribution are realized.

[0227] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0228] Get audio data and corresponding text transcription;

[0229] Segmenting the audio data into phoneme-level speech units, word-level speech units, and sentence-level speech units;

[0230] Extracting acoustic features of each speech unit, wherein the acoustic features include pitch, speech rate and rhythm;

[0231] Use the pre-trained model to analyze the emotional intensity of each speech unit and generate an emotional distribution vector;

[0232] processing the text transcription to obtain text embedding information;

[0233] The sentiment distribution vector, the text embedding information and the acoustic feature are combined and input into a variable converter in a speech synthesis framework to generate a prosodic feature;

[0234] Generate a mel-spectrogram based on the prosodic features by a decoder;

[0235] The mel-spectrogram is converted into a time-domain audio signal by a vocoder, and the time-domain audio signal is output as the target emotional speech.

[0236] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0237] Get audio data and corresponding text transcription;

[0238] Segmenting the audio data into phoneme-level speech units, word-level speech units, and sentence-level speech units;

[0239] Extracting acoustic features of each speech unit, wherein the acoustic features include pitch, speech rate and rhythm;

[0240] Use the pre-trained model to analyze the emotional intensity of each speech unit and generate an emotional distribution vector;

[0241] processing the text transcription to obtain text embedding information;

[0242] The sentiment distribution vector, the text embedding information and the acoustic feature are combined and input into a variable converter in a speech synthesis framework to generate a prosodic feature;

[0243] Generate a mel-spectrogram based on the prosodic features by a decoder;

[0244] The mel-spectrogram is converted into a time-domain audio signal by a vocoder, and the time-domain audio signal is output as the target emotional speech.

[0245] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0246] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0247] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0248] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A speech synthesis method based on hierarchical emotion distribution, characterized in that: The following steps are involved: Get audio data and corresponding text transcription; Segmenting the audio data into phoneme-level speech units, word-level speech units, and sentence-level speech units; Extracting acoustic features of each speech unit, wherein the acoustic features include pitch, speech rate and rhythm; Use the pre-trained model to analyze the emotional intensity of each speech unit and generate an emotional distribution vector; processing the text transcription to obtain text embedding information; The sentiment distribution vector, the text embedding information and the acoustic feature are combined and input into a variable converter in a speech synthesis framework to generate a prosodic feature; Generate a mel-spectrogram based on the prosodic features by a decoder; The mel-spectrogram is converted into a time-domain audio signal by a vocoder, and the time-domain audio signal is output as the target emotional speech.

2. The speech synthesis method based on hierarchical emotion distribution as claimed in claim 1, characterized in that: The audio data is segmented into phoneme-level speech units, word-level speech units, and sentence-level speech units, including: aligning the audio data with the text transcription using a forced alignment tool; According to the alignment result, the audio data is segmented into phoneme-level speech units; Segmenting the audio data into word-level speech units based on word boundaries in the text transcription; The audio data is segmented into sentence-level speech units according to sentence boundaries in the text transcription.

3. The speech synthesis method based on hierarchical emotion distribution as claimed in claim 1, characterized in that: Extracting acoustic features of each speech unit, the acoustic features including pitch, speech rate and rhythm, including: For each phoneme-level speech unit, the fundamental frequency contour is determined to extract the pitch acoustic features; For each word-level speech unit, the duration is measured to extract the acoustic features of speech rate; For each sentence-level speech unit, the energy profile is obtained to extract rhythmic acoustic features; Extract additional acoustic features including formant and harmonic-to-noise ratio from phoneme-level speech units and word-level speech units; The extracted acoustic features are associated with the corresponding speech units.

4. The speech synthesis method based on hierarchical emotion distribution as claimed in claim 1, characterized in that: The pre-trained model is used to analyze the emotional intensity of each speech unit and generate an emotional distribution vector, including: Obtain an emotion recognition model pre-trained based on an emotion dataset; Inputting the acoustic features of each speech unit into the emotion recognition model to determine the emotion intensity score of each speech unit on a predefined emotion category; Based on the emotion intensity score of each speech unit, an emotion distribution vector representing the emotion state is formed, and each dimension of the emotion distribution vector represents a predefined emotion category.

5. The speech synthesis method based on hierarchical emotion distribution as claimed in claim 1, characterized in that: Processing the text transcription to obtain text embedding information includes: converting the text transcription into a phoneme sequence using a grapheme-to-phoneme conversion model; Encoding the phoneme sequence through a text encoder to generate text embedding information; extracting linguistic features including part of speech, syntax and semantics from the text transcription; The linguistic features are fused into the text embedding information, and context information of the text transcription is embedded into the fused text embedding information to generate enhanced text embedding information.

6. The speech synthesis method based on hierarchical emotion distribution as described in claim 1, characterized in that: The sentiment distribution vector, the text embedding information and the acoustic feature are combined and input into a variable converter in a speech synthesis framework to generate a prosodic feature, including: Aligning the sentiment distribution vector with the corresponding text embedding information at the phoneme level, word level and sentence level; fusing the aligned sentiment distribution vector, text embedding information and the acoustic features to form a combined feature representation; Applying an attention mechanism to the combined feature representation to enhance the association between the sentiment distribution vector and the text embedding information; The combined feature representation enhanced by the attention mechanism is input into the variable transformer in the speech synthesis framework to generate prosodic features.

7. The speech synthesis method based on hierarchical emotion distribution as claimed in claim 1, characterized in that: The mel-spectrogram is converted into a time-domain audio signal by a vocoder, comprising: Analyzing the emotional features of the target emotional speech based on the prosodic features; Determining a vocoder model based on the emotional characteristics of the target emotional speech, and adjusting the sampling rate and frequency resolution of the vocoder model; configuring a generation step size of the vocoder model to optimize speech generation speed and quality; Using the configured vocoder model, convert the Mel-spectrogram into a preliminary time-domain audio signal; The preliminary time-domain audio signal is post-processed to generate a final time-domain audio signal, wherein the post-processing includes dynamic range compression, frequency equalization and gain control.

8. A speech synthesis device based on hierarchical emotion distribution, characterized in that: The speech synthesis device based on hierarchical emotion distribution includes: A data acquisition module, used to acquire audio data and corresponding text transcription; A speech segmentation module, used for segmenting the audio data into phoneme-level speech units, word-level speech units and sentence-level speech units; An acoustic feature extraction module, used to extract the acoustic features of each speech unit, wherein the acoustic features include pitch, speech rate and rhythm; The sentiment analysis module is used to analyze the sentiment intensity of each speech unit using a pre-trained model and generate a sentiment distribution vector; a text embedding module for processing the text transcription to obtain text embedding information; A feature fusion and variable converter module, used for combining the sentiment distribution vector, the text embedding information and the acoustic feature, and inputting the result into a variable converter in a speech synthesis framework to generate a prosodic feature; A decoder module, configured to generate a mel-spectrogram based on the prosodic features through a decoder; The vocoder module is used to convert the mel-spectrogram into a time-domain audio signal through a vocoder, and output the time-domain audio signal as the target emotional speech.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a speech synthesis program based on hierarchical emotion distribution stored in the memory and executable on the processor. When the speech synthesis program based on hierarchical emotion distribution is executed by the processor, the steps of the speech synthesis method based on hierarchical emotion distribution as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a speech synthesis program based on hierarchical emotion distribution, and when the speech synthesis program based on hierarchical emotion distribution is executed by the processor, the steps of the speech synthesis method based on hierarchical emotion distribution as described in any one of claims 1-7 are implemented.

Images