Voice generation method and device based on double-layer style modeling, equipment and medium

Abstract

The invention provides a voice generation method and device based on double-layer style modeling, equipment and a medium, and relates to the technical field of voice synthesis. According to the method, the text is converted into the phoneme sequence, the rhythm mark sequence is extracted, and the two sequences are fused into the rhythm and phoneme fusion feature by adopting the gating fusion mechanism, so that the naturalness and expressive force of the speech are enhanced. The text style description is converted into a style vector by utilizing a semantic embedding model, and local style features and global style features are extracted, so that finer style changes can be captured. Local style features are injected into the melody and phoneme fusion features through a modulation mechanism, phoneme-level stylized features are obtained, and high consistency of styles is achieved. And the phoneme-level stylized features and the global style features are input into the decoder, and the target voice audio highly matched with the requirements of the user is generated, so that the voice style regulation and control accuracy is improved, and the method can better serve the application scenarios with specific requirements on the voice style, such as financial intelligent customer service and medical health auxiliary inquiry.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, and in particular to a speech generation method, apparatus, device and medium based on two-layer style modeling. Background Technology

[0002] In the financial and healthcare sectors, text-to-speech (TTS) synthesis systems are increasingly widely used, for example, in automated report generation, customer service chatbots, and electronic health records. However, existing TTS systems still have significant shortcomings in achieving natural, controllable, and expressive speech generation. These shortcomings limit the performance of speech synthesis systems when handling complex emotions, technical terms, and sensitive information.

[0003] Traditional end-to-end models (such as Tacotron2, FastSpeech, VITS, etc.) can generate speech with high naturalness, but they mainly focus on the clarity and coherence of speech. They have weak control over speaking style (such as emotion, gender, age, tone, speaking speed, etc.) and cannot flexibly adjust the expression of speech according to the user's intention.

[0004] While some existing models (such as StyleSpeech and LanStyleTTS) have introduced a "style embedding" mechanism to achieve local control of speech prosody, their control level is mainly limited to the phoneme or prosody level. They lack the ability to model higher-level paralinguistic features, such as the speaker's emotions, identity characteristics, age, and accent.

[0005] Another emerging approach attempts to leverage large language models (LLMs) for semantically driven style generation (such as CosyVoice and Spark-TTS). These models control speech style through natural language prompts, offering some flexibility, but also suffer from the following significant problems: LLM inference is complex and has a large number of parameters, resulting in slow inference speed and high memory consumption, making it unsuitable for deployment on edge devices; style and semantics are highly coupled within the model, making it difficult for the system to interpret the style control mechanism, and the generated results are extremely sensitive to the wording of the prompts.

[0006] In intelligent financial customer service, the voice style needs to be dynamically switched according to the risk level of the financial product (e.g., conservative customers require low-risk prompts with a calm tone, while aggressive customers can tolerate higher-risk terms with an aggressive tone). However, existing systems cannot independently control semantic content and style parameters, resulting in a monotonous customer experience. In medical and health assisted consultation, it is necessary to generate reassuring voices that are appropriate for the age and emotional state of different patient groups (e.g., using a light tone for children and a slow, gentle pace for elderly patients). However, existing models lack the ability to model paralinguistic features, making it difficult to meet the needs of personalized medical communication.

[0007] Therefore, improving the accuracy of style control in text-to-speech synthesis systems when generating speech has become an urgent technical problem to be solved in application scenarios such as intelligent financial customer service and medical health assisted diagnosis. Summary of the Invention

[0008] This application provides a speech generation method, apparatus, device, and medium based on two-layer style modeling, aiming to solve the technical problem that the speech style accuracy generated by text-to-speech synthesis systems in related technologies is low, which cannot meet the needs of personalized style speech generation in application scenarios such as financial intelligent customer service and medical health assisted diagnosis, so as to improve the style control accuracy of text-to-speech synthesis systems when generating speech.

[0009] In a first aspect, this application provides a speech generation method based on two-layer style modeling, the speech generation method based on two-layer style modeling comprising the following steps: Retrieve the original text and text style description; The original text is converted into a phoneme sequence, and the prosodic marker sequence is extracted; A gated fusion mechanism is used to fuse the phoneme sequence and the prosodic marker sequence to obtain prosodic phoneme fusion features; Based on the semantic embedding model, the text style description is converted into a style vector, and local style features and global style features are extracted from the style vector; By using a modulation mechanism, the local stylistic features are injected into the phonemic fusion features to obtain phoneme-level stylized features; The factor-level stylization features and the global style features are input into a pre-built decoder to generate the target speech audio.

[0010] Secondly, this application also provides a speech generation device based on two-layer style modeling, the speech generation device based on two-layer style modeling comprising: The data acquisition module is used to acquire the raw text and text style description; A phoneme conversion module is used to convert the original text into a phoneme sequence and extract a prosodic marker sequence; The phoneme-prosody fusion module is used to fuse the phoneme sequence and the prosodic marker sequence using a gated fusion mechanism to obtain prosodic-phoneme fusion features; The style feature extraction module is used to convert the text style description into a style vector based on a semantic embedding model, and extract local style features and global style features from the style vector; The feature modulation module is used to inject the local style features into the phoneme fusion features through a modulation mechanism to obtain phoneme-level stylized features; The speech generation module is used to input the factor-level stylization features and the global style features into a pre-built decoder to generate target speech audio.

[0011] Thirdly, this application also provides a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the speech generation method based on two-layer style modeling as described above.

[0012] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the speech generation method based on two-layer style modeling as described above.

[0013] This application provides a speech generation method, apparatus, computer device, and storage medium based on two-layer style modeling. The method, by acquiring the original text and style description, can clearly define the target text content and desired style features for speech synthesis. The text is converted into a phoneme sequence, and a prosodic marker sequence is extracted. A gated fusion mechanism is used to fuse the phoneme sequence and the prosodic marker sequence. The resulting phoneme-prosodic fusion features not only contain basic speech information but also incorporate rhythmic and prosodic information, enhancing the naturalness and expressiveness of the speech. A semantic embedding model is used to convert the text style description into a style vector, and local and global style features are extracted from it, enabling the capture and simulation of more subtle style changes. A modulation mechanism injects the local style features into the phoneme-prosodic fusion features, obtaining phoneme-level stylized features that accurately reflect the style description of the text, thereby achieving high style consistency during speech synthesis. By inputting phoneme-level stylistic features and global style features into the decoder, the generated target speech audio is not only accurate in content, but also highly matched with the user's needs in style. This improves the accuracy of style control in the text-to-speech synthesis system when generating speech, ensuring the style accuracy and personalization of the final speech output, making the generated speech more natural and personalized. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of an application environment for a speech generation method based on two-layer style modeling in one embodiment of the present invention; Figure 2 A flowchart illustrating an embodiment of a speech generation method based on two-layer style modeling provided in this application; Figure 3 A schematic diagram of the data processing flow for a speech generation method based on two-layer style modeling provided in this application; Figure 4 This is a schematic diagram of the structure of a speech generation device based on two-layer style modeling provided in this application. Figure 5 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.

[0016] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0019] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The speech generation method based on two-layer style modeling provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. When the server receives a speech generation request from the client, it can obtain the original text and text style description; convert the original text into a phoneme sequence and extract the prosodic marker sequence; use a gated fusion mechanism to fuse the phoneme sequence and the prosodic marker sequence to obtain phoneme-prosodic fusion features; based on a semantic embedding model, convert the text style description into a style vector and extract local style features and global style features from the style vector; inject the local style features into the phoneme-prosodic fusion features through a modulation mechanism to obtain phoneme-level stylization features; and input the factor-level stylization features and the global style features into a pre-built decoder to generate the target speech audio.

[0022] This invention addresses the technical problem of low accuracy in speech style generation by text-to-speech synthesis systems, which fails to meet the personalized speech generation requirements in applications such as intelligent financial customer service and medical health assisted diagnosis. By acquiring the original text and style description, the target text content and desired style features for speech synthesis can be clearly defined. The text is converted into a phoneme sequence, and a prosodic marker sequence is extracted. A gated fusion mechanism is then used to fuse the phoneme and prosodic marker sequences. The resulting phoneme-prosodic fusion features not only contain basic speech information but also incorporate rhythm and prosodic information, enhancing the naturalness and expressiveness of the speech. A semantic embedding model is used to convert the text style description into a style vector, from which local and global style features are extracted, enabling the capture and simulation of more subtle style changes. A modulation mechanism injects the local style features into the phoneme-prosodic fusion features, obtaining phoneme-level stylized features that accurately reflect the style description of the text, thereby achieving high style consistency during speech synthesis. By inputting phoneme-level stylistic features and global style features into the decoder, the generated target speech audio is not only accurate in content, but also highly matched with the user's needs in style. This improves the accuracy of style control in the text-to-speech synthesis system when generating speech, ensuring the style accuracy and personalization of the final speech output, making the generated speech more natural and personalized.

[0023] The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0024] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of a speech generation method based on two-layer style modeling provided in this application.

[0025] like Figure 2 As shown, the speech generation method based on two-layer style modeling includes steps S101 to S106.

[0026] S101. Obtain the original text and text style description.

[0027] This embodiment provides a style-controllable speech generation model based on two-layer style modeling. By introducing a hierarchical modeling structure that simultaneously introduces a prosody level and a paralinguistic level into the speech generation process, it achieves high-fidelity, interpretable, and controllable generation from text to speech.

[0028] In one embodiment, a user interface or application programming interface (API) can be designed to allow users to input or upload text content to be converted into speech. Simultaneously, an interface or API can be provided that allows users to input text style descriptions, such as "pleasant," "middle-aged male," or "conference report," which will be used to guide the speech synthesis style.

[0029] The original text refers to the text content that the user wants to convert into speech. It can be any form of written language, including but not limited to sentences or paragraphs (such as content in articles, books, and news reports), dialogues or scripts (such as dialogue text in movies, TV series, and radio dramas), instructions or announcements (such as text used for voice assistants, automatic response systems, or public address systems), tags or prompts (such as short instructions used for navigation, educational, or training materials), etc.

[0030] Text style description refers to the user's description of the style or characteristics of the generated speech. It can include paralinguistic features such as emotion, tone, speech rate, and volume, as well as specific speaker attributes such as gender, age, and accent. It can also include the speaking context, such as daily conversation, family gathering, meeting, speech, broadcast, and recording.

[0031] For example, emotions can include happiness, sadness, anger, surprise, fear, disgust, neutrality, etc. Intonation can include rising, falling, interrogative, declarative, singing, etc. Speaking speed can include fast, slow, moderate, or dynamically varying according to the content. Volume can include loud, soft, moderate, or dynamically adjusted according to the content. Speaker attributes can include gender (e.g., male, female, neutral), (e.g., age, child, teenager, adult, elderly), and accent (e.g., accent of a specific region or country, such as a southern American accent, a London accent, etc.). Speaking occasions can include formal occasions (e.g., meetings, speeches, formal announcements), informal occasions (e.g., conversations between friends, family gatherings, casual chats), and specific environments (e.g., radio, telephone, recording studio, outdoors, etc.). Voice style can include narrative (used for storytelling or narrating events), conversational (simulating everyday conversation or role-playing), imperative (used for giving instructions or guidance), and performative (used for acting in plays, movies, or radio dramas), etc. In addition, the generated voice can also be used for educational purposes (such as teaching materials and voice tutorials), entertainment (such as audiobooks, game dubbing, and animation dubbing), and voice assistance (such as voice assistance for visually impaired people).

[0032] These text style descriptions can be used as input parameters to guide the speech generation model in producing speech output that better meets user expectations, thereby providing richer and more personalized speech synthesis services.

[0033] In one embodiment, the user can enter raw text through a text box in the client's graphical user interface (GUI) or through a command-line interface (CLI), and enter a text style description through another text box in the GUI or through the CLI.

[0034] This embodiment provides a style-controllable speech generation model based on two-layer style modeling. This model, by introducing a hierarchical modeling structure of prosody and paralinguistic layers, achieves high-fidelity, interpretable, and controllable generation from text to speech. Users can input text content and style descriptions, such as emotion, intonation, and speech rate, through a graphical user interface or command-line interface, so that the speech generation model can generate speech output with a specific style.

[0035] S102. Convert the original text into a phoneme sequence and extract the prosodic marker sequence.

[0036] For the original text, preprocessing is required. Specifically, text cleaning can be performed to remove or replace special characters and symbols. Then, the cleaned text is segmented into words or phrases, which helps determine the phrasing and rhythm of the speech synthesis. Simultaneously, punctuation marks, such as periods and commas, are identified and processed according to predefined rules. Punctuation marks play an important grammatical and semantic role in text, affecting the rhythm and intonation of speech.

[0037] For the preprocessed raw text, a phoneme dictionary or phoneme generation model can be used to map each word to its corresponding phoneme sequence. Simultaneously, the grammatical structure of the text is analyzed to determine stress and intonation patterns, and changes in speech rate and rhythm are predicted based on the text content and context, thereby extracting prosodic sequences.

[0038] Further, a phoneme conversion rule library matching the language type of the original text is loaded from a preset database; based on the phoneme conversion rule library, the original text is phoneme converted to generate the phoneme sequence; using a time warping algorithm, the phoneme sequence is temporally aligned with a reference prosodic template, and the prosodic parameters corresponding to each phoneme in the phoneme sequence are extracted, wherein the prosodic parameters include energy intensity, fundamental frequency profile, and duration distribution; the prosodic parameters corresponding to each phoneme in the phoneme sequence are mapped to predefined prosodic markers to generate the prosodic marker sequence.

[0039] In one embodiment, a multilingual language model can be pre-built using a BILSTM (Bidirectional Long Short-Term Memory) network, collecting text datasets in multiple languages, and training the multilingual language model to recognize the features of different languages. An IPA (International Phonetic Alphabet) phoneme conversion rule base is prepared for each language.

[0040] The system takes pre-processed raw text as input, performs language identification using a pre-trained BLSTM model, outputs the identification result, and determines the language type of the raw text. Based on the identified language type, it loads the corresponding IPA phoneme conversion rule library from the database.

[0041] Specifically, feature vectors, such as word embeddings and character embeddings, can be extracted from the preprocessed text. These extracted feature vectors are then input into a BLSTM model for language type inference. The model outputs a recognition result, which can be a probability distribution representing the likelihood of the original text belonging to each language. Based on the output of the BLSTM model, the language with the highest probability can be selected as the language type of the text.

[0042] Based on the language type, query the phoneme conversion rule library stored in the database, load the corresponding IPA phoneme conversion rule library, and parse the IPA phoneme conversion rule library to understand the rules and mapping relationships for converting text into IPA phonemes.

[0043] The preprocessed original text is then vectorized. This involves segmenting the text into sequences by character and mapping each character to a fixed-dimensional trainable embedding vector, forming a character embedding vector. Simultaneously, sine / cosine positional encoding or learned positional embedding is used to generate position vectors, which are then added element-wise to the character embedding vectors to obtain the text vector.

[0044] Based on the phoneme conversion rule library, characters or words in the text are mapped to corresponding phoneme representations to construct a preliminary phoneme sequence.

[0045] Load a pre-trained CTC-Attention model capable of handling text-to-phoneme sequence conversion tasks. Input the text vector into the CTC-Attention model, including character embedding vectors and position vectors. The character embedding vector maps each character in the text to a vector in a high-dimensional space, capturing the semantic and syntactic features of the character. The position vector provides the model with information about the character's location within the text, facilitating the model's understanding of the text's sequential structure. Simultaneously, the initial phoneme sequence is also input into the CTC-Attention model.

[0046] Next, the CTC-Attention model processes the input representation through a multi-layer Transformer encoder. The encoder consists of multiple self-attention layers and feedforward neural network layers, which capture long-range dependencies within the text and learn complex interactions between characters. The self-attention mechanism allows the model to consider other characters in the entire text sequence when processing each character, thus better understanding the context. The feedforward network layers further process this information, enhancing the model's expressive power.

[0047] Based on the encoder's output, the CTC-Attention model uses the CTC (Connection Temporal Classification) mechanism to generate phoneme sequences. CTC is a special loss function that allows the model to handle repetition and insertion operations when generating sequences. CTC finds the most probable phoneme sequence through a dynamic programming algorithm, and the phoneme sequence may contain repeating phonemes or blank symbols.

[0048] Meanwhile, the CTC-Attention model also utilizes an attention mechanism to further refine the correspondence between text and phonemes. The attention mechanism generates an attention weight matrix by calculating the correlation between each character in the text sequence and each phoneme in the phoneme sequence.

[0049] The attention weight matrix reflects the influence of each character in the text on the generation of each phoneme. The CTC-Attention model uses the attention weight matrix to strengthen the feature representation of the text portion most relevant to the generation of the current phoneme. Using this feature representation, the CTC-Attention model generates a phoneme sequence through the decoder. In the decoder, the CTC-Attention model employs either greedy decoding or beam search strategies to generate the phoneme sequence. Greedy decoding simply selects the phoneme with the highest probability at each time step, while beam search considers multiple possible phoneme sequences to find the sequence with the highest overall probability, generating the final phoneme sequence.

[0050] In one embodiment, the reference prosodic template is obtained; based on the dynamic time warping algorithm, the phoneme sequence is temporally aligned with the reference prosodic template to obtain the temporal boundary of each phoneme; according to the temporal boundary corresponding to each phoneme, the energy intensity, fundamental frequency profile and duration distribution of each phoneme are extracted to obtain the prosodic parameters.

[0051] In one embodiment, after generating the phoneme sequence, the DTW (Dynamic Time Warping) algorithm is used to align the phoneme sequence with a reference prosodic template to obtain the time boundary of each phoneme. The DTW algorithm is a method for calculating the similarity between two time series of potentially different lengths; it finds the best match between them by non-linearly aligning the two sequences. The reference prosodic template is a predefined set of phoneme sequences representing different prosodic features (such as stress, intonation, and speech rate).

[0052] The phoneme sequence and reference prosodic template are preprocessed, including normalization and feature extraction, to ensure they are suitable for DTW alignment. Specifically, the features in the phoneme sequence and reference prosodic template are normalized by scaling the feature values ​​to the [0, 1] interval or by Z-score normalization to give them a uniform scale and range, thus eliminating the influence of dimensions and improving the robustness of the algorithm. Features that facilitate alignment, including phoneme energy, fundamental frequency, duration, and formants, are extracted from the phoneme sequence and reference prosodic template using signal processing techniques or machine learning methods (such as Fourier transform, Mel-frequency cepstral coefficients (MFCC), etc.). Noise and outliers that may affect the alignment results, such as silence segments and non-speech phonemes, are removed.

[0053] Define a distance metric (such as Euclidean distance, Manhattan distance, cosine similarity, etc.) to calculate the similarity between corresponding elements in a phoneme sequence and a reference template. Calculate the distance matrix between the phoneme sequence and the reference template based on the chosen distance metric. Each element of the distance matrix represents the distance between corresponding elements in the two sequences.

[0054] The cumulative cost matrix for the DTW algorithm is initialized on the distance matrix, typically set to 0 in the top-left corner and infinity for the remaining elements. Starting from the top-left corner, each element in the cumulative cost matrix is ​​recursively computed, where the value of each element is the minimum of its three adjacent elements (left, top, and top-left) plus the distance to the current element. Starting from the bottom-right corner of the cumulative cost matrix, the algorithm backtracks along the path of minimum cumulative cost to find the optimal alignment path, which represents the best matching between the phoneme sequence and the reference template.

[0055] Based on the optimal alignment path, the start and end positions of each phoneme in the time series are extracted, i.e., time boundaries, which represent the precise position of the phoneme in the speech signal. The extracted time boundaries are combined with the phoneme sequence to generate the alignment result.

[0056] In this embodiment, a pre-trained BLSTM model identifies the language type of the text and loads the corresponding IPA phoneme conversion rule base from the database based on the recognition results. The CTC-Attention model is used to convert text features into phoneme sequences, and the DTW algorithm is used to align them with a reference prosodic template to accurately extract the prosodic parameters of the phonemes. These prosodic parameters are then mapped onto predefined prosodic markers to generate a prosodic marker sequence. This not only improves the naturalness and expressiveness of the speech synthesis but also enhances the model's adaptability to different language and style features, thereby generating speech output that better meets user expectations.

[0057] In one embodiment, prosodic parameters such as energy intensity, fundamental frequency profile, and duration distribution of each phoneme are extracted from the audio signal based on time boundaries.

[0058] Specifically, the audio signal is segmented into short frames using a window function (such as a Hamming or Hanning window), each frame containing 20-40 milliseconds of audio data. For each frame, the amplitude is calculated by taking the absolute value of the signal. A Fast Fourier Transform (FFT) is then used to transform each frame from the time domain to the frequency domain, and the Power Spectral Density (PSD) is calculated to reflect the power distribution of each frequency component in the signal. The PSD of each frame is integrated to obtain the energy value of that frame, where the energy intensity of a phoneme is the sum of the power of all frequency components within the frame. To eliminate energy differences between different phonemes, the calculated energy value can be normalized, for example, using maximum value normalization or Z-score normalization.

[0059] The audio signal undergoes preprocessing, including denoising, pre-emphasis, and framing, to highlight the fundamental frequency component. A fundamental frequency detection algorithm is then used to extract the fundamental frequency profile of each phoneme from the audio signal. This algorithm can include autocorrelation methods, the YIN algorithm, or neural network fundamental frequency detectors. Specifically, the autocorrelation method calculates the autocorrelation function of the signal and finds the location of the first significant peak, which corresponds to the period of the fundamental frequency; the YIN algorithm calculates the instantaneous phase of the signal using a cumulative averaging method and estimates the fundamental frequency using the phase difference; and the neural network fundamental frequency detector directly predicts the fundamental frequency value from audio frames by training a deep learning model, such as a convolutional neural network (CNN) or a recurrent neural network (RNN). After the fundamental frequency is calculated using the fundamental frequency detection algorithm, the fundamental frequency detection results of consecutive frames can be smoothed using a moving average filter or a Kalman filter to generate the fundamental frequency profile.

[0060] Based on the phoneme time boundaries determined by the DTW alignment results, the duration distribution of each phoneme is calculated. Specifically, the time boundaries of each phoneme, defined by the DTW alignment results, define the start and end positions of each phoneme in the audio signal. The duration of each phoneme is calculated, i.e., the difference between the end and start times of the phoneme. By analyzing the duration data of all phonemes, statistics such as mean duration, standard deviation, minimum, and maximum values ​​can be calculated to understand the duration distribution. To eliminate duration differences between different speakers or under different recording conditions, the duration data can be normalized, for example, by using relative duration or proportional duration.

[0061] In one embodiment, after extracting prosodic parameters such as energy intensity, fundamental frequency profile, and duration distribution of each phoneme from the audio signal according to time boundaries, the extracted prosodic parameters are input into a Gaussian Mixture Module (GMM) model for cluster analysis to generate a prosodic tag sequence. The Gaussian Mixture Module (GMM) is a probabilistic model used for cluster analysis of data, assuming that all data points are mixtures of a finite number of Gaussian distributions.

[0062] In one embodiment, audio data containing different prosodic features (such as stressed, unstressed, rising, falling, etc.) can be collected and manually labeled to obtain a labeled dataset of prosodic parameters. Prosodic parameters, including energy intensity, fundamental frequency profile, and duration distribution, are extracted from the collected audio data. The extracted prosodic parameters are preprocessed, including normalization, outlier removal, and missing value imputation, to improve the efficiency and accuracy of model training. Based on the type of prosodic label, the number of clusters (i.e., the number of Gaussian distributions) in the GMM model is initialized. The GMM model is trained using the Expectation-Maximization (EM) algorithm to learn the parameters of each Gaussian distribution, including mean, variance, and mixing coefficients. The model performance is evaluated using methods such as cross-validation, including clustering accuracy and adjusting the RAND index (ARI), and the model parameters are adjusted as needed.

[0063] After the GMM model is trained, the extracted prosodic parameters (energy intensity, fundamental frequency profile, and duration distribution, etc.) can be used as input features. The trained GMM model is then used to perform cluster analysis on the input features, assigning the most likely prosodic marker to each phoneme. Based on the output of the GMM model, the prosodic marker corresponding to each cluster is interpreted, such as stressed or unstressed pronunciations.

[0064] Based on the clustering results of the GMM model, a corresponding prosodic label is assigned to each phoneme. The assigned prosodic labels are then constructed into a prosodic label sequence according to the order of the phonemes. The generated prosodic label sequence is then optimized, such as by smoothing and removing redundant labels, to improve the naturalness and intelligibility of the sequence.

[0065] This embodiment, through a series of steps including text preprocessing, phoneme mapping, prosodic parameter extraction, and cluster analysis, generates natural and expressive speech output, improving the naturalness and intelligibility of speech synthesis. It also ensures that the generated speech accurately reflects different language styles, thus providing users with a richer and more realistic voice interaction experience. Furthermore, by using machine learning and deep learning technologies, the system can adapt to different languages ​​and speaking styles, exhibiting excellent flexibility and scalability.

[0066] S103. Using a gating fusion mechanism, the phoneme sequence and the prosodic marker sequence are fused to obtain the prosodic phoneme fusion feature.

[0067] like Figure 3 As shown, after obtaining the phoneme sequence and prosodic marker sequence, a gated fusion mechanism (GTU) is adopted to fuse prosodic information at the phoneme level, thereby achieving refined control of speech rate, stress, and pitch.

[0068] Further, the phoneme sequence is mapped to a first embedding vector, and the prosodic marker sequence is mapped to a second embedding vector; a linear transformation is performed on the first embedding vector, and an activation function is applied to the linear transformation result to generate candidate features; a linear transformation is performed on the second embedding vector, and an activation function is applied to the linear transformation result to generate a gating signal; the candidate features and the gating signal are multiplied element-wise to obtain the prosodic phoneme fusion features.

[0069] In one embodiment, a phoneme embedding table can be constructed from a large amount of speech data using an unsupervised learning algorithm (such as principal component analysis, PCA) or a pre-trained model. This table maps each phoneme to a vector of a fixed dimension. The phoneme embedding table can be static or dynamic, i.e., the vector representation of the phonemes can be dynamically adjusted based on contextual information.

[0070] For the input phoneme sequence, each phoneme is mapped to its corresponding vector representation by looking up the phoneme embedding table to obtain the first embedding vector; if a phoneme in the phoneme sequence is not in the embedding table, the closest phoneme can be found by similarity matching, or a special unknown phoneme vector can be used.

[0071] In one embodiment, a set of prosodic markers is defined, such as stressed, unstressed, rising, falling, and level tones. For each marker in the prosodic marker sequence, it is converted into a numerical code using one-hot encoding to obtain a second embedding vector. One-hot encoding assigns a binary vector to each unique prosodic marker, where only one position is 1 and the rest are 0. For example, if there are three prosodic markers (stressed, unstressed, and rising), then stressed can be encoded as [1, 0, 0], unstressed as [0, 1, 0], and rising as [0, 0, 1].

[0072] Alternatively, a neural network model can be trained to learn to map prosodic markers to a dense vector representation. This neural network model maps the prosodic markers to a low-dimensional continuous vector space, obtaining a second embedding vector.

[0073] A gated fusion mechanism is employed to fuse the first embedding vector corresponding to the phoneme sequence and the second embedding vector corresponding to the prosodic marker sequence. The weight matrix in the gated fusion mechanism can be initialized using a random initialization method. , and bias , Parameters are initialized, and these parameters can be learned and adjusted during training. Biases are typically initialized to small constant values, such as 0 or very small positive numbers (e.g., 0.01), to avoid having an excessive impact on the output of the activation function at the beginning of training.

[0074] For the first embedding vector Applying weight matrix and bias Perform a linear transformation:

[0075] in, Represents the first embedding vector. This represents the result of a linear transformation of the first embedding vector.

[0076] For the second embedding vector Applying weight matrix and bias Perform a linear transformation:

[0077] in, Represents the first embedding vector. This represents the result of a linear transformation of the first embedding vector.

[0078] For the transformed vector and Activation functions are typically applied, using the hyperbolic tangent (tanh) and sigmoid functions to generate candidate features and gate signals, respectively.

[0079]

[0080] in, Candidate features representing phonemes; This indicates a gating signal used to control the degree to which phoneme features are allowed to pass through.

[0081] The candidate features of phonemes are multiplied element-wise with the gated signal to fuse prosodic information.

[0082] in, This indicates the characteristics of phoneme fusion in rhythm. Candidate features representing phonemes; Indicates a gating signal; This indicates dot product.

[0083] This embodiment generates candidate features and gating signals through linear transformation and activation functions. Then, these features are fused with the gating signals through element-wise multiplication to obtain phoneme fusion features. These features are used to guide the speech synthesis model to generate speech with precise control over speech rate, stress, and tone, thereby improving the naturalness and expressiveness of speech synthesis and providing more refined style control capabilities. This makes the generated speech more in line with user expectations and application scenario requirements.

[0084] S104. Based on the semantic embedding model, convert the text style description into a style vector, and extract local style features and global style features from the style vector.

[0085] Adopt a lightweight semantic embedding model (such as MPNet) to convert a text-style style description (such as "a young woman speaks with a pleasant mood") into a numerical paralinguistic style vector, and then extract local features from the extracted style vector through two linear layers and global features , which are used to guide the overall performance of speech generation.

[0086] A lightweight semantic embedding model suitable for processing style descriptions can be selected, such as MPNet, BERT, or other pre-trained Transformer models. For example, MPNet is a Transformer-based model designed for processing natural language and can capture the deep semantic information of text. If the existing models cannot meet specific requirements, a large amount of text data with style labels can be collected and the selected lightweight semantic embedding model can be fine-tuned to adapt to the style description task.

[0087] Furthermore, perform word segmentation on the text style description to obtain segmented text; input the segmented text into the pre-trained semantic embedding model for vectorization processing to obtain the style vector; based on the first linear layer, extract the local style features from the style vector; based on the second linear layer, extract the global style features from the style vector.

[0088] Select a suitable word segmentation tool according to the language of the text to perform word segmentation on the text style description. Input the text style description into the word segmentation tool and cut it into individual words or phrases. For example, the text "a young woman speaks with a pleasant mood" will be segmented into ["young", "woman", "with", "pleasant", "mood", "speaks"].

[0089] Perform normalization processing on the text, including lowercasing, removing stop words, punctuation marks, etc., to reduce the interference of irrelevant information. Specifically, convert all words to lowercase to reduce the differences caused by case, for example, convert "Young" to "young". Stop words in the text (such as "de", "shi", "zai", etc.) usually do not carry important information, and a stop word list can be constructed or used to filter and remove such stop words from the text. Punctuation marks may interfere with the model's understanding, and regular expressions or string processing functions can be used to remove punctuation marks from the text.

[0090] Define a preprocessing function that integrates the above word segmentation and standardization steps, taking the original text as input and outputting the processed text. Apply this preprocessing function to all text style descriptions to ensure that all input data have a consistent format.

[0091] The preprocessed text style description is input into a semantic embedding model to obtain the embedding representation of each word or the entire sequence. Specifically, a suitable pre-trained model for text embedding can be selected, such as BERT, MPNet, or other Transformer architectures. The preprocessed text style description (e.g., "a young woman speaks with a pleasant emotion") is input into the model, and forward propagation is performed to obtain the embedding representation of each word. The embedding vector of each word is extracted from the output of the semantic embedding model. Pooling operations (such as average pooling or max pooling) are performed on the word embeddings. The selected pooling operation is then applied to the extracted word embeddings to obtain the style vector. For example, average pooling can be used to calculate the average of all word embeddings to obtain a single vector representation of the entire sequence, or max pooling can be used to select the largest value from all word embeddings, which helps to capture the most salient features in the sequence.

[0092] A first linear layer can be designed to map the style vector's dimension from its original dimension to a lower dimension to extract local style features. A second linear layer can then be designed to map the style vector's dimension to another dimension to extract global style features.

[0093] The style vector, representing the text style description, is obtained from a semantic embedding model (such as MPNet). This style vector is a numerical representation of the text style description by the model, capturing the semantic information of the text. The style vector is then input into the first and second linear layers. The weight matrices and bias vectors of the two linear layers are initialized, typically to zero or small random values.

[0094] The style vector is input into the first linear layer, matrix multiplication is calculated, and a bias is added to obtain the local style features. The same style vector is input into the second linear layer, matrix multiplication is performed, and a bias is added to obtain the global style features. .

[0095] This embodiment uses a lightweight semantic embedding model (such as MPNet) to process text style descriptions, converting them into numerical style vectors. Then, it uses two linear layers to extract local and global style features, which not only captures the deep semantic information of the text, but also effectively guides the speech synthesis process through the extracted style features, generating speech output with specific style features.

[0096] S105. By using a modulation mechanism, the local stylistic features are injected into the phoneme fusion features to obtain phoneme-level stylization features.

[0097] In one embodiment, the modulation mechanism (Feature-wise Linear Modulation, FiLM) transmits information by altering certain properties of a signal. In this embodiment, the modulation mechanism is used for style control in speech synthesis. Continuing... Figure 3 As shown, style information (such as intonation and emotion) is injected into the speech synthesis model through a modulation mechanism. Different features (such as local style features and phoneme fusion features) are fused together through the modulation mechanism, so that the generated speech has a specific style, in order to generate richer speech representations.

[0098] Furthermore, based on the local style features, modulation parameters are calculated; based on the modulation mechanism, the modulation parameters are used to perform modulation calculations on the phoneme fusion features to generate the phoneme-level stylization features.

[0099] In one embodiment, a third linear layer can be defined for integrating local style features. Convert to scaled vector The weight matrix and bias of the third linear layer are as follows: and Define a fourth linear layer to process local style features. Convert to bias vector The weight matrix and bias of the third linear layer are as follows: and .

[0100] The weight matrix obtained through training and and bias and Used to generate style-specific modulation parameters, i.e., scaling vectors. and bias vector During training, the weight matrix of the modulation layer is updated and optimized using the backpropagation algorithm. and and bias and To minimize the loss function.

[0101] Local style features The input is fed into the third linear layer, where the scaling vector is computed. :

[0102] in, This is the weight matrix. For bias parameters, As a local stylistic feature, For scaling vectors.

[0103] Local style features The input is fed into the fourth linear layer to compute the bias vector. :

[0104] in, This is the weight matrix. For bias parameters, As a local stylistic feature, This is the bias vector.

[0105] Acquiring phoneme fusion features from gated fusion units (GTUs) This feature already contains phoneme and prosodic information. The calculated modulation parameters are then used. and , on the characteristics of phoneme fusion Affine transformations are performed, and stylistic information is injected into the features of each phoneme through element-wise multiplication (Hadamard product) and addition, generating phoneme-level stylized features. :

[0106] in, Indicates phoneme-level stylization features. This indicates the characteristics of phoneme fusion in rhythm. and For modulation parameters, Used to represent element-wise multiplication (Hadamard product), that is... and Multiply the corresponding elements together, then add... .

[0107] This embodiment effectively injects local style features into the phonology-phoneme fusion features through a modulation mechanism, thereby generating phoneme-level stylized features. This allows the speech synthesis model to dynamically adjust the generated speech according to specific style descriptions (such as intonation and emotion), improving the naturalness and expressiveness of the speech and making it more in line with user expectations and application scenario needs. Through optimization of modulation layer parameters during training, the model can learn how to better fuse style and phoneme information, ultimately generating speech output with rich style features.

[0108] S106. Input the factor-level stylization features and the global style features into a pre-built decoder to generate the target speech audio.

[0109] In one embodiment, a decoder model based on the Diffusion Transformer (DiT) framework can be selected or designed. This model should be able to process stylized features and generate high-quality speech. The DiT model is a deep learning model that combines diffusion models and the Transformer architecture for generation tasks such as image, audio, or text generation. It generates data by simulating diffusion processes in the physical world.

[0110] For DiT decoders, appropriate loss functions can be selected based on the requirements of the synthesis task, such as mean squared error, Mel-spectral distance, and perceptual loss. Mean squared error is used to directly compare the differences between generated and target speech in the time or frequency domains; Mel-spectral distance measures the similarity between the Mel spectra of generated and target speech; and perceptual loss uses neural networks to simulate human evaluation of audio perceptual differences. In some cases, multiple loss functions can be combined to simultaneously optimize different aspects of the DiT decoder model, such as sound quality, naturalness, and style consistency, depending on the specific application requirements.

[0111] Continue as Figure 3 As shown, the input features and target speech are fed into the model, and the forward propagation process is executed to generate predicted speech. The loss function is calculated based on the predicted and target speech. Using the gradient of the loss function with respect to the DiT decoder model parameters, the model parameters are updated via backpropagation to reduce the loss. The forward and backpropagation processes are repeated until the DiT decoder model converges or reaches a predetermined number of training epochs, resulting in a trained DiT decoder model suitable for speech synthesis.

[0112] Further, the factor-level stylistic features and the global style features are fused to obtain fused features; the fused features are input into the decoder, and the decoder gradually converts the fused features into speech representations until the Mel spectrum is output; the Mel spectrum is converted into a time-domain waveform through a pre-trained neural vocoder to obtain the target speech audio.

[0113] Phoneme-level stylization features can be used With global style features The features are spliced ​​and merged into a unified feature representation. Or, define weights and The two are then weighted and merged: The integrated features are then normalized to ensure they are on the same scale. Normalization methods can include min-max normalization, Z-score normalization, and L2 normalization.

[0114] Fusion features These features, serving as the initial input features, comprise both phoneme-level stylized features and global stylized features. The time step during the diffusion process of the DiT decoder is defined. This determines the number of iterations required to generate the speech.

[0115] In the DiT decoder, through features Noise is gradually added to simulate the process of the data distribution transforming into a Gaussian noise distribution. At each time step... The Transformer structure of the DiT decoder is used to refine the feature representation, reduce noise, and progressively recover the speech signal. Transformer layers are applied to process the features. It includes attention mechanisms and feedforward networks to capture long-range dependencies between features. At each time step, the decoder learns to predict and remove noise, gradually recovering clear speech features.

[0116] Specifically, global style features copy This ensures that each time step... Each has corresponding style information. The copied global style features are then compared with the initial hidden state. Adding each digit sequentially forms a conditional implicit sequence. This is to ensure the consistency of style information during the generation process. This serves as the initial state for the diffusion Transformer decoder. An N-step inverse denoising process is performed, where in each step, the decoder utilizes the output from the previous time step and the current global style features. The input conditions for cross-attention are used to refine the hidden states. In each step, the decoder progressively reduces noise and recovers clearer speech features until a Mel spectrum is generated.

[0117] After N denoising steps, the decoder outputs the final Mel spectrum, which is the frequency domain representation of the speech signal, containing pitch and timbre information. A pre-trained neural vocoder, such as WaveNet, WaveGlow, or a Mel spectrum-based vocoder, is selected, capable of converting the Mel spectrum into a time-domain waveform. The Mel spectrum is input into the neural vocoder, which, through the learned mapping relationship, outputs the final speech waveform as the target speech audio. The generated target speech audio can be smoothed to reduce noise and distortion, and audio enhancement techniques, such as dynamic range compression or equalization, can be applied to improve the naturalness and intelligibility of the speech.

[0118] This embodiment employs a lightweight structure and a cacheable style embedding mechanism, resulting in inference speeds tens of times faster than schemes based on large language models, making it suitable for real-time and low-computing-power devices. Two-layer style modeling enables independent control of local prosody and global paralinguistic features, allowing for precise adjustment of multi-dimensional styles such as speech rate, intonation, gender, emotion, age, and accent, generating highly natural and consistent speech. The explicit modulation structure improves interpretability and maintainability, ensuring stable output under different prompts, demonstrating good robustness and consistency. Based on a unified phoneme coding design, it possesses cross-language adaptability and can be widely applied in voice assistants, audiobooks, game dubbing, and multilingual interactive scenarios, exhibiting significant application value and promising prospects.

[0119] This embodiment provides a speech generation method based on two-layer style modeling. This method, by acquiring the original text and style description, can clearly define the target text content and desired style features for speech synthesis. The text is converted into a phoneme sequence and a prosodic marker sequence is extracted. A gated fusion mechanism is then used to fuse the phoneme sequence and the prosodic marker sequence. The resulting phoneme-prosodic fusion features not only contain basic speech information but also incorporate rhythmic and prosodic information, enhancing the naturalness and expressiveness of the speech. A semantic embedding model is used to convert the text style description into a style vector, from which local and global style features are extracted, enabling the capture and simulation of more subtle style changes. A modulation mechanism injects the local style features into the phoneme-prosodic fusion features, obtaining phoneme-level stylized features that accurately reflect the style description of the text, thereby achieving high style consistency during speech synthesis. By inputting phoneme-level stylistic features and global style features into the decoder, the generated target speech audio is not only accurate in content, but also highly matched with the user's needs in style. This improves the accuracy of style control in the text-to-speech synthesis system when generating speech, ensuring the style accuracy and personalization of the final speech output, making the generated speech more natural and personalized.

[0120] Please see Figure 4 , Figure 4 This is a schematic diagram of the current embodiment of a speech generation device based on two-layer style modeling provided in this application. The speech generation device based on two-layer style modeling is used to execute the aforementioned speech generation method based on two-layer style modeling.

[0121] like Figure 4 As shown, the speech generation device 200 based on dual-layer style modeling includes: a data acquisition module 201, a phoneme conversion module 202, a phoneme and rhythm fusion module 203, a style feature extraction module 204, a feature modulation module 205, and a speech generation module 206.

[0122] Data acquisition module 201 is used to acquire raw text and text style description; Phoneme conversion module 202 is used to convert the original text into a phoneme sequence and extract the prosodic marker sequence; The phoneme-prosody fusion module 203 is used to fuse the phoneme sequence and the prosodic marker sequence using a gated fusion mechanism to obtain prosodic-phoneme fusion features; The style feature extraction module 204 is used to convert the text style description into a style vector based on a semantic embedding model, and extract local style features and global style features from the style vector; The feature modulation module 205 is used to inject the local style features into the phoneme fusion features through a modulation mechanism to obtain phoneme-level stylization features; The speech generation module 206 is used to input the factor-level stylization features and the global style features into a pre-built decoder to generate target speech audio.

[0123] In one embodiment, the phoneme conversion module 202 includes: The conversion rule base loading unit is used to load a phoneme conversion rule base that matches the language type of the original text from a preset database; A phoneme sequence generation unit is used to perform phoneme conversion on the original text based on the phoneme conversion rule library to generate the phoneme sequence; The prosodic parameter extraction unit is used to perform temporal alignment between the phoneme sequence and the reference prosodic template using a time warping algorithm, and to extract the prosodic parameters corresponding to each phoneme in the phoneme sequence, wherein the prosodic parameters include energy intensity, fundamental frequency profile and duration distribution. The prosodic marker sequence generation unit is used to map the prosodic parameter corresponding to each phoneme in the phoneme sequence to a predefined prosodic marker to generate the prosodic marker sequence.

[0124] In one embodiment, the prosodic parameter extraction unit includes: A reference prosodic template acquisition subunit is used to acquire the reference prosodic template; The time boundary acquisition sub-unit is used to align the phoneme sequence with the reference prosodic template in time based on the dynamic time warping algorithm to obtain the time boundary of each phoneme. The phonological parameter acquisition subunit is used to extract the energy intensity, fundamental frequency profile, and duration distribution of each phoneme based on the time boundary corresponding to each phoneme, and obtain the phonological parameters.

[0125] In one embodiment, the style feature extraction module 204 includes: A text segmentation unit is used to segment the text style description to obtain segmented text. The style vector generation unit is used to input the segmented text into the pre-trained semantic embedding model for vectorization processing to obtain the style vector; A local style feature extraction unit is used to extract the local style features from the style vector based on the first linear layer; A global style feature extraction unit is used to extract the global style features from the style vector based on the second linear layer.

[0126] In one embodiment, the phoneme-rhythm fusion module 203 includes: The feature mapping unit is used to map the phoneme sequence to a first embedding vector and the prosodic marker sequence to a second embedding vector; The candidate feature generation unit is used to perform a linear transformation on the first embedding vector and apply an activation function to the linear transformation result to generate candidate features; The gate signal generation unit is used to perform a linear transformation on the second embedded vector and apply an activation function to the linear transformation result to generate a gate signal; The rhythmic phoneme fusion feature generation unit is used to perform element-wise multiplication of the candidate features with the gated signal to obtain rhythmic phoneme fusion features.

[0127] In one embodiment, the feature modulation module 205 includes: A modulation parameter calculation unit is used to calculate modulation parameters based on the local style features; A modulation calculation unit is used to perform modulation calculations on the phoneme fusion features based on the modulation mechanism and the modulation parameters, and generate the phoneme-level stylization features.

[0128] In one embodiment, the speech generation module 206 includes: The feature fusion unit is used to fuse the factor-level stylization features and the global style features to obtain fused features; The feature conversion unit is used to input the fused features into the decoder, and the decoder gradually converts the fused features into speech representation until the Mel spectrum is output; The speech audio generation unit is used to convert the Mel spectrum into a time-domain waveform using a pre-trained neural vocoder to obtain the target speech audio.

[0129] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the device and each module described above can be referred to the corresponding processes in the aforementioned embodiments of the speech generation method based on two-layer style modeling, and will not be repeated here.

[0130] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 5 It runs on the computer device shown.

[0131] Please see Figure 5 , Figure 5 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0132] See Figure 5 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0133] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any speech generation method based on two-level style modeling.

[0134] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0135] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When executed by a processor, the computer program enables the processor to perform any speech generation method based on two-level style modeling.

[0136] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0137] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0138] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: Retrieve the original text and text style description; The original text is converted into a phoneme sequence, and the prosodic marker sequence is extracted; A gated fusion mechanism is used to fuse the phoneme sequence and the prosodic marker sequence to obtain prosodic phoneme fusion features; Based on the semantic embedding model, the text style description is converted into a style vector, and local style features and global style features are extracted from the style vector; By using a modulation mechanism, the local stylistic features are injected into the phonemic fusion features to obtain phoneme-level stylized features; The factor-level stylization features and the global style features are input into a pre-built decoder to generate the target speech audio.

[0139] In one embodiment, the processor, when implementing the conversion of the original text into a phoneme sequence and the extraction of the prosodic marker sequence, is configured to: Load a phoneme conversion rule library that matches the language type of the original text from a preset database; Based on the phoneme conversion rule library, the original text is converted into phonemes to generate the phoneme sequence; Using a time warping algorithm, the phoneme sequence is temporally aligned with a reference prosodic template, and the prosodic parameters corresponding to each phoneme in the phoneme sequence are extracted. The prosodic parameters include energy intensity, fundamental frequency profile, and duration distribution. The prosodic parameters corresponding to each phoneme in the phoneme sequence are mapped to predefined prosodic markers to generate the prosodic marker sequence.

[0140] In one embodiment, when the processor implements the time warping algorithm to perform temporal alignment between the phoneme sequence and the reference prosodic template, and extracts the prosodic parameters corresponding to each phoneme in the phoneme sequence, it is configured to: Obtain the reference prosody template; Based on the dynamic time warping algorithm, the phoneme sequence is temporally aligned with the reference prosodic template to obtain the time boundary of each phoneme; Based on the time boundary corresponding to each phoneme, the energy intensity, fundamental frequency profile, and duration distribution of each phoneme are extracted to obtain the rhythm parameters.

[0141] In one embodiment, when implementing the semantic embedding model to convert the text style description into a style vector and extract local style features and global style features from the style vector, the processor is configured to: The text style description is segmented into words to obtain segmented text; The segmented text is input into the pre-trained semantic embedding model for vectorization processing to obtain the style vector; Based on the first linear layer, the local style features are extracted from the style vector; The global style features are extracted from the style vector based on the second linear layer.

[0142] In one embodiment, when the processor implements the gated fusion mechanism to fuse the phoneme sequence and the prosodic marker sequence to obtain prosodic phoneme fusion features, it is configured to: The phoneme sequence is mapped to a first embedding vector, and the prosodic marker sequence is mapped to a second embedding vector; A linear transformation is performed on the first embedding vector, and an activation function is applied to the result of the linear transformation to generate candidate features; A linear transformation is performed on the second embedding vector, and an activation function is applied to the result of the linear transformation to generate a gating signal; The candidate features are multiplied element-wise with the gating signal to obtain the phoneme fusion features.

[0143] In one embodiment, when the processor implements the process of injecting the local style features into the phoneme fusion features through a modulation mechanism to obtain phoneme-level stylized features, it is configured to: Based on the aforementioned local style features, the modulation parameters are calculated; Based on the modulation mechanism, the modulation parameters are used to perform modulation calculations on the phoneme fusion features to generate the phoneme-level stylized features.

[0144] In one embodiment, when the processor generates target speech audio by inputting the factor-level stylization features and the global style features into a pre-built decoder, it is configured to: The factor-level stylization features and the global style features are fused to obtain fused features; The fused features are input into the decoder, which then progressively converts them into a speech representation until a Mel spectrum is output. The target speech audio is obtained by converting the Mel spectrum into a time-domain waveform using a pre-trained neural vocoder.

[0145] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the speech generation methods based on two-layer style modeling provided in the embodiments of this application.

[0146] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMediaCard (SMC), SecureDigital (SD) card, or FlashCard equipped on the computer device.

[0147] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A speech generation method based on two-layer style modeling, characterized in that, The method includes: Retrieve the original text and text style description; The original text is converted into a phoneme sequence, and the prosodic marker sequence is extracted; A gated fusion mechanism is used to fuse the phoneme sequence and the prosodic marker sequence to obtain prosodic phoneme fusion features; Based on the semantic embedding model, the text style description is converted into a style vector, and local style features and global style features are extracted from the style vector; By using a modulation mechanism, the local stylistic features are injected into the phonemic fusion features to obtain phoneme-level stylized features; The factor-level stylization features and the global style features are input into a pre-built decoder to generate the target speech audio.

2. The speech generation method based on two-layer style modeling according to claim 1, characterized in that, The process of converting the original text into a phoneme sequence and extracting the prosodic marker sequence includes: Load a phoneme conversion rule library that matches the language type of the original text from a preset database; Based on the phoneme conversion rule library, the original text is converted into phonemes to generate the phoneme sequence; Using a time warping algorithm, the phoneme sequence is temporally aligned with a reference prosodic template, and the prosodic parameters corresponding to each phoneme in the phoneme sequence are extracted. The prosodic parameters include energy intensity, fundamental frequency profile, and duration distribution. The prosodic parameters corresponding to each phoneme in the phoneme sequence are mapped to predefined prosodic markers to generate the prosodic marker sequence.

3. The speech generation method based on two-layer style modeling according to claim 2, characterized in that, The time warping algorithm is used to align the phoneme sequence with the reference prosodic template in time, and to extract the prosodic parameters corresponding to each phoneme in the phoneme sequence, including: Obtain the reference prosody template; Based on the dynamic time warping algorithm, the phoneme sequence is temporally aligned with the reference prosodic template to obtain the time boundary of each phoneme; Based on the time boundary corresponding to each phoneme, the energy intensity, fundamental frequency profile, and duration distribution of each phoneme are extracted to obtain the rhythm parameters.

4. The speech generation method based on two-layer style modeling according to claim 1, characterized in that, The semantic embedding model converts the text style description into a style vector, and extracts local and global style features from the style vector, including: The text style description is segmented into words to obtain segmented text; The segmented text is input into the pre-trained semantic embedding model for vectorization processing to obtain the style vector; Based on the first linear layer, the local style features are extracted from the style vector; The global style features are extracted from the style vector based on the second linear layer.

5. The speech generation method based on two-layer style modeling according to claim 1, characterized in that, The method employs a gated fusion mechanism to fuse the phoneme sequence and the prosodic marker sequence to obtain prosodic phoneme fusion features, including: The phoneme sequence is mapped to a first embedding vector, and the prosodic marker sequence is mapped to a second embedding vector; A linear transformation is performed on the first embedding vector, and an activation function is applied to the result of the linear transformation to generate candidate features; A linear transformation is performed on the second embedding vector, and an activation function is applied to the result of the linear transformation to generate a gating signal; The candidate features are multiplied element-wise with the gating signal to obtain the phoneme fusion features.

6. The speech generation method based on two-layer style modeling according to claim 1, characterized in that, The process of injecting the local stylistic features into the phonemic fusion features through a modulation mechanism to obtain phoneme-level stylized features includes: Based on the aforementioned local style features, the modulation parameters are calculated; Based on the modulation mechanism, the modulation parameters are used to perform modulation calculations on the phoneme fusion features to generate the phoneme-level stylized features.

7. The speech generation method based on two-layer style modeling according to claim 1, characterized in that, The step of inputting the factor-level stylization features and the global style features into a pre-built decoder to generate the target speech audio includes: The factor-level stylization features and the global style features are fused to obtain fused features; The fused features are input into the decoder, which then progressively converts them into a speech representation until a Mel spectrum is output. The target speech audio is obtained by converting the Mel spectrum into a time-domain waveform using a pre-trained neural vocoder.

8. A speech generation device based on two-layer style modeling, characterized in that, The speech generation device based on dual-style modeling includes: The data acquisition module is used to acquire the raw text and text style description; A phoneme conversion module is used to convert the original text into a phoneme sequence and extract a prosodic marker sequence; The phoneme-prosody fusion module is used to fuse the phoneme sequence and the prosodic marker sequence using a gated fusion mechanism to obtain prosodic-phoneme fusion features; The style feature extraction module is used to convert the text style description into a style vector based on a semantic embedding model, and extract local style features and global style features from the style vector; The feature modulation module is used to inject the local style features into the phoneme fusion features through a modulation mechanism to obtain phoneme-level stylized features; The speech generation module is used to input the factor-level stylization features and the global style features into a pre-built decoder to generate target speech audio.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the speech generation method based on two-layer style modeling as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the speech generation method based on two-layer style modeling as described in any one of claims 1 to 7.

Images