Voice generation method and device based on distribution prediction, equipment and medium

Abstract

The invention relates to the technical field of speech synthesis, can be applied to business system platforms of communication, medical health, financial science and technology and the like, and discloses a speech generation method, device, equipment and medium based on distribution prediction. And inputting the context features and the historical waveform feature sequence into a next distribution prediction module, predicting probability distribution for generating a voice waveform at a next moment, generating voice waveform points based on the probability distribution, updating the historical waveform feature sequence, performing loop execution until a termination condition is met, and finally performing combination to generate a target voice waveform. According to the method, context coding and historical waveform features are combined, the long-distance dependence modeling capability is improved, the voice coherence and naturalness are ensured, voice generation is optimized through distributed prediction, the reasoning efficiency is improved, and the real-time synthesis requirement is met.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis technology, and in particular to a speech generation method, apparatus, device and storage medium based on distribution prediction. Background Art

[0002] In the field of speech synthesis, autoregressive models have made significant progress, with representative methods including WaveNet and Tacotron2. These models progressively generate speech waveforms, with the output at each moment dependent on previously generated waveform points, resulting in more natural synthesized speech. However, existing technologies still have limitations. Due to the high computational complexity of autoregressive generation, waveforms must be generated frame by frame during the inference phase, resulting in slow generation speed. This inefficiency can impact user experience, particularly in real-time voice interaction scenarios. Furthermore, existing speech synthesis methods primarily rely on maximum likelihood estimation for distribution modeling, resulting in a lack of diversity in the generated speech, making it difficult to meet the demands for personalized or emotional speech in diverse application scenarios. Furthermore, challenges remain in modeling long-range dependencies, which can lead to issues such as incoherent intonation and unnatural rhythmic changes during speech synthesis, impacting the stability of speech quality.

[0003] In the healthcare sector, speech synthesis technology is used in applications such as remote consultations, rehabilitation training, and assisted diagnosis and treatment. Speech clarity, coherence, and personalized expression are crucial to the user experience. Existing technologies are prone to instability in intonation and prosody during long-duration speech synthesis. For example, in applications such as medical record entry and voice-based diagnosis and treatment assistants, the synthesized speech may suffer from inadequate contextual information processing, leading to unnatural speech flow and pauses. Furthermore, speech clarity and noise immunity are particularly important in telemedicine and rehabilitation training scenarios. Existing methods are susceptible to noise interference during speech waveform generation, resulting in unwanted noise inclusion in the speech signal or unclear speech features, hindering communication between patients and medical staff. Furthermore, medical applications also have different requirements for speech style. For example, patient instructions require a soft and easy-to-understand voice, while medical report readings require a more formal and professional style. However, existing technologies lack flexibility in speech style control, making it difficult to adjust speech features to suit the needs of different medical scenarios, limiting the widespread application of speech synthesis technology in the healthcare sector.

[0004] In the FinTech sector, speech synthesis technology is widely used in scenarios such as intelligent customer service, financial assistants, and voice auditing. For these applications, inference speed and speech naturalness are key factors in determining system performance. However, existing autoregressive speech synthesis methods suffer from low inference efficiency and are prone to response delays when handling high-concurrency requests, impacting the user experience. Furthermore, speech accuracy and stability are crucial for tasks such as financial risk warnings and business explanations. Existing technologies can suffer from information attenuation when generating long speech segments, resulting in unclear sentences or unnatural punctuation. Furthermore, different financial scenarios require different speech styles. For example, customer service conversations require a more approachable voice, while risk warnings require a more rigorous voice. Due to limitations in their generation methods, existing technologies struggle to flexibly adjust speech styles to specific scenarios, which limits the applicability of speech synthesis.

[0005] Overall, existing autoregressive speech synthesis methods still have limitations in applications in industries such as finance and healthcare, including slow inference speed, insufficient diversity in generated speech, and limited ability to model long-range dependencies. These technical bottlenecks not only affect the quality and efficiency of speech synthesis but also restrict its application in high-concurrency, real-time scenarios. Therefore, improving speech synthesis inference efficiency, enhancing speech generation diversity, and optimizing long-range dependency modeling have become key issues that need to be addressed in the practical application of speech synthesis technology. Summary of the Invention

[0006] The main purpose of the present invention is to provide a speech generation method, device, equipment and storage medium based on distribution prediction, aiming to solve the technical problems of low reasoning efficiency and limited long-distance dependency modeling capabilities in speech synthesis in the existing technology, which affect the fluency and naturalness of speech.

[0007] To achieve the above object, the present invention provides a speech generation method based on distribution prediction, comprising:

[0008] Encode the input text through the context encoder to generate context features containing global temporal dependencies;

[0009] Initialize the historical waveform feature sequence to an empty sequence or preset initial mark;

[0010] Inputting the context features and the historical waveform feature sequence into a next distribution prediction module to predict and generate a probability distribution of the speech waveform at the next moment;

[0011] Generate a speech waveform point at the next moment according to the probability distribution of the speech waveform at the next moment;

[0012] Encoding the speech waveform point at the next moment as a new waveform feature, and updating the new waveform feature to the historical waveform feature sequence;

[0013] Repeat the probability distribution prediction of the next distribution prediction module, the speech waveform point generation at the next moment and the historical waveform feature sequence update steps until the preset termination condition is met, stop the repetitive execution, and combine the generated speech waveform points into the target speech waveform.

[0014] Furthermore, to achieve the above-mentioned object, the present invention provides a speech generation device based on distribution prediction, comprising:

[0015] The context encoding module is used to encode the input text through the context encoder to generate context features containing global temporal dependencies;

[0016] A historical feature management module, used to initialize the historical waveform feature sequence to an empty sequence or a preset initial mark;

[0017] A distribution prediction module is used to input the context features and the historical waveform feature sequence into the next distribution prediction module to predict and generate the probability distribution of the speech waveform at the next moment;

[0018] A speech waveform point generation module, configured to generate a speech waveform point at the next moment according to the probability distribution of the speech waveform at the next moment;

[0019] A waveform feature encoding module is used to encode the speech waveform point at the next moment into a new waveform feature, and update the new waveform feature into the historical waveform feature sequence;

[0020] The speech synthesis control module is used to repeatedly execute the probability distribution prediction, speech waveform point generation at the next moment and historical waveform feature sequence update steps of the next distribution prediction module until a preset termination condition is met, stop the repeated execution, and combine the generated speech waveform points into a target speech waveform.

[0021] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a distribution prediction-based speech generation program stored in the memory and runnable on the processor, and when the distribution prediction-based speech generation program is executed by the processor, the steps of the distribution prediction-based speech generation method as described above are implemented.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a speech generation program based on distribution prediction is stored. When the speech generation program based on distribution prediction is executed by a processor, the steps of the speech generation method based on distribution prediction as described above are implemented.

[0023] Beneficial effects: The present invention relates to the field of speech synthesis technology and can be applied to business system platforms such as communications, medical health, and financial technology. A speech generation method based on distribution prediction is disclosed, comprising: encoding input text, generating context features containing global temporal dependencies, initializing a historical waveform feature sequence to an empty sequence or a preset initial marker, inputting the context features and the historical waveform feature sequence into a next distribution prediction module, predicting and generating the probability distribution of the speech waveform at the next moment, generating speech waveform points at the next moment based on the predicted probability distribution, encoding the generated speech waveform points into new waveform features and updating them into the historical waveform feature sequence, repeatedly performing probability distribution prediction, speech waveform point generation, and historical waveform feature sequence update, stopping execution when a termination condition is met, and combining the generated speech waveform points into a target speech waveform. The present invention obtains context features containing global temporal dependencies through a context encoder, and performs autoregressive distribution prediction in combination with the historical waveform feature sequence, so that long-distance dependencies can be effectively modeled during speech synthesis, thereby improving the coherence and naturalness of speech. At the same time, the speech waveform points at the next moment are generated through probability distribution prediction and continuously updated based on the historical waveform feature sequence, so that speech generation not only considers short-term information but also maintains global consistency. In addition, the termination condition is used to control the generation process, improve the inference efficiency, reduce the computational overhead, and meet the needs of real-time speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0025] Figure 1 Schematic diagram of an application environment of a speech generation method based on distribution prediction in one embodiment of the present invention;

[0026] Figure 2 1 is a flow chart of an embodiment of a method for speech generation based on distribution prediction according to the present invention;

[0027] Figure 3 Schematic diagram of functional modules of a preferred embodiment of a speech generation device based on distribution prediction of the present invention;

[0028] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0029] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0031] The speech generation method based on distribution prediction provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the user end communicates with the server end through a network. The server end can encode the input text through the user end, generate context features containing global temporal dependencies, initialize the historical waveform feature sequence to an empty sequence or a preset initial mark, input the context features and the historical waveform feature sequence into the next distribution prediction module, predict and generate the probability distribution of the speech waveform at the next moment, generate the speech waveform points at the next moment according to the predicted probability distribution, encode the generated speech waveform points into new waveform features and update them to the historical waveform feature sequence, repeat the probability distribution prediction, speech waveform point generation and historical waveform feature sequence update, stop execution when the termination condition is met, and combine the generated speech waveform points into the target speech waveform. The present invention obtains context features containing global temporal dependencies through a context encoder, and performs autoregressive distribution prediction in combination with the historical waveform feature sequence, so that long-distance dependencies can be effectively modeled in the speech synthesis process, thereby improving the coherence and naturalness of the speech. At the same time, the speech waveform points at the next moment are generated by probability distribution prediction, and are continuously updated based on the historical waveform feature sequence, so that speech generation not only considers short-term information but also maintains global consistency. Furthermore, a termination condition is used to control the generation process, improving inference efficiency, reducing computational overhead, and meeting the requirements of real-time speech synthesis. The user end includes, but is not limited to, various personal computers, laptops, smartphones, tablet computers, and portable wearable devices. The server end can be implemented as a standalone server or a server cluster consisting of multiple servers. The present invention is described in detail below using specific embodiments.

[0032] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a method for speech generation based on distribution prediction provided by the present invention. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0033] like Figure 2 As shown, the speech generation method based on distribution prediction proposed by the present invention includes the following steps:

[0034] S10, encodes the input text through the context encoder to generate context features containing global temporal dependencies;

[0035] In this embodiment, encoding the input text is one of the most critical steps in the speech generation process. Its core goal is to convert discrete text data into continuous feature representations that can be used for subsequent prediction of speech waveforms, while ensuring that the model can capture the global temporal dependencies in the text.

[0036] Input text is typically natural language text, consisting of multiple words, phonemes, and grammatical structures. Due to the complexity of natural languages, text structures vary significantly across languages. For example, Chinese is a character-based language, while English is a vocabulary- and phoneme-based language. Therefore, encoding text requires first considering the basic units of text, namely phonemes, words, and subwords, and then performing encoding based on these units.

[0037] The context encoder is a core component for processing input text. Its purpose is to model the structured text information, ensuring that during speech generation, the model can learn the dependencies between different words, phrases, and sentences, thereby generating coherent speech output. Global temporal dependencies refer to long-range connections between different time steps in a text sequence. For example, the words at the beginning of a sentence may influence the intonation at the end. Therefore, the encoding process needs to be able to capture both local information and long-range dependencies.

[0038] Text encoding can be preprocessed based on a phoneme segmentation strategy. A phoneme is the smallest unit of pronunciation of speech, and different languages ​​have different sets of phonemes. For example, English contains 44 phonemes, while Japanese contains only a little over 20. For different languages, a phoneme segmentation method based on linguistic rules can be used, or a data-driven approach can be used to automatically segment phoneme sequences. To more efficiently represent phoneme sequences, each phoneme is typically mapped to a high-dimensional vector during the encoding process. This process can be achieved through an embedding layer, which maps discrete text symbols into a continuous vector space, ensuring that subsequent neural networks can more effectively learn text patterns.

[0039] To further extract temporal features from text, a bidirectional long short-term memory (BiLSTM) network can be used to model the local temporal features of phoneme sequences. BiLSTM has two hidden layers, one forward and one backward, which capture the relationship between the current phoneme and its preceding and following phonemes, ensuring the modeling of local dependencies. For example, in Chinese speech synthesis, the combination of initials and finals has a significant impact on the tone of speech, and BiLSTM can effectively capture this synergistic relationship between adjacent phonemes.

[0040] Building on this foundation, to further capture global dependencies, a multi-head self-attention mechanism can be introduced. The core idea of ​​the multi-head self-attention mechanism is to calculate attention weights between any two time steps in the text and perform weighted aggregation of text features based on these weights. This ensures that the model takes into account long-range contextual information when predicting the current phoneme. For example, in English, the intonation pattern at the end of a sentence may be determined by the question word at the beginning. The self-attention mechanism can learn this long-range dependency and adjust the speech synthesis strategy accordingly.

[0041] Finally, the context encoder concatenates the local features extracted by BiLSTM with the global features modeled by the self-attention mechanism to obtain contextual features containing complete temporal information, providing high-quality input for subsequent speech waveform prediction.

[0042] During the specific implementation process, different technical means can be used to optimize the text encoding effect to adapt to different languages, application scenarios and computing resource constraints.

[0043] In some implementations, a context encoder based on an end-to-end Transformer architecture can be used. This approach leverages a fully attention-based architecture to simultaneously capture both short- and long-range dependencies while offering high parallel computational efficiency. Compared to the traditional BiLSTM architecture, the Transformer architecture offers advantages when processing long texts and is suitable for long text-to-speech tasks, such as news and novel reading.

[0044] In another implementation, a CNN-based text encoding approach can be used. For example, a TCN (temporal convolutional network) can replace BiLSTM to extract local temporal features. TCN has better parallel computing capabilities and demonstrates strong local modeling capabilities in speech synthesis tasks. Therefore, using TCN can improve model inference speed in low-power devices or embedded systems.

[0045] Furthermore, in multilingual scenarios, language category markers can be introduced before the embedding layer to ensure that the model can dynamically adjust the phoneme encoding method based on the characteristics of different languages. For example, in a multilingual speech synthesis system, independent embedding tables can be used to store phonemes in different languages, and language category information can be incorporated into the context encoding process to ensure the naturalness of speech synthesis.

[0046] For low-resource languages, data augmentation techniques can be used to introduce pseudo-data during training, such as phoneme-based concatenation data or data synthesized using TTS (text-to-speech), to enhance the model's generalization capabilities. For example, in language environments with limited corpus data, such as Tibetan or Vietnamese, transfer learning can be performed using high-resource languages ​​(such as Chinese or English), allowing the context encoder to more effectively learn the patterns of phoneme sequences.

[0047] Example: In the healthcare field, voice interaction systems can be applied to intelligent medical assistants, such as generating voice medical records based on text input. After a doctor enters a medical record summary, the text can be converted into high-quality speech for patient review or archiving as a voice medical record. Because medical terms are highly contextually related, for example, "hypertension" may appear alongside "cardiovascular disease," the context encoder effectively captures the connections between these terms, ensuring that the synthesized speech content is accurate and conforms to medical conventions.

[0048] In the financial sector, this technology can be applied to intelligent voice broadcast systems, such as automated voice responses for bank customer service or financial news broadcasts. Financial terminology requires high pronunciation accuracy, and many financial terms require dynamic adjustment in different scenarios. For example, the term "credit risk" may have different phonetic expressions in a credit assessment context. The system can identify key terms in financial text and adjust speech synthesis strategies based on the context, enhancing the intelligence of customer interactions.

[0049] In news reading and audiobook applications, it can be used for personalized speech synthesis, matching different styles of articles with appropriate speech expressions. For example, in a news broadcast scenario, the context encoder can analyze the semantic structure of the news and adjust the speech rate and pitch to make the broadcast content more consistent with the style of the news report. In the audiobook synthesis scenario, the encoder can recognize the dialogue between characters in the novel and assign personalized speech characteristics to different characters, making the audiobook listening experience richer.

[0050] By encoding the input text through a context encoder, the model can simultaneously capture local temporal features and global temporal dependencies, ensuring a more natural rhythm and coherence in the generated speech. The BiLSTM structure is used to learn local dependencies, capturing the synergy between adjacent phonemes, while the multi-head self-attention mechanism effectively models long-range dependencies, ensuring sensitivity to global textual information during speech generation. This improves the encoding quality of text information in speech synthesis tasks, making subsequent speech waveform prediction more accurate and enhancing the naturalness and expressiveness of speech.

[0051] S20, initializing the historical waveform feature sequence to an empty sequence or a preset initial mark;

[0052] In this embodiment, the purpose of initializing the historical waveform feature sequence is to provide an initial state for the speech generation model, enabling it to begin generating speech waveforms without historical data. In speech synthesis tasks, each generated waveform point is influenced by previous historical waveform points. Therefore, a reasonable initial state is essential to ensure the stability of the generation process.

[0053] The historical waveform feature sequence is a structure that stores waveform features from past moments. It is primarily used to maintain the temporal continuity of speech, allowing the current waveform to be predicted based on past speech features. In practice, this sequence can be initialized in two ways:

[0054] Initializing with an empty sequence: When the historical waveform feature sequence is empty, the model relies solely on the contextual features of the input text to generate the first speech waveform point. This approach is typically used in scenarios where speech synthesis relies entirely on text information, such as text-to-speech (TTS) systems. In this case, the model requires strong global modeling capabilities to ensure the naturalness of the initial synthesized speech.

[0055] Initialization with preset initial markers: To provide stable initial conditions, preset marker values ​​can be added to the historical waveform feature sequence. These marker values ​​can be fixed initial waveform features, such as zero vectors, silent waveform features, or average waveform features statistically derived from training data. This approach can reduce model instability in the initial stages and avoid incoherence issues caused by a lack of historical information. For example, in an end-to-end speech synthesis system, standard silent segments can be used as initial waveform features to ensure that the generated audio does not exhibit sudden changes or unstable noise due to initialization issues.

[0056] In the specific implementation, the historical waveform feature sequence is usually stored as a data structure with increasing time steps. Each time a new waveform feature is generated, the latest feature is added to the sequence and the earliest feature is removed to keep the sequence length within a reasonable range.

[0057] In some implementations, the historical waveform feature sequence can be managed using a fixed-length buffer. This buffer stores past waveform features in a first-in, first-out (FIFO) manner, with the oldest waveform feature removed each time a new waveform feature is added. This approach effectively controls computational costs, ensuring that the system does not consume excessive memory or computing resources due to the accumulation of historical data during the speech synthesis process.

[0058] In another implementation, a probability-based initialization strategy can be employed. This involves using statistical learning methods to extract typical waveform features from existing training data and using these as the initial state. For example, the mean vector of silent waveform segments in the training data can be calculated and used to populate the initial historical waveform feature sequence, ensuring that the model is in a stable state at the start of inference. This approach is particularly important when generating highly stable speech, such as customer service or navigation speech.

[0059] In some special application scenarios, such as speech style transfer or personalized speech synthesis, the historical waveform feature sequence can be initialized based on the speech characteristics of the target speaker. For example, when generating speech from a specific speaker, initial waveform features can be extracted from the target speaker's recorded data to ensure that the generated speech has the personalized characteristics of that speaker from the beginning.

[0060] By providing two initialization methods, the historical waveform feature sequence can be adapted to different speech synthesis scenarios. For fully text-based speech synthesis tasks, empty sequence initialization provides a more flexible generation space, while pre-setting initial markers improves the stability of initial generation. This avoids the incoherent sound quality issues caused by improper initialization, making the initial stage of speech generation more natural, while also reducing computational overhead and improving inference efficiency.

[0061] S30, inputting the context features and the historical waveform feature sequence into a next distribution prediction module to predict and generate a probability distribution of the speech waveform at the next moment;

[0062] In this embodiment, contextual features and historical waveform feature sequences are used to predict the probability distribution of the speech waveform at the next moment. Contextual features are high-dimensional representations encoded from the input text, containing information about the global temporal dependencies of speech. Historical waveform feature sequences are time series data obtained by encoding the generated waveforms, providing a reference for prediction at the current moment.

[0063] After inputting contextual features and a sequence of historical waveform features into the distribution prediction module, the two pieces of information must be combined for calculation. Contextual features provide global information derived from the textual level, such as speech pitch, rate, and prosody, while the sequence of historical waveform features contains information about the generated waveform, which is used to maintain speech coherence. In the distribution prediction module, a multi-layer neural network is typically used to process the input features to predict the speech waveform distribution at the next moment.

[0064] The distribution prediction module usually includes multiple sub-modules, mainly involving time series feature enhancement, cross-modal alignment, mixed density modeling, etc.

[0065] Temporal feature enhancement: Because historical waveform feature sequences may contain information spanning long time periods, they require processing using temporal convolution or recurrent neural networks to extract key temporal features. By encoding historical waveform feature sequences, the model's ability to model long-range dependencies is enhanced, thereby improving the fluency of generated speech.

[0066] Cross-modal alignment: Contextual features and historical waveform feature sequences differ significantly in data distribution, necessitating cross-modal attention mechanisms for feature alignment. A multi-head self-attention mechanism is typically used to calculate similarity weights between contextual features and historical waveform features, and the two features are fused based on these weights. This ensures that predictions fully utilize contextual information while maintaining consistency with the style of the existing waveform.

[0067] Mixture Density Modeling: Speech waveforms are highly random, making it difficult to accurately simulate the diversity of human speech using a single deterministic prediction. To address this issue, a Mixture Density Network (MDN) can be used to model the probability distribution of speech waveforms. MDNs can predict the parameters of multiple Gaussian distributions, including the mean, variance, and mixing weights, thereby probabilistically generating more natural speech waveforms.

[0068] In some application scenarios, temporal feature enhancement can be achieved using a one-dimensional convolutional neural network (1D-CNN). By extracting features from historical waveform feature sequences using multiple convolution kernels with different receptive fields, the modeling capability of long-term dependencies can be enhanced. For example, in a speech synthesis task with a 16kHz sampling rate, a multi-layer convolutional structure with kernel sizes of 5, 10, and 20 can be used to capture short-, medium-, and long-term waveform variation information.

[0069] In another implementation, a bidirectional long short-term memory (BiLSTM) network can be used to enhance temporal features. BiLSTM can simultaneously model forward and backward temporal dependencies, and is better at capturing the collaborative relationships between phonemes than conventional LSTM or GRU structures.

[0070] Cross-modal alignment can be achieved using a Transformer-based attention mechanism. Attention weights are calculated between contextual features and historical waveform feature sequences, and weighted feature fusion is performed based on these weights. For example, in systems like FastSpeech2, a multi-head attention-based feature mapping layer is used to ensure that the synthesized speech waveform accurately aligns with the rhythmic information of the text input.

[0071] Mixture density modeling can be implemented using a multi-Gaussian mixture model (GMM). By predicting the parameters of the Gaussian mixture distribution at different time steps, more coherent and natural waveform data can be generated. In practice, the number of Gaussian kernels in the mixture distribution can be adjusted based on specific application requirements. For example, a speech dialogue system may only require 5-10 Gaussian distributions, while music synthesis tasks may require higher-dimensional modeling capabilities.

[0072] By combining contextual features with historical waveform feature sequences, the distribution prediction module improves the naturalness and coherence of generated speech while ensuring the accuracy of speech content. Temporal feature enhancement effectively models long-term dependencies, avoiding instability in the speech generation process. Cross-modal alignment ensures a match between text and waveforms, making the rhythm and prosody of synthesized speech more consistent with the laws of natural speech. Mixture density modeling increases the diversity of waveform generation, making speech more natural and reducing the stiffness of sound quality caused by single-source predictions.

[0073] S40, generating a speech waveform point at the next moment according to the probability distribution of the speech waveform at the next moment;

[0074] In this embodiment, the method for generating speech waveform points during speech synthesis plays a crucial role in the naturalness, coherence, and diversity of speech. Traditional methods typically use deterministic predictions, directly obtaining a fixed value from the model output. This lacks flexibility and results in a lack of randomness and natural expression in the generated speech. In contrast, waveform point generation through probability distribution can more accurately model the complex variations of speech signals, resulting in more natural and fluent speech.

[0075] The probability distribution of the speech waveform at the next moment refers to the statistical distribution of possible speech waveform points at the next moment, given the current speech state, as predicted by the prediction model. This distribution typically consists of a mean, standard deviation, and mixing weights. The mean reflects the expected value of the waveform point, the standard deviation measures the uncertainty of the prediction, and the mixing weights determine the contribution of different distribution components. Common probability distribution modeling methods include Gaussian mixture models and distribution prediction networks. These methods offer significant advantages in ensuring smoothness and diversity in speech synthesis.

[0076] Sampling random noise from a standard normal distribution introduces randomness, resulting in more diverse waveform points. Because speech signals are inherently random, waveforms generated solely through maximum likelihood estimation are often overly smooth and lack the detailed characteristics of real speech. By sampling noise from a standard normal distribution (mean zero, variance one) and calculating waveform points based on the distribution parameters, we can effectively improve the expressiveness of speech, making speech synthesis more realistic.

[0077] Element-wise multiplication of the standard deviation vector and subsequent addition to the mean vector is part of the reparameterization technique used to calculate the final waveform points. The standard deviation vector represents the dispersion of the predicted distribution, while the mean vector represents the central tendency of the model's predictions. This element-wise multiplication and addition ensures that each waveform point is adjusted based on its uncertainty, rather than being fixed along the mean prediction. This better models subtle variations in the speech signal and improves the diversity of the synthesized speech.

[0078] The residual convolutional network (RCN) is used to post-process the generated raw waveform points to remove noise and enhance speech features. Because the speech synthesis process may introduce unnecessary noise components, a RCNN using a residual structure effectively preserves key speech information and suppresses irrelevant noise. The multi-scale convolutional kernel structure extracts speech features at different scales, making the resulting speech clearer and more natural.

[0079] The mean and standard deviation of historical waveform features are used for amplitude adjustment to ensure continuity and volume consistency of the speech signal. During speech generation, waveform points at different time steps may have significant amplitude differences. Directly using unprocessed waveform points can cause volume fluctuations, affecting the listening experience. Normalization smoothes volume variations between time steps, ensuring speech stability and audibility.

[0080] Generating waveform points using a probability distribution effectively improves the naturalness of speech synthesis, making the generated speech more similar to the random characteristics of real speech. The use of reparameterization techniques ensures the stability of the training process, avoiding training difficulties caused by non-differentiable operations introduced by sampling. Furthermore, combined with the denoising processing of the residual convolutional network, it can reduce background noise while preserving the clarity of the speech signal, ensuring speech quality. Amplitude normalization improves the volume stability of speech synthesis, making the final speech sound more fluent and smooth.

[0081] S50, encoding the speech waveform point at the next moment into a new waveform feature, and updating the new waveform feature into the historical waveform feature sequence;

[0082] In this embodiment, encoding the next speech waveform point into a new waveform feature and updating the new waveform feature into the historical waveform feature sequence is a key step in ensuring the fluency of generated speech, information integrity, and modeling stability in the speech synthesis system. This process primarily involves feature extraction of the speech signal, storage and management of historical information, and a mechanism for updating time series data. The technical characteristics and implementation methods of each step have a significant impact on the naturalness and computational efficiency of speech synthesis.

[0083] Speech waveforms are continuous time-series signals. Each waveform point at each moment only contains instantaneous amplitude information, which cannot effectively capture context-dependent features. Therefore, a feature encoder is needed to convert them into high-dimensional feature vectors to more comprehensively describe the speech's spectral structure, phoneme information, and intonation.

[0084] Feature encoders typically use one-dimensional convolutional neural networks (1D-CNN) for feature extraction. Unlike traditional Fourier transforms (FFTs) or Mel-spectrum analysis, one-dimensional convolution can directly model in the time domain and extract short-term sequence patterns through sliding convolution kernels, enhancing the model's perception of waveform changes. The main functions of the convolution layer are: to obtain temporal patterns of different scales through convolution kernels of different sizes (such as 3, 5, and 7), so that both short-term and long-term dependencies can be effectively captured; to increase the receptive field through dilated convolution, allowing the convolution layer to capture historical information from farther away without excessively increasing computational complexity; and to extract features from different frequency ranges through multi-channel convolution to avoid information loss.

[0085] During convolution extraction, nonlinear activation functions (such as ReLU, LeakyReLU, and Swish) are used to enhance the model's expressiveness, imbuing feature transformations with nonlinear mapping capabilities. The ReLU activation function effectively suppresses background noise and improves speech signal quality, while the Swish activation function, due to its self-gating properties, performs better in continuous signal processing.

[0086] The feature vectors output by convolutional networks are typically high-dimensional, for example, mapping a 1-dimensional waveform point to a 64- or 128-dimensional feature space. These high-dimensional features not only contain instantaneous information but also embed the relative position of the waveform point in the entire time series, thereby improving the model's ability to perceive global patterns.

[0087] Speech synthesis is a dynamic prediction process. New waveform points at each time step need to be stored so that subsequent generation calculations can be optimized using complete historical information. Because speech signals have long-term dependencies, efficiently storing and updating historical information is a core issue.

[0088] When updating the historical waveform feature sequence, it is necessary to ensure the temporal integrity of the data while avoiding the computational burden brought by redundant storage. To achieve this, a first-in-first-out storage structure can be used so that new waveform features are always appended to the end of the historical sequence in chronological order, thereby maintaining data consistency. At the same time, in order to prevent the unlimited growth of historical data and the increase in computational complexity, a sliding window mechanism can be used to limit the length of the historical sequence. The size of the sliding window can be set according to a fixed time length, for example, at a 16kHz sampling rate, 1 second corresponds to 16,000 frames, or it can be limited based on a fixed number of frames, such as a maximum storage of 1024 frames. When the length of the historical waveform feature sequence exceeds the set threshold, the earliest waveform feature is automatically deleted to ensure that new data can be stored normally and to reduce the storage resource occupation of useless data.

[0089] Because speech signal amplitudes can vary significantly at different time steps, directly using unnormalized data can affect model stability. Therefore, when updating the historical waveform feature sequence, feature normalization is necessary to stabilize the data distribution. Layer normalization can be used to maintain overall stability of features at each time step, thereby reducing the impact of data changes on the model. Alternatively, batch normalization can be used to standardize features across samples in small-batch inference tasks, improving model stability and generalization.

[0090] The storage method of historical waveform features will also affect computational efficiency. To reduce storage overhead and improve access efficiency, sparse storage can be used to store only the features of key points and ignore the data of redundant points, thereby reducing data storage requirements. At the same time, a quantized storage method can be used to convert high-precision floating-point data into low-precision data types, such as converting 32-bit floating-point numbers into 8-bit integers, to reduce the computational burden brought by storage. In addition, for long sequence data, time step sampling can be used for storage, such as storing historical waveform features at a sampling rate of 1 / 2, or using a variable step size storage method so that the data storage method can adapt to different computational requirements.

[0091] In different application scenarios, the update strategy for historical waveform features needs to be adjusted to accommodate varying computational requirements. In real-time speech synthesis tasks, whenever a new waveform feature is generated, the historical sequence is immediately updated to ensure that the prediction for the next moment is optimized based on the latest information. In batch speech generation applications, a batch of waveform points can be generated first, and then the historical sequence is updated all at once. This reduces the computational overhead associated with frequent updates and improves overall computational efficiency. In certain computationally demanding tasks, skipped storage can be used, for example, storing data every three time steps. This reduces data storage while maintaining long-term dependencies, thereby improving the computational efficiency and stability of the model.

[0092] By transforming speech waveform points using a feature encoder, high-dimensional features can be effectively extracted, making the representation of speech information more compact and enhancing noise suppression. By updating the historical waveform feature sequence, speech prediction at the next moment can utilize complete historical information, thereby improving the coherence of speech synthesis. Storing historical information using a time-series splicing method ensures data integrity, while a sliding window update mechanism avoids storing redundant information and improves computational efficiency.

[0093] S60, repeatedly executing the probability distribution prediction of the next distribution prediction module, the speech waveform point generation at the next moment, and the historical waveform feature sequence update steps until the preset termination condition is met, stopping the repeated execution, and combining the generated speech waveform points into the target speech waveform.

[0094] In this embodiment, during the speech generation process, it is necessary to continuously perform distribution prediction, speech waveform point generation, and updating of the historical waveform feature sequence to ensure that the generated speech signal has coherence and temporal consistency. This process is a cyclical execution process, and each iteration depends on the prediction result of the previous moment to ensure that the speech signal can gradually approach the natural fluency of real speech. At each time step, the historical waveform features are analyzed by the distribution prediction module, and the probability distribution of the speech waveform at the next moment is calculated in combination with the context features. Then, the speech waveform points at the current moment are generated by probability distribution sampling, so that the generated waveform has a certain diversity, rather than a single deterministic value. The generated waveform points will be encoded as new waveform features and spliced ​​into the historical waveform feature sequence as a reference for subsequent predictions.

[0095] In order to ensure that the speech generation process can be terminated correctly, a preset termination condition needs to be set. When the termination condition is met, the iterative calculation is stopped, and all the speech waveform points that have been generated are combined into a complete speech signal. Common termination conditions can be limited according to the number of speech waveform points that have been generated. For example, when the number of frames of the historical waveform feature sequence reaches the length of the target speech, the generation is stopped. In addition, energy threshold detection can be used. When the short-term energy of the speech signal is lower than the preset silence threshold, it is determined that the speech generation process is nearing its end, and the generation is stopped to avoid unnecessary computational overhead. Another way is to use text information for alignment to ensure that the length of the generated speech matches the timing information of the input text, preventing the speech signal from being too long or too short and affecting the speech quality.

[0096] Once the generation of speech waveform points reaches the termination condition, the generated waveform data requires further processing to improve the quality of the output speech. The generated waveform points are spliced ​​together in chronological order to form a complete speech waveform, and then post-processed, such as resampling, to ensure that the final output waveform matches the target sampling rate. Furthermore, amplitude normalization can be used to ensure that the amplitude of the speech signal is within a reasonable range, improving the loudness consistency of the synthesized speech. In some cases, denoising filtering can be used to remove possible synthesis artifacts, thereby improving the intelligibility and clarity of the speech. Ultimately, after this series of processing, the generated speech waveform can be directly played in the target application or further used in downstream speech synthesis tasks.

[0097] Example: In healthcare, speech synthesis technology can be used to assist patients with speech disorders with rehabilitation training, improve the interactive capabilities of medical assistive devices, and provide personalized speech synthesis services for patients who cannot speak. For patients with aphasia or language development disorders, a dedicated speech synthesis system can be built based on this method, enabling them to generate natural, personalized speech by inputting text or selecting preset phrases, helping them communicate with medical staff and family members. In this application scenario, the system first analyzes the patient's speech sample to extract key speech features. It then uses a context encoder to encode the input text, generating contextual features that incorporate global temporal dependencies. Subsequently, when the historical waveform feature sequence is empty, a preset initial marker or the patient's speech features is used as initialization data, enabling speech synthesis to be generated based on personalized speech. The contextual features and the historical waveform feature sequence are then fed into the next distribution prediction module to calculate the probability distribution of the speech waveform at the next moment, ensuring that the generated speech matches the patient's timbre. By sampling this probability distribution, the speech waveform points at the next moment are generated, and denoising and amplitude adjustment techniques are used to ensure the stability and clarity of the speech signal. The generated speech waveform points are further encoded into new waveform features and updated to the historical waveform feature sequence, ensuring that subsequent speech generation remains natural and fluent. This process continues until the complete speech output is generated, ultimately forming a continuous, personalized speech waveform, ensuring that the patient's pronunciation conforms to their original speech characteristics and improving the effectiveness of rehabilitation training.

[0098] In the FinTech sector, speech synthesis is widely used in scenarios such as intelligent customer service, intelligent voice broadcasting, and personalized speech synthesis. For example, in banking customer service systems, it can be used to synthesize professional and approachable intelligent voice assistants, providing users with a more natural experience during phone interactions. When users inquire about financial products, the system extracts semantic information from the text or voice commands entered by the customer using a context encoder, and optimizes the professionalism by incorporating financial industry-specific terminology. When generating speech, historical waveform features can be initialized using standard industry customer service speech data, making the synthesized speech more professional in intonation and emotional expression. During the speech generation process, the speech waveform point at each moment is predicted based on contextual and historical waveform features to ensure speech continuity and fluency. Furthermore, by modeling the probability distribution of speech using a mixture density network, the system generates more natural speech output that is more consistent with the characteristics of the financial industry. Furthermore, the introduction of a generative adversarial network can enhance the realism of speech, giving customers a more humanistic experience when interacting with the voice assistant. This approach not only improves the quality of voice customer service but also increases the level of automation in financial services.

[0099] In online education and intelligent learning systems, this method can be used to build personalized intelligent voice explanation assistants. For example, in English oral learning applications, users can enter text or select specific sentences. The system analyzes the text content using a context encoder and determines a phoneme segmentation strategy based on the target speech category to ensure accurate pronunciation. During speech synthesis, historical waveform features can be initialized using standard pronunciation samples or adjusted based on the user's learning progress, allowing the generated speech to adapt to the user's listening training needs. As speech waveform points are gradually generated, the system can provide speech with different speaking speeds and intonations in different contexts, making it easier for learners to adapt to different accents and expressions. Furthermore, in programming courses, this method can be used to automatically generate code explanations, allowing learners to understand complex programming concepts through high-quality voice explanations. By controlling the fluency, pitch, and pauses of the speech, the system can optimize the learner's listening experience and improve the learning effect of online courses.

[0100] By continuously executing autoregressive probability distribution predictions, speech waveform point generation, and historical waveform feature sequence updates, the speech synthesis process dynamically adapts to the context of the input text, achieving highly natural speech generation. By properly setting termination conditions, the length and fluency of the speech can be effectively controlled, preventing overly long or short speech segments from affecting the synthesis effect.

[0101] The present invention relates to the field of speech synthesis technology and can be applied to business system platforms such as communications, medical health, and financial technology. A speech generation method based on distribution prediction is disclosed, comprising: encoding input text, generating context features containing global temporal dependencies, initializing a historical waveform feature sequence to an empty sequence or a preset initial marker, inputting the context features and the historical waveform feature sequence into a next distribution prediction module, predicting and generating a probability distribution of the speech waveform at the next moment, generating speech waveform points at the next moment based on the predicted probability distribution, encoding the generated speech waveform points into new waveform features and updating them into the historical waveform feature sequence, repeatedly performing probability distribution prediction, speech waveform point generation, and historical waveform feature sequence update, stopping execution when a termination condition is met, and combining the generated speech waveform points into a target speech waveform. The present invention obtains context features containing global temporal dependencies through a context encoder and performs autoregressive distribution prediction in combination with the historical waveform feature sequence, so that long-range dependencies can be effectively modeled during speech synthesis, improving the coherence and naturalness of speech. At the same time, the speech waveform points at the next moment are generated through probability distribution prediction and continuously updated based on the historical waveform feature sequence, so that speech generation not only considers short-term information but also maintains global consistency. In addition, the termination condition is used to control the generation process, improve the inference efficiency, reduce the computational overhead, and meet the needs of real-time speech synthesis.

[0102] In one embodiment, the above step S10 includes:

[0103] S101, determining a phoneme division strategy according to a target speech category, and dividing an input text into phoneme sequences based on the phoneme division strategy;

[0104] S102, mapping each phoneme in the phoneme sequence into a high-dimensional vector through an embedding layer to generate a phoneme vector sequence;

[0105] S103, inputting the phoneme vector sequence into a bidirectional long short-term memory network, and extracting local temporal features of the phoneme vector sequence through the bidirectional long short-term memory network;

[0106] S104, inputting the local temporal features into a multi-head self-attention mechanism layer, determining the attention weight between any two time steps in the phoneme sequence, and performing weighted aggregation on the local temporal features based on the attention weight to generate a global temporal dependency relationship;

[0107] S105 , combining the local temporal features and the global temporal dependency to generate the contextual features.

[0108] In this embodiment, encoding the input text to extract contextual features of global temporal dependencies is a key step in ensuring the natural and fluent nature of the generated speech. This process involves a complete information processing flow, from text parsing to high-dimensional feature extraction and then to temporal modeling, enabling the speech synthesis system to accurately capture language structure and pronunciation patterns.

[0109] The first step in text encoding is to determine the phoneme segmentation strategy, which determines how to convert the text into a phoneme sequence. The target phonetic category influences the method of phoneme segmentation. For example, in Chinese, phonemes typically consist of initials, finals, and tones, while in English, phonemes are directly composed of phonemes. The phoneme segmentation strategy can be based on a pretrained pronunciation dictionary or employ an automatic phoneme segmentation method based on statistical learning.

[0110] The conversion of phoneme sequences relies on an embedding layer, which maps discrete phonemes into continuous, high-dimensional vector representations. This mapping captures the similarities between phonemes, enabling the model to better understand the phonetic relationships between them. The embedding layer is typically implemented using a trainable lookup table and can also be initialized with pre-trained speech data to improve model generalization.

[0111] A high-dimensional vector is a vector that represents data in a high-dimensional numerical space. In speech synthesis tasks, high-dimensional vectors are used to represent the characteristics of language units such as phonemes, words, and sentences. Their dimensions are often greater than, or much greater than, the dimensions of the original input data. For example, a phoneme itself may be a discrete character or token, but after being transformed by the embedding layer, it becomes a continuous numerical vector with dozens or even hundreds of dimensions.

[0112] In deep learning, discrete inputs (such as phonemes and words in text) cannot be fed directly into a neural network. Instead, they must be converted into computable numerical representations. An embedding layer is typically used to map phonemes into a high-dimensional vector space. For example, a phoneme "a" can be mapped to a 256-dimensional vector. This mapping is typically implemented using a lookup table, whose parameters are continuously optimized during training to bring similar phonemes closer together in the vector space, thereby improving the model's learning ability.

[0113] Low-dimensional vector representations may not fully capture the relationship between phonemes, while high-dimensional representations can store more phonetic information. For example, a 64-dimensional vector may not be sufficient to distinguish complex pronunciation features, while a 256-dimensional or 512-dimensional vector can contain more phoneme co-occurrence patterns, pronunciation methods, contextual information, and so on. Each dimension of a high-dimensional vector can represent different features. For example, one dimension may represent the co-occurrence probability of a phoneme, while another dimension may represent the tonal characteristics of a phoneme. Such a distributed representation is more expressive than traditional one-hot encoding and can capture the similarities between phonemes.

[0114] Local temporal features are extracted using a bidirectional long short-term memory (BiLSTM) network. This bidirectional structure ensures that the model leverages both contextual information and overcomes the limitations of unidirectional LSTM in capturing contextual dependencies. LSTM effectively models both short-term and long-term dependencies through a gating mechanism and avoids the vanishing gradient problem when processing speech sequences.

[0115] Modeling global temporal dependencies relies on a multi-head self-attention mechanism, which captures long-range dependencies by calculating attention weights between different time steps in a phoneme sequence. The self-attention mechanism identifies key phonemes in speech and the correlations between them, enabling the model to more accurately generate pronunciation patterns that align with speech logic. A weighted aggregation approach ensures that the attention mechanism not only focuses on local phonemes but also comprehensively considers the pronunciation patterns of the entire sentence.

[0116] Finally, by combining local temporal features with global temporal dependencies, complete contextual features are generated. This fusion process ensures that the model possesses both short-term information and long-term dependencies, ensuring both local pronunciation accuracy and global natural fluency during speech synthesis.

[0117] This embodiment uses a multi-head self-attention mechanism to enhance the system's ability to perceive different types of speech features, making synthesized speech more expressive in different contexts. Furthermore, the combination of BiLSTM to extract local temporal features enables more precise processing of speech details, especially in the generation of pronunciation features such as high-frequency consonants, connected speech, and stress.

[0118] In one embodiment, the above step S30 includes:

[0119] S301, inputting the context feature and the historical waveform feature sequence into a next distribution prediction module;

[0120] S302, performing time convolution coding processing on the historical waveform feature sequence through the time series feature enhancement layer of the next distribution prediction module to generate time series enhancement features;

[0121] S303, performing multi-head attention interaction processing on the contextual features and the temporal enhancement features through the cross-modal alignment layer of the next distribution prediction module to generate a fused alignment feature;

[0122] S304, determining the mean vector, standard deviation vector, and mixing weight parameters of a multi-Gaussian mixture distribution according to the fused alignment features through the mixture density network of the next distribution prediction module;

[0123] S305, performing scaling processing on the blending weight parameter based on a preset temperature coefficient through the distribution sharpening layer of the next distribution prediction module to generate a sharpened blending weight;

[0124] S306, constructing a mixed Gaussian distribution according to the mean vector, standard deviation vector and sharpened mixed weights through the distribution construction layer of the next distribution prediction module, and using the mixed Gaussian distribution as the probability distribution of the speech waveform at the next moment.

[0125] In this embodiment, in the speech generation task, the input contextual features and the historical waveform feature sequence jointly determine the probability distribution of the speech waveform at the next moment. To achieve this goal, the next distribution prediction module performs multi-level processing on the input features to capture short-term dynamic characteristics and long-term temporal dependencies, ultimately modeling a probability distribution that conforms to the laws of natural speech.

[0126] Contextual features and historical waveform feature sequences are two core inputs in the speech generation process. Contextual features are used to model the linguistic structure and prosody of speech, while historical waveform feature sequences provide temporal correlation information, ensuring that the generated waveform maintains speech coherence and naturalness. The fusion of these two features enables the prediction module to effectively simulate the dynamic changes of natural speech.

[0127] The temporal feature enhancement layer processes the historical waveform feature sequence through temporal convolutional coding to extract its temporal features. Temporal convolutional networks (TCNs) can capture patterns between different time steps and, through the action of convolutional kernels, aggregate local temporal features. Compared to RNNs, temporal convolutions offer parallel computing capabilities, higher computational efficiency, and the ability to learn long-term dependencies through a hierarchical structure.

[0128] The cross-modal alignment layer uses multi-head attention to interactively process contextual features and temporal enhancement features. This ensures that speech waveform generation not only relies on historical information but also fully incorporates the semantic features of the current speech content. This multi-head attention mechanism focuses on the relationships between features from different perspectives, making speech waveform generation more accurate and avoiding information loss or the generation of incoherent speech fragments.

[0129] The mixture density network determines the probability distribution of the speech waveform at the next moment. It calculates the mean vector, standard deviation vector, and mixing weight parameters of a multi-Gaussian mixture distribution based on the fused alignment features. The Gaussian mixture model (GMM) flexibly fits complex probability distributions and effectively models different waveform patterns in speech modeling, enabling synthesized speech to more realistically reflect the variations in natural speech.

[0130] The distribution sharpening layer scales the blending weight parameters to adjust the kurtosis of the distribution, improving the model's ability to smoothly transition between different phonemes. The temperature coefficient setting determines the model's sensitivity to the probability distribution. Higher temperature values ​​result in a more even probability distribution, while lower temperature values ​​result in a sharper probability distribution, helping to enhance the clarity of certain phonemes.

[0131] The distribution construction layer ultimately constructs an adjustable mixed Gaussian distribution based on the mean vector, standard deviation vector, and sharpened mixing weights. This probability distribution is used to sample the speech waveform at the next moment, ensuring that the generated waveform conforms to the statistical laws of natural speech, improving speech authenticity and intelligibility.

[0132] Through the above steps, this embodiment can effectively improve the naturalness of the speech synthesis system, so that the generated waveform can more accurately match the timbre, rhythm and sound quality of the target speech. The multi-head attention mechanism combined with the design of the temporal enhancement layer enables the model to fully utilize the historical waveform features and solve the problem that the traditional autoregressive method is insufficient in modeling long-term dependencies. The introduction of the mixed density network makes the generated speech waveform more diverse, while the design of the distribution sharpening layer improves the stability of speech generation and avoids the waveform being too smooth or distorted. This overall design improves the accuracy of speech synthesis and reduces problems such as sound quality distortion and intermittence that may occur during the generation process.

[0133] In one embodiment, the above step S40 includes:

[0134] S401, sampling random noise from a standard normal distribution, and multiplying the random noise element-by-element by the standard deviation vector of the probability distribution of the speech waveform at the next moment, and then adding the result to the mean vector of the probability distribution to generate original waveform points;

[0135] S402: Inputting the original waveform points into a residual convolutional network, suppressing non-speech band noise in the original waveform points and retaining speech band features through a multi-scale convolution kernel in the residual convolutional network, thereby generating denoised waveform points;

[0136] S403: performing amplitude adjustment processing on the denoised waveform points according to the mean and standard deviation of the historical waveform feature sequence to generate the speech waveform points at the next moment.

[0137] In this embodiment, during the gradual generation of speech waveforms, sampling based on a probability distribution effectively ensures waveform diversity. Combined with steps like denoising and amplitude adjustment, the resulting synthesized waveform more closely resembles the sound quality and timing characteristics of natural speech. This process, involving key steps such as random sampling, noise suppression, and amplitude normalization, ensures the stability and naturalness of speech generation.

[0138] Sampling random noise from a standard normal distribution is a key step in introducing randomness into the generation process. The standard normal distribution is a symmetric distribution with mean zero and variance one, and is widely used in neural network training and generation tasks to model data perturbations. Sampling random noise prevents the speech synthesis process from falling into a deterministic mode and improves the dynamics and naturalness of the speech waveform. During the generation process, the random noise is element-wise multiplied by the standard deviation vector of the probability distribution and then added to the mean vector to form the original waveform points. This operation, based on a reparameterization technique, ensures that the generated waveform matches the predicted distribution while maintaining parameter optimizability during training.

[0139] The original waveform points may contain noise that is not part of speech, so a residual convolutional network is needed for denoising. The residual convolutional network uses skip connections between layers to improve the convergence and stability of the model while preserving key information. The multi-scale convolution kernel is the core of the network, which allows the model to capture features at different scales, enabling the model to adapt to different frequency components of speech. For example, smaller convolution kernels can extract short-term local features, while larger convolution kernels can capture long-term dependency information. In this way, the residual convolutional network can effectively suppress non-speech band noise while retaining the key features of the speech band, improving the clarity of the generated speech.

[0140] In order to ensure that the amplitude range of synthesized speech is consistent with natural speech, the denoised waveform points need to be amplitude adjusted. The mean and standard deviation of the historical waveform feature sequence provide an adaptive amplitude adjustment reference, so that the speech waveform points generated at the current moment will not have excessively large or small amplitude deviations. Common methods for amplitude adjustment include Min-Max Normalization and Z-score normalization. The former prevents numerical overflow by adjusting the data range to a specific interval (such as -1 to 1), while the latter uses standardization operations (subtracting the mean and dividing by the standard deviation) to match the statistical characteristics of the generated waveform with the historical waveform characteristics, ensuring timing consistency.

[0141] This embodiment introduces a moderate amount of randomness through random sampling from a standard normal distribution, improving the naturalness of speech synthesis. This prevents the synthesized speech from exhibiting a completely fixed pattern and avoids the perceived stiffness of the synthesized sound. Combined with the denoising capabilities of the residual convolutional network, it effectively reduces non-speech frequency band noise, improves speech clarity, and makes the synthesized speech more natural and fluent. Furthermore, through amplitude adjustment processing, the generated waveform can match the characteristics of historical waveforms, ensuring the coherence and volume consistency of speech generation, thereby improving the quality of speech synthesis.

[0142] In one embodiment, the above step S50 includes:

[0143] S501: Input the speech waveform point at the next moment into a waveform feature encoder, generate a high-dimensional acoustic feature vector through a one-dimensional convolution layer and a nonlinear activation function, and use the high-dimensional acoustic feature vector as a new waveform feature;

[0144] S502, splicing the new waveform feature to the end of the historical waveform feature sequence in time sequence to form an updated historical waveform feature sequence;

[0145] S503: If the length of the updated historical waveform feature sequence exceeds a preset maximum sequence length, the waveform feature at the earliest moment in the historical waveform feature sequence is removed.

[0146] In this embodiment, the key goal of updating the historical waveform feature sequence during speech waveform generation is to ensure that the model can continuously utilize the latest speech information while limiting computational overhead and preventing the storage of redundant data. This process involves waveform point encoding, feature concatenation, and sequence length control.

[0147] Speech waveform point coding converts low-dimensional raw waveform points into high-dimensional acoustic features so that subsequent prediction models can use richer temporal information for modeling. The waveform feature encoder plays a core role in this process and usually adopts a one-dimensional convolution (1D-CNN) structure to capture local temporal dependencies. Compared with directly using raw waveform points, the convolution layer can extract local patterns, improve the generalization ability of the model, and reduce noise interference. The role of the nonlinear activation function is to introduce nonlinear mapping to make the neural network more expressive. Common activation functions include ReLU, Leaky ReLU, and Swish. The choice of each activation function will affect the dynamic range of the final encoded feature.

[0148] High-dimensional acoustic feature vectors are generated to more effectively represent the frequency, energy, and time-varying characteristics of a waveform. Typically, these vectors have a much higher dimensionality than the input waveform points, providing richer information. For example, a single waveform point sampled at 16kHz might contain only a single value, while a high-dimensional acoustic feature vector might contain multiple channels, each corresponding to a characteristic in a different frequency or time window.

[0149] Updating the historical waveform feature sequence ensures that past waveform information can always be used for modeling during speech synthesis. Using a time-series concatenation approach, the latest waveform features are added to the end of the historical sequence, allowing the sequence to continuously retain the context of the speech. This concatenation approach is typically based on a FIFO (first-in, first-out) storage structure, ensuring that data is organized in chronological order, allowing the next prediction process to utilize the latest time-series information.

[0150] When the length of a historical waveform feature sequence exceeds a set maximum threshold, the sequence needs to be truncated to ensure that computational and storage overhead does not grow indefinitely. Setting the maximum sequence length can be based on various criteria, such as a fixed time window (e.g., 1 second of waveform data) or a fixed number of frames (e.g., 1024 frames). Deleting the earliest waveform features is a common strategy to ensure that the data in the sequence is always up-to-date while preventing information redundancy.

[0151] This embodiment converts speech waveform points through a waveform feature encoder, improving the model's ability to model the temporal sequence of audio data, enabling subsequent prediction models to utilize richer features for calculations. Time-series splicing is used to update historical waveform feature sequences, allowing the model to continuously reference past waveform data, thereby improving the coherence of speech synthesis. Furthermore, by limiting the length of historical waveform feature sequences, unlimited storage overhead is avoided, computational efficiency is improved, and the model can run efficiently in different computing environments.

[0152] In one embodiment, the above step S60 includes:

[0153] S601, repeatedly executing the steps of probability distribution prediction of the next distribution prediction module, generating speech waveform points at the next moment, and updating the historical waveform feature sequence;

[0154] S602, after each generation of the speech waveform point at the next moment, detecting whether the generated speech waveform point sequence is consistent with the time-series alignment length of the input text, or detecting whether the short-term energy of the generated speech waveform point sequence is lower than a preset silence threshold, or detecting whether the length of the historical waveform feature sequence reaches the total number of frames after frame rate conversion of the target speech duration;

[0155] S603, if any of the following conditions is met: the length of the generated speech waveform point sequence is consistent with the time-series alignment length of the input text; the short-term energy of the generated speech waveform point sequence is lower than a preset silence threshold; or the length of the historical waveform feature sequence reaches the total number of frames after the target speech duration is converted to a frame rate, then the repeated execution is terminated;

[0156] S604, splicing the speech waveform points generated at the next moment each time in the order of generation time to form an uncompressed time domain waveform sequence;

[0157] S605 , resampling and amplitude normalization processing are performed on the time domain waveform sequence to generate a final target speech waveform.

[0158] In this embodiment, the cyclical execution of prediction and synthesis steps during speech waveform generation is crucial for ensuring the integrity and fluency of speech synthesis. By continuously predicting the next speech waveform point and dynamically updating the historical waveform feature sequence, the generated speech is gradually refined. Setting the termination condition is crucial for determining whether speech waveform synthesis is complete, while post-processing of the generated waveform further improves speech quality, making it adaptable to different application scenarios.

[0159] The system continuously performs probability distribution prediction, speech waveform point generation, and historical waveform feature sequence updates to ensure speech integrity. This process builds an autoregressive generative model, in which the speech waveform point at each moment depends on the historical waveform features at the previous moment and the probability distribution prediction at the current moment. By continuously repeating this cycle, the system ensures that the generated speech waveform has sufficient temporal consistency and content integrity.

[0160] After each new speech waveform point is generated, the system checks whether the currently generated speech meets the termination criteria to ensure that the synthesis process ends at the appropriate time, improving computational efficiency. The termination criteria can be set based on various criteria, including alignment with the temporal length of the input text, meeting the silence threshold, and reaching the total number of target speech frames. These criteria are tailored to actual speech synthesis needs and can adapt to different application scenarios. For example, in a text-to-speech system, the termination criteria primarily depends on the length of the input text, while in streaming speech synthesis, the silence threshold may be a more important factor.

[0161] All generated speech waveform points are spliced ​​in chronological order to form an uncompressed time domain waveform sequence. This process ensures the temporal continuity of the synthesized speech, allowing the final output speech waveform to be seamless without breakpoints or mutations.

[0162] The final time-domain waveform sequence is resampled and amplitude normalized to conform to specific audio standards. Resampling can be used to adjust the sampling rate of speech signals to suit different playback devices or speech applications. For example, a 48kHz sampling rate can be converted to 16kHz to accommodate online speech processing systems. Amplitude normalization eliminates volume differences between different speech segments, making speech sound more natural and improving the user experience.

[0163] This embodiment ensures the stability of the speech synthesis process by continuously executing probability distribution prediction, speech waveform point generation, and historical waveform feature sequence updates, ensuring that the generated speech waveform accurately reflects the content of the input text. The adoption of reasonable termination conditions improves the accuracy and efficiency of speech generation, avoids unnecessary computational overhead, and ensures the integrity of the speech output. The generated waveform data is spliced ​​in time sequence to ensure temporal coherence of the speech and avoid sudden changes and interruptions in the speech signal.

[0164] In one embodiment, before step S30, the method further includes:

[0165] S2001, extracting training text and corresponding real speech waveform from the training data set;

[0166] S2002, generating a training global context feature of the training text by a context encoder, and generating a training history waveform feature sequence based on the real speech waveform;

[0167] S2003, inputting the training global context feature sequence and the training historical waveform feature sequence into a next distribution prediction module to generate a training probability distribution of the speech waveform at the next moment;

[0168] S2004, sampling and generating training speech waveform points at the next moment from the training probability distribution, and accumulating the generated training speech waveform points until a complete training speech segment is formed;

[0169] S2005, inputting the complete training speech segment into the discriminator of the generative adversarial network to obtain a naturalness score of the training generated speech;

[0170] S2006, inputting the real speech waveform into the discriminator to obtain a naturalness score of the real speech waveform;

[0171] S2007, determining a generative adversarial loss based on a difference between the naturalness score of the training generated speech and the naturalness score of the real speech waveform;

[0172] S2008: Update the parameters of the next distribution prediction module based on the generative adversarial loss and contrastive loss function.

[0173] In this embodiment, during speech synthesis training, to improve the naturalness and stability of the generated speech, a large amount of training data is required for model optimization. This process involves not only preprocessing the text and speech data, but also learning probability distributions based on the training data, adversarial training optimization, and updating model parameters. These steps work together to enable the model to generate high-quality speech waveforms.

[0174] Extracting training data is the starting point for optimizing speech synthesis models. This typically involves obtaining training text and its corresponding real speech waveforms from large-scale speech databases. The quality of training data directly impacts the model's learning performance. Therefore, during data extraction, it is important to ensure that the data covers diverse speech styles, speaker characteristics, and various contextual variations. Data cleaning and alignment are also key steps, such as removing background noise, standardizing speech length, and ensuring accurate alignment between text and speech.

[0175] The contextual information of the text is modeled using a context encoder to extract the semantic features and temporal dependencies of the text in speech expression. The process of generating training global context features is similar to the inference stage, but the input here is the training text rather than the predicted text during the synthesis process. After generating the training global context features, it is necessary to construct a training history waveform feature sequence based on the actual speech waveform. This step ensures the accuracy of the historical information during the training process, thereby improving the reliability of predicting the next speech waveform point.

[0176] The contextual features and historical waveform feature sequences are fed into the next distribution prediction module to generate a training probability distribution for the speech waveform at the next moment. This process learns the statistical properties of real speech and, based on these properties, constructs a parameterized probability model, such as a Gaussian mixture model or other variational distribution. The difference between the training phase and the inference phase is that the input in the training phase includes not only the generated features but also real speech data, which guides the model for more precise parameter adjustments.

[0177] The training speech waveform point for the next moment is sampled from the training probability distribution and accumulated until a complete training speech segment is formed. Because real speech signals are continuous, the generated training data must also conform to this property. This accumulation process ensures that the generated data is long enough for subsequent evaluation of the generated speech quality.

[0178] The entire training speech segment is fed into the generative adversarial network's discriminator, which evaluates the naturalness of the generated speech. The discriminator is typically a binary classification neural network, whose inputs include synthesized speech and real speech, and whose goal is to distinguish between the two. During training, the model continuously adjusts the generator to make the generated speech as close to real speech as possible, thereby improving the naturalness score.

[0179] The real speech waveform is also fed into the discriminator to obtain a naturalness score for the real speech. The difference between the two scores is used to calculate the generative adversarial loss, a loss function that measures the similarity between the generated speech and the real speech and is used to optimize the model, making the generated speech closer to the real speech. Furthermore, to enhance training stability, a contrastive loss function is incorporated. This loss function aligns the distribution of the generated and real speech in feature space, ensuring that the model does not experience mode collapse during the generation process.

[0180] Based on the generative adversarial loss and contrastive loss functions, the parameters of the next distribution prediction module are updated. Parameter updates are typically performed using backpropagation and gradient descent, such as Adam or SGD. The learning rate can be dynamically adjusted during training to ensure stable model convergence and prevent overfitting.

[0181] In different training scenarios, optimizations can be made to the training data sampling method, discriminator structure, loss calculation method, and other aspects. For large datasets, batch training can be used, inputting multiple speech segments at a time to improve training efficiency. For low-resource speech synthesis tasks, data augmentation techniques, such as pitch shifting and speech rate adjustment, can be used to expand the dataset and improve the model's generalization capabilities.

[0182] The choice of discriminator can also affect the quality of speech synthesis. Common discriminators include convolutional neural networks (CNNs) and variational autoencoders (VAEs). CNNs are suitable for distinguishing short-term speech features, while VAEs can be used to learn the feature space mapping between generated and real speech. For high-quality speech synthesis tasks, a cascaded discriminator can be used—one that utilizes both low-frequency and high-frequency features to improve speech clarity and naturalness.

[0183] The loss calculation method can be further optimized to improve training stability. For example, a perceptual loss function can be used to optimize the model based not only on numerical error but also on human auditory perception when optimizing the generated speech. Furthermore, a speech style transfer module can be added to make the synthesized speech more consistent with the characteristics of a specific speaker, enhancing personalization.

[0184] This embodiment extracts text and speech from training data and models it with real speech waveforms, ensuring that the speech synthesis model can learn the temporal dependencies of speech, improving speech intelligibility and clarity. The introduction of generative adversarial networks makes the generated speech more natural and reduces the monotonous synthesized tone problem that is common in traditional methods. The parameter update method based on generative adversarial loss and contrastive loss functions makes the training process more stable, effectively avoids the problem of pattern collapse, and improves the diversity of speech synthesis.

[0185] In one embodiment, a speech generation device based on distribution prediction is provided, and the speech generation device based on distribution prediction corresponds one-to-one to the speech generation method based on distribution prediction in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of a speech generation device based on distribution prediction according to the present invention. It includes a context encoding module 10, a historical feature management module 20, a distribution prediction module 30, a speech waveform point generation module 40, a waveform feature encoding module 50, and a speech synthesis control module 60. Each functional module is described in detail below:

[0186] A context encoding module 10 is used to encode the input text through a context encoder to generate context features containing global temporal dependencies;

[0187] A historical feature management module 20 is used to initialize the historical waveform feature sequence to an empty sequence or a preset initial mark;

[0188] The distribution prediction module 30 is used to input the context features and the historical waveform feature sequence into the next distribution prediction module to predict and generate the probability distribution of the speech waveform at the next moment;

[0189] A speech waveform point generating module 40 is configured to generate a speech waveform point at the next moment according to the probability distribution of the speech waveform at the next moment;

[0190] The waveform feature encoding module 50 is used to encode the speech waveform point at the next moment into a new waveform feature, and update the new waveform feature into the historical waveform feature sequence;

[0191] The speech synthesis control module 60 is used to repeatedly execute the probability distribution prediction, speech waveform point generation at the next moment and historical waveform feature sequence update steps of the next distribution prediction module until the preset termination condition is met, stop the repeated execution, and combine the generated speech waveform points into the target speech waveform.

[0192] In one embodiment, the context encoding module 10 is specifically configured to:

[0193] Determining a phoneme segmentation strategy according to the target speech category, and segmenting the input text into phoneme sequences based on the phoneme segmentation strategy;

[0194] Mapping each phoneme in the phoneme sequence into a high-dimensional vector through an embedding layer to generate a phoneme vector sequence;

[0195] Inputting the phoneme vector sequence into a bidirectional long short-term memory network, and extracting local temporal features of the phoneme vector sequence through the bidirectional long short-term memory network;

[0196] Inputting the local temporal features into a multi-head self-attention mechanism layer, determining the attention weight between any two time steps in the phoneme sequence, and performing weighted aggregation on the local temporal features based on the attention weight to generate a global temporal dependency relationship;

[0197] The local temporal features and the global temporal dependency are concatenated to generate the contextual features.

[0198] In one embodiment, the distribution prediction module 30 is specifically configured to:

[0199] Inputting the context features and the historical waveform feature sequence into a next distribution prediction module;

[0200] Performing time convolution coding processing on the historical waveform feature sequence through the time series feature enhancement layer of the next distribution prediction module to generate time series enhancement features;

[0201] The context features and the temporal enhancement features are subjected to multi-head attention interaction processing through the cross-modal alignment layer of the next distribution prediction module to generate fused alignment features;

[0202] Determining the mean vector, standard deviation vector, and mixing weight parameters of a multi-Gaussian mixture distribution according to the fused alignment features through a mixture density network of the next distribution prediction module;

[0203] Scaling the blending weight parameters based on a preset temperature coefficient by a distribution sharpening layer of the next distribution prediction module to generate a sharpened blending weight;

[0204] Through the distribution construction layer of the next distribution prediction module, a mixed Gaussian distribution is constructed according to the mean vector, standard deviation vector and sharpened mixing weights, and the mixed Gaussian distribution is used as the probability distribution of the speech waveform at the next moment.

[0205] In one embodiment, the speech waveform point generation module 40 is specifically configured to:

[0206] Sampling random noise from a standard normal distribution, and multiplying the random noise element-by-element by the standard deviation vector of the probability distribution of the speech waveform at the next moment, and then adding the resulting sum to the mean vector of the probability distribution to generate an original waveform point;

[0207] Inputting the original waveform points into a residual convolutional network, suppressing non-speech band noise in the original waveform points and retaining speech band features through a multi-scale convolution kernel in the residual convolutional network, thereby generating denoised waveform points;

[0208] According to the mean and standard deviation of the historical waveform feature sequence, the denoised waveform point is amplitude-adjusted to generate the speech waveform point at the next moment.

[0209] In one embodiment, the waveform feature encoding module 50 is specifically configured to:

[0210] Inputting the speech waveform point at the next moment into a waveform feature encoder, generating a high-dimensional acoustic feature vector through a one-dimensional convolution layer and a nonlinear activation function, and using the high-dimensional acoustic feature vector as a new waveform feature;

[0211] Splicing the new waveform feature to the end of the historical waveform feature sequence in time sequence to form an updated historical waveform feature sequence;

[0212] If the length of the updated historical waveform feature sequence exceeds the preset maximum sequence length, the waveform feature at the earliest moment in the historical waveform feature sequence is removed.

[0213] In one embodiment, the speech synthesis control module 60 is specifically configured to:

[0214] Repeat the steps of probability distribution prediction of the next distribution prediction module, generation of speech waveform points at the next moment, and updating of historical waveform feature sequences;

[0215] After each generation of the speech waveform point at the next moment, detecting whether the generated speech waveform point sequence is consistent with the time-series alignment length of the input text, or detecting whether the short-time energy of the generated speech waveform point sequence is lower than a preset silence threshold, or detecting whether the length of the historical waveform feature sequence reaches the total number of frames after the target speech duration is converted by frame rate;

[0216] If any of the following conditions is met: the generated speech waveform point sequence is consistent with the time-series alignment length of the input text, or the short-time energy of the generated speech waveform point sequence is lower than the preset silence threshold, or the length of the historical waveform feature sequence reaches the total number of frames after the target speech duration is converted by frame rate, the repeated execution is terminated;

[0217] The speech waveform points generated at the next moment are spliced ​​together in the order of generation time to form an uncompressed time domain waveform sequence;

[0218] The time domain waveform sequence is resampled and amplitude normalized to generate a final target speech waveform.

[0219] In one embodiment, the distribution prediction module 30 is specifically configured to:

[0220] Extract training text and corresponding real speech waveform from the training dataset;

[0221] Generate a training global context feature of the training text through a context encoder, and generate a training history waveform feature sequence based on the real speech waveform;

[0222] Inputting the training global context feature sequence and the training historical waveform feature sequence into the next distribution prediction module to generate a training probability distribution of the speech waveform at the next moment;

[0223] Sampling the training speech waveform points at the next moment from the training probability distribution, and accumulating the generated training speech waveform points until a complete training speech segment is formed;

[0224] Inputting the complete training speech segment into the discriminator of the generative adversarial network to obtain a naturalness score of the training generated speech;

[0225] Inputting the real speech waveform into the discriminator to obtain a naturalness score of the real speech waveform;

[0226] determining a generative adversarial loss based on a difference between a naturalness score of the training generated speech and a naturalness score of a real speech waveform;

[0227] Based on the generative adversarial loss and the contrastive loss function, the parameters of the next distribution prediction module are updated.

[0228] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a speech generation method based on distribution prediction.

[0229] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a speech generation method based on distribution prediction.

[0230] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0231] Encode the input text through the context encoder to generate context features containing global temporal dependencies;

[0232] Initialize the historical waveform feature sequence to an empty sequence or preset initial mark;

[0233] Inputting the context features and the historical waveform feature sequence into a next distribution prediction module to predict and generate a probability distribution of the speech waveform at the next moment;

[0234] Generate a speech waveform point at the next moment according to the probability distribution of the speech waveform at the next moment;

[0235] Encoding the speech waveform point at the next moment as a new waveform feature, and updating the new waveform feature to the historical waveform feature sequence;

[0236] Repeat the probability distribution prediction of the next distribution prediction module, the speech waveform point generation at the next moment and the historical waveform feature sequence update steps until the preset termination condition is met, stop the repetitive execution, and combine the generated speech waveform points into the target speech waveform.

[0237] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0238] Encode the input text through the context encoder to generate context features containing global temporal dependencies;

[0239] Initialize the historical waveform feature sequence to an empty sequence or preset initial mark;

[0240] Inputting the context features and the historical waveform feature sequence into a next distribution prediction module to predict and generate a probability distribution of the speech waveform at the next moment;

[0241] Generate a speech waveform point at the next moment according to the probability distribution of the speech waveform at the next moment;

[0242] Encoding the speech waveform point at the next moment as a new waveform feature, and updating the new waveform feature to the historical waveform feature sequence;

[0243] Repeat the probability distribution prediction of the next distribution prediction module, the speech waveform point generation at the next moment and the historical waveform feature sequence update steps until the preset termination condition is met, stop the repetitive execution, and combine the generated speech waveform points into the target speech waveform.

[0244] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0245] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0246] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0247] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A speech generation method based on distribution prediction, characterized in that: The following steps are involved: Encode the input text through the context encoder to generate context features containing global temporal dependencies; Initialize the historical waveform feature sequence to an empty sequence or preset initial mark; Inputting the context features and the historical waveform feature sequence into a next distribution prediction module to predict and generate a probability distribution of the speech waveform at the next moment; Generate a speech waveform point at the next moment according to the probability distribution of the speech waveform at the next moment; Encoding the speech waveform point at the next moment as a new waveform feature, and updating the new waveform feature to the historical waveform feature sequence; Repeat the probability distribution prediction of the next distribution prediction module, the speech waveform point generation at the next moment and the historical waveform feature sequence update steps until the preset termination condition is met, stop the repetitive execution, and combine the generated speech waveform points into the target speech waveform.

2. The speech generation method based on distribution prediction according to claim 1, wherein The input text is encoded through the context encoder to generate context features containing global temporal dependencies, including: Determining a phoneme segmentation strategy according to the target speech category, and segmenting the input text into phoneme sequences based on the phoneme segmentation strategy; Mapping each phoneme in the phoneme sequence into a high-dimensional vector through an embedding layer to generate a phoneme vector sequence; Inputting the phoneme vector sequence into a bidirectional long short-term memory network, and extracting local temporal features of the phoneme vector sequence through the bidirectional long short-term memory network; Inputting the local temporal features into a multi-head self-attention mechanism layer, determining the attention weight between any two time steps in the phoneme sequence, and performing weighted aggregation on the local temporal features based on the attention weight to generate a global temporal dependency relationship; The local temporal features and the global temporal dependency are concatenated to generate the contextual features.

3. The speech generation method based on distribution prediction according to claim 1, wherein Inputting the context feature and the historical waveform feature sequence into a next distribution prediction module to predict and generate a probability distribution of the speech waveform at the next moment, including: Inputting the context features and the historical waveform feature sequence into a next distribution prediction module; Performing time convolution coding processing on the historical waveform feature sequence through the time series feature enhancement layer of the next distribution prediction module to generate time series enhancement features; The context features and the temporal enhancement features are subjected to multi-head attention interaction processing through the cross-modal alignment layer of the next distribution prediction module to generate fused alignment features; Determining the mean vector, standard deviation vector, and mixing weight parameters of a multi-Gaussian mixture distribution according to the fused alignment features through a mixture density network of the next distribution prediction module; Scaling the blending weight parameters based on a preset temperature coefficient by a distribution sharpening layer of the next distribution prediction module to generate a sharpened blending weight; Through the distribution construction layer of the next distribution prediction module, a mixed Gaussian distribution is constructed according to the mean vector, standard deviation vector and sharpened mixing weights, and the mixed Gaussian distribution is used as the probability distribution of the speech waveform at the next moment.

4. The speech generation method based on distribution prediction according to claim 1, wherein Generating a speech waveform point at the next moment according to the probability distribution of the speech waveform at the next moment includes: Sampling random noise from a standard normal distribution, and multiplying the random noise element-by-element by the standard deviation vector of the probability distribution of the speech waveform at the next moment, and then adding the resulting sum to the mean vector of the probability distribution to generate an original waveform point; Inputting the original waveform points into a residual convolutional network, suppressing non-speech band noise in the original waveform points and retaining speech band features through a multi-scale convolution kernel in the residual convolutional network, thereby generating denoised waveform points; According to the mean and standard deviation of the historical waveform feature sequence, the denoised waveform point is amplitude-adjusted to generate the speech waveform point at the next moment.

5. The speech generation method based on distribution prediction according to claim 1, wherein: Encoding the speech waveform point at the next moment into a new waveform feature, and updating the new waveform feature into the historical waveform feature sequence, including: Inputting the speech waveform point at the next moment into a waveform feature encoder, generating a high-dimensional acoustic feature vector through a one-dimensional convolution layer and a nonlinear activation function, and using the high-dimensional acoustic feature vector as a new waveform feature; Splicing the new waveform feature to the end of the historical waveform feature sequence in time sequence to form an updated historical waveform feature sequence; If the length of the updated historical waveform feature sequence exceeds the preset maximum sequence length, the waveform feature at the earliest moment in the historical waveform feature sequence is removed.

6. The speech generation method based on distribution prediction according to claim 1, wherein: Repeating the probability distribution prediction of the next distribution prediction module, generating speech waveform points at the next moment, and updating the historical waveform feature sequence until a preset termination condition is met, stopping the repeated execution, and combining the generated speech waveform points into a target speech waveform, including: Repeat the steps of probability distribution prediction of the next distribution prediction module, generation of speech waveform points at the next moment, and updating of historical waveform feature sequences; After each generation of the speech waveform point at the next moment, detecting whether the generated speech waveform point sequence is consistent with the time-series alignment length of the input text, or detecting whether the short-time energy of the generated speech waveform point sequence is lower than a preset silence threshold, or detecting whether the length of the historical waveform feature sequence reaches the total number of frames after frame rate conversion of the target speech duration; If any of the following conditions is met: the generated speech waveform point sequence is consistent with the time-series alignment length of the input text, or the short-time energy of the generated speech waveform point sequence is lower than the preset silence threshold, or the length of the historical waveform feature sequence reaches the total number of frames after the target speech duration is converted by frame rate, the repeated execution is terminated; The speech waveform points generated at the next moment are spliced ​​together in the order of generation time to form an uncompressed time domain waveform sequence; The time domain waveform sequence is resampled and amplitude normalized to generate a final target speech waveform.

7. The speech generation method based on distribution prediction according to claim 1, wherein: Before inputting the context feature and the historical waveform feature sequence into the next distribution prediction module to predict and generate the probability distribution of the speech waveform at the next moment, the method further includes: Extract training text and corresponding real speech waveform from the training dataset; Generate a training global context feature of the training text through a context encoder, and generate a training history waveform feature sequence based on the real speech waveform; Inputting the training global context feature sequence and the training historical waveform feature sequence into the next distribution prediction module to generate a training probability distribution of the speech waveform at the next moment; Sampling the training speech waveform points at the next moment from the training probability distribution, and accumulating the generated training speech waveform points until a complete training speech segment is formed; Inputting the complete training speech segment into the discriminator of the generative adversarial network to obtain a naturalness score of the training generated speech; Inputting the real speech waveform into the discriminator to obtain a naturalness score of the real speech waveform; determining a generative adversarial loss based on a difference between a naturalness score of the training generated speech and a naturalness score of a real speech waveform; Based on the generative adversarial loss and the contrastive loss function, the parameters of the next distribution prediction module are updated.

8. A speech generation device based on distribution prediction, characterized in that: The speech generation device based on distribution prediction includes: The context encoding module is used to encode the input text through the context encoder to generate context features containing global temporal dependencies; A historical feature management module, used to initialize the historical waveform feature sequence to an empty sequence or a preset initial mark; A distribution prediction module is used to input the context features and the historical waveform feature sequence into the next distribution prediction module to predict and generate the probability distribution of the speech waveform at the next moment; A speech waveform point generation module, configured to generate a speech waveform point at the next moment according to the probability distribution of the speech waveform at the next moment; A waveform feature encoding module is used to encode the speech waveform point at the next moment into a new waveform feature, and update the new waveform feature into the historical waveform feature sequence; The speech synthesis control module is used to repeatedly execute the probability distribution prediction, speech waveform point generation at the next moment and historical waveform feature sequence update steps of the next distribution prediction module until a preset termination condition is met, stop the repeated execution, and combine the generated speech waveform points into a target speech waveform.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a distribution prediction-based speech generation program stored in the memory and capable of running on the processor. When the distribution prediction-based speech generation program is executed by the processor, the steps of the distribution prediction-based speech generation method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a speech generation program based on distribution prediction, which, when executed by a processor, implements the steps of the speech generation method based on distribution prediction as described in any one of claims 1 to 7.

Images