A Deep Learning-Based Real-Time Multilingual TTS Synthesis Method

Abstract

This invention discloses a real-time multilingual TTS synthesis method based on deep learning. This method achieves high-quality real-time conversion of multilingual text to speech through a deep neural network model. First, a multilingual acoustic feature library and pronunciation rule library are constructed, and an end-to-end neural network architecture is used to extract semantic features from the text. Then, an improved attention mechanism is used to achieve accurate synthesis of speech prosody and pronunciation, ensuring the naturalness and coherence of multilingual pronunciation. Simultaneously, a lightweight inference engine is designed to optimize computational resource allocation and meet the requirements of real-time synthesis. Compared with traditional TTS methods, this invention significantly improves the fluency and naturalness of multilingual speech synthesis, greatly reduces computational latency, and provides efficient and reliable technical support for intelligent voice interaction and multilingual content production.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to a real-time multilingual TTS synthesis method based on deep learning. Background Technology

[0002] Multilingual speech synthesis technology, as one of the core technologies in the fields of artificial intelligence and speech processing, plays a crucial role in applications such as global information exchange, intelligent assistants, and barrier-free communication. TTS stands for Text-to-Speech, a technology that automatically converts written text into spoken output. With the increasing prevalence of applications such as intelligent voice broadcasting and risk disclosure, technologies using deep learning methods to achieve text-to-speech conversion, supporting multiple languages, generating speech in real time, and accurately and naturally synthesizing multilingual speech have become key drivers for improving business processing efficiency and ensuring business compliance. However, current mainstream multilingual speech synthesis methods generally have significant shortcomings. Traditional statistical parameter-based synthesis methods have limited performance in handling the complex speech features of different languages, making it difficult to capture subtle differences between languages. While existing neural network methods have made progress in single-language scenarios, they often face problems such as unstable speech quality and insufficient pronunciation accuracy in multilingual scenarios. More importantly, these methods have excessively high computational complexity in practical deployment, making it difficult to meet the performance requirements of real-time applications. The main technical challenges of multilingual speech synthesis stem from the complexity of speech feature extraction and semantic understanding. Different languages ​​possess unique acoustic features and pronunciation rules. A single feature extraction method cannot effectively cover the speech representation needs of multiple languages, directly impacting the accurate conversion of textual semantic information into speech signals. Insufficient semantic feature extraction further exacerbates the difficulty of speech prosody control. Traditional sequence modeling methods are prone to attentional distraction and information loss when processing long speech sequences, resulting in poor performance in terms of prosodic naturalness and pronunciation coherence in synthesized speech. Therefore, how to construct a system that can accurately extract semantic features from multiple languages, precisely control speech prosody generation, and perform lightweight, real-time multilingual speech synthesis while ensuring synthesis quality has become a critical problem urgently needing to be solved in this field. Summary of the Invention

[0003] This invention provides a real-time multilingual TTS synthesis method based on deep learning, mainly including:

[0004] After acquiring multilingual text input, word embedding encoding is performed through a pre-trained language model, and semantic features in the text sequence are deeply extracted. Based on the detected language identifiers, the feature extraction branch of the corresponding language is activated to generate a semantic feature vector containing language-specific information.

[0005] Sequence modeling is performed using a recurrent neural network based on the semantic feature vectors. The direction of information flow is controlled by a gating mechanism. If there is a change in intonation between the semantic features at the current moment and the previous moment, the prosodic control parameters are updated to obtain a sequence encoding result that integrates temporal dependencies.

[0006] The sequence encoding results are weighted by an attention weight allocation mechanism to assign importance weights at different positions. Key prosodic nodes are determined by these weights to obtain the distribution of key time points for prosodic control.

[0007] Based on the distribution of the key time points, an acoustic modeling network is used to generate Mel spectrum features, and a convolutional neural network is used to extract local acoustic patterns to obtain spectrum generation results adapted to multilingual characteristics.

[0008] The frequency domain information is mapped to the time domain space by the frequency domain information through the fast Fourier transform to obtain time domain speech waveform data.

[0009] The time-domain speech waveform data is post-processed and optimized to eliminate discontinuous jumps between waveforms, determine the multilingual speech synthesis output, and optimize memory usage efficiency based on resource allocation to obtain a lightweight speech synthesis result that meets real-time requirements.

[0010] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0011] This invention discloses a real-time multilingual TTS synthesis method based on deep learning. It extracts semantic features of multilingual text through a pre-trained language model and a multi-head attention mechanism, establishes a sequence model using a recurrent neural network and integrates temporal dependencies, determines key time points for prosodic control using attention weight allocation, generates spectral features adapted to multilingual characteristics based on an acoustic modeling network, converts the data into time-domain speech waveforms using a lightweight vocoder algorithm, and finally performs post-processing optimization to obtain continuous and natural multilingual speech output. This invention can effectively process multilingual mixed text, achieve language-independent high-quality speech synthesis, and ensure real-time performance through dynamic batch processing and model compression techniques, providing an efficient and reliable solution for multilingual voice interaction systems. Attached Figure Description

[0012] Figure 1 This is a flowchart of a real-time multilingual TTS synthesis method based on deep learning according to the present invention. Detailed Implementation

[0013] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] like Figure 1 This embodiment of a deep learning-based multilingual TTS real-time synthesis method specifically includes:

[0015] S101. After obtaining multilingual text input, word embedding encoding is performed through a pre-trained language model to deeply extract semantic features from the text sequence. Based on the detected language identifiers, the feature extraction branch of the corresponding language is activated to generate a semantic feature vector containing language-specific information.

[0016] After acquiring multilingual text input, a pre-trained language model is used to perform word embedding encoding on each word in the text, generating an initial word vector representation. Positional encoding is then added based on the word's position in the sentence, resulting in an initial embedding representation incorporating positional information. A multi-head attention mechanism is used to calculate the attention weights between each word and other words in the initial embedding representation. These attention weights are then used to weight and sum the word vectors, extracting semantic dependencies in the text sequence to obtain an intermediate semantic representation containing contextual information. A softmax classifier is used to calculate the probability distribution of each language category in the intermediate semantic representation. The language identifier is determined based on the maximum probability, and a feature extraction branch for the corresponding language is activated based on this identifier. This feature extraction branch is a fully connected network layer pre-trained for a specific language. The activated language-specific feature extraction branch performs a linear transformation on the intermediate semantic representation to extract the grammatical structure patterns and lexical collocation rules of the language. A ReLU activation function is then used for non-linear mapping to generate a semantic feature vector containing language-specific information.

[0017] Specifically, in multilingual text processing, word embedding encoding is a fundamental step in understanding text semantics. Pre-trained language models learn vector representations of words from large-scale corpora, with each word mapped to a point in a high-dimensional space. The addition of positional encoding addresses the problem of missing word order information by generating a unique encoding vector for each position using sine and cosine functions, ensuring that the same word has different representations at different positions.

[0018] In one possible implementation, the initial embedding representation, after being processed by a multi-head attention mechanism, can capture the dependencies between lexical units. The calculation of attention weights involves the dot product operation of three vectors: query, key, and value. The correlation strength between each pair of lexical units is obtained by scaling the dot product attention formula. Multiple attention heads process different semantic subspaces in parallel, extracting features from multiple dimensions such as lexical, syntactic, and semantic dimensions, and concatenating them to form a rich intermediate semantic representation.

[0019] It should be noted that language recognition calculates the probability distribution using a softmax classifier. This classifier maps intermediate semantic representations to a language category space, with each dimension corresponding to a probability value for a language. The language corresponding to the highest probability is the recognition result. This probability-based method can handle the potential language mixing in text and provides reliable language identifiers.

[0020] Specifically, language-specific feature extraction branches are dedicated network layers designed for the characteristics of different languages. The English branch may focus more on root and affix variations, the Chinese branch emphasizes character combinations and contextual relationships, and the Arabic branch needs to handle right-to-left writing order. Each branch contains multiple fully connected networks, with weight parameters pre-trained on the corresponding language corpus, enabling the extraction of grammatical patterns unique to that language.

[0021] For example, when processing a German text containing technical terms, the German feature extraction branch identifies compound word structures, breaking down long compound words into morphemes for understanding. The linear transformation matrix adjusts feature weights according to German grammar rules, reinforcing grammatical markers such as nouns and verb tenses. The ReLU activation function introduces non-linearity, enabling the network to learn complex language pattern mappings.

[0022] In one embodiment, the final generated semantic feature vector contains a fused representation of the original semantic information and language-specific features. Each dimension of the vector encodes a specific linguistic attribute, such as part-of-speech distribution, syntactic structure, semantic roles, etc.

[0023] S102. Sequence modeling is performed using a recurrent neural network based on the semantic feature vector. The direction of information flow is controlled by a gating mechanism. If there is a change in intonation between the semantic features at the current moment and the previous moment, the prosodic control parameters are updated to obtain the sequence encoding result that integrates temporal dependencies.

[0024] Based on the semantic feature vector, a long short-term memory network is used to sequentially model the features at each time step. The matrix product of the current semantic feature vector and the input weight matrix is ​​calculated, and after processing by the sigmoid activation function, the input gate control value is obtained to determine the degree of new information reception and generate candidate memory unit states. Based on the candidate memory unit states and the memory unit states at the previous time step, the forget gate weight matrix is ​​multiplied by the current semantic feature vector and activated by the sigmoid function to obtain the forget gate control value. This value is then multiplied by the memory unit state at the previous time step to selectively retain historical information. Finally, it is added to the candidate states adjusted by the input gate to obtain the current memory unit state. Based on the current memory unit state, the current semantic feature vector is processed by the output gate weight matrix and activated by the sigmoid function to obtain the output gate control value. Simultaneously, the fundamental frequency component is extracted from the semantic feature vector. If the difference between the fundamental frequency component and the fundamental frequency at the previous time step exceeds a preset threshold, the corresponding component value in the prosody control parameter vector is adjusted. The output gate control value is weighted and adjusted using the prosody control parameter vector, and then multiplied with the value of the current memory unit state after mapping by the tanh function to obtain a hidden state vector containing temporal dependency information and prosody modulation information, which serves as the sequence encoding result that fuses temporal dependencies.

[0025] Specifically, Long Short-Term Memory (LSTM) networks play a crucial role in sequence modeling, their core function being the precise control of information flow through gating mechanisms. Semantic feature vectors, as input, are first multiplied by the input weight matrix. This process essentially performs a linear transformation on the features, mapping the original feature space to a gating space. The application of the sigmoid activation function restricts the output value to between 0 and 1, forming a valve-like control signal.

[0026] In one possible implementation, the input gate control value determines the degree to which new information is received. When processing a speech-to-text sequence, if a significant semantic shift occurs at the current moment, the input gate will produce a control value close to 1, fully accepting new semantic information. Conversely, for redundant or noisy information, the input gate will produce a smaller control value, restricting its entry into the memory unit. The candidate memory unit states contain filtered new information, laying the foundation for subsequent information fusion.

[0027] It's worth noting that the forget gate design addresses the vanishing gradient problem in long sequence processing. The forget gate generates a selective forgetting signal by multiplying the current input by a specialized forgetting weight matrix. This mechanism is particularly important when processing long dialogues or texts, automatically identifying and retaining crucial contextual information while downplaying minor details. The previous memory unit state is multiplied by the forget gate control value, enabling dynamic filtering of historical information.

[0028] Specifically, the extraction of the fundamental frequency component involves frequency domain analysis of the semantic feature vector. The fundamental frequency of a speech signal reflects the speaker's pitch variations and is significant in emotion recognition and intonation analysis. The fundamental frequency component can be separated from the feature vector using Fast Fourier Transform or autocorrelation analysis. When a sudden change in the fundamental frequency is detected, the system determines that a question, surprise, or emotional shift may have occurred, triggering an update of the prosodic control parameters.

[0029] For example, the prosodic control parameter vector contains multiple dimensions, each corresponding to a different prosodic feature. The pitch change rate reflects the rising or falling trend of the intonation, the pause duration encodes speech rhythm information, and the stress pattern marks the position of emphasis. These parameters adjust the control value of the output gate in a weighted manner, so that the final hidden state vector not only contains semantic information but also incorporates rich prosodic features.

[0030] In one embodiment, the tanh function maps the memory cell states to a range of -1 to 1, enhancing the network's expressive power. The prosodic modulated output gate control value is multiplied by the memory states mapped by the tanh function to produce the sequence encoding result.

[0031] S103. The sequence encoding result is weighted by an attention weight allocation mechanism, and importance weights are assigned to different positions. Key prosodic nodes are determined by the weights to obtain the key time point distribution of prosodic control.

[0032] A self-attention weight is calculated for each position vector of the sequence encoding result using an attention weight allocation mechanism. The position vector is used as the query, key, and value. The dot product of the query and key is calculated and divided by the square root of the feature dimension. After softmax normalization, the attention weight value for each position is obtained. This attention weight value is then used to weight the value vector at the corresponding position to obtain the weighted feature representation and its corresponding attention weight value for each position. A weight sequence is constructed based on the attention weight values ​​for each position. A sliding window with a preset window width is applied to this weight sequence, and a subsequence of weight values ​​within each window is extracted. If the weight value at the center of the window is greater than the weight values ​​at all other positions within the window, then the center position is determined to be a local peak candidate point. The attention weight value of the local peak candidate point is compared with a preset prosodic threshold. If the weight value exceeds the threshold, the candidate point is marked as a key prosodic node, and its time position index and weight value are recorded. The time position indices of all key prosodic nodes are summarized to generate a key time point distribution arranged in chronological order.

[0033] Specifically, self-attention mechanisms play a central role in prosodic analysis. Essentially, they enable each position in a sequence to pay attention to all other positions, thereby capturing long-distance dependencies. Each position vector in the sequence encoding result contains semantic and prosodic information at that moment. By processing it simultaneously as a query, key, and value, self-association of information is achieved.

[0034] In one possible implementation, the dot product operation measures the similarity between different positions. A larger dot product value indicates a strong semantic or prosodic correlation when the vectors of two positions are oriented similarly in a high-dimensional space. Dividing by the square root of the feature dimension is a scaling technique to prevent the dot product from becoming too large in high dimensions, causing the softmax function to enter the saturation region and maintaining stable gradient propagation.

[0035] It's important to note that softmax normalization transforms the original similarity scores into a probability distribution, ensuring that the sum of all attention weights at each position is 1. This normalization process allows the attention mechanism to adaptively allocate attention, which is particularly important for prosodic analysis. When processing recitation audio, rhyme positions often receive higher attention weights, while transition words tend to have relatively lower weights.

[0036] Specifically, the sliding window technique takes into account the local characteristics of rhythm. Rhythmic rhythms are typically characterized by alternating strong and weak beats in specific areas, rather than a uniform global distribution. The selection of the preset window width needs to balance locality and continuity; too narrow a width may miss important rhythmic transitions, while too wide a width may confuse different rhythmic units. The comparison operation within the window identifies the most prominent local position by comparing the weight values ​​of the center position with those of the surrounding positions one by one.

[0037] For example, when processing a speech recording, local peaks often correspond to stressed syllables or intonation transitions. When a speaker emphasizes a keyword, the attention weight at that location is significantly higher than before or after it, forming a local peak. This peak detection mechanism can automatically locate the emphasis in speech, providing important reference for subsequent speech synthesis or sentiment analysis.

[0038] In one embodiment, the prosodic threshold setting needs to consider the differences in languages ​​and speaking styles. Chinese, with its rich tonal variations, may require a lower threshold to capture subtle prosodic changes; while English, relying more on stress patterns, may require a higher threshold to filter out true stress positions. Positions exceeding the threshold are marked as key prosodic nodes, which collectively constitute the prosodic skeleton. The distribution of key time points is generated by aggregating the temporal indices of all prosodic nodes. This distribution not only records the location of prosodic emphasis but also preserves their temporal relationships. In speech synthesis applications, these key time points can guide prosodic modeling, applying appropriate pitch variations and duration adjustments at corresponding positions to make the synthesized speech more natural and fluent.

[0039] S104. Based on the distribution of the key time points, an acoustic modeling network is used to generate Mel spectrum features, and local acoustic patterns are extracted through a convolutional neural network to obtain a spectrum generation result adapted to multilingual characteristics.

[0040] Based on the key time point distribution, the time index sequence and corresponding prosodic markers are input into the acoustic modeling network. The discrete time index is converted into a continuous vector representation through an embedding layer. This is then mapped through a three-layer fully connected network to generate the basic acoustic parameters for each time point. Finally, a Mel filter bank transform is used to obtain the Mel spectral feature sequence. This Mel spectral feature sequence is processed by a convolutional neural network, using convolutional kernels of different sizes to extract short-term and long-term time-frequency patterns. The convolution operation captures the harmonic structure and temporal transition features between frequency bands. After pooling, a multi-scale local acoustic pattern representation is obtained. Based on this multi-scale local acoustic pattern representation, the Euclidean distance between adjacent time frames is calculated and compared with the average distance of the preceding frames. If the current distance exceeds a preset multiple of the average distance, it is identified as a potential language switching point. The language type conversion is determined by comparing the change in the spectral envelope slope before and after the switching point. Based on the language type conversion result, the acoustic parameter set of the corresponding target language is retrieved, containing the language's unique fundamental frequency range and formant distribution template. These parameters are applied to the Mel spectral features after the switching point, and parameter interpolation smooths the transition region, generating a spectrum generation result adapted to multilingual characteristics.

[0041] Specifically, the core function of acoustic modeling networks is to transform abstract temporal information into concrete acoustic representations. Each time index in the key time point distribution represents a prosodic turning point, and these discrete values ​​need to be mapped to a continuous vector space through embedding layers. The embedding process is similar to a table lookup operation, with each time position corresponding to a learnable vector. This representation can capture the relative relationships between time positions and the prosodic importance.

[0042] In one possible implementation, the three-layer fully connected network is designed following the principle of layer-by-layer feature abstraction. The first layer extends the embedding vectors to a higher dimension, increasing their expressive power; the second layer performs a nonlinear transformation to learn the complex mapping relationship between time and acoustic features; and the third layer compresses the high-dimensional features down to the dimension of the Mel spectrum. The application of the Mel filter bank is based on the characteristics of human hearing, with dense distribution in the low-frequency part and sparse distribution in the high-frequency part, better simulating human auditory perception.

[0043] It's worth noting that convolutional neural networks (CNNs) have unique advantages in processing Mel-frequency spectra. Different sized convolutional kernels perform different feature extraction tasks: small kernels capture local spectral details, such as the precise location of formants; large kernels extract the global spectral envelope, reflecting the overall timbre characteristics. In the extraction of time-frequency patterns, convolution operations can identify harmonic structures, i.e., the regular relationship between the fundamental frequency and its integer multiples of frequency components.

[0044] Specifically, calculating the Euclidean distance involves a comprehensive comparison of multi-dimensional feature vectors. When processing mixed English and Chinese speech, the switch point from English to Chinese often manifests as a significant change in the spectral envelope. English typically has stronger energy in the mid-to-high frequency range, while Chinese tonal variations lead to a more complex fundamental frequency trajectory. By calculating the difference in feature vectors between adjacent frames and comparing it with historical averages, the boundaries of this language transition can be effectively identified.

[0045] For example, changes in the slope of the spectral envelope reflect the pronunciation characteristics of different languages. The four-tone system of Mandarin Chinese causes significant fluctuations in the fundamental frequency over a short period, resulting in a dynamic spectral envelope; while English relies more on slow changes in stress and intonation, leading to a relatively stable spectral envelope. By analyzing the difference in slope before and after the switching point, it is possible to determine whether the language is switching from a tonal language to a stressed language, or vice versa.

[0046] In one embodiment, the choice of acoustic parameter set directly affects the final spectrum generation quality. The fundamental frequency range parameter determines the pitch variation space; the fundamental frequency range of Chinese male voices is typically 80-180 Hz, while English may be slightly higher. The formant distribution template records the vowel space characteristics of different languages. The vowel system of Chinese is relatively simple, and the formant positions are more concentrated; English has a rich vowel system, and the formant distribution is more dispersed. Parameter interpolation technology plays a key role in the transition region of language switching. Through linear or nonlinear interpolation methods, the spectral characteristics are smoothly transitioned, avoiding abrupt jumps and generating natural and smooth multilingual spectra.

[0047] S105. Perform time-domain signal conversion on the spectrum generation result, and map the frequency domain information to the time domain space through fast Fourier transform to obtain time-domain speech waveform data.

[0048] A lightweight vocoder is used to process the generated spectrum. The phase component is estimated iteratively from the amplitude information using the Griffin-Lim algorithm. The estimated phase is combined with the original amplitude spectrum to form a complex spectral representation, resulting in a complete frequency domain signal containing both amplitude and phase information. An inverse fast Fourier transform is performed on the complete frequency domain signal, converting the frequency domain complex sequence of each frame into a corresponding time-domain sampling point sequence. Weighted superposition is performed using the overlapping portions of frame shifts to eliminate discontinuities at frame boundaries, generating a smoothly transitioning time-domain waveform sequence. The root mean square energy value within a sliding window is calculated based on the time-domain waveform sequence. If the energy ratio of adjacent windows exceeds a preset dynamic range threshold, the high-energy segment is amplitude compressed using a hyperbolic tangent function to maintain the relative dynamic characteristics of the signal, obtaining continuous time-domain speech waveform data.

[0049] Specifically, the core task of lightweight vocoders is to reconstruct time-domain speech signals from spectral information. A key challenge in this process is the recovery of phase information. Spectral generation typically only contains amplitude information, while complete signal reconstruction requires phase components. The Griffin-Lim algorithm addresses this problem through iterative optimization; its basic principle is to use time-frequency domain consistency constraints to estimate the most reasonable phase distribution.

[0050] In one possible implementation, the algorithm first assigns a random initial phase to the amplitude spectrum, forming an initial complex spectrum. After obtaining the time-domain signal through inverse Fourier transform, a forward Fourier transform is performed, preserving the original amplitude spectrum but updating the phase information. This iterative process is repeated, with each iteration making the phase estimate closer to the true value. After multiple iterations, the phase gradually converges to a stable state, at which point the complex spectrum can reconstruct a natural speech signal.

[0051] It should be noted that the inverse fast Fourier transform maps the complex representation in the frequency domain back to the sampling points in the time domain. Each complex number in the frequency domain contains the amplitude and phase information of that frequency component. Through the inverse transform, these frequency components are superimposed in the time domain to form a complete waveform. However, frame-by-frame processing can lead to discontinuities at frame boundaries. Adjacent frames may have phase jumps or amplitude abrupt changes at the boundaries, and direct splicing will produce noticeable clicking sounds or distortion.

[0052] Specifically, the overlay technique addresses this issue by creating an overlapping region between adjacent frames. Assuming a frame length of 512 samples and a frame shift of 256 samples, adjacent frames will have a 50% overlap. Within the overlapping region, the signal from the preceding frame gradually attenuates, while the signal from the following frame gradually increases, and the two are weighted according to a specific window function. This gradual transition ensures signal continuity and eliminates abrupt changes at frame boundaries.

[0053] For example, the root mean square energy calculation within a sliding window reflects the local intensity of the signal. When processing mixed speech containing whispers and normal speech, the energy difference can be tens of times. The energy value of the whisper segment is very low, while the energy value of the normal speech segment is high. This huge dynamic range can cause difficulties for subsequent processing and playback. By calculating the square root of the mean of the sum of squares of all sampling points within each window, the root mean square energy value of that window can be obtained, which can accurately characterize the local signal intensity.

[0054] In one embodiment, the hyperbolic tangent function, as a compression function, possesses ideal characteristics. When the input value is small, the function is approximately linear, preserving the details of low-energy signals; when the input value is large, the function tends to saturate, limiting the amplitude of high-energy signals. This non-linear mapping prevents signal overflow while preserving the natural dynamics of speech. The compressed signal maintains intelligibility while avoiding sudden volume changes during playback, achieving a comfortable listening experience.

[0055] S106. Post-processing optimization is performed on the time-domain speech waveform data to eliminate discontinuous jumps between waveforms, determine the multilingual speech synthesis output, and optimize memory usage efficiency according to resource allocation to obtain a lightweight speech synthesis result that meets real-time requirements.

[0056] The time-domain speech waveform data is analyzed by a feature fusion module to identify the connection positions of adjacent phoneme segments. The mean difference of sampled values ​​within a preset window before and after the connection point is calculated. If the mean difference exceeds a smoothing threshold, linear interpolation is applied in the connection region to generate a sequence of transitional sampled points, replacing the sampled values ​​at the original abrupt change positions, thus obtaining a preliminary optimized waveform that eliminates discontinuous jumps. Based on the preliminary optimized waveform, short-time energy values ​​at each moment are calculated using a sliding window. The energy ratio of the windows before and after the boundary is compared at the phoneme boundary. If the ratio exceeds the abrupt change detection threshold, a half-cosine window function is applied in the boundary region to weight and adjust the waveform amplitude, achieving a gradual transition of energy and generating a smooth multilingual speech waveform. According to the feature distribution of the multilingual speech waveform, the processing parameters of subsequent speech segments are dynamically adjusted. Speech segments with similar features are merged and processed through a batch caching mechanism, reusing intermediate calculation results. At the same time, the cache occupancy rate is monitored. If it exceeds the memory capacity threshold, a circular reuse mechanism is used to release the early cache, maintaining the continuity of the processing flow and obtaining a lightweight speech synthesis result that meets real-time requirements.

[0057] Specifically, the feature fusion module plays a crucial role in speech post-processing, its main task being to identify and repair unnatural phenomena caused by phoneme splicing. The identification of phoneme connection points relies on speech segmentation information, and these points often exhibit waveform discontinuities. By setting analysis windows before and after the connection points and calculating the statistical characteristics of the sampled values ​​within the windows, the degree of discontinuity can be quantified.

[0058] In one possible implementation, the calculation of the mean difference reflects the overall offset of the signals on both sides of the connection point. When a high-energy phoneme is directly connected to a low-energy phoneme, an obvious step will appear at the connection point. The preset window usually selects a duration of 5 - 10 milliseconds, which contains enough sampling points to characterize local features. Linear interpolation creates a smooth transition path in the connection area, and the new sampling values are distributed between the original values at both ends according to a linear ratio, effectively eliminating abrupt jumps.

[0059] It should be noted that short-time energy calculation is a basic technology in speech signal processing. By summing the squares of all sampling points within a window, the energy value of that period can be obtained. The sliding window moves on the entire waveform with a fixed step size, generating a trajectory of energy changing over time. This energy trajectory clearly shows the pattern of the strength change of speech, and phoneme boundaries usually correspond to points of rapid energy change.

[0060] Specifically, the half-cosine window function has ideal characteristics in energy transition processing. This function smoothly rises from 0 to 1 or falls from 1 to 0 at both ends of the window, and remains stable in the middle part. When applied to the phoneme boundary area, the waveform of the previous phoneme gradually decays according to a decreasing cosine curve, and the waveform of the next phoneme gradually increases according to an increasing cosine curve. This gradual change processing eliminates the popping or clicking sounds caused by sudden energy changes.

[0061] Exemplarily, when processing a Chinese-English mixed speech like "你好world", there are significant acoustic feature differences between the Chinese character "好" and the English word "world". Through feature distribution analysis, typical patterns of different language segments can be identified. The batch caching mechanism classifies speech segments with similar Mel-spectrum features. For example, all Chinese tonal phonemes share certain intermediate calculation results. When processing a new Chinese phoneme, the cached feature transformation matrix can be directly reused, avoiding repeated calculations.

[0062] In one embodiment, the cyclic reuse mechanism manages the cache space using a first-in, first-out strategy. The speech segment data processed earlier is retained in the cache for the longest time. When a new processing request arrives and the cache is approaching saturation, the earliest cache entry is released. This mechanism ensures that the most recently and frequently used data is preferentially retained, while maintaining the continuous operation ability of the system. By dynamically monitoring the cache occupancy rate and processing delay, the system can find the best balance between quality and efficiency, achieving real-time lightweight speech synthesis.

[0063] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A real-time multilingual TTS synthesis method based on deep learning, characterized in that, The method includes: acquiring multilingual text input; performing word embedding encoding using a pre-trained language model; extracting semantic features of the text sequence using a multi-head attention mechanism; activating the feature extraction branch of the corresponding language based on language identifiers to generate a semantic feature vector; performing sequence modeling of the semantic feature vector using a recurrent neural network; controlling information flow using a triple gating mechanism; adjusting prosodic control parameters based on intonation changes to generate a sequence encoding result; wherein the semantic feature vector is sequence modeled using a long short-term memory network, input gate control values ​​are calculated, and candidate memory unit states are generated; based on the candidate memory unit states and the previous memory unit states, a forget gate control value is calculated, multiplied by the previous memory unit states, to generate the current memory state. The unit state is determined by processing the semantic feature vector through the output gate weight matrix to generate output gate control values, adjusting prosodic control parameters based on the fundamental frequency component, and generating a hidden state vector. The sequence encoding result is weighted using an attention weight allocation mechanism, and key prosodic nodes are determined using a soft attention calculation method to generate a key time point distribution. Based on the key time point distribution, Mel spectral features are generated through an acoustic modeling network, local acoustic patterns are extracted, acoustic parameters are adjusted based on language switching boundaries, and a spectrum generation result is generated. A lightweight vocoder is used to perform time-domain signal conversion on the spectrum generation result to generate time-domain speech waveform data. The time-domain speech waveform data is post-processed through a feature fusion module to generate multilingual speech synthesis output.

2. The deep learning-based multilingual TTS real-time synthesis method according to claim 1, characterized in that, The process of acquiring multilingual text input, performing word embedding encoding using a pre-trained language model, extracting semantic features from the text sequence using a multi-head attention mechanism, activating the feature extraction branch for the corresponding language based on the language identifier, and generating a semantic feature vector includes: performing word embedding encoding on each word in the multilingual text input using a pre-trained language model to generate an initial word vector, adding positional encoding to generate an initial embedding representation; calculating the attention weights of each word in the initial embedding representation using a multi-head attention mechanism, extracting semantic dependencies through weighted summation to generate an intermediate semantic representation; calculating the language category probability distribution of the intermediate semantic representation using a softmax classifier, determining the language identifier based on the maximum probability, and activating the feature extraction branch for the corresponding language; performing a linear transformation on the intermediate semantic representation through the feature extraction branch to extract language-specific grammatical structures and lexical collocation features, and performing a non-linear mapping using the ReLU activation function to generate a semantic feature vector.

3. The real-time multilingual TTS synthesis method based on deep learning according to claim 1, characterized in that, The step of adjusting prosodic control parameters based on fundamental frequency components to generate a hidden state vector includes: extracting the fundamental frequency components of the semantic feature vector, calculating the difference between the fundamental frequency at the current time and the fundamental frequency at the previous time, adjusting the component values ​​in the prosodic control parameter vector by comparing the difference with a preset threshold, and weighting the output gate control value and multiplying it with the value of the current memory unit state after mapping by the tanh function to generate a hidden state vector.

4. The real-time multilingual TTS synthesis method based on deep learning according to claim 1, characterized in that, The step of weighting the sequence encoding results through an attention weight allocation mechanism, determining key prosodic nodes using a soft attention calculation method, and generating a key time point distribution includes: calculating self-attention weights for each position vector of the sequence encoding results; generating attention weight values ​​by dividing the dot product of the query, key, and value by the square root of the feature dimension and normalizing it using softmax; weighting the value vectors using the attention weight values ​​to generate a weighted feature representation; constructing a weight sequence of the attention weight values; extracting subsequences using a sliding window; marking a local peak candidate point if the weight value at the center of the window is greater than other positions; comparing the attention weight values ​​of the local peak candidate points with a preset prosodic threshold; marking a key prosodic node if it exceeds the threshold; recording the time position index; and generating a key time point distribution arranged in chronological order.

5. The real-time multilingual TTS synthesis method based on deep learning according to claim 1, characterized in that, The step of using a lightweight vocoder to perform time-domain signal conversion on the spectrum generation result to generate time-domain speech waveform data includes: iteratively estimating the phase component of the spectrum generation result, and generating a complex spectrum representation by combining the amplitude spectrum; performing an inverse fast Fourier transform on the complex spectrum representation to generate a time-domain sampling point sequence, and generating a time-domain waveform sequence by weighting the overlapping of frame shift positions.

6. The real-time multilingual TTS synthesis method based on deep learning according to claim 1, characterized in that, The step of post-processing the time-domain speech waveform data through the feature fusion module to generate multilingual speech synthesis output includes: analyzing the time-domain speech waveform data through the feature fusion module to identify the connection positions of adjacent phoneme segments, eliminating discontinuous jumps based on the connection positions of adjacent phoneme segments, and obtaining a preliminary optimized waveform; based on the preliminary optimized waveform, calculating the short-time energy value at each moment through a sliding window, comparing the energy ratio of the windows before and after the boundary at the phoneme boundary position, and if the energy ratio exceeds the abrupt change detection threshold, then weighting the waveform amplitude in the boundary region to achieve a gradual transition of energy and generate multilingual speech synthesis output.

Images