Multi-language TTS real-time synthesis method based on deep learning

Abstract

The invention discloses a multi-language TTS real-time synthesis method based on deep learning. According to the method, through a deep neural network model, high-quality real-time conversion from a multi-language text to voice is realized. The method comprises the following steps: firstly, constructing a multi-language acoustic feature library and a pronunciation rule library, and extracting text semantic features by adopting an end-to-end neural network architecture; and then, an improved attention mechanism is utilized to realize accurate synthesis of voice rhythm and pronunciation, and naturalness and coherence of multi-language pronunciation are ensured. Meanwhile, a lightweight reasoning engine is designed, calculation resource allocation is optimized, and the real-time synthesis requirement is met. Compared with a traditional TTS method, the method has the advantages that the fluency and naturalness of multilingual speech synthesis are remarkably improved, the calculation delay is greatly reduced, and efficient and reliable technical support is provided for intelligent speech interaction and multilingual content production.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a multilingual TTS real-time synthesis method based on deep learning. Background Art

[0002] Multilingual speech synthesis, a core technology in artificial intelligence and speech processing, plays a vital role in applications such as global information exchange, intelligent assistants, and accessible communications. Text-to-speech (TTS), which automatically converts written text into spoken output, is gaining popularity in applications such as intelligent voice announcements and risk disclosure. Using deep learning methods to achieve text-to-speech conversion, and supporting multiple languages ​​and capable of real-time speech generation, this technology, which accurately and naturally synthesizes speech in multiple languages, has become a key driver for improving business efficiency and ensuring compliance. Current mainstream multilingual speech synthesis methods suffer from significant shortcomings. Traditional statistical parameter-based synthesis methods have limited expressiveness when processing the complex speech features of different languages ​​and struggle to capture subtle differences between them. While existing neural network methods have made progress in single-language scenarios, they often suffer from unstable speech quality and inaccurate pronunciation in multilingual scenarios. Furthermore, these methods are computationally complex in practical deployments, making them difficult to meet the performance requirements of real-time applications. The technical challenges facing multilingual speech synthesis primarily stem from the complexity of speech feature extraction and semantic understanding. Different languages ​​have unique acoustic characteristics and pronunciation patterns. A single feature extraction method cannot effectively cover the speech representation needs of multiple languages, which directly affects the accurate conversion of text semantic information into speech signals. Insufficient semantic feature extraction further exacerbates the difficulty of speech prosody control. Traditional sequence modeling methods are prone to attention distraction and information loss when processing long-sequence speech generation, resulting in poor performance of synthesized speech in terms of rhythmic naturalness and pronunciation coherence. Therefore, how to build a system that can accurately extract multilingual semantic features, precisely control speech prosody generation, and lightweight and real-time process multilingual speech synthesis while ensuring synthesis quality has become a key issue that needs to be urgently addressed in this field. Summary of the Invention

[0003] The present invention provides a multilingual TTS real-time synthesis method based on deep learning, which mainly includes:

[0004] After obtaining multilingual text input, it uses a pre-trained language model to perform word embedding encoding, deeply extracts semantic features from the text sequence, activates the feature extraction branch of the corresponding language based on the detected language identifier, and generates a semantic feature vector containing language-specific information;

[0005] Sequence modeling is performed using a recurrent neural network based on the semantic feature vectors, and the direction of information flow is controlled by a gating mechanism. If there is a change in intonation between the semantic features at the current moment and the previous moment, the prosody control parameters are updated to obtain a sequence encoding result that integrates temporal dependencies;

[0006] The sequence encoding results are weighted by an attention weight distribution mechanism to assign importance weights to different positions, and key rhythmic nodes are determined by the weights to obtain the distribution of key time points for rhythmic control;

[0007] An acoustic modeling network is used to generate Mel-spectrogram features based on the distribution of the key time points, and a convolutional neural network is used to extract local acoustic patterns to obtain a spectrum generation result that adapts to multilingual characteristics;

[0008] Performing time domain signal conversion on the spectrum generation result, mapping the frequency domain information to the time domain space through fast Fourier transform, and obtaining time domain speech waveform data;

[0009] The time-domain speech waveform data is post-processed and optimized to eliminate discontinuous jumps between waveforms, determine multilingual speech synthesis output, and optimize memory usage efficiency based on resource allocation to obtain a lightweight speech synthesis result that meets real-time requirements.

[0010] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0011] This invention discloses a multilingual TTS real-time synthesis method based on deep learning. It extracts the semantic features of multilingual text using a pre-trained language model and a multi-head attention mechanism. A recurrent neural network is used to establish a sequence model and integrate temporal dependencies. Attention weight distribution is used to determine key time points for prosody control. An acoustic modeling network is used to generate spectral features adapted to multilingual characteristics. These features are then converted into time-domain speech waveforms using a lightweight vocoder algorithm. Finally, post-processing optimization is performed to produce continuous and natural multilingual speech output. This invention can effectively process mixed multilingual text, achieving high-quality, language-independent speech synthesis. It also ensures real-time performance through technologies such as dynamic batch processing and model compression, providing an efficient and reliable solution for multilingual speech interaction systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a flowchart of a multilingual TTS real-time synthesis method based on deep learning of the present invention. DETAILED DESCRIPTION

[0013] The technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0014] like Figure 1 In this embodiment, a multi-language TTS real-time synthesis method based on deep learning may specifically include:

[0015] S101. After obtaining multilingual text input, perform word embedding encoding through a pre-trained language model, deeply extract semantic features in the text sequence, activate the feature extraction branch of the corresponding language based on the detected language identifier, and generate a semantic feature vector containing language-specific information.

[0016] After obtaining multilingual text input, each word in the text is word-embedded using a pre-trained language model to generate an initial word vector representation, and a positional encoding is added based on the word's position in the sentence to obtain an initial embedding representation that incorporates position information. A multi-head attention mechanism is used to calculate the attention weights between each word and other words in the initial embedding representation, and the word vectors are weighted and summed using the attention weights to extract the semantic dependencies in the text sequence, thereby obtaining an intermediate semantic representation containing contextual information. The probability distribution of each language category is calculated for the intermediate semantic representation using a softmax classifier, a language identifier is determined based on the maximum probability value, and a feature extraction branch for the corresponding language is activated based on the language identifier. The feature extraction branch is a fully connected network layer pre-trained for a specific language. The intermediate semantic representation is linearly transformed using the activated language-specific feature extraction branch to extract the grammatical structure patterns and vocabulary collocation rules of the language, and nonlinear mapping is performed using the ReLU activation function to generate a semantic feature vector containing language-specific information.

[0017] Specifically, in multilingual text processing, word embedding is fundamental to understanding text semantics. Pre-trained language models learn from large-scale corpora to obtain vector representations of word units, with each word unit mapped to a point in a high-dimensional space. Positional encoding addresses the issue of missing word order information. Using sine and cosine functions, a unique encoding vector is generated for each position, allowing the same word unit to have different representations at different positions.

[0018] In one possible implementation, the initial embedding representation is processed through a multi-head attention mechanism to capture dependencies between tokens. The attention weights are calculated by performing a dot product operation on the query, key, and value vectors. The strength of the association between each pair of tokens is determined by using a scaled dot product attention formula. Multiple attention heads process different semantic subspaces in parallel, extracting features from multiple dimensions—lexical, syntactic, and semantic—and concatenating them to form a rich intermediate semantic representation.

[0019] It should be noted that language identification uses a softmax classifier to calculate a probability distribution. This classifier maps the intermediate semantic representation to a language category space, with each dimension corresponding to a probability value for a language. The language with the highest probability is the identification result. This probabilistic approach can handle the potential for language mixing in text and provide reliable language identifiers.

[0020] Specifically, language-specific feature extraction branches are specialized network layers designed for different languages. The English branch might focus more on root and affix variations, the Chinese branch emphasizes character combinations and contextual relationships, and the Arabic branch needs to process right-to-left writing order. Each branch consists of a multi-layer fully connected network with weight parameters pre-trained on the corresponding language's corpus, capable of extracting unique grammatical patterns.

[0021] For example, when processing a German text containing technical terms, the German feature extraction branch identifies compound word structures and breaks down long compound words into morphemes for comprehension. The linear transformation matrix adjusts feature weights according to German grammatical rules, emphasizing grammatical markers such as noun gender and verb tense. The ReLU activation function introduces nonlinearity, enabling the network to learn complex language pattern mappings.

[0022] In one embodiment, the resulting semantic feature vector contains a fusion representation of the original semantic information and language-specific features. Each dimension of the vector encodes a specific linguistic attribute, such as part-of-speech distribution, syntactic structure, and semantic role.

[0023] S102. Perform sequence modeling through a recurrent neural network based on the semantic feature vector, control the direction of information flow through a gating mechanism, and update the prosody control parameters if there is a change in intonation between the semantic feature at the current moment and the previous moment, to obtain a sequence encoding result that integrates temporal dependencies.

[0024] According to the semantic feature vector, the features of each moment are sequentially modeled through the long short-term memory network, the matrix product of the semantic feature vector at the current moment and the input weight matrix is ​​calculated, and the input gate control value is obtained after processing with the sigmoid activation function to determine the degree of reception of new information and generate the candidate memory unit state. Based on the candidate memory unit state and the memory unit state at the previous moment, the forget gate weight matrix is ​​multiplied with the current semantic feature vector and activated by the sigmoid function to obtain the forget gate control value, which is multiplied with the memory unit state at the previous moment to achieve selective retention of historical information, and then added to the candidate state after input gate adjustment to obtain the memory unit state at the current moment. According to the memory unit state at the current moment, the current semantic feature vector is processed by the output gate weight matrix, and the output gate control value is obtained after sigmoid activation. At the same time, the fundamental frequency component is extracted from the semantic feature vector. If the difference between the fundamental frequency component and the fundamental frequency at the previous moment exceeds a preset threshold, the corresponding component value in the rhythm control parameter vector is adjusted. The rhythm control parameter vector is used to perform weighted adjustment on the output gate control value, and multiplied by the value of the memory unit state at the current moment after being mapped by the tanh function, to obtain a hidden state vector containing timing dependency information and rhythm modulation information as a sequence encoding result that integrates the timing dependency relationship.

[0025] Specifically, long-short-term memory networks (LSTMs) play a key role in sequence modeling. Their core lies in precisely controlling the flow of information through a gating mechanism. The semantic feature vector, taken as input, is first matrix-multiplied with the input weight matrix. This process essentially performs a linear transformation on the features, mapping the original feature space into a gated space. The application of the sigmoid activation function constrains the output value between 0 and 1, forming a control signal similar to a valve.

[0026] In one possible implementation, the input gate control value determines the extent to which new information is accepted. When processing a speech-to-text sequence, if a significant semantic transition occurs at the current moment, the input gate generates a control value close to 1, fully accepting the new semantic information. Conversely, for redundant or noisy information, the input gate generates a smaller control value, limiting its entry into the memory cell. The candidate memory cell state contains the filtered new information, laying the foundation for subsequent information fusion.

[0027] It's important to note that the forget gate design addresses the vanishing gradient problem in long sequence processing. The forget gate generates a selective forgetting signal by multiplying the current input with a specialized forget weight matrix. This mechanism is particularly important when processing long conversations or long texts, automatically identifying and retaining key contextual information while downplaying minor details. The previous moment's memory cell state is multiplied by the forget gate control value, enabling dynamic filtering of historical information.

[0028] Specifically, extracting the fundamental frequency component involves frequency-domain analysis of the semantic feature vector. The fundamental frequency of a speech signal reflects the speaker's pitch variations and is crucial for emotion recognition and intonation analysis. Using fast Fourier transforms or autocorrelation analysis, the fundamental frequency component can be isolated from the feature vector. When a sudden change in the fundamental frequency is detected, the system identifies it as a possible sign of doubt, exclamation, or emotional shift, triggering an update of the prosody control parameters.

[0029] For example, the prosodic control parameter vector contains multiple dimensions, each corresponding to a different prosodic feature. The pitch change rate reflects the upward or downward trend of intonation, the pause duration encodes information about speech rhythm, and the stress pattern marks the location of emphasis. These parameters are weighted to adjust the control value of the output gate, ensuring that the final hidden state vector not only contains semantic information but also incorporates rich prosodic features.

[0030] In one embodiment, the tanh function maps the memory cell state to a range of -1 to 1, enhancing the network's expressive power. The rhythmically modulated output gate control value is multiplied by the memory state mapped by the tanh function to produce a sequence encoding result.

[0031] S103 , weighting the sequence encoding results through an attention weight distribution mechanism, assigning importance weights to different positions, determining key rhythmic nodes through the weights, and obtaining the distribution of key time points for rhythmic control.

[0032] An attention weight allocation mechanism calculates a self-attention weight for each position vector of the sequence encoding result. The position vector is used as the query, key, and value. The dot product of the query and key is calculated and divided by the square root of the feature dimension. After softmax normalization, an attention weight value is obtained for each position. The value vector at the corresponding position is weighted using this attention weight value to obtain a weighted feature representation for each position and its corresponding attention weight value. A weight sequence is constructed based on the attention weight values ​​of each position. A sliding window of a preset window width is applied to this weight sequence. A subsequence of weight values ​​within each window is extracted. If the weight value at the center of the window is greater than the weight values ​​of all other positions in the window, the center position is determined to be a local peak candidate. The attention weight value of each local peak candidate is compared with a preset prosodic threshold. If the weight value exceeds the threshold, the candidate is marked as a key prosodic node. Its time position index and weight value are recorded. The time position indexes of all key prosodic nodes are summarized to generate a chronological distribution of key time points.

[0033] Specifically, the self-attention mechanism plays a core role in prosodic analysis. Its essence is to enable each position in the sequence to pay attention to all other positions, thereby capturing long-range dependencies. Each position vector in the sequence encoding result contains semantic and prosodic information at that moment. By treating it as a query, key, and value simultaneously, self-association of information is achieved.

[0034] In one possible implementation, the dot product operation measures the similarity between different locations. When the vectors at two locations have similar directions in high-dimensional space, the dot product value is large, indicating a strong semantic or prosodic connection between the two locations. Dividing by the square root of the feature dimension is a scaling technique that prevents excessive dot product values ​​in high dimensions from saturating the softmax function and maintains stable gradient propagation.

[0035] It's important to note that softmax normalization converts the raw similarity scores into a probability distribution, so that the sum of all attention weights at each position is 1. This normalization allows the attention mechanism to adaptively distribute attention, which is particularly important for prosodic analysis. When processing recitation audio, rhyme positions tend to receive higher attention weights, while transition words receive relatively lower weights.

[0036] Specifically, the application of sliding window technology takes into account the local characteristics of rhythm. Rhythmic rhythms are often characterized by localized alternations of strength and weakness, rather than a globally uniform distribution. The selection of the preset window width requires a balance between locality and continuity. A narrow window width may miss important rhythmic transitions, while a wide window width may confuse different rhythmic units. The comparison operation within the window identifies the most prominent local location by comparing the weights of the central location with those of the surrounding locations.

[0037] For example, when processing a speech recording, local peaks often correspond to stressed syllables or turning points in intonation. When the speaker emphasizes a key word, the attention weight at that location becomes significantly higher than that at preceding and following locations, forming a local peak. This peak detection mechanism automatically locates the key points of emphasis in speech, providing important reference for subsequent speech synthesis or sentiment analysis.

[0038] In one embodiment, the setting of the prosodic threshold needs to take into account the differences between different languages ​​and speaking styles. Chinese has rich tonal variations, and a lower threshold may be required to capture subtle prosodic changes; while English relies more on stress patterns, and a higher threshold may be required to screen out the true stress positions. Positions exceeding the threshold are marked as key prosodic nodes, which together constitute the skeleton structure of the prosody. The generation of the key time point distribution is achieved by summarizing the time indexes of all prosodic nodes. This distribution not only records the position of the prosodic emphasis, but also preserves their temporal relationship. In speech synthesis applications, these key time points can guide prosodic modeling, apply appropriate pitch changes and duration adjustments at the corresponding positions, and make the synthesized speech more natural and fluent.

[0039] S104: Generate mel-spectrogram features using an acoustic modeling network according to the key time point distribution, extract local acoustic patterns using a convolutional neural network, and obtain a spectrum generation result adapted to multilingual characteristics.

[0040] Based on the distribution of key time points, a time index sequence and corresponding prosodic markers are input into an acoustic modeling network. An embedding layer converts the discrete time indexes into continuous vector representations. Basic acoustic parameters at each moment are mapped through a three-layer fully connected network. Mel-spectrographic feature sequences are then transformed through a Mel filter bank transform. This Mel-spectrographic feature sequence is processed using a convolutional neural network, using convolution kernels of different sizes to extract short-term and long-term time-frequency patterns. The convolution operation captures the harmonic structure between frequency bands and temporal transition characteristics. Pooling is performed to obtain a multi-scale local acoustic pattern representation. Based on this multi-scale local acoustic pattern representation, the Euclidean distance between adjacent time frames is calculated and compared with the average distance of the previous frame. If the current distance exceeds a preset multiple of the average distance, it is identified as a potential language switch point. The language type transition is determined by comparing the change in the spectral envelope slope before and after the switch point. Based on the language type conversion results, the acoustic parameter set corresponding to the target language is retrieved, including the language-specific fundamental frequency range and formant distribution template. These parameters are applied to the Mel-spectrographic features after the switch point. Parameter interpolation is used to smooth the transition region, generating a spectrum generation result that adapts to multilingual characteristics.

[0041] Specifically, the core function of the acoustic modeling network is to transform abstract temporal information into a concrete acoustic representation. Each time index in the key time point distribution represents a rhythmic turning point. These discrete values ​​need to be mapped into a continuous vector space through an embedding layer. The embedding process is similar to a table lookup operation, with each time position corresponding to a learnable vector. This representation captures the relative relationship between time positions and their prosodic importance.

[0042] In one possible implementation, the design of a three-layer fully connected network follows the principle of layer-by-layer feature abstraction. The first layer expands the embedding vector to higher dimensions, increasing its expressive power. The second layer performs nonlinear transformations to learn the complex mapping between time and acoustic features. The third layer compresses the high-dimensional features to the mel-spectrogram dimension. The application of the mel-spectrogram filter bank is based on the characteristics of human hearing, with a dense distribution in low frequencies and a sparse distribution in high frequencies to better simulate human auditory perception.

[0043] It's important to note that convolutional neural networks have unique advantages when processing mel-spectrograms. Convolution kernels of different sizes perform different feature extraction tasks: small kernels capture local spectral details, such as the precise location of formants; large kernels extract the global spectral envelope, reflecting the overall timbre characteristics. During the extraction of time-frequency patterns, the convolution operation can identify harmonic structure, namely the regular relationship between the fundamental frequency and its integer multiple frequency components.

[0044] Specifically, calculating the Euclidean distance involves a holistic comparison of multidimensional feature vectors. When processing mixed English and Chinese speech, the switching point from English to Chinese often manifests as a significant change in the spectral envelope. The English spectrum typically has stronger energy in the mid- and high-frequency regions, while the tonal variations in Chinese lead to a more complex fundamental frequency trajectory. By calculating the difference in feature vectors between adjacent frames and comparing it with a historical average, the boundaries of these language transitions can be effectively identified.

[0045] For example, changes in the slope of the spectral envelope reflect the pronunciation characteristics of different languages. Chinese's four-tone system causes the fundamental frequency to fluctuate significantly over a short period of time, resulting in a dynamic spectral envelope. In contrast, English relies more on slow changes in stress and intonation, resulting in a relatively stable spectral envelope. By analyzing the difference in slope before and after the switching point, it is possible to determine whether the language is switching from a tonal to a stressed language, or vice versa.

[0046] In one embodiment, the selection of the acoustic parameter set directly affects the quality of the final spectrum generation. The fundamental frequency range parameter determines the pitch variation space. The fundamental frequency range of Chinese male voices is usually 80-180 Hz, while that of English may be slightly higher. The resonance peak distribution template records the vowel space characteristics of different languages. The vowel system of Chinese is relatively simple, and the resonance peak position is relatively concentrated; English has rich vowels, and the resonance peak distribution is more dispersed. Parameter interpolation technology plays a key role in the transition area of ​​language switching. Through linear or nonlinear interpolation methods, the spectrum characteristics are smoothly transitioned to avoid abrupt jumps, and a natural and smooth multi-language spectrum is generated.

[0047] S105 , performing time domain signal conversion on the spectrum generation result, mapping the frequency domain information to the time domain space through fast Fourier transform, and obtaining time domain speech waveform data.

[0048] A lightweight vocoder is used to process the spectrum generation result, and the phase component is iteratively estimated from the spectrum amplitude information using the Griffin-Lim algorithm. The estimated phase is combined with the original amplitude spectrum to form a complex spectrum representation, thereby obtaining a complete frequency domain signal containing amplitude and phase information. An inverse fast Fourier transform is performed on the complete frequency domain signal, and the frequency domain complex sequence of each frame is converted into a corresponding time domain sampling point sequence. The overlapping parts of the frame shift positions are weighted and superimposed to eliminate the discontinuity of the frame boundary and generate a smoothly transitioned time domain waveform sequence. The root mean square energy value in the sliding window is calculated based on the time domain waveform sequence. If the energy ratio of adjacent windows exceeds the preset dynamic range threshold, the high energy segment is amplitude compressed using a hyperbolic tangent function to maintain the relative dynamic characteristics of the signal and obtain continuous time domain speech waveform data.

[0049] Specifically, the core task of a lightweight vocoder is to reconstruct the time-domain speech signal from the spectral information. The key challenge in this process lies in recovering the phase information. The spectrum generated typically only contains amplitude information, while complete signal reconstruction requires the phase component. The Griffin-Lim algorithm addresses this problem through iterative optimization. Its fundamental principle is to use consistency constraints in the time-frequency domain to estimate the most reasonable phase distribution.

[0050] In one possible implementation, the algorithm first assigns a random initial phase to the magnitude spectrum, forming an initial complex spectrum. After obtaining the time-domain signal through an inverse Fourier transform, a forward Fourier transform is performed, preserving the original magnitude spectrum but updating the phase information. This iterative process is repeated, with each iteration bringing the phase estimate closer to the true value. After multiple iterations, the phase gradually converges to a stable state, and the complex spectrum at this point can reconstruct a natural-sounding speech signal.

[0051] It's important to note that the inverse fast Fourier transform maps the complex number representation in the frequency domain back to the time domain sampling points. Each frequency domain complex number contains the amplitude and phase information of that frequency component. Through the inverse transform, these frequency components are superimposed in the time domain to form a complete waveform. However, framing can lead to discontinuities at frame boundaries. Adjacent frames may experience phase jumps or amplitude abrupt changes at their boundaries, and direct splicing can produce noticeable clicks or distortion.

[0052] Specifically, the overlap overlay technique addresses this issue by creating an overlapping region between adjacent frames. Assuming a frame length of 512 samples and a frame shift of 256 samples, there is a 50% overlap between two adjacent frames. In this overlapping region, the signal of the preceding frame gradually attenuates while the signal of the succeeding frame gradually increases. The two are then weighted together according to a specific window function. This gradual transition ensures signal continuity and eliminates abrupt changes at frame boundaries.

[0053] For example, the RMS energy calculation within a sliding window reflects the local strength of the signal. When processing mixed speech containing whispers and normal speech, the energy difference can reach tens of times. Whispered segments have very low energy values, while normal speech segments have higher energy values. This huge dynamic range can create difficulties in subsequent processing and playback. By calculating the mean of the sum of squares of all sampling points within each window and then taking the square root, the RMS energy value for that window can accurately represent the local signal strength.

[0054] In one embodiment, the hyperbolic tangent function exhibits ideal properties as a compression function. When the input value is small, the function becomes approximately linear, preserving the details of low-energy signals. When the input value is large, the function tends to saturate, limiting the amplitude of high-energy signals. This nonlinear mapping prevents signal overflow while preserving the natural dynamics of speech. The compressed signal maintains intelligibility while avoiding sudden volume changes during playback, achieving a comfortable listening experience.

[0055] S106 , performing post-processing optimization on the time-domain speech waveform data to eliminate discontinuous jumps between waveforms, determining a multilingual speech synthesis output, and optimizing memory usage efficiency based on resource allocation to obtain a lightweight speech synthesis result that meets real-time requirements.

[0056] The feature fusion module analyzes the time-domain speech waveform data, identifies the connection locations of adjacent phoneme segments, and calculates the mean difference between the sampling values ​​within a preset window before and after the connection point. If the mean difference exceeds a smoothing threshold, linear interpolation is applied to the connection region to generate a sequence of transition sampling points, replacing the sampling values ​​at the original mutation location, to obtain a preliminary optimized waveform that eliminates discontinuous jumps. Based on this preliminary optimized waveform, a sliding window is used to calculate the short-term energy value at each moment. At the phoneme boundary, the energy ratio of the window before and after the boundary is compared. If the ratio exceeds the mutation detection threshold, a half-cosine window function is applied to the boundary region to perform weighted adjustment on the waveform amplitude, achieving a gradual energy transition and generating a smooth multilingual speech waveform. Based on the feature distribution of the multilingual speech waveform, the processing parameters of subsequent speech segments are dynamically adjusted. Speech segments with similar features are merged and processed using a batch caching mechanism, and intermediate calculation results are reused. The cache occupancy rate is monitored. If the memory capacity threshold is exceeded, a round-robin mechanism is used to release the earlier caches, maintaining the continuity of the processing flow and obtaining lightweight speech synthesis results that meet real-time requirements.

[0057] Specifically, the feature fusion module plays a key role in speech post-processing, primarily identifying and repairing artifacts caused by phoneme splicing. Identifying phoneme join locations relies on speech segmentation information, and these locations often exhibit waveform discontinuities. By setting an analysis window before and after the join point and calculating the statistical features of the samples within the window, the degree of discontinuity can be quantified.

[0058] In a possible implementation, the calculation of the mean difference reflects the overall offset of the signals on both sides of the connection point. When a high-energy phoneme is directly connected to a low-energy phoneme, an obvious step will appear at the connection point. The preset window usually selects a duration of 5 - 10 milliseconds, which contains enough sampling points to characterize local features. Linear interpolation creates a smooth transition path in the connection area, and the new sampling values are distributed between the original values at both ends according to a linear ratio, effectively eliminating abrupt jumps.

[0059] It should be noted that short-time energy calculation is a basic technology in speech signal processing. By summing the squares of all sampling points within the window, the energy value of this period can be obtained. The sliding window moves on the entire waveform with a fixed step size, generating a trajectory of energy changing over time. This energy trajectory clearly shows the pattern of the strength change of speech, and phoneme boundaries usually correspond to points where the energy changes rapidly.

[0060] Specifically, the half-cosine window function has ideal characteristics in energy transition processing. This function smoothly rises from 0 to 1 or falls from 1 to 0 at both ends of the window, and remains stable in the middle part. When applied to the phoneme boundary area, the waveform of the previous phoneme gradually decays according to a descending cosine curve, and the waveform of the next phoneme gradually increases according to an ascending cosine curve. This gradual change processing eliminates the popping or clicking sounds caused by sudden energy changes.

[0061] Exemplarily, when processing a Chinese-English mixed speech like "你好world", there are significant acoustic feature differences between the Chinese character "好" and the English word "world". Through feature distribution analysis, typical patterns of different language segments can be identified. The batch caching mechanism classifies speech segments with similar Mel-spectrum features. For example, all Chinese tonal phonemes share some intermediate calculation results. When processing a new Chinese phoneme, the cached feature transformation matrix can be directly reused to avoid repeated calculations.

[0062] In one embodiment, the cyclic reuse mechanism manages the cache space using a first-in-first-out strategy. The speech segment data processed earlier remains in the cache for the longest time. When a new processing request arrives and the cache is approaching saturation, the earliest cache entry is released. This mechanism ensures that the most recently and frequently used data is preferentially retained, while maintaining the continuous operation ability of the system. By dynamically monitoring the cache occupancy rate and processing delay, the system can find the best balance between quality and efficiency to achieve real-time lightweight speech synthesis.

[0063] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to specific embodiments. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multilingual TTS real-time synthesis method based on deep learning, characterized in that: The method comprises: Acquire multilingual text input, perform word embedding encoding through a pre-trained language model, use a multi-head attention mechanism to extract semantic features of the text sequence, activate the feature extraction branch of the corresponding language based on the language identifier, and generate a semantic feature vector; perform sequence modeling on the semantic feature vector through a recurrent neural network, use a triple gating mechanism to control information flow, adjust prosody control parameters based on intonation changes, and generate a sequence encoding result; perform weighted processing on the sequence encoding result through an attention weight allocation mechanism, use a soft attention calculation method to determine key prosody nodes, and generate a key time point distribution; based on the key time point distribution, generate Mel spectrum features through an acoustic modeling network, extract local acoustic patterns, adjust acoustic parameters based on language switching boundaries, and generate a spectrum generation result; use a lightweight vocoder to perform time domain signal conversion on the spectrum generation result to generate time domain speech waveform data; post-process the time domain speech waveform data through a feature fusion module to generate a multilingual speech synthesis output.

2. The multilingual TTS real-time synthesis method based on deep learning according to claim 1 is characterized in that: The method of obtaining multilingual text input, performing word embedding encoding using a pre-trained language model, extracting semantic features of the text sequence using a multi-head attention mechanism, activating the feature extraction branch of the corresponding language based on the language identifier, and generating a semantic feature vector includes: Each word unit of the multilingual text input is word-embedded using a pre-trained language model to generate an initial word vector, and position encoding is added to generate an initial embedding representation; a multi-head attention mechanism is used to calculate the attention weight of each word unit in the initial embedding representation, and semantic dependencies are extracted through weighted summation to generate an intermediate semantic representation; a softmax classifier is used to calculate the language category probability distribution of the intermediate semantic representation, a language identifier is determined based on the maximum probability, and a feature extraction branch for the corresponding language is activated; a linear transformation is performed on the intermediate semantic representation through the feature extraction branch to extract language-specific grammatical structure and vocabulary collocation features, and a ReLU activation function is used for nonlinear mapping to generate a semantic feature vector.

3. The multilingual TTS real-time synthesis method based on deep learning according to claim 1 is characterized in that: The semantic feature vector is sequentially modeled by a recurrent neural network and a triple gating mechanism is used to control the flow of information, including: The semantic feature vector is sequence modeled through a long short-term memory network, the input gate control value is calculated, and a candidate memory unit state is generated; based on the candidate memory unit state and the memory unit state at the previous moment, the forget gate control value is calculated, and multiplied by the memory unit state at the previous moment to generate the memory unit state at the current moment; the semantic feature vector is processed through an output gate weight matrix to generate an output gate control value, and the rhythm control parameter is adjusted based on the fundamental frequency component to generate a hidden state vector.

4. The multilingual TTS real-time synthesis method based on deep learning according to claim 3 is characterized in that: The step of adjusting the prosody control parameter based on the fundamental frequency component to generate a hidden state vector includes: Extract the fundamental frequency component of the semantic feature vector, calculate the difference between the fundamental frequency at the current moment and the fundamental frequency at the previous moment, and adjust the component value in the rhythm control parameter vector by comparing the difference with a preset threshold; perform weighted adjustment on the output gate control value, and multiply it with the value of the memory unit state at the current moment after mapping by the tanh function to generate a hidden state vector.

5. The multilingual TTS real-time synthesis method based on deep learning according to claim 1, characterized in that: The sequence encoding result is weighted by the attention weight distribution mechanism, the key rhythmic nodes are determined by the soft attention calculation method, and the key time point distribution is generated, including: A self-attention weight is calculated for each position vector of the sequence encoding result, and an attention weight value is generated based on the dot product of the query, key and value divided by the square root of the feature dimension and normalized by softmax. The value vector is weighted by the attention weight value to generate a weighted feature representation; a weight sequence of the attention weight values ​​is constructed, and a sliding window is used to extract a subsequence. If the weight value of the window center position is greater than that of other positions, it is marked as a local peak candidate point; the attention weight value of the local peak candidate point is compared with a preset rhythmic threshold. If it exceeds the threshold, it is marked as a key rhythmic node, the time position index is recorded, and a distribution of key time points arranged in chronological order is generated.

6. The multilingual TTS real-time synthesis method based on deep learning according to claim 1, characterized in that: The step of using a lightweight vocoder to perform time-domain signal conversion on the spectrum generation result to generate time-domain speech waveform data includes: Iteratively estimate the phase component of the spectrum generation result and generate a complex spectrum representation in combination with the amplitude spectrum; perform inverse fast Fourier transform on the complex spectrum representation to generate a time domain sampling point sequence, and generate a time domain waveform sequence through frame shift position overlap weighting.

7. The multilingual TTS real-time synthesis method based on deep learning according to claim 1, characterized in that: The post-processing of the time-domain speech waveform data by the feature fusion module to generate a multilingual speech synthesis output includes: Analyzing the time-domain speech waveform data through a feature fusion module to identify the connection positions of adjacent phoneme segments, eliminating discontinuous jumps based on the connection positions of adjacent phoneme segments, and obtaining a preliminary optimized waveform; Based on the preliminary optimized waveform, the short-term energy value at each moment is calculated through a sliding window. The energy ratio of the window before and after the phoneme boundary is compared at the phoneme boundary. If the energy ratio exceeds the mutation detection threshold, the waveform amplitude is weightedly adjusted in the boundary area to achieve a gradual transition of energy and generate multilingual speech synthesis output.

Images