A real-time speech synthesis method based on specific voice timbre

Abstract

This invention relates to a real-time speech synthesis method based on specific human voice timbre. The method is unique in that it features real-time synthesis, high voice realism, and low training sample requirements, optimizing model construction efficiency and performance. The implementation stages include: data preprocessing, data vectorization, model construction, model training, and model inference. The technologies involved in this invention include: a human voice timbre recognition model, a deep gated recurrent neural network, and an attention mechanism. The technical solution proposed in this invention enables the voice model to simultaneously possess human-like speech fluency and a highly human-like pronunciation style within limited computational performance and development cycle.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, and in particular to a real-time speech synthesis method based on the timbre of a specific person. Background Technology

[0002] Text-to-speech conversion technology is a core research area in the field of speech signal processing, and its research and application are a focus of attention for both academia and industry.

[0003] Text-to-speech (TTS) technology originated in the late 1970s. Initially, it was implemented by splicing sound waveform fragments based on language rules to synthesize human voice audio. In English speech synthesis, DECTalk, developed by MIT, can produce seven different timbres, maintaining clarity and naturalness even at a speaking speed of 350 words per minute. In Chinese speech synthesis, Matsushita Electronics of Japan, based on DECTalk technology, has also developed a system that can convert Chinese text into spoken audio. When using waveform splicing technology to synthesize speech, the acoustic characteristics of the original spliced ​​units are well preserved, leading to its widespread application in speech synthesis with limited vocabulary, such as bus announcements and domain-specific text-to-speech systems. However, this approach has drawbacks: it requires manual processing of large amounts of corpus data and the development of grammatical and pronunciation rules based on specific language knowledge. Furthermore, it is difficult to control the rhythm of the voice through context, resulting in a lower degree of naturalness in the generated human voice.

[0004] Meanwhile, parametric synthesis methods based on vocal tract models are also developing. Depending on the different ways of describing vocal tract characteristics, two technical routes have emerged: linear prediction synthesis and formant synthesis. With technological advancements, parametric speech synthesis technology based on statistical models has begun to emerge, with Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs) gaining prominence. These models enable the automatic training and construction of text-to-speech systems, and are therefore also known as trainable speech synthesis. The basic idea is to model the acoustic parameters of the input speech data and construct a corresponding synthesis system based on the trained statistical model.

[0005] Since the beginning of the 21st century, speech processing technology based on deep learning theory has achieved comprehensive breakthroughs, with significantly improved performance. Corresponding research has made key progress in areas such as model design refinement, parameter extraction and optimization, and system adaptive techniques. End-to-end deep learning speech synthesis technology is built upon the synthesis approach of traditional parameterized models. Theoretically, it can synthesize highly human-like speech. However, due to the excessively large number of parameters in deep learning models, a large amount of training data is required for model convergence, making deep learning solutions often difficult to implement practically. Summary of the Invention

[0006] The purpose of this invention is to provide a real-time speech synthesis method based on the timbre of a specific person. This method can generate a speech audio model that matches the timbre of a specific person in real time based on a given text, given a speech data sample of a specific person and limited computing resources. This method takes speech style quantization technology and speech synthesis technology as its starting point, and optimizes the efficiency and performance of the model based on the modeling method of the timbre of a specific person and the frequency domain information waveform reconstruction method in the speaker recognition technology.

[0007] To achieve the above objectives, the technical solution of the present invention is: a real-time speech synthesis method based on a specific person's timbre, comprising:

[0008] In the data preprocessing stage, the raw audio data is standardized.

[0009] In the data vectorization stage, Chinese text data is converted into Pinyin representation, then into the corresponding vector encoding sequence of Pinyin, and a bandpass filter bank is used to perform sliding window filtering on the audio corresponding to the text to obtain the spectrum of the audio.

[0010] In the model building phase, a neural network model for synthesizing the voice of a specific person is constructed. The model is implemented in three parts: timbre discrimination model, spectrum synthesis model, and audio reconstruction model.

[0011] During the model training phase, the timbre discrimination model, the spectrum synthesis model, and the audio reconstruction model were fitted respectively. The timbre discrimination model was fitted using the speaker identification task objective. The spectrum synthesis model was fitted based on the timbre vectors output by the Chinese text and the timbre discrimination model. The audio reconstruction model was fitted based on real Chinese audio and the corresponding bandpass filter spectrum.

[0012] In the model inference stage, Chinese text and the timbre vector of a specific person are input into the spectrum synthesis model, and then the output spectrum is input into the audio reconstruction model to finally synthesize a speech segment with the timbre of a specific person.

[0013] Compared to existing technologies, this invention offers the following advantages: The audio synthesized by the method of this invention significantly outperforms sound waveform splicing schemes in terms of speech fluency. In timbre imitation tasks, it enables real-time computation and provides realistic timbre transfer performance. Most existing timbre imitation schemes require lengthy computation times and cannot meet the needs of real-time synthesis. Furthermore, this patent only requires approximately ten minutes of audio footage of a specific person to complete timbre transfer, while other schemes often require hours of audio data. When users utilize the model fitting and construction scheme proposed in this invention, computational and time costs can be reduced. With limited data samples, timbre transfer and speech synthesis for specific individuals can be achieved, reducing the high dependence on human intervention in scenarios such as broadcasting, live-streaming e-commerce, and tourism narration. Moreover, due to the modeling of gated recurrent neural networks in this invention, voice synthesis becomes a realistic possibility on devices with limited computing power, such as mobile devices. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the speech synthesis process of the present invention.

[0015] Figure 2 This is a schematic diagram of the model training phase of the present invention.

[0016] Figure 3 This is a schematic diagram of the reasoning stage of the model in this invention. Detailed Implementation

[0017] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0018] This invention discloses a real-time speech synthesis method based on a specific person's voice timbre, comprising:

[0019] In the data preprocessing stage, the raw audio data is standardized.

[0020] In the data vectorization stage, Chinese text data is converted into Pinyin representation, then into the corresponding vector encoding sequence of Pinyin, and a bandpass filter bank is used to perform sliding window filtering on the audio corresponding to the text to obtain the spectrum of the audio.

[0021] In the model building phase, a neural network model for synthesizing the voice of a specific person is constructed. The model is implemented in three parts: timbre discrimination model, spectrum synthesis model, and audio reconstruction model.

[0022] During the model training phase, the timbre discrimination model, the spectrum synthesis model, and the audio reconstruction model were fitted respectively. The timbre discrimination model was fitted using the speaker identification task objective. The spectrum synthesis model was fitted based on the timbre vectors output by the Chinese text and the timbre discrimination model. The audio reconstruction model was fitted based on real Chinese audio and the corresponding bandpass filter spectrum.

[0023] In the model inference stage, Chinese text and the timbre vector of a specific person are input into the spectrum synthesis model, and then the output spectrum is input into the audio reconstruction model to finally synthesize a speech segment with the timbre of a specific person.

[0024] The following is a detailed implementation process of the present invention.

[0025] like Figure 1 As shown, the present invention provides a real-time speech synthesis method based on a specific person's timbre, comprising the following steps:

[0026] Step 1: Data Preprocessing Stage

[0027] The model input data defined in this patent uses real human recordings. These recordings commonly contain audio defects such as background noise, electrical noise, swallowing sounds, and long gaps in the audio. To ensure the effectiveness of the data features and the stability of model convergence, the original audio needs to be normalized. The first step is to normalize the amplitude of the audio data to ensure that the human voices in the dataset are at the same loudness level. The second step is to use a bandpass filter to filter and suppress noise signals in the background and electrical noise frequency bands. The third step is to perform a window integration operation on the audio signal to calculate the energy spectrum, divide the blank audio segments by the signal energy within the window, and then trim the segments. The fourth step is to resample the audio segments in the human voice signal frequency band to further normalize the data and remove high-frequency signals outside the human voice frequency band.

[0028] Step 2: Data Vectorization Stage

[0029] After preprocessing, the raw audio data has undergone preliminary vectorization. It still needs to be window-filtered using a nonlinear filter bank to obtain the corresponding nonlinear spectrum. In step 2, the Chinese text data needs to be vectorized. The first step is to convert the Chinese text data into pinyin representation; the second step is to convert the pinyin representation into the corresponding vector code in the pinyin dictionary; and the third step is to combine the pinyin vector code units into a vector sequence.

[0030] Step 3: Model Building Stage

[0031] In step 3, a neural network model is constructed that is used for the specific voice synthesis technology. The model is implemented in three parts: timbre discrimination model, spectrum synthesis model, and audio reconstruction model.

[0032] In the timbre discrimination model, the goal is to quantize the timbre of a specific speaker into a vector representation. This quantization algorithm must be independent of the text content; that is, audio recordings of different speakers narrating the same content need to be mapped to vector spaces with similar geometric distances. Even audio recordings of the same text from different speakers must have a sufficiently large geometric distance to distinguish the timbre of different speakers. To achieve this quantization objective, this patent constructs a timbre quantization model based on the concepts of cosine distance and perceptron. In calculating the cosine distance (CD), this patent defines and calculates both semantic cosine distance (SCD) and timbre cosine distance (TCD), using gradient descent to adjust the weights of the two cosine distances on timbre quantization. SCD is a vector embedding description of audio data from different speakers within the same Chinese text, while TCD is a vector embedding description of audio data from different Chinese texts spoken by the same speaker.

[0033] The following defines SCD and TCD:

[0034]

[0035]

[0036]

[0037]

[0038] in,

[0039] V: The complete set of vectorized speech datasets;

[0040] M: Number of speakers in the dataset

[0041] N: Number of sample statements in the dataset

[0042] m∈[1,M];

[0043] n∈[1,N];

[0044] V mn : Audio vectors of m speakers in a speech dataset, based on n samples;

[0045] ωs Semantic cosine distance weights;

[0046] β s Semantic cosine distance bias;

[0047] ω t : Timbre cosine distance weight;

[0048] β t : Timbre cosine distance offset;

[0049] In the spectral synthesis model, the goal is to synthesize the frequency domain representation of the audio signal corresponding to the given Chinese text. To achieve this, the first step is to extract the semantic representation from the Chinese text vector sequence. This patent employs word embedding technology and a bidirectional recurrent neural network (RNN) to extract the semantic representation from the Chinese vector sequence. The second step is to obtain the specific voice timbre vector representation output by the timbre discrimination model, and then merge the timbre representation with the Chinese semantic representation to obtain a timbre and semantic embedding representation. The third step is to decode the corresponding spectral representation vector based on the timbre and semantic embedding representation. In this patent, the decoding of the spectral vector uses a position-based attention mechanism and a long short-term memory (LSTM) recurrent neural network for modeling. The position-based encoding models the word order, while the recurrent neural network models the contextual dependencies in speech. The decoding result is the spectral representation vector corresponding to the Chinese text.

[0050] In the audio reconstruction model, the goal is to decode the temporal representation of the spectrum, i.e., the amplitude spectrum of the audio signal in the time domain, based on the spectral representation vector output by the spectral synthesis model. To achieve this, this patent employs downsampling convolution technology to perform contextual sampling of the spectral representation. The sampled features are then input into a decoder unit constructed from a gated recurrent neural network, which generates the temporal representation of the audio signal.

[0051] Step 4: Model Training Phase

[0052] like Figure 2 As shown, in the model training phase, the first stage uses speaker identification as the task objective to fit the timbre recognition model. The second stage, based on the Chinese text, the corresponding audio, and the timbre vector output by the timbre recognition model, fits the spectrum synthesis model. The total audio duration required to fit a specific person's timbre is approximately 10 minutes, achieving the timbre transfer function. Compared to other solutions, requiring only a shorter duration of audio data is one of the unique features of this patent. In the third stage, the audio reconstruction model is fitted based on real Chinese audio and the corresponding nonlinear spectrum.

[0053] Step 5: Model Inference Stage

[0054] like Figure 3As shown, by inputting text and timbre vectors into the spectral synthesis model, a nonlinear spectral input audio reconstruction model is obtained, which can then synthesize speech segments with specific vocal timbres. The model structure constructed using gated recurrent neural network units in this invention reduces computational overhead and enables audio synthesis with extremely low latency, which is one of the original features of this invention.

[0055] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for real-time speech synthesis based on a specific character's voice tone, characterized by, The method comprises the following steps: a data preprocessing stage for normalizing the original audio data; a data vectorization stage for converting the Chinese text data into pinyin representation, then into a vector coding sequence corresponding to the pinyin, and using a band-pass filter set to perform sliding window filtering on the audio corresponding to the text to obtain the spectrum corresponding to the audio; a model construction stage for constructing a neural network model for synthesizing the voice of a specific person, which is implemented in three parts, namely a timbre discrimination model, a spectrum synthesis model, and an audio reconstruction model; a model training stage for fitting the timbre discrimination model, the spectrum synthesis model, and the audio reconstruction model respectively; the timbre discrimination model is fitted using the task target of speaker recognition; the spectrum synthesis model is fitted based on the Chinese text and the timbre vector output by the timbre discrimination model; and the audio reconstruction model is fitted based on the real Chinese audio and the band-pass filtered spectrum corresponding to the audio; a model inference stage for inputting the Chinese text and the timbre vector of a specific person into the spectrum synthesis model, then inputting the output spectrum into the audio reconstruction model, and finally synthesizing a voice segment with the timbre of the specific person.

2. The method according to claim 1, wherein the method is characterized by, The specific implementation of the data preprocessing stage is as follows: The original audio is recorded by a real person, and the normalization of the original audio data is performed, that is, first, the amplitude of the original audio data is standardized to ensure that the human voice in the audio data set is at the same loudness level; second, a band-pass filter is selected to filter and suppress noise signals in the background and current noise frequency bands; third, the audio signal is subjected to window integration operation to calculate the energy spectrum, the blank audio segments are divided by the signal energy within the window, and the segments are cropped; and fourth, the audio segments in the audio data set are resampled in the human voice signal frequency band to further standardize the data and remove high-frequency signals beyond the human voice frequency band.

3. The method of real-time speech synthesis based on a specific character's voice tone according to claim 1, characterized in that, The specific implementation of converting the Chinese text data into pinyin representation, then into a vector coding sequence corresponding to the pinyin, is as follows: first, the Chinese text data is converted into pinyin representation; second, the pinyin representation is converted into a vector coding corresponding to the pinyin in the pinyin dictionary; and third, the pinyin vector coding units are combined into a vector sequence.

4. The method of real-time speech synthesis based on a specific character's voice tone according to claim 1, characterized in that, The specific implementation of using a band-pass filter set to perform sliding window filtering on the audio corresponding to the text to obtain the spectrum corresponding to the audio is as follows: after the preprocessing of the original audio data, a nonlinear filter set is used to perform window filtering on the audio to obtain the nonlinear spectrum corresponding to the audio.

5. The method of real-time speech synthesis based on a specific character's voice tone according to claim 1, characterized in that, The specific implementation of the model construction stage is as follows: In the voice recognition model, the purpose of constructing the model is to quantify the voice of a specific person into a vector representation, and the quantization algorithm must be independent of the text content, that is, the audio of the same speaker telling different content needs to be mapped into a vector space with close geometric distance, and there must be a large enough geometric distance between the audio of different speakers telling the same text content, so as to distinguish the voices of different people; therefore, a voice quantization model is constructed based on the idea of cosine distance and perception machine; in the calculation of cosine distance CD, semantic cosine distance SCD and voice cosine distance TCD are defined and calculated respectively, and gradient descent method is used to adjust the influence weight of the two cosine distances on voice quantization, wherein SCD is a vector embedding description of audio data of different speakers under the same Chinese text, and TCD is a vector embedding description of audio data of different Chinese texts under the same speaker; SCD and TCD are defined as follows: Wherein, V: vectorized speech data set; M: number of speakers in the data set N: number of example sentences in the data set m∈[1,M]; n∈[1,N]; V mn : audio vector of m speakers in the speech data set speaking according to n examples; ω s : semantic cosine distance weight; β s : semantic cosine distance biasing; ω t : timbre cosine distance weight; β t : timbre cosine distance biasing; In the frequency spectrum synthesis model, the purpose of the model is to synthesize the frequency domain representation of the audio signal corresponding to the text based on the given Chinese text; based on this purpose, the first step is to extract the semantic representation in the Chinese text vector sequence, that is, to extract the semantic representation in the Chinese vector sequence by using word embedding technology and bidirectional memory recurrent neural network; the second step is to obtain the voice vector representation of a specific person output by the voice recognition model, and the voice representation and the Chinese semantic representation are combined to obtain the voice and semantic embedding representation; the third step is to decode the corresponding frequency spectrum representation vector based on the voice and semantic embedding representation; the decoding of the frequency spectrum representation vector is modeled by using the attention mechanism based on position encoding and long short term memory recurrent neural network, wherein the position encoding realizes the modeling of the word order, and the bidirectional memory recurrent neural network realizes the modeling of the context dependence in the speech; the result of decoding is the frequency spectrum representation vector corresponding to the Chinese text; In the audio reconstruction model, the purpose of the model is to decode the time domain representation of the spectrum based on the frequency spectrum representation vector output by the frequency spectrum synthesis model, that is, the amplitude spectrum of the audio in the time domain; Based on this purpose, down-sampling convolution technology is used to sample the context of the spectrum representation, and then the sampled features are input into the decoder unit constructed by the gated recurrent neural network to generate the time domain representation of the audio signal.

6. The method of real-time speech synthesis based on a specific character's voice tone according to claim 1, characterized in that, The model training stage is specifically implemented as follows: In the first stage, the voice recognition model is fitted by using the task target of speaker recognition; in the second stage, the frequency spectrum synthesis model is fitted based on the Chinese text and the corresponding audio and the voice vector output by the voice recognition model; in the third stage, the audio reconstruction model is fitted based on the real Chinese audio and the nonlinear spectrum corresponding to the audio.

Images