Speech synthesis method and system

Abstract

The invention relates to the technical field of speech synthesis, and relates to a speech synthesis method and system. The method comprises the following steps: acquiring an original voice signal; preprocessing the original voice signal, wherein the preprocessing comprises pre-emphasis, framing and windowing; performing fast Fourier transform on each frame of signal after windowing to obtain a frequency spectrum of each frame; performing Mel filtering on the obtained frequency spectrum of each frame to obtain a Mel frequency spectrum; logarithm taking is carried out on the Mel frequency spectrum, discrete cosine transform is carried out, and therefore a Mel frequency cepstrum coefficient is obtained; inputting the Mel-frequency cepstrum coefficient into a preset vocoder so as to obtain voice signal parameter representation; and inputting the voice signal parameter representation into a preset decoder so as to obtain an audio waveform. By adopting the method provided by the invention, the quality and naturalness of the synthesized speech can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis technology, and more specifically, to a speech synthesis method and system. Background Art

[0002] At present, speech synthesis technology has made some progress, but there are still many technical routes and methods. Some traditional methods are based on rules and simple statistical models, such as synthesizing speech by setting fixed speech parameters and patterns. These rules may be based on a simple understanding of the basic acoustic principles of speech, such as the basic pronunciation rules of vowels and consonants. Some other methods use speech synthesis technology based on hidden Markov models (HMM). In the HMM method, by modeling the acoustic features of speech, the speech signal is regarded as a random process consisting of hidden states and observation sequences. The model infers hidden states based on the observation sequence (i.e., the acoustic features of speech) and generates speech based on these hidden states. However, these methods all have certain limitations in practical applications.

[0003] Methods based on rules and simple statistical models often fail to accurately capture the complex features of speech, causing the synthesized speech to appear stiff and unnatural in terms of timbre, intonation, and rhythm. For example, these methods are unable to simulate the emotional expression and intonation changes of human speech. Due to their fixed parameters and patterns, rule-based methods cannot adapt to the characteristics of different speech signals, resulting in a lack of natural changes in intonation. Methods based on statistical models may not capture the complex acoustic characteristics of speech sufficiently, resulting in the synthesized speech being not rich in timbre and lacking a sense of rhythm in rhythm.

[0004] Although the method based on hidden Markov model can model the acoustic features of speech to a certain extent, it still cannot fully restore the richness and naturalness of real speech due to the complexity and limitations of the model. In the HMM model, the simplified processing of speech features may lead to the loss of some important speech information, thus affecting the speech quality. For example, the model may not accurately capture some subtle emotional changes and subtle adjustments in intonation in speech.

[0005] In addition, the speech synthesis process of rule-based and simple statistical model-based methods is slow due to the high computational complexity, which cannot meet the needs of real-time applications. For example, in some interactive systems that require instant speech feedback, the slow synthesis speed will seriously affect the user experience. The calculation process of the hidden Markov model-based method involves a large number of matrix operations and probability calculations, which greatly limits the synthesis speed.

[0006] Even if some optimization measures are taken to improve efficiency, such as simplifying the model structure or using parallel computing technology, it is often at the expense of speech quality, and it is impossible to achieve high-quality and efficient speech synthesis at the same time. For example, simplifying the model structure may lead to a decrease in the ability to capture speech features, thus affecting speech quality. Summary of the invention

[0007] In order to solve the technical problem that the speech quality synthesized by the speech synthesis method in the prior art is poor, the present invention provides solutions in the following aspects.

[0008] In a first aspect, the present invention provides a speech synthesis method, comprising:

[0009] Obtaining the original speech signal;

[0010] Preprocessing the original speech signal, the preprocessing comprising: pre-emphasis, framing and windowing; the pre-emphasis comprising: obtaining the ratio of the energy of the high frequency part of the speech signal to the total energy of the speech signal, if the ratio is greater than a preset ratio threshold, making the pre-emphasis coefficient less than a preset first pre-emphasis coefficient threshold, otherwise making the pre-emphasis coefficient greater than a preset second pre-emphasis coefficient threshold; the first pre-emphasis coefficient threshold is less than the second pre-emphasis coefficient threshold;

[0011] Perform fast Fourier transform on each frame of the windowed signal to obtain the spectrum of each frame;

[0012] Perform Mel filtering on the spectrum of each frame to obtain a Mel spectrum;

[0013] Take the logarithm of the Mel spectrum and perform discrete cosine transform to obtain the Mel frequency cepstrum coefficients;

[0014] Inputting the Mel-frequency cepstral coefficients into a preset vocoder to obtain a speech signal parameter representation;

[0015] The speech signal parameter representation is input into a preset decoder to obtain an audio waveform.

[0016] The beneficial effects are as follows: the speech synthesis method of the present invention takes into account, in the pre-emphasis link, that there are more high-frequency components in female voice signals, while there are fewer high-frequency components in male voice signals. For speech signals with more high-frequency components, a smaller pre-emphasis coefficient is used when emphasizing them, and for speech signals with fewer high-frequency components, a larger pre-emphasis coefficient is used when emphasizing them. This enables the speech synthesis method of the present invention to better adapt to the synthesis requirements of speech signals of different genders, and improves the quality and naturalness of the synthesized speech.

[0017] Preferably, the first pre-emphasis coefficient threshold is 0.92, and the second pre-emphasis coefficient threshold is 0.97.

[0018] Preferably, when the original speech signal is divided into frames, the method for determining the frame length includes:

[0019] Detecting whether the frequency change amplitude of the original voice signal within a preset time length is greater than a preset frequency change amplitude threshold, and whether the number of frequency changes is greater than a preset frequency change number threshold;

[0020] If the number of frequency changes is greater than the preset frequency change number threshold and the frequency change amplitude is greater than the preset frequency change amplitude threshold, the frame length is set to be less than or equal to the preset first frame length threshold; otherwise, the frame length is set to be greater than or equal to the preset second frame length threshold; the first frame length threshold is less than the second frame length threshold.

[0021] The beneficial effects are as follows: if the number of frequency changes of the original speech signal is greater than the preset frequency change number threshold and the frequency change amplitude is greater than the preset frequency change amplitude threshold, it means that the original speech signal contains a speech signal with more rapidly changing information. For such a speech signal containing more rapidly changing information, a shorter frame length is selected in the frame operation, so that the rapid changes of the speech can be captured more accurately; if the number of frequency changes of the original speech signal is less than the preset frequency change number threshold, or the frequency change amplitude is greater than the preset frequency change amplitude threshold, it means that the original speech signal is relatively stable and its signal characteristics change little. By adopting a longer frame length, more information can be included, which helps to improve the accuracy of frequency analysis; therefore, the frame length determination method of this embodiment can improve the accuracy of the calculation of the Mel-frequency cepstral coefficients, can better adapt to the synthesis requirements of speech signals of different age groups, and further improve the quality and naturalness of the synthesized speech.

[0022] Preferably, the first frame length threshold value is 20 milliseconds, and the second frame length threshold value is 40 milliseconds.

[0023] Preferably, when windowing the original speech signal, a Hamming window or a Hanning window is selected for the high-frequency part of the speech signal; and a Blackman window or a Casey window is selected for the low-frequency part of the speech signal.

[0024] The beneficial effects are as follows: in speech signals, high-frequency components (such as consonants) often contain more information, which is essential for speech recognition and understanding. Therefore, in order to accurately capture these high-frequency components, a higher signal resolution is required; in addition, high-frequency signals are susceptible to attenuation and interference during transmission, so it is necessary to enhance the detection and analysis capabilities of these signals by improving resolution. The low-frequency components (such as vowels) in speech signals are relatively stable and do not change much. Therefore, when processing low-frequency signals, more attention should be paid to the spectrum leakage problem to avoid energy being dispersed to other frequency points, affecting the accuracy of the analysis. Therefore, for the high-frequency part of the speech signal, a Hamming window or a Hanning window is selected so that the information contained in the speech signal is accurately captured. For the low-frequency part of the speech signal, a Blackman window or a Kayser window is selected to ensure the accuracy of the speech signal analysis, thereby improving the accuracy of the calculation of the Mel-frequency cepstral coefficients.

[0025] Preferably, when performing a fast Fourier transform on the windowed signal, for the high-frequency part of the windowed signal, the selected FFT point number is greater than a preset first FFT point number threshold, and for the low-frequency part of the windowed signal, the selected FFT point number is less than a preset second FFT point number threshold, and the first FFT point number threshold is greater than the second FFT point number threshold.

[0026] The beneficial effects are as follows: for the high-frequency part of the speech signal, a higher frequency resolution is required, and a higher FFT point number is selected. When processing the high-frequency part of the speech signal, a higher FFT point number can provide a finer frequency resolution. Using a high FFT point number, the high frequency band can be divided into more small frequency bands, and the subtle frequency changes and energy distribution of these fricative sounds in the high frequency part can be captured more accurately. This helps to accurately identify the characteristic differences between different fricative sounds. For applications such as speech recognition and speech synthesis, it can improve the ability to distinguish speech details, thereby improving the accuracy of the system. For the low-frequency part of the speech signal, it does not require too high a frequency resolution, and a lower FFT point number is sufficient to accurately describe its basic low-frequency characteristics such as the resonance peak frequency. Using a low FFT point number can clearly present the position and relative strength of these resonance peaks without over-subdividing the low-frequency spectrum, providing sufficient information for speech analysis and processing, while avoiding unnecessary computing resource consumption caused by too high a point number, avoiding excessive computational complexity, and improving the efficiency of MFCCs calculation.

[0027] Preferably, the first FFT point threshold is 4096 points, and the second FFT point threshold is 512 points.

[0028] Preferably, the vocoder adopts a WaveNet model, which includes a convolution layer, a pooling layer, a fully connected layer and a normalization layer; wherein the number of convolution kernels of the convolution layer and the pooling layer is set to 16 to 32, the number of convolution kernels of the fully connected layer is set to 32 to 64, and the number of convolution kernels of the normalization layer is set to 64 to 128; the convolution kernel adopts a convolution kernel of size 3x3 or 5x5.

[0029] The beneficial effects are as follows: as the network gradually deepens, the number of convolution kernels increases, and each layer can capture dependencies of different ranges. As the number of layers increases, the model can understand and process more complex audio patterns. This layer-by-layer enhancement method enables the model to have higher accuracy when processing complex audio data. In addition, it enables the vocoder to output parameter representations of speech signals more quickly when processing MFCCs input. Furthermore, as the number of network layers deepens and the number of convolution kernels increases, the WaveNet model is more robust to noise and deformation of the input data. Even if there are certain quality problems with the input data, the model can still stably output a relatively accurate parameter representation of the speech signal.

[0030] Preferably, the method for the decoder to decode the speech signal parameter representation comprises:

[0031] Obtaining the type represented by the speech signal parameter;

[0032] In response to the speech signal parameter being represented as a linear prediction coding coefficient, an LPC synthesis method is used to reconstruct the audio waveform; in response to the speech signal parameter being represented as a parameter based on a probability distribution, a sampling method is used to obtain the actual audio waveform from the probability distribution.

[0033] In a second aspect, the present invention provides a speech synthesis system, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the speech synthesis method of the present invention is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0035] Figure 1 is a flowchart schematically illustrating a speech synthesis method according to an embodiment of the present invention;

[0036] Figure 2 FIG. 1 is a schematic diagram schematically showing the structure of a speech synthesis system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0038] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0039] Speech synthesis method embodiment:

[0040] like Figure 1 As shown, the speech synthesis method of the present invention comprises:

[0041] S101: Acquire an original speech signal.

[0042] The original speech signal can be obtained through a recording device or a communication device.

[0043] S102, preprocessing the original speech signal, specifically: preprocessing the original speech signal, the preprocessing includes: pre-emphasis, framing and windowing; the pre-emphasis includes: obtaining the ratio of the energy of the high-frequency part of the speech signal to the total energy of the speech signal, if the ratio is greater than a preset ratio threshold, making the pre-emphasis coefficient less than a preset first pre-emphasis coefficient threshold, otherwise, making the pre-emphasis coefficient greater than a preset second pre-emphasis coefficient threshold; the first pre-emphasis coefficient threshold is less than the second pre-emphasis coefficient threshold.

[0044] The high-frequency part refers to the speech signal with a frequency of 2000Hz and above; the low-frequency part refers to the speech signal with a frequency of less than 500Hz.

[0045] Pre-emphasis is usually implemented using a first-order digital filter, and its formula is: y(n)=x(n)-ax(n-1), where x(n) is the original speech signal sample, y(n) is the pre-emphasized signal sample, and a is the pre-emphasis coefficient.

[0046] Since there are more high-frequency components in female voice signals and fewer high-frequency components in male voice signals, a smaller pre-emphasis coefficient is used when emphasizing voice signals with more high-frequency components, and a larger pre-emphasis coefficient is used when emphasizing voice signals with fewer high-frequency components. This allows the speech synthesis method of the present invention to better adapt to the synthesis requirements of speech signals of different genders and improve the quality and naturalness of the synthesized speech.

[0047] In this embodiment, the first pre-emphasis coefficient threshold is 0.92, and the second pre-emphasis coefficient threshold is 0.97. In other embodiments, the first pre-emphasis coefficient threshold and the second pre-emphasis coefficient threshold can take other appropriate values, but it should be ensured that the first pre-emphasis coefficient threshold and the second pre-emphasis coefficient threshold are both values ​​between 0.9 and 1.

[0048] In this embodiment, when windowing the original speech signal, a Hamming window or a Hanning window is selected for the high frequency part of the speech signal; and a Blackman window or a Casey window is selected for the low frequency part of the speech signal.

[0049] The commonly used Hamming window function formula is: Where w(n) represents the value of the Hamming window at the nth sampling point, n is the sampling point index in the window, n=0,1,...,L-1; L is the window length (usually the same as the frame length). Apply this window function to each frame of speech signal, that is, x w (n) = x(n)w(n), where x(n) is the original frame signal, x w (n) is the frame signal after windowing.

[0050] In speech signals, high-frequency components (such as consonants) often contain more information, which is crucial for speech recognition and understanding. Therefore, in order to accurately capture these high-frequency components, a higher signal resolution is required; in addition, high-frequency signals are susceptible to attenuation and interference during transmission, so it is necessary to enhance the detection and analysis capabilities of these signals by improving resolution. The low-frequency components (such as vowels) in speech signals are relatively stable and do not change much. Therefore, when processing low-frequency signals, more attention should be paid to the spectrum leakage problem to avoid energy being dispersed to other frequency points, affecting the accuracy of the analysis. Therefore, for the high-frequency part of the speech signal, a Hamming window or a Hanning window is selected so that the information contained in the speech signal is accurately captured. For the low-frequency part of the speech signal, a Blackman window or a Kasey window is selected to ensure the accuracy of the speech signal analysis, thereby improving the accuracy of the calculation of the Mel-frequency cepstral coefficients.

[0051] Framing is to divide the continuous speech signal into fixed lengths. Assuming the total length of the speech signal is n, the frame length is l, and the frame shift is s (generally the frame shift is less than the frame length), the framing formula can be expressed as: for the kth frame, its starting position is ks and the ending position is min(ks+l,n). In this way, the speech signal is divided into multiple frames for subsequent processing.

[0052] S103 , performing fast Fourier transform on each frame of the windowed signal, so as to obtain a frequency spectrum of each frame.

[0053] Fast Fourier Transform (FFT) is an algorithm for efficiently calculating Discrete Fourier Transform (DFT). Its basic principle is to use the periodicity and symmetry of complex numbers to reduce the computational complexity of DFT (Discrete Fourier Transform) from O(N 2 ) is reduced to O(NlogN), where N is the number of data points (usually the number of points per frame in speech signal processing). The specific FFT algorithm implementation is relatively complex, and there are many classic algorithm implementation methods, such as the radix 2-FFT algorithm.

[0054] When performing a fast Fourier transform on the windowed signal, for the high-frequency portion of the windowed signal, the selected FFT point number is greater than a preset first FFT point threshold, and for the low-frequency portion of the windowed signal, the selected FFT point number is less than a preset second FFT point threshold, and the first FFT point threshold is greater than the second FFT point threshold. In this embodiment, the first FFT point threshold is 4096 points, and the second FFT point threshold is 512 points. In other embodiments, the first FFT point threshold and the second FFT point threshold may also take other appropriate values.

[0055] For high-frequency speech signals, higher frequency resolution is required. Selecting a higher FFT point number can provide finer frequency resolution when processing high-frequency speech signals. For example, for fricative sounds in speech (such as / s / , / f / , etc.), their spectral energy is mainly concentrated in the high-frequency area, and the high-frequency components are complex and changeable. Using a high FFT point number, the high-frequency band can be divided into more small frequency bands, and the subtle frequency changes and energy distribution of these fricative sounds in the high-frequency part can be captured more accurately. This helps to accurately identify the characteristic differences between different fricative sounds. For applications such as speech recognition and speech synthesis, it can improve the ability to distinguish speech details, thereby improving the accuracy of the system. For low-frequency speech signals, it does not require too high frequency resolution. For example, vowels ( / a / , / o / , / u / , etc.), their main energy is concentrated in the low-frequency area, and the frequency changes relatively slowly. A lower FFT point number is sufficient to accurately describe its basic low-frequency characteristics such as formant frequency. Formant is the frequency peak generated by the resonance of the vocal tract when vowels are pronounced, which is of great significance to the timbre and intelligibility of speech. Using a low FFT point count can clearly present the position and relative strength of these resonance peaks without over-segmenting the low-frequency spectrum, providing sufficient information for speech analysis and processing, while avoiding unnecessary consumption of computing resources due to excessively high points, avoiding excessive computational complexity, and improving the efficiency of MFCCs calculations. Due to efficiency optimization in the speech synthesis process, the time required to synthesize a standard length of speech (such as 10 seconds) is effectively controlled under ordinary computer hardware environments. For example, the synthesis time can be controlled to no more than 2 seconds, meeting the needs of real-time applications, such as in real-time interactive applications such as intelligent voice assistants and voice navigation systems, which can quickly provide high-quality voice feedback and improve user experience.

[0056] S104 , performing Mel filtering on the obtained frequency spectrum of each frame, so as to obtain a Mel spectrum.

[0057] The design of the Mel filter bank is based on the Mel frequency scale. First, the linear frequency f is converted to the Mel frequency m, the formula is: Then, a series of triangular filters are designed based on the Mel frequency. The center frequency and bandwidth of each filter are determined according to the characteristics of the speech signal and the auditory perception characteristics of the human ear. Let the center frequency of the i-th filter be m i , bandwidth is B i , then for the input spectrum X(k) (k is the frequency index), the output Y after passing through the i-th filter is i (K) can be calculated by the following formula: where h(jk) is the impulse response of the jkth filter, and start and end are the frequency index ranges related to the filter bandwidth.

[0058] The spectrum is filtered by a Mel filter bank. The parameters of the Mel filter bank are optimized according to the characteristics of human auditory perception and speech signals. For example, for high-pitched speech signals, the center frequency is set higher and the bandwidth is set narrower; for low-pitched speech signals, the center temperature is set lower and the bandwidth is set wider.

[0059] S105 , taking the logarithm of the Mel frequency spectrum and performing discrete cosine transform to obtain Mel frequency cepstrum coefficients.

[0060] By accurately extracting the MFCCs features of speech signals, we can better capture key information such as the spectral envelope of speech, providing a more accurate basis for subsequent speech synthesis. Compared with traditional speech feature extraction methods, MFCCs can more accurately describe the acoustic characteristics of speech, making the timbre of synthesized speech more natural and closer to real human speech.

[0061] S106: Input the Mel-frequency cepstral coefficients into a preset vocoder to obtain a speech signal parameter representation.

[0062] In this embodiment, the vocoder may adopt a deep learning model. In other embodiments, the vocoder may not adopt a deep learning model, but may be composed of an excitation source model and a vocal tract model. The excitation source model is used to simulate the excitation signal generated by speech. For voiced sounds, a periodic pulse sequence is usually used for simulation, and its period is determined according to the fundamental pitch period of the speech. For unvoiced sounds, a noise sequence is used for simulation. For example, for voiced sounds, the fundamental pitch period in the speech signal can be detected, and then a pulse sequence of the corresponding period can be generated as an excitation signal. The detection of the fundamental pitch period can be carried out by various methods such as the autocorrelation method.

[0063] The vocal tract model is used to simulate the filtering effect of the vocal tract on the excitation signal. The commonly used vocal tract model is the linear predictive coding (LPC) model. Its basic principle is to construct a vocal tract filter by predicting the relationship between the current speech sample and the past speech sample. Assume that the current speech sample is y(nk) and the past speech sample is y(nk) (k = 1, 2, 3, ..., p, p is the prediction order), then the output of the vocal tract filter can be expressed as: where a k is the prediction coefficient, and e(n) is the excitation signal. k It can be determined by the least squares method and the like.

[0064] S107: Input the speech signal parameter representation into a preset decoder to obtain an audio waveform.

[0065] The speech synthesis method of the present embodiment takes into account, in the pre-emphasis link, that female voice signals have more high-frequency components, while male voice signals have fewer high-frequency components. For speech signals with more high-frequency components, a smaller pre-emphasis coefficient is used when emphasizing them, and for speech signals with fewer high-frequency components, a larger pre-emphasis coefficient is used when emphasizing them. This allows the speech synthesis method of the present invention to better adapt to the synthesis requirements of speech signals of different genders, thereby improving the quality and naturalness of the synthesized speech.

[0066] It can be seen from the above embodiments that the vocoder can adopt a deep learning model. In one embodiment, the vocoder adopts a WaveNet model. The training method of the WaveNet model includes:

[0067] S201, build and load the training set,

[0068] Audio data can be collected from various data sources, such as web crawlers, API interfaces, databases, and public audio datasets. The collected audio data should be diverse, including audio samples from speakers of different ages, speakers of different genders, different environments, different emotions, different speaking speeds and tones, in order to improve the generalization ability of the model. After collecting the audio data, it is preprocessed, and the preprocessing includes: denoising, missing value processing, and data normalization processing; after preprocessing, labels are created for the audio data, and the labels include text content for speech recognition, category labels for audio classification, emotion labels, etc. S202. Define the architecture of the WaveNet model and set initial values ​​for the parameters of the WaveNet model;

[0069] In this embodiment, the WaveNet model is implemented using the TensorFlow deep learning framework or the PyTorch deep learning framework. The WaveNet model in this embodiment includes a convolution layer, a pooling layer, a fully connected layer, and a normalization layer. In order to accurately capture adjacent frequency and time point information when processing speech spectrum features, the convolution kernel size of the WaveNet model uses small-sized convolution kernels such as 3x3 and 5x5. The number of convolution kernels of the convolution layer and the pooling layer is set to 16 to 32, the number of convolution kernels of the fully connected layer is set to 32 to 64, and the number of convolution kernels of the normalization layer is set to 64 to 128.

[0070] As the network becomes deeper, the number of convolution kernels increases, and each layer can capture dependencies of different ranges. As the number of layers increases, the model can understand and process more complex audio patterns. This layer-by-layer enhancement method enables the model to have higher accuracy when processing complex audio data. In addition, it enables the vocoder to output parameter representations of speech signals more quickly when processing MFCCs input. Furthermore, as the number of network layers increases and the number of convolution kernels increases, the WaveNet model is more robust to noise and deformation of the input data. Even if there are certain quality problems with the input data, the model can still stably output relatively accurate parameter representations of speech signals.

[0071] S203, setting loss function and optimizer;

[0072] The loss function can adopt the mean square error loss function; the optimizer adopts the gradient descent method.

[0073] S204, training the model, including: setting training parameters, and iterating training until a preset number of training times is reached or the loss function value reaches a preset threshold.

[0074] The model is trained iteratively on the training subset, with each iteration using a batch of data from the training set for forward and backward propagation.

[0075] During backpropagation, the gradients are calculated with respect to the loss function and the optimizer is used to update the parameters of the model.

[0076] Using the WaveNet deep learning model as a vocoder can learn more complex speech features and parameter mapping relationships, and it can better capture the emotional information in the speech. In the speech synthesis process, it can generate speech with corresponding emotional expressions based on the input emotion-related information (such as emotional tags in the text), improving the adaptability to different speech emotional expressions, making the synthesized speech more vivid and natural.

[0077] In one embodiment, when the original speech signal is divided into frames, the method for determining the frame length includes:

[0078] S201, detecting whether the frequency change amplitude of the original voice signal within a preset time length is greater than a preset frequency change amplitude threshold, and whether the number of frequency changes is greater than a preset frequency change number threshold;

[0079] S202. If the number of frequency changes is greater than a preset frequency change number threshold and the frequency change amplitude is greater than a preset frequency change amplitude threshold, then the frame length is set to be less than or equal to a preset first frame length threshold; otherwise, the frame length is set to be greater than or equal to a preset second frame length threshold; the first frame length threshold is less than the second frame length threshold.

[0080] In this embodiment, the value of the first frame length threshold is 20 milliseconds, and the value of the second frame length threshold is 40 milliseconds.

[0081] If the number of frequency changes of the original speech signal is greater than the preset frequency change number threshold and the frequency change amplitude is greater than the preset frequency change amplitude threshold, it means that the original speech signal contains a speech signal with more rapidly changing information. For such a speech signal containing more rapidly changing information, selecting a shorter frame length in the frame operation can more accurately capture the rapid changes of the speech; if the number of frequency changes of the original speech signal is less than the preset frequency change number threshold, or the frequency change amplitude is greater than the preset frequency change amplitude threshold, it means that the original speech signal is relatively stable and its signal characteristics change little. By adopting a longer frame length, more information can be included, which helps to improve the accuracy of frequency analysis; therefore, the frame length determination method of this embodiment can improve the accuracy of the calculation of the Mel-frequency cepstral coefficients, can better adapt to the synthesis requirements of speech signals of different age groups, and further improve the quality and naturalness of the synthesized speech.

[0082] In one embodiment, the method for a decoder to decode the speech signal parameter representation comprises:

[0083] S301, obtaining the type represented by the speech signal parameter;

[0084] S302. In response to the speech signal parameter being represented as a linear prediction coding coefficient, an LPC synthesis method is used to reconstruct an audio waveform. In response to the speech signal parameter being represented as a parameter based on a probability distribution, a sampling method is used to obtain an actual audio waveform from the probability distribution.

[0085] When the vocoder outputs linear predictive coding (LPC) coefficients, the decoder uses the LPC synthesis method to reconstruct the audio waveform. Let the LPC coefficient be a k (k=1,2,3,...,p, p is the prediction order), the excitation signal is e(n), then the reconstructed audio waveform p(n) can be calculated by the following formula: This formula is similar to the formula in the vocal tract model, but here the audio waveform is reconstructed based on the LPC coefficients output by the vocoder; k is an integer variable used to represent the past time offset, μ represents the number of speech samples, and a(nk) is the prediction order determined based on the characteristics of the speech signal.

[0086] When the vocoder output is a parameter based on probability distribution, the decoder uses a sampling algorithm (such as Monte Carlo sampling) to obtain the actual audio waveform from the probability distribution. For example, for the output P(x) based on probability distribution, the basic principle of Monte Carlo sampling is to randomly select sample points X i(i=1,2,3,...,N, N is the number of samples), and then estimate the actual audio waveform based on these sample points. Specifically, the audio waveform can be constructed by calculating the weighted average of the sample points. For example, assuming that the sample point obtained by sampling is, and its corresponding probability is, then the estimated audio waveform can be calculated by the following formula: In the formula, y represents the amplitude of the estimated audio waveform, i represents the sample point, N is the number of samples, and x is i is the i-th sample point obtained by sampling, and p is the probability corresponding to the i-th sample point.

[0087] The decoder uses a suitable method to restore the audio waveform based on the vocoder output, ensuring the quality and accuracy of the final generated audio, making the intonation of the synthesized speech smoother and in line with the rhythm of human language.

[0088] Speech synthesis system embodiment:

[0089] The present invention also provides a speech synthesis system. Figure 2 As shown, the speech synthesis system includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the speech synthesis method described in the above embodiment is implemented.

[0090] The speech synthesis system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface, whose configuration and functions are known in the art and thus will not be described in detail here.

[0091] In the present invention, the aforementioned memory may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus or device. For example, a computer-readable storage medium may be any suitable magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc., or any other medium that may be used to store the required information and may be accessed by an application, a module, or both. Any such computer storage medium may be part of a device or accessible or connectable to a device. Any application or module described in the present invention may be implemented using computer-readable / executable instructions that may be stored or otherwise maintained by such a computer-readable medium.

[0092] In the description of this specification, "plurality" or "several" means at least two, such as two, three or more, etc., unless otherwise clearly and specifically defined.

[0093] Although this specification has shown and described a number of embodiments of the present invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will conceive of many modifications, changes and alternatives without departing from the ideas and spirit of the present invention. It should be understood that in the practice of the present invention, various alternatives to the embodiments of the present invention described herein may be employed.

Claims

1. A speech synthesis method, characterized in that: include: Obtaining the original speech signal; Preprocessing the original speech signal, the preprocessing comprising: pre-emphasis, framing and windowing; the pre-emphasis comprising: obtaining the ratio of the energy of the high frequency part of the speech signal to the total energy of the speech signal, if the ratio is greater than a preset ratio threshold, making the pre-emphasis coefficient less than a preset first pre-emphasis coefficient threshold, otherwise making the pre-emphasis coefficient greater than a preset second pre-emphasis coefficient threshold; the first pre-emphasis coefficient threshold is less than the second pre-emphasis coefficient threshold; Perform fast Fourier transform on each frame of the windowed signal to obtain the spectrum of each frame; Perform Mel filtering on the spectrum of each frame to obtain a Mel spectrum; Take the logarithm of the Mel spectrum and perform discrete cosine transform to obtain the Mel frequency cepstrum coefficients; Inputting the Mel-frequency cepstral coefficients into a preset vocoder to obtain a speech signal parameter representation; The speech signal parameter representation is input into a preset decoder to obtain an audio waveform.

2. The speech synthesis method according to claim 1, characterized in that: The first pre-emphasis coefficient threshold is 0.92, and the second pre-emphasis coefficient threshold is 0.

97.

3. The speech synthesis method according to claim 1, characterized in that: When the original speech signal is divided into frames, the method for determining the frame length includes: Detecting whether the frequency change amplitude of the original voice signal within a preset time length is greater than a preset frequency change amplitude threshold, and whether the number of frequency changes is greater than a preset frequency change number threshold; If the number of frequency changes is greater than the preset frequency change number threshold and the frequency change amplitude is greater than the preset frequency change amplitude threshold, the frame length is set to be less than or equal to the preset first frame length threshold; otherwise, the frame length is set to be greater than or equal to the preset second frame length threshold; the first frame length threshold is less than the second frame length threshold.

4. The speech synthesis method according to claim 3, characterized in that: The first frame length threshold value is 20 milliseconds, and the second frame length threshold value is 40 milliseconds.

5. The speech synthesis method according to claim 1, wherein: When windowing the original speech signal, a Hamming window or a Hanning window is selected for the high-frequency part of the speech signal; and a Blackman window or a Casey window is selected for the low-frequency part of the speech signal.

6. The speech synthesis method according to claim 1, wherein: When performing a fast Fourier transform on the windowed signal, for the high-frequency part of the windowed signal, the selected FFT point number is greater than a preset first FFT point number threshold, and for the low-frequency part of the windowed signal, the selected FFT point number is less than a preset second FFT point number threshold, and the first FFT point number threshold is greater than the second FFT point number threshold.

7. The speech synthesis method according to claim 6, characterized in that: The first FFT point threshold is 4096 points, and the second FFT point threshold is 512 points.

8. The speech synthesis method according to claim 1, wherein: The vocoder adopts a WaveNet model, which includes a convolution layer, a pooling layer, a fully connected layer and a normalization layer; wherein the number of convolution kernels of the convolution layer and the pooling layer is set to 16 to 32, the number of convolution kernels of the fully connected layer is set to 32 to 64, and the number of convolution kernels of the normalization layer is set to 64 to 128; the convolution kernel adopts a convolution kernel of size 3x3 or 5x5.

9. The speech synthesis method according to any one of claims 1 to 8, characterized in that: The method for the decoder to decode the speech signal parameter representation comprises: Obtaining the type represented by the speech signal parameter; In response to the speech signal parameter being represented as a linear prediction coding coefficient, an LPC synthesis method is used to reconstruct the audio waveform; in response to the speech signal parameter being represented as a parameter based on a probability distribution, a sampling method is used to obtain the actual audio waveform from the probability distribution.

10. A speech synthesis system, characterized in that: The invention comprises a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the speech synthesis method according to any one of claims 1 to 9 is implemented.

Images