Rhythm controllable speech synthesis method based on VITS and related device

Abstract

The invention provides a rhythm controllable speech synthesis method based on VITS and a related device, and relates to the technical field of speech synthesis. The method comprises the steps of extracting rhythm feature information from an input text; performing semantic coding on the input text to generate text context feature representation; constructing a rhythm controller, and performing independent modeling on the rhythm features to generate a prediction result; fusing the text context feature representation with the prediction result to generate rhythm control embedding; generating a multi-speaker speech spectrum based on a VITS model in combination with the rhythm control embedding and speaker embedding vectors; converting the multi-speaker speech spectrum into a time domain speech signal through a decoder; by adjusting the rhythm characteristic parameters, independent control of duration, pitch period, energy, pause and rhythm of the synthesized speech is realized, so that the subjective quality of the generated speech signal is improved.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, and in particular, to a prosody controllable speech synthesis method and related device based on VITS. Background Art

[0002] Speech synthesis is a very challenging task. In the actual application scenarios of audio content production, such as audiobooks, audio-visual videos, news broadcasts, etc., matching voices to corresponding texts and videos requires professional recording equipment and personnel for recording and editing. How to reduce costs and time consumption, make the generated voice more anthropomorphic, achieve the "cadence" effect of human vocalization, and improve the user experience are precisely the difficulties of the speech synthesis problem. Due to technical limitations, traditional speech generation technologies are difficult to achieve ideal results. With the booming development of deep learning in Text-to-Speech, the quality of the generated voice has also improved in recent years, but the research results are difficult to be put into practical applications. The generation effect of multi-speaker Chinese speech synthesis is also less than satisfactory, and there is still a large gap from the sound quality of artificial dubbing. Existing models consider single prosody features. The present invention focuses on the refined control of multiple prosody features in speech synthesis, and pays attention to the fusion method of the extracted prosody features, and draws on the inner and outer layer function ideas of KAN to fuse prosody features including duration, fundamental frequency period, energy, pause, and rhythm.

[0003] Therefore, how to optimize the speech generation effect has become an urgent technical problem to be solved. Summary of the Invention

[0004] To optimize the speech generation effect, this application provides a prosody controllable speech synthesis method and related device based on VITS.

[0005] In a first aspect, a prosody controllable speech synthesis method based on VITS provided by this application adopts the following technical solution: A prosody controllable speech synthesis method based on VITS includes: S1: Extract prosody feature information from the input text, where the prosody feature information includes prosody features of duration, fundamental frequency period, energy, pause, and rhythm; S2: Perform semantic encoding on the input text through a BERT pre-trained model to generate a text context feature representation; S3: Construct a prosody controller including a duration predictor, an energy predictor, a fundamental frequency period predictor, a pause predictor, and a rhythm predictor, and independently model the prosody features through the prosody controller and generate a prediction result; S4: Fuse the text context feature representation and the prediction result to generate a fused prosody control embedding; S5: Based on the VITS model, combine the prosody control embedding and the speaker embedding vector to generate a multi-speaker speech spectrum; S6: Convert the multi-speaker speech spectrum into a time-domain speech signal through a decoder; S7: By adjusting the prosody feature parameters, achieve independent control of the duration, pitch period, energy, pause, and rhythm of the synthesized speech.

[0006] Optionally, in step S1: The duration extracts the phoneme duration through a forced alignment algorithm and trains a duration predictor using maximum likelihood estimation; The pitch period is decomposed into multi-scale components through continuous wavelet transform and quantized into control conditions; The energy is encoded after calculating the L2 norm of the Mel spectrum amplitude and quantizing; The pause generates a pause duration sequence by analyzing text punctuation and phrase structure; The rhythm captures semantic associations through a BERT pre-trained model and adjusts the pronunciation speed in combination with positional embeddings.

[0007] Optionally, in step S3: The prosody controller consists of a one-dimensional convolutional network, a bidirectional long short-term memory network, and a linear layer. The outputs of each predictor are superimposed through a feature fusion module, and the Kolmogorov-Arnold representation theorem is applied to achieve the fusion of multivariate features into a unary basis function. The specific formula is: ; where are weights, is the basis function.

[0008] Optionally, in step S5: The posterior encoder of the VITS model takes the Mel spectrum as input to generate a latent variable z, and converts the latent variable distribution into a prior distribution through a flow model; The speaker embedding vector is concatenated with the text feature and the prosody control embedding and then input into the decoder to generate a multi-speaker speech spectrum.

[0009] Optionally, in step S7: The control of the duration adjusts the speech speed by changing the output multiple of the duration predictor; The control of the pitch period adjusts the pitch by modifying the amplitude of the wavelet component; The control of the pause specifies the pause duration through the markup symbols in the input text; The control of the rhythm dynamically adjusts the pronunciation speed through the semantic weights output by BERT.

[0010] Optionally, it further includes a training step: Using the AISHELL-3 or BZNSYP Chinese speech dataset, extract the duration, pitch period, energy, and pause duration of the real speech as the training targets; By minimizing the reconstruction error between the predicted values and the true values of each prosodic feature, jointly optimize the parameters of the text encoder, prosody controller, and VITS model.

[0011] Optionally, the decoder adopts the HiFiGAN V1 structure to convert the speech spectrum into a time-domain waveform through multi-level upsampling.

[0012] In a second aspect, the present application provides a prosody-controllable speech synthesis device based on VITS, including: An extraction module, configured to extract prosodic feature information from the input text, where the prosodic feature information includes prosodic features such as duration, pitch period, energy, pause, and rhythm; A semantic encoding module, configured to perform semantic encoding on the input text through a BERT pre-trained model to generate a text context feature representation; A predictor construction module, configured to construct a prosody controller including a duration predictor, an energy predictor, a pitch period predictor, a pause predictor, and a rhythm predictor, and independently model the prosodic features through the prosody controller and generate a prediction result; A fusion module, configured to fuse the text context feature representation and the prediction result to generate a fused prosody control embedding; A speech spectrum module, configured to generate a multi-speaker speech spectrum based on the VITS model, in combination with the prosody control embedding and the speaker embedding vector; A conversion module, configured to convert the multi-speaker speech spectrum into a time-domain speech signal through a decoder; A control module, configured to independently control the duration, pitch period, energy, pause, and rhythm of the synthesized speech by adjusting prosodic feature parameters.

[0013] In a third aspect, the present application provides a computer device, where the device includes: a memory and a processor, and when the processor runs the computer instructions stored in the memory, it executes the method described above.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, including instructions, which when run on a computer, cause the computer to execute the method described above.

[0015] In summary, the present application extracts prosodic feature information from the input text; performs semantic encoding on the input text through the BERT pre-trained model to generate text context feature representations; constructs a prosody controller including a duration predictor, an energy predictor, a fundamental period predictor, a pause predictor, and a rhythm predictor, and independently models the prosodic features through the prosody controller to generate prediction results; fuses the text context feature representations with the prediction results to generate a fused prosody control embedding; based on the VITS model, combines the prosody control embedding and the speaker embedding vector to generate a multi-speaker speech spectrum; converts the multi-speaker speech spectrum into a time-domain speech signal through a decoder; and realizes independent control of the duration, fundamental period, energy, pause, and rhythm of the synthesized speech by adjusting the prosodic feature parameters. Description of the Drawings

[0016] Figure 1 It is a schematic structural diagram of a computer device for the hardware operating environment involved in the solution of the embodiment of the present application; Figure 2 It is a schematic flowchart of the first embodiment of the prosody-controllable speech synthesis method based on VITS of the present application; Figure 3 It is a schematic diagram of the pause design of the prosody-controllable speech synthesis method based on VITS of the present application; Figure 4 It is an internal structural diagram of the prosody controller of the prosody-controllable speech synthesis method based on VITS of the present application; Figure 5 It is a structural block diagram of the first embodiment of the prosody-controllable speech synthesis device based on VITS of the present application. Detailed Description of the Embodiment

[0017] In order to make the objectives, technical solutions, and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] Refer to Figure 1 , Figure 1 which is a schematic structural diagram of a computer device for the hardware operating environment involved in the solution of the embodiment of the present application.

[0019] As Figure 1As shown in the figure, the computer device may include: a processor 1001, such as a Central Processing Unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed Random Access Memory (RAM) or a stable Non-Volatile Memory (NVM), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0020] Those skilled in the art can understand that Figure 1 the structure shown in does not constitute a limitation on the computer device, and it may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0021] As Figure 1 shown, in the memory 1005 as a storage medium, there may be included an operating system, a network communication module, a user interface module, and a prosody-controllable speech synthesis program based on VITS.

[0022] In Figure 1 the computer device shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; in the present application, the processor 1001 and the memory 1005 may be arranged in the computer device, and the computer device calls the prosody-controllable speech synthesis program stored in the memory 1005 through the processor 1001 and executes the prosody-controllable speech synthesis method provided in the embodiments of the present application.

[0023] The embodiments of the present application provide a prosody-controllable speech synthesis method based on VITS. Referring to Figure 2 , Figure 2 is a schematic flowchart of the first embodiment of the prosody-controllable speech synthesis method based on VITS in the present application.

[0024] In this embodiment, the prosody-controllable speech synthesis method based on VITS includes the following steps: S1: Extract prosodic feature information from the input text, where the prosodic feature information includes prosodic features such as duration, fundamental period, energy, pause, and rhythm.

[0025] It should be noted that in step S1: the duration is extracted by the forced alignment algorithm to obtain the phoneme duration, and a duration predictor is trained using maximum likelihood estimation; the fundamental period is decomposed into multi-scale components by continuous wavelet transform and quantified as control conditions; the energy is encoded after calculating the L2 norm of the Mel spectrum amplitude and quantization; the pause generates a pause duration sequence by analyzing text punctuation and phrase structure; the rhythm captures semantic associations through the BERT pre-trained model and adjusts the pronunciation speed in combination with positional embeddings.

[0026] In a specific implementation, the extraction of duration includes: The duration of each input phoneme in the text can be calculated by summing each row in the estimated alignment matrix. The row sum is to count the number of Mel frames occupied by each phoneme, which is mapped to the duration. Associate discrete phonemes with continuous Mel spectra to match the text and speech lengths. Design a duration predictor, usually trained by maximum likelihood estimation. Define the correspondence between the latent variable z and the distribution as the alignment A, and the dimension of A is (number of phonemes number of Mel frames), calculate the exact log-likelihood of the data: ; ; The first term: the log probability of the latent space the second term: the Jacobian determinant of the invertible transformation Let , , then: ; .

[0027] Among them, is the number of frames of the Mel spectrogram. The above formula realizes the invertible mapping from a simple Gaussian distribution to complex speech. Since the text and speech are monotonically aligned, we assume that the alignment A is monotonically increasing: emphasizing the consistency of speech-text timing and avoiding phoneme reversal. represents that given the text condition and alignment, the j-th element of the latent variable z follows a Gaussian distribution, with the mean predicted by and the variance predicted by . Find the parameters that maximize the log-likelihood (acoustic model parameters, such as neural network weights) and the alignment A such that the log probability of the speech data x is maximized given the text condition: ; Assume that the generation of the speech frame x follows a parametric distribution with a probability density of , the logarithmic transformation converts the product into a sum for easy optimization. The traditional method is not differentiable, and this formula realizes the gradient backpropagation through A.

[0028] The extraction of the fundamental period includes: Using continuous wavelet transform to decompose the continuous fundamental contour, and quantizing each frame of it into 256 possible values on the logarithmic scale as the control condition, which is converted into a control embedding through the control condition encoder. Given a continuous fundamental contour function F0, it is converted into a fundamental period spectrogram , using the continuous wavelet transform CWT: ; where is the Mexican hat wavelet, is x the fundamental value at the position of, and t are the scale and position of the wavelet respectively. The Mexican hat wavelet has the second derivative property to detect instantaneous changes, and can realize multi-resolution analysis through its scaling and translation t , is the energy normalization factor. The traditional STFT cannot adapt to the non-stationarity of the fundamental frequency, and the CWT has adaptive resolution in both the time-frequency domain. The original fundamental contour F0 can be calculated through the inverse continuous wavelet transform : ; Assume that F0 is decomposed into 10 scales, then F0 can be represented by 10 independent components: ; i = 1, …, 10 , = 5ms, logarithmic scale division: , the coefficient fits the sensitivity of the human ear to the change of the fundamental frequency. Given 10 wavelet components , can be reconstructed through the following formula: ; If , then its contribution to the reconstruction , separating the macro trend and micro fluctuations of the fundamental frequency is more robust than directly using the original F0.

[0029] The extraction of energy includes: Calculate the energy of each short-time Fourier transform frame by calculating the L2 norm of the mel_spectrogram amplitude and quantifying it on a logarithmic scale, and encode it using the control condition encoder. Let the short-time energy of the nth frame of the speech signal be denoted as En, and its calculation formula is: ; where M is the frame length, are the sample points in this frame. The short-time energy reflects the sum of the squares of the speech amplitudes, and the squaring operation enhances the discrimination of high-energy sections.

[0030] Example parameters: Let the frame length M = 400 (25ms @ 16kHz), and the sample values = [0.1, -0.2, 0.05,...]: .

[0031] The extraction of pauses includes: First, analyze the input text and split the whole sentence into phrase segments according to ",", and actually use a sequence of elements including pinyin and tones. Only text is used as an example in the sample diagram to briefly express the pause idea. Secondly, convert the processed text string into an ID sequence corresponding to the symbols in the text. Define the pause time: divide it into long pauses and short pauses, and there are different pause durations at different marked places.

[0032] The extraction of rhythm includes: Rhythm can be simply defined as the pronunciation speed of each Chinese character. If the speed is consistent, it will appear smooth. For interrogative tones, there are usually changes such as lengthening of the speech rate and rising of the pitch, so rhythm control is indispensable. Introduce a BERT-based prosody pre-training model to improve rhythm and prosody. By analogy with the above pause design, assuming there is already a position embedding, the pronunciation at the corresponding position can be lengthened or shortened. Among them, the schematic diagram of the pause design is as Figure 3 shown, and the meaning represented by the pause sequence is the corresponding pause duration.

[0033] S2: Semantically encode the input text through the BERT pre-training model to generate text context feature representations.

[0034] S3: Construct a prosody controller including a duration predictor, an energy predictor, a fundamental period predictor, a pause predictor, and a rhythm predictor, and independently model the prosody features through the prosody controller and generate prediction results.

[0035] In a specific implementation, in step S3: the prosody controller is composed of a one-dimensional convolutional network, a bidirectional long short-term memory network, and a linear layer. The outputs of each predictor are embedded and superimposed through a feature fusion module, and the Kolmogorov-Arnold representation theorem is applied to achieve the fusion of multi-variate features into a single-variable basis function. Each multi-variate continuous function can be represented as the superposition of a finite number of single-variable continuous functions. The specific formula is: ; where corresponds to a learnable basis function, that is, the inner function of the fusion module, is the outer function, which approximates the multi-variate function with a single-variable function. In actual application, it is transformed into a trainable neural network structure (i.e., the KAN fusion module). The inner and outer functions are designed as different MLP networks. The multi-variate function is decomposed into an internal summation, so that each input dimension x_p is independently mapped to the feature space. n is the number of features. In actual settings, the output results of each prosody feature through the inner function are added and then averaged as the input of the outer function, and directly output after passing through the outer function, without the superposition design in the formula, so as to improve the training speed.

[0036] Therefore, the specific formula in this embodiment is: ; where is the weight, is the basis function.

[0037] In a specific implementation, this embodiment designs a prosody controller. The internal structure diagram of the prosody controller is as shown in Figure 4 and includes duration prediction, energy prediction, pitch period prediction, pause prediction, and rhythm prediction. The predictor structure is composed of a one-dimensional convolutional network and a linear layer, with a ReLU activation function and dropout operation.

[0038] The duration predictor (abbreviated as D hereinafter) outside the prosody predictor uses the phoneme duration extracted by forced alignment MAS as the training target, inputs the output h_text of the text encoder, and the output duration is sent to the prosody predictor. Together with h_text, it passes through the Length Regulator (length normalization) and is used as the key information for subsequent output with other predictors. It is hereinafter denoted as x . x are respectively input into the remaining predictors.

[0039] The input of the pause predictor is the output of the phrase structure encoder, which performs word-level processing on the context representation; the pause predictor uses syntactic representation to predict the speaker-related pause sequence. The pause predictor consists of two bidirectional long short-term memory (Bi-LSTM) layers and a one-dimensional convolutional network, with ReLU activation, layer normalization, and dropout operations. The final linear layer projects the hidden representation into a word-level pause sequence. In addition, the pause predictor is optimized by the cross-entropy loss between the probability distribution and the target pause label sequence.

[0040] S4: Fuse the text context feature representation and the prediction result to generate a fused prosody control embedding.

[0041] It should be noted that the corresponding embeddings obtained by all predictors are respectively input into the inner function , and after adding and averaging the outputs of all inner functions, the fusion of the outer function is used as the final output of the prosody controller.

[0042] Among them, and : The Kolmogorov-Arnold representation theorem means that every multivariate continuous function can be represented as a superposition of a finite number of univariate continuous functions. The key points in understanding the Kolmogorov-Arnold representation theorem: 1. Continuity: For any point on the function, no matter from which direction to approach, the values are equal; (1) The multivariate function (the target function to be fitted) is continuous; (2) The decomposed univariate functions (hereinafter simply referred to as basis functions, or univariate basis functions) are also continuous; 2. Finiteness: The number of decomposed univariate basis functions is finite.

[0043] This representation method can be used for feature fusion, and the specific steps are as follows: 1. Inner function Design: Use MLP + LayerNorm to replace the spline function.

[0044] 2. Inner combination: It is designed as an MLP structure including a linear layer, a LayerNorm layer, and a SiLU activation function, simplifying the summation part in the theorem to the average after adding features, and reducing the computational complexity.

[0045] 3. Outer function : The outer function is designed to have a similar MLP structure to the inner function, realizing non-linear mapping through MLP, extending the pure linear requirement of the theorem to adapt to speech tasks, adding residual connections to maintain information integrity. Adding a LayerNorm layer can stabilize the feature scale and solve problems such as distribution shift and gradient explosion during training caused by the MLP replacing the spline.

[0046] S5: Based on the VITS model, combine the prosody control embedding and the speaker embedding vector to generate a multi-speaker speech spectrum.

[0047] It should be noted that in step S5: The posterior encoder of the VITS model takes the Mel spectrum as input to generate the latent variable z, and converts the latent variable distribution into a prior distribution through a flow model; the speaker embedding vector is concatenated with the text feature and the prosody control embedding and then input into the decoder to generate a multi-speaker speech spectrum.

[0048] In a specific implementation, the steps of Chinese speech synthesis include: establishing a multi-speaker Chinese speech synthesis model based on VITS, including multiple modules such as an encoder, a discriminator, a decoder, a random duration predictor, and a prosody controller.

[0049] The text encoder adopts a Transformer structure, converts the input text into an embedding vector, adds positional encoding to retain the order information of the text, extracts the context information of the text through multiple Transformer layers, and generates a high-dimensional feature representation.

[0050] Fuse the prosody features and text features, further extract and model the prosody features through multiple convolutional layers, and generate the final prosody feature prediction value through a prediction layer.

[0051] Introduce a speaker embedding vector so that the model can distinguish the timbre and prosody features of different speakers: (1) Convert the speaker information into an embedding vector; (2) Fuse the speaker embedding vector with the text features and prosody features.

[0052] Take the latent variable h_text generated by the text encoder and the duration information predicted by the duration predictor D(duration) as the input of the prosody controller. Denote each predictor simply as E(energy), P(pitch), R(rhythm), and T(pause). In addition, the energy, fundamental pitch period, etc. extracted from the speech dataset are used as the target input of the E, P, R, and T predictors. Output the result of each predictor as an embedding, and continuously fuse it with the LR'd x as the output of the prosody controller. During inference, appropriate prosody feature parameter control of the generated embedding can be achieved by adjusting the corresponding parameters.

[0053] Then, through the linear projection layer, the mean and variance for constructing the prior distribution are generated. A noise is sampled from a Gaussian distribution on (0, 1) and input together with h_text into the period predictor to generate a list of periods. According to the duration, the corresponding mapping is constructed, and the corresponding Gaussian distribution is copied to match the audio length.

[0054] During training, the posterior encoder takes the linear spectrum as input and outputs the latent variable z. The audio features are transformed to size - 192 through one - dimensional convolution, which is consistent with the size of the text embedding; in the inference stage, the latent variable z is generated by the flow model. The flow layer transforms the z of the Q distribution into the P distribution, increasing the modeling ability. Finally, through the decoder of HiFiGAN V1, training with slices is to make the length of each sample consistent. The decoder is a process of a group of upsamplings, and finally restores to an audio segment of size 256.

[0055] S6: Convert the multi - speaker speech spectrum into a time - domain speech signal through a decoder.

[0056] In a specific implementation, the decoder adopts the HiFiGAN V1 structure and converts the speech spectrum into a time - domain waveform through multi - level upsampling.

[0057] S7: Independently control the duration, fundamental period, energy, pause, and rhythm of the synthesized speech by adjusting prosodic feature parameters.

[0058] In step S7: The control of the duration realizes the change of the speech rate by adjusting the output multiple of the duration predictor; the control of the fundamental period adjusts the pitch by modifying the amplitude of the wavelet component; the control of the pause specifies the pause duration through the marking symbols in the input text; the control of the rhythm dynamically adjusts the pronunciation speed through the semantic weights output by BERT.

[0059] In a specific implementation, the steps of prosody control include: appropriately adjusting the embedding output by the corresponding predictor through input parameters to achieve the purpose of prosody control. The length of the speech duration directly affects the speech rate. A shorter duration means a faster speech rate, while a longer duration results in a slower speech rate. For speech rate control, it is reflected in multiplying the output of the duration predictor by a certain value. A faster speech rate is shown when the multiplied value is less than 1, and a slower speech rate is shown when the multiplied value is greater than 1.

[0060] The fundamental period reflects the time interval between two adjacent openings and closings of the vocal cords. Among them, the fundamental frequency (f0 / F0) is the reciprocal of the fundamental period, corresponding to the frequency of the vocal cord vibration of the human voice and representing the pitch of the sound. Usually, the faster the vocal cords vibrate, the higher the fundamental frequency.

[0061] The short-time energy reflects the strength of the signal at different moments and represents the strength of the sound signal.

[0062] Appropriate pauses between adjectives and nouns can help listeners identify the modification relationship. Natural speech contains many subtle pauses, and the reproduction of these pauses in synthetic speech can make it sound more natural and realistic. At the same time, pauses are also essential for expressing emotions such as surprise and hesitation. By precisely controlling pauses, the speech synthesis system can generate more natural, clear, and expressive speech output. The control of pauses is reflected in different markers of the input text: _pause = ["sil", "eos", "sp", "#0", "#1", "#2", "#3"] Among them, sil: the text start marker, encoded as index 0; eos: silence; sp: comma, short silence, encoded as index 2; it is designed that "#0" is a short pause marker and "#1" is a long pause marker, and the pause duration can be manually input.

[0063] In speech synthesis, rhythm refers to the relative timing and intensity patterns of various syllables, words, or sentences in speech, which determines the musicality and fluency of language, is closely related to pauses, speaking speed, and intensity, and together constitutes the prosodic structure of language.

[0064] It can be understood that it also includes a training step: using the AISHELL-3 or BZNSYP Chinese speech dataset, extracting the duration, fundamental period, energy, and pause duration of real speech as training targets; by minimizing the reconstruction error between the predicted values and the real values of each prosodic feature, jointly optimizing the parameters of the text encoder, prosody controller, and VITS model.

[0065] In specific implementation, the steps of prosody controller training include: in the preprocessing of the dataset, first filter out the rhotacized sounds not supported by this project, extract and save information such as energy, fundamental period, and spectrum, and use these prosodic feature elements extracted as the training targets of the corresponding predictors.

[0066] In the training stage, use forced alignment to calculate the phoneme duration information, input the duration information parameters obtained by the duration predictor into the prosody controller, and use the output after the Length Regulator as the input of the remaining rhythm, energy, and fundamental period predictors. Use the extracted intensity, fundamental period, etc. information as the targets of their respective predictors for training. By minimizing the reconstruction error of prosodic features, train the prosody controller to accurately model prosodic features. The specific steps are as follows: (1) Energy reconstruction error: Calculate the error between the predicted energy and the extracted real energy; (2) Pitch period reconstruction error: Calculate the error between the predicted pitch period and the extracted true pitch period; (3) Rhythm reconstruction error: Calculate the error between the predicted rhythm and the extracted true rhythm; (4) Pause reconstruction error: Calculate the error between the predicted pause and the extracted true pause; Optimization: Optimize the prosody controller parameters by minimizing the above reconstruction errors.

[0067] It should be noted that in this embodiment, the data processing method is as follows: Use the AISHELL-3 Chinese Mandarin speech database as the training and test data, which can be used as a multi-speaker synthesis system. The number of speakers in this dataset is 174. Each dataset contains full-bandwidth and mono audio, with a speech duration of 85 hours and a sampling frequency of 44.1 kHz. Use the BZNSYP-Beibiao Chinese standard female single-speaker dataset to simply replace AISHELL-3 as the alternative dataset during development, which is convenient for directly used in model training and reduces the workload of data preprocessing. It contains 10,000 audio data, with a total duration of about 10 hours and a sampling rate of 22,050 Hz. During the data preprocessing stage, the sampling rate is unified to 16,000 Hz.

[0068] In this embodiment, prosody feature information is extracted from the input text; the input text is semantically encoded by the BERT pre-trained model to generate text context feature representations; a prosody controller including a duration predictor, an energy predictor, a pitch period predictor, a pause predictor, and a rhythm predictor is constructed, and the prosody features are independently modeled by the prosody controller to generate prediction results; the text context feature representations are fused with the prediction results to generate fused prosody control embeddings; based on the VITS model, combined with the prosody control embeddings and speaker embedding vectors, multi-speaker speech spectra are generated; the multi-speaker speech spectra are converted into time-domain speech signals by a decoder; by adjusting the prosody feature parameters, independent control of the duration, pitch period, energy, pause, and rhythm of the synthesized speech is achieved.

[0069] In addition, an embodiment of the present application also proposes a computer-readable storage medium, on which a program for prosody-controllable speech synthesis based on VITS is stored. When the program for prosody-controllable speech synthesis based on VITS is executed by a processor, the steps of the method for prosody-controllable speech synthesis based on VITS as described above are implemented.

[0070] Refer to Figure 5 , Figure 5 which is the structural block diagram of the first embodiment of the prosody-controllable speech synthesis device based on VITS of the present application.

[0071] As Figure 5 shown, the prosody-controllable speech synthesis device based on VITS proposed in the embodiment of the present application includes: An extraction module 10, configured to extract prosody feature information from the input text, where the prosody feature information includes prosody features such as duration, fundamental period, energy, pause, and rhythm; A semantic encoding module 20, configured to perform semantic encoding on the input text through a BERT pre-trained model to generate a text context feature representation; A predictor construction module 30, configured to construct a prosody controller including a duration predictor, an energy predictor, a fundamental period predictor, a pause predictor, and a rhythm predictor, and independently model the prosody features through the prosody controller and generate a prediction result; A fusion module 40, configured to fuse the text context feature representation and the prediction result to generate a fused prosody control embedding; A voice spectrum module 50, configured to generate a multi-speaker voice spectrum based on the VITS model, in combination with the prosody control embedding and a speaker embedding vector; A conversion module 60, configured to convert the multi-speaker voice spectrum into a time-domain voice signal through a decoder; A control module 70, configured to independently control the duration, fundamental period, energy, pause, and rhythm of the synthesized speech by adjusting prosody feature parameters.

[0072] It should be understood that the above is only for illustration and does not impose any limitation on the technical solution of the present application. In specific applications, those skilled in the art can set according to needs, and the present application does not make any restrictions in this regard.

[0073] In this embodiment, prosody feature information is extracted from the input text; semantic encoding is performed on the input text through a BERT pre-trained model to generate a text context feature representation; a prosody controller including a duration predictor, an energy predictor, a fundamental period predictor, a pause predictor, and a rhythm predictor is constructed, and the prosody features are independently modeled through the prosody controller and a prediction result is generated; the text context feature representation and the prediction result are fused to generate a fused prosody control embedding; based on the VITS model, in combination with the prosody control embedding and a speaker embedding vector, a multi-speaker voice spectrum is generated; the multi-speaker voice spectrum is converted into a time-domain voice signal through a decoder; and the duration, fundamental period, energy, pause, and rhythm of the synthesized speech are independently controlled by adjusting prosody feature parameters.

[0074] It should be noted that the above-described work process is only illustrative and does not limit the protection scope of this application. In actual applications, those skilled in the art can select some or all of them according to actual needs to achieve the purpose of the solution of this embodiment, and no limitation is made here.

[0075] In addition, for the technical details not described in detail in this embodiment, reference can be made to the method for prosody-controllable speech synthesis based on VITS provided in any embodiment of this application, and details will not be repeated here.

[0076] In addition, it should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or system including that element.

[0077] The serial numbers of the embodiments of the present application above are only for description and do not represent the superiority or inferiority of the embodiments.

[0078] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as Read-Only Memory (ROM) / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application. The above is only the preferred embodiment of this application, and does not limit the patent scope of this application accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of this application, or directly or indirectly applied in other related technical fields, is equally included in the patent protection scope of this application.

Claims

1. A prosody-controllable speech synthesis method based on VITS, characterized in that, Including: S1: Extract prosodic feature information from the input text, where the prosodic feature information includes prosodic features such as duration, fundamental period, energy, pause, and rhythm; S2: Semantically encode the input text through the BERT pre-trained model to generate text context feature representations; S3: Construct a prosody controller including a duration predictor, an energy predictor, a fundamental period predictor, a pause predictor, and a rhythm predictor, and independently model the prosodic features through the prosody controller and generate prediction results; S4: Fuse the text context feature representations with the prediction results to generate fused prosody control embeddings; S5: Based on the VITS model, combine the prosody control embeddings and speaker embedding vectors to generate multi-speaker speech spectrograms; S6: Convert the multi-speaker speech spectrograms into time-domain speech signals through a decoder; S7: Realize independent control of the duration, fundamental period, energy, pause, and rhythm of the synthesized speech by adjusting prosodic feature parameters.

2. The prosody controllable speech synthesis method based on VITS according to claim 1, characterized in that In the step S1: The duration extracts the phoneme duration through the forced alignment algorithm and trains the duration predictor using maximum likelihood estimation; The fundamental period is decomposed into multi-scale components through continuous wavelet transform and quantized into control conditions; The energy is encoded after calculating the L2 norm of the Mel spectrogram amplitude and quantizing; The pause generates a pause duration sequence by analyzing text punctuation marks and phrase structures; The rhythm captures semantic associations through the BERT pre-trained model and adjusts the pronunciation speed in combination with position embeddings.

3. The prosody-controllable speech synthesis method based on VITS according to claim 1, characterized in that, In the step S3: The prosody controller is composed of a one-dimensional convolutional network, a bidirectional long short-term memory network, and a linear layer. The outputs of each predictor are stacked through a feature fusion module, and the Kolmogorov-Arnold representation theorem is applied to realize the fusion of multivariate features into a unary basis function. The specific formula is: ； Among them, is the weight, is the basis function.

4. The prosody-controllable speech synthesis method based on VITS according to claim 1, wherein, In the step S5: The posterior encoder of the VITS model takes the Mel spectrogram as input to generate latent variables z, and converts the latent variable distribution into a prior distribution through a flow model; The speaker embedding vector is concatenated with the text features and prosody control embeddings and input into the decoder to generate multi-speaker speech spectrograms.

5. The prosody-controllable speech synthesis method based on VITS according to claim 1, characterized in that In the step S7: The control of the duration realizes the change of speech speed by adjusting the output multiple of the duration predictor; The control of the fundamental period adjusts the pitch by modifying the amplitude of the wavelet components; The control of the pause specifies the pause duration through the marking symbols in the input text; The control of the rhythm dynamically adjusts the pronunciation speed through the semantic weights output by BERT.

6. The prosody controllable speech synthesis method based on VITS according to claim 1, wherein It also includes a training step: Use the AISHELL-3 or BZNSYP Chinese speech dataset to extract the duration, fundamental period, energy, and pause duration of real speech as training targets; Jointly optimize the parameters of the text encoder, prosody controller, and VITS model by minimizing the reconstruction error between the predicted values and the real values of each prosodic feature.

7. The prosody controllable speech synthesis method based on VITS according to claim 1, wherein The decoder adopts the HiFiGAN V1 structure and converts the speech spectrogram into a time-domain waveform through multi-level upsampling.

8. A prosody-controllable speech synthesis device based on VITS, characterized in that, Including: An extraction module for extracting prosodic feature information from the input text, where the prosodic feature information includes prosodic features of duration, fundamental period, energy, pause, and rhythm; A semantic encoding module for semantically encoding the input text through a BERT pre-trained model to generate a text context feature representation; A predictor construction module for constructing a prosody controller including a duration predictor, an energy predictor, a fundamental period predictor, a pause predictor, and a rhythm predictor, and independently modeling the prosodic features through the prosody controller and generating prediction results; A fusion module for fusing the text context feature representation and the prediction results to generate a fused prosody control embedding; A voice spectrum module for generating a multi-speaker voice spectrum based on the VITS model, combining the prosody control embedding and the speaker embedding vector; A conversion module for converting the multi-speaker voice spectrum into a time-domain voice signal through a decoder; A control module for independently controlling the duration, fundamental period, energy, pause, and rhythm of the synthesized speech by adjusting prosodic feature parameters.

9. A computer device, characterized in that, The device includes: a memory, a processor, and when the processor runs the computer instructions stored in the memory, it executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Including instructions that, when run on a computer, cause the computer to execute the method according to any one of claims 1 to 7.

Images