Method and apparatus for training vocoder and speech synthesis

Abstract

The present disclosure provides a method and device for training a vocoder and speech synthesis, and relates to the field of deep learning technology, and in particular to the field of speech technology. The specific implementation scheme is: obtaining a training sample set, wherein the training sample includes a spectrogram, fundamental frequency features, and voiced features; selecting a training sample from the training sample set; performing the following training steps: inputting the spectrogram in the selected training sample into the vocoder to obtain an intermediate feature; inputting the intermediate feature into the fundamental frequency prediction network to obtain a fundamental frequency prediction value; calculating a loss value based on the fundamental frequency features, voiced features, and fundamental frequency prediction values ​​in the selected training sample; if the loss value is less than a predetermined threshold, the vocoder training is completed. This implementation can generate a vocoder that solves problems that affect sound quality, such as vibrato and muteness, thereby improving the quality of speech synthesis.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning technology, in particular to the field of speech technology, and specifically to a method and device for training a vocoder and speech synthesis. Background Art

[0002] The speech synthesis vocoder reconstructs the speech waveform from acoustic features such as fundamental frequency and spectrum, and is an indispensable part of the speech synthesis system. The accuracy, naturalness and synthesis speed of the synthesized speech are important indicators to measure the performance of the vocoder. In recent years, with the successful application of deep neural network models in acoustic modeling, the accuracy and naturalness of speech synthesis have been effectively improved. However, due to the poor restoration of the fundamental frequency of the vocoder, problems such as vibrato and muteness that affect the sound quality will occur. Summary of the invention

[0003] The present disclosure provides a method, apparatus, device, storage medium and computer program product for training a vocoder and speech synthesis.

[0004] According to a first aspect of the present disclosure, a method for training a vocoder is provided, comprising: obtaining a training sample set, wherein the training sample comprises a spectrogram, fundamental frequency features, and voiced features; selecting a training sample from the training sample set; performing the following training steps: inputting the spectrogram in the selected training sample into the vocoder to obtain an intermediate feature; inputting the intermediate feature into a fundamental frequency prediction network to obtain a fundamental frequency prediction value; calculating a loss value based on the fundamental frequency features, voiced features, and fundamental frequency prediction values ​​in the selected training sample; and determining that the vocoder training is completed if the loss value is less than a predetermined threshold.

[0005] According to a second aspect of the present disclosure, a speech synthesis method is provided, comprising: acquiring acoustic features to be processed; inputting the acoustic features into a vocoder trained according to the method of the first aspect, and outputting synthesized speech.

[0006] According to a third aspect of the present disclosure, there is provided a device for training a vocoder, comprising: an acquisition unit configured to acquire a training sample set, wherein the training sample comprises a spectrogram, fundamental frequency features, and voiced features; a training unit configured to select training samples from the training sample set; and performing the following training steps: inputting the spectrogram in the selected training sample into the vocoder to obtain intermediate features; inputting the intermediate features into a fundamental frequency prediction network to obtain fundamental frequency prediction values; calculating a loss value based on the fundamental frequency features, voiced features, and fundamental frequency prediction values ​​in the selected training sample; and determining that the vocoder training is completed if the loss value is less than a predetermined threshold.

[0007] According to a fourth aspect of the present disclosure, a speech synthesis device is provided, comprising: an acquisition unit configured to acquire acoustic features to be processed; a synthesis unit configured to input the acoustic features into a vocoder trained by the device described in any one of the third aspects, and output synthesized speech.

[0008] According to the fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect or the second aspect.

[0009] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect or the second aspect.

[0010] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the method described in the first aspect or the second aspect when executed by a processor.

[0011] The method and device for training a vocoder provided by the embodiment of the present disclosure utilizes fundamental frequency prediction to assist in the supervision of the vocoder, adopts an insertable structure, is only used in the training phase, does not affect the main structure of the neural network vocoder, and does not increase the amount of calculation. It solves the problems of vibrato and mute that affect the sound quality, thereby improving the quality of speech synthesis.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;

[0015] Figure 2 is a flow chart of one embodiment of a method for training a vocoder according to the present disclosure;

[0016] Figure 3 is a schematic diagram of an application scenario of the method for training a vocoder according to the present disclosure;

[0017] Figure 4 is a flow chart of an embodiment of a method for speech synthesis according to the present disclosure;

[0018] Figure 5 is a structural schematic diagram of an embodiment of an apparatus for training a vocoder according to the present disclosure;

[0019] Figure 6 is a structural schematic diagram of an embodiment of a speech synthesis device according to the present disclosure;

[0020] Figure 7 It is a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] Figure 1 An exemplary system architecture 100 is shown to which the method for training a vocoder, the apparatus for training a vocoder, the method for speech synthesis, or the apparatus for speech synthesis according to the embodiments of the present disclosure can be applied.

[0023] like Figure 1 As shown, the system architecture 100 may include terminals 101, 102, a network 103, a database server 104, and a server 105. The network 103 is used to provide a medium for communication links between the terminals 101, 102, the database server 104, and the server 105. The network 103 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0024] The user 110 can use the terminals 101 and 102 to interact with the server 105 through the network 103 to receive or send messages, etc. Various client applications can be installed on the terminals 101 and 102, such as model training applications, speech synthesis applications, shopping applications, payment applications, web browsers, and instant messaging tools.

[0025] The terminals 101 and 102 here can be hardware or software. When the terminals 101 and 102 are hardware, they can be various electronic devices with voice acquisition devices, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Dynamic Image Experts Compression Standard Audio Layer 3), laptops and desktop computers, etc. When the terminals 101 and 102 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0026] When the terminals 101 and 102 are hardware, a voice collection device may be installed thereon. The voice collection device may be any device capable of collecting voice, such as a microphone, etc. The user 110 may use the voice collection device on the terminals 101 and 102 to collect the voice of himself or others.

[0027] The database server 104 may be a database server that provides various services. For example, a training sample set may be stored in the database server. The training sample set includes a large number of training samples. The training samples may include a spectrogram, fundamental frequency features, and voiced features. In this way, the user 110 may also select a training sample from the training sample set stored in the database server 104 through the terminals 101 and 102.

[0028] The server 105 may also be a server that provides various services, such as a background server that supports various applications displayed on the terminals 101 and 102. The background server may train the initial vocoder using the training samples in the training sample set sent by the terminals 101 and 102, and may send the training results (such as the generated vocoder) to the terminals 101 and 102. In this way, the user may use the generated vocoder for speech synthesis.

[0029] The database server 104 and server 105 here can also be hardware or software. When they are hardware, they can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When they are software, they can be implemented as multiple software or software modules (for example, used to provide distributed services), or as a single software or software module. No specific limitation is made here. The database server 104 and server 105 can also be servers of a distributed system, or servers combined with blockchain. The database server 104 and server 105 can also be cloud servers, or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology.

[0030] It should be noted that the method for training a vocoder or the method for speech synthesis provided in the embodiments of the present disclosure is generally executed by the server 105. Accordingly, the device for training a vocoder or the device for speech synthesis is generally also disposed in the server 105.

[0031] It should be pointed out that, in the case where the server 105 can implement the related functions of the database server 104 , the database server 104 may not be provided in the system architecture 100 .

[0032] It should be understood that Figure 1 The number of terminals, networks, database servers and servers in the embodiment is only for illustration. Any number of terminals, networks, database servers and servers may be provided according to the implementation requirements.

[0033] Continue to see Figure 2 , which shows a process 200 of an embodiment of a method for training a vocoder according to the present disclosure. The method for training a vocoder may include the following steps:

[0034] Step 201: Obtain a training sample set.

[0035] In this embodiment, the execution body of the method for training the vocoder (eg Figure 1 The server 105 shown in FIG. 104 may obtain the training sample set in a variety of ways. For example, the execution subject may obtain the training sample set from a database server (e.g. Figure 1 For example, the user can obtain the existing training sample set stored in the database server 104 shown in FIG. Figure 1 In this way, the execution subject can receive the training samples collected by the terminal and store these training samples locally, thereby generating a training sample set.

[0036] Here, the training sample set may include at least one training sample. Among them, the training sample may include a spectrogram, fundamental frequency features, and clear and voiced features. If the original signal is a sound signal, then the two-dimensional signal obtained by STFT (Short Time Fourier Transform) is the so-called spectrogram. The spectrogram is actually rotating the spectrogram of each frame to the left by 90 degrees, using the depth of color to represent the amplitude. The larger the amplitude, the darker the color. Then, the color information of each frame is listed in the order of time (frame). Therefore, the horizontal axis of the spectrogram is time (frame), the vertical axis is frequency, and the color is the amplitude of the frequency. The spectrogram is often a large picture. In order to obtain the sound features of the appropriate size, it is often transformed into a mel spectrum through a mel-scale filter bank, and the log is taken to obtain a log mel spectrogram.

[0037] Pitch is one of the three major properties of sound (volume, pitch, timbre). Except for some extreme cases, pitch is determined by the fundamental frequency of the sound, so the terms "pitch" and "fundamental frequency" are often used interchangeably in literature. Sounds produced by regular vibrations generally have a fundamental frequency, such as vowels and voiced consonants in speech, and the sounds produced by musical instruments that can play melodies. Some sounds do not have a fundamental frequency and they sound like noise, such as unvoiced consonants in speech and the sounds produced by percussion instruments.

[0038] Pitch estimation and pitch tracking are widely used in sound processing. Its most direct application is to identify the melody of music. It can also be used in speech processing, such as assisting speech recognition of tonal languages ​​(such as Chinese) and identifying emotions in speech.

[0039] Since the fundamental frequency of sound often changes over time, fundamental frequency extraction usually divides the signal into frames (the frame length is usually tens of milliseconds) and then extracts the fundamental frequency frame by frame. The methods for extracting the fundamental frequency of a frame of sound can be roughly divided into time domain method and frequency domain method. The time domain method takes the waveform of the sound as input, and its basic principle is to find the minimum positive period of the waveform. Of course, the periodicity of the actual signal can only be approximate. The frequency domain method will first perform a Fourier transform on the signal to obtain a spectrum (only the amplitude spectrum is taken, and the phase spectrum is discarded). There will be peaks on the spectrum at integer multiples of the fundamental frequency. The basic principle of the frequency domain method is to find the greatest common divisor of these peak frequencies.

[0040] Considering that the fundamental frequency is not present in every frame, it is often necessary to determine whether the fundamental frequency exists before, after, or at the same time as extracting the fundamental frequency. This is called unvoiced / voiced decision (U / V decision), also known as unvoiced and voiced feature, or UV feature for short. 0 means unvoiced sound, and 1 means voiced sound.

[0041] The mature fundamental frequency extraction methods such as SWIPE and CREPE can be used to extract fundamental frequency features and voiced and unvoiced features from the original speech signal corresponding to the spectrogram. For the convenience of training, the fundamental frequency features are converted to the log domain and recorded as log-f0. The UV feature is used to indicate whether the frame-level data is unvoiced or voiced.

[0042] Step 202: Select training samples from the training sample set.

[0043] In this embodiment, the execution subject may select a training sample from the training sample set obtained in step 201, and perform the training steps from step 203 to step 208. The selection method and the number of selected training samples are not limited in the present disclosure. For example, at least one training sample may be randomly selected, or a training sample with more voiced sounds may be selected from the training samples, or the same training sample may be repeatedly selected.

[0044] Step 203, input the selected spectrogram in the training sample into the vocoder to obtain intermediate features.

[0045] In this embodiment, the vocoder may adopt a mainstream neural network vocoder that uses progressive upsampling, and may include multiple upsampling modules (each upsampling module also includes an upsampling layer and a residual module), such as MELGAN and HiFiGAN. Using frame-level acoustic features (e.g., Mel-spectrogram) as input, each upsampling module performs integer multiple upsampling (Upsampling Layer) on the result of the previous level, and then passes through a residual module (ResBlock) to obtain intermediate features and output them to the next level.

[0046] Step 204: input the intermediate features into the fundamental frequency prediction network to obtain the fundamental frequency prediction value.

[0047] In this embodiment, the intermediate features can also be input into the fundamental frequency prediction network. The fundamental frequency prediction network may include at least one fundamental frequency prediction module, each fundamental frequency prediction module includes a downsampling module and an output layer (convolution layer). The downsampling module downsamples the intermediate features output by the upsampling module to the frame level. The output layer includes a multi-layer network of convolutions with large convolution kernels with a step size, which is not limited to the output layer, and the output layer is responsible for compressing the number of channels to one dimension, which is consistent with the dimension of the fundamental frequency sequence. The fundamental frequency prediction network can predict the fundamental frequency features according to the input intermediate features to obtain the fundamental frequency prediction value.

[0048] The intermediate features output by any upsampling module in the vocoder can be input into a baseband prediction module for prediction. The intermediate features output by the last upsampling module can also be selected to be input into a baseband prediction module for prediction.

[0049] Step 205: Calculate the loss value based on the fundamental frequency feature, the clear and voiced feature, and the fundamental frequency prediction value in the selected training sample.

[0050] In this embodiment, the loss function is calculated for the output of the fundamental frequency prediction network using the real fundamental frequency data (fundamental frequency features and voiced features in the training sample) acquired in advance. The loss function may adopt the absolute value error (L1 loss) or the square error (L2 loss). The loss function may also adopt the combination of L1 loss and L2 loss, which has the advantages of both and is conducive to improving the prediction accuracy of the fundamental frequency.

[0051] The specific calculation method is as follows:

[0052]

[0053] Among them, Loss represents the loss value, u i Represents the UV features of each frame, 0 represents unvoiced sound, 1 represents voiced sound, f i represents the true fundamental frequency (log-f0) of each frame, represents the output of the fundamental frequency prediction module, N represents the number of speech frames in the spectrogram. α represents the weight of the L1 loss, and β represents the weight of the L2 loss. When α=0, only the L2 loss is used. When β=0, only the L1 loss is used.

[0054] Step 206: If the loss value is less than a predetermined threshold, it is determined that the vocoder training is completed.

[0055] In this embodiment, if the loss value is less than a predetermined threshold, it means that the vocoder training is completed and can be used for speech synthesis.

[0056] Step 207: If the loss value is greater than or equal to the predetermined threshold, relevant parameters of the vocoder and relevant parameters of the baseband prediction network are adjusted, and steps 203-207 are continued.

[0057] In this embodiment, if the loss value is greater than or equal to the predetermined threshold, it means that the vocoder has not been trained yet, and the relevant parameters of the vocoder and the relevant parameters of the baseband prediction network need to be adjusted. For example, the weights in each convolution layer in the vocoder and the baseband prediction network are modified by back propagation technology. And the method can return to step 202 and reselect training samples from the training sample set. Thus, the above training steps can be continued. The selected training samples can also be reused to continue the above training steps.

[0058] It should be noted that the selection method here is not limited in the present disclosure. For example, when there are a large number of training samples in the training sample set, the execution subject can select training samples that have not been selected from them.

[0059] The method for training a vocoder in this embodiment uses fundamental frequency prediction to perform auxiliary supervision of the vocoder, adopts an inserted structure, is only used in the training phase, does not affect the main structure of the neural network vocoder, and does not increase the amount of calculation. Thus, a vocoder that can be used for speech synthesis can be obtained. The problem of vibrato and mute that affect the sound quality during speech synthesis is solved, thereby improving the quality of speech synthesis.

[0060] In some optional implementations of the present embodiment, the vocoder includes at least one upsampling module, the fundamental frequency prediction network includes at least one fundamental frequency prediction module, and the at least one upsampling module is connected to the fundamental frequency prediction module; and the intermediate features are input into the fundamental frequency prediction network to obtain the fundamental frequency prediction value, including: for at least one upsampling module connected to the fundamental frequency prediction module, the intermediate features output by the upsampling module are input into the fundamental frequency prediction module connected thereto, to obtain the fundamental frequency prediction value of the fundamental frequency prediction module; and the loss value is calculated based on the fundamental frequency features, unvoiced features, and fundamental frequency prediction values ​​in the selected training samples, including: for at least one fundamental frequency prediction module, the loss value of the fundamental frequency prediction module is calculated based on the fundamental frequency prediction value of the fundamental frequency prediction module, the fundamental frequency features, and unvoiced features in the selected training samples; and the weighted sum of the loss values ​​of the fundamental frequency prediction modules is used as the final loss value.

[0061] like Figure 3 The network structure of the vocoder and the network structure of the baseband prediction network shown in the figure, the vocoder includes at least one upsampling module, and each upsampling module includes an upsampling layer (Upsampling Layer) and a residual module (ResBlock). The vocoder may also include a preprocessing network (PreNet) and a post-processing network (PostNet). The preprocessing network can be used to extract some features from the mel-spectrogram. The post-processing network can convert the intermediate features output by the upsampling module into a speech waveform (wave). The upsampling multiple can be set according to demand. For example, if the sampling frequency of speech synthesis is 16K, a total of 160 times upsampling is required, and multiple upsampling modules can be connected in series to form 160 times upsampling. The outputs of some or all of the upsampling modules are respectively connected to the baseband prediction modules (each baseband prediction module is only connected to one upsampling module). Figure 3 Only the case where each upsampling module corresponds to one baseband prediction module is shown. In fact, some upsampling modules may not be connected to the baseband prediction module. Each baseband prediction module includes a downsampling module (Downsampling Block) and an output layer (Output Layer). The downsampling multiple of the downsampling module corresponds to the upsampling layer it is connected to. Figure 3As shown, 8 times downsampling corresponds to 8 times upsampling. 40 times downsampling corresponds to 8 times upsampling * 5 times upsampling, and 160 times downsampling corresponds to 8 times upsampling * 5 times upsampling * 4 times upsampling. Each fundamental frequency prediction module is independent. Each fundamental frequency prediction module predicts the fundamental frequency according to the input intermediate features, obtains the fundamental frequency prediction value, and then compares it with the reference value (fundamental frequency feature, voiced feature) in the training sample, calculates the loss value of each branch, and then accumulates the loss values ​​of all branches as the final loss value. If the loss value does not converge to the predetermined threshold, the relevant parameters of the vocoder (not limited to the upsampling module, but also including the preprocessing network and the post-processing network) and the relevant parameters of each fundamental frequency prediction module are adjusted. Using multiple fundamental frequency prediction modules for auxiliary supervised training can speed up the convergence of the vocoder and improve the quality of speech synthesis.

[0062] In some optional implementations of this embodiment, each upsampling module includes an upsampling layer and a residual module, and each baseband prediction module includes a downsampling module and an output layer. The upsampling layer is used for frame-level feature amplification and replication, and the residual module is used for synthesizing the high frequency of the audio. The downsampling module and the output layer are used together to extract baseband features from the intermediate features. This structure facilitates any combination of upsampling modules and baseband modules during training. Baseband modules with various downsampling multiples can be prepared in advance and trimmed according to the loss values ​​of different branches. For example, if the loss value of the branch where the baseband module with 8-fold downsampling is located is less than a certain value, the baseband module can be removed from the baseband prediction network. Only the parameters of the other two baseband modules are adjusted. This can speed up the training speed.

[0063] In some optional implementations of this embodiment, the loss value is calculated based on the fundamental frequency characteristics, clear and voiced characteristics, and fundamental frequency prediction values ​​in the selected training samples, including: calculating the absolute error loss value and the square error loss value based on the fundamental frequency characteristics, clear and voiced characteristics, and fundamental frequency prediction values ​​in the selected training samples; and taking the weighted sum of the absolute error loss value and the square error loss value as the final loss value. The weights of the two loss values ​​can be adjusted according to the quality of the output audio. The combination of the two loss values ​​can have the advantages of both, which is conducive to improving the prediction accuracy of the fundamental frequency.

[0064] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an application scenario of the method for training a vocoder according to this embodiment. Figure 3In the application scenario, a training sample set is prepared in advance, and each training sample includes a spectrogram, a fundamental frequency feature, and a voiced feature. Among them, the spectrogram is used as the input of the vocoder, and the fundamental frequency feature and the voiced feature are used as the supervision signal of the vocoder. The training sample is selected from the training sample set. The spectrogram is input into the vocoder, and after some features are obtained through the preprocessing network, upsampling is performed step by step. The intermediate features output by the upsampling module are not only input to the next level sampling module, but also output to the fundamental frequency prediction module in the fundamental frequency prediction network ( Figure 3 It is only schematically provided that a corresponding fundamental frequency prediction module is provided for each upsampling module. In fact, it is also possible to provide only one fundamental frequency prediction module). The fundamental frequency prediction module is used to extract the fundamental frequency feature from the intermediate feature, and then compare it with the fundamental frequency feature and the clear and voiced feature in the training sample to calculate the loss value. Each fundamental frequency prediction module corresponds to a branch, and the loss value on the branch is calculated. The loss values ​​on different branches are then accumulated to obtain the final loss value. The relevant parameters of the vocoder and the relevant parameters of the fundamental frequency prediction network are adjusted by the gradient descent method so that the loss value converges to less than the predetermined threshold, and the vocoder training is completed.

[0065] See also Figure 4 , which shows a process 400 of an embodiment of a speech synthesis method provided by the present disclosure. The speech synthesis method may include the following steps:

[0066] Step 401: Acquire acoustic features to be processed.

[0067] In this embodiment, the execution subject of the speech synthesis method (for example Figure 1 The server 105 shown in FIG. 104 can obtain the acoustic features to be processed in a variety of ways. For example, the execution subject can obtain the acoustic features from the database server (e.g. Figure 1 For example, the execution subject may also receive a terminal (eg, Figure 1 The terminal 101, 102 shown in the figure) or other devices performs text analysis on the text collected, and generates the corresponding language, phoneme (or syllable), rhythm, and emotion information according to the input text information. The synthesized language, phoneme, rhythm, emotion label and other features are used as input, and the text features are converted into the acoustic features (mel-spectrogram, etc.) of the corresponding audio through model learning.

[0068] Step 402, input the acoustic features into a vocoder and output synthesized speech.

[0069] In this embodiment, the execution subject may input the acoustic features obtained in step 401 into a vocoder to generate synthesized speech. The vocoder may be a vocoder as described above. Figure 2The specific generation process can be found in Figure 2 The relevant description of the embodiments will not be repeated here.

[0070] It should be noted that the speech synthesis method of this embodiment can be used to test the vocoders generated by the above embodiments. Then, the vocoders can be continuously optimized according to the test results. This method can also be a practical application method of the vocoders generated by the above embodiments. Using the vocoders generated by the above embodiments to perform speech synthesis helps to improve the sound quality of the synthesized speech, such as removing vibrato and mute.

[0071] Continue to see Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for training a vocoder. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0072] like Figure 5 As shown, the apparatus 500 for training a vocoder in this embodiment may include: an acquisition unit 501 and a training unit 502. The acquisition unit 501 is configured to acquire a training sample set, wherein the training sample includes a spectrogram, a fundamental frequency feature, and a voiced feature; the training unit 502 is configured to perform the following training steps: selecting a training sample from the training sample set; inputting the spectrogram in the selected training sample into the vocoder to obtain an intermediate feature; inputting the intermediate feature into a fundamental frequency prediction network to obtain a fundamental frequency prediction value; calculating a loss value based on the fundamental frequency feature, the voiced feature, and the fundamental frequency prediction value in the selected training sample; if the loss value is less than a predetermined threshold, determining that the vocoder training is completed.

[0073] In some optional implementations of this embodiment, the training unit 502 is further configured to: if the loss value is greater than or equal to a predetermined threshold, adjust relevant parameters of the vocoder and relevant parameters of the fundamental frequency prediction network, and continue to perform the training step.

[0074] In some optional implementations of this embodiment, the vocoder includes at least one upsampling module, the fundamental frequency prediction network includes at least one fundamental frequency prediction module, and at least one upsampling module is connected to the fundamental frequency prediction module; and the training unit 502 is further configured to: for at least one upsampling module connected to the fundamental frequency prediction module, input the intermediate features output by the upsampling module into the fundamental frequency prediction module connected thereto, to obtain the fundamental frequency prediction value of the fundamental frequency prediction module; for at least one fundamental frequency prediction module, calculate the loss value of the fundamental frequency prediction module based on the fundamental frequency prediction value of the fundamental frequency prediction module, the fundamental frequency features in the selected training samples, and the clear and voiced features; and take the weighted sum of the loss values ​​of the fundamental frequency prediction modules as the final loss value.

[0075] In some optional implementations of this embodiment, each upsampling module includes an upsampling layer and a residual module, and each baseband prediction module includes a downsampling module and an output layer.

[0076] In some optional implementations of this embodiment, the training unit 502 is further configured to: calculate the absolute error loss value and the square error loss value based on the fundamental frequency characteristics, clear and voiced characteristics, and fundamental frequency prediction values ​​in the selected training samples; and take the weighted sum of the absolute error loss value and the square error loss value as the final loss value.

[0077] Continue to see Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a speech synthesis device. Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0078] like Figure 6 As shown, the speech synthesis device 600 of this embodiment may include: an acquisition unit 601, configured to acquire acoustic features to be processed; a synthesis unit 602, configured to input the acoustic features into a vocoder trained by the device 500, and output synthesized speech.

[0079] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0080] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0081] An electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in process 200 or 400.

[0082] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method described in process 200 or 400.

[0083] A computer program product includes a computer program, wherein the computer program implements the method described in flow 200 or 400 when executed by a processor.

[0084] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0085] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0086] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0087] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a method for training a vocoder. For example, in some embodiments, the method for training a vocoder may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the method for training a vocoder described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform a method for training a vocoder in any other appropriate manner (e.g., by means of firmware).

[0088] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0089] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0090] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0091] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0092] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0093] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0094] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0095] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a vocoder, comprising: Obtaining a training sample set, wherein the training sample includes a spectrogram, fundamental frequency features, and voiced features; A training sample is selected from the training sample set, and the following training steps are performed: the spectrogram in the selected training sample is input into the vocoder to obtain an intermediate feature; the intermediate feature is input into the fundamental frequency prediction network to obtain a fundamental frequency prediction value; a loss value is calculated based on the fundamental frequency feature, voiced feature, and fundamental frequency prediction value in the selected training sample; if the loss value is less than a predetermined threshold, it is determined that the vocoder training is completed.

2. The method according to claim 1, wherein: The method further comprises: If the loss value is greater than or equal to a predetermined threshold, relevant parameters of the vocoder and relevant parameters of the fundamental frequency prediction network are adjusted, and the training step is continued.

3. The method according to claim 1, wherein: The vocoder includes at least one up-sampling module, the baseband prediction network includes at least one baseband prediction module, and the at least one up-sampling module is connected to the baseband prediction module; as well as Inputting the intermediate features into a fundamental frequency prediction network to obtain a fundamental frequency prediction value includes: For an up-sampling module connected to at least one fundamental frequency prediction module, inputting an intermediate feature output by the up-sampling module into the fundamental frequency prediction module connected thereto to obtain a fundamental frequency prediction value of the fundamental frequency prediction module; And the calculation of the loss value based on the fundamental frequency feature, the clear and voiced feature, and the fundamental frequency prediction value in the selected training sample includes: For at least one fundamental frequency prediction module, calculating a loss value of the fundamental frequency prediction module based on a fundamental frequency prediction value of the fundamental frequency prediction module, fundamental frequency features in the selected training samples, and clear and voiced features; The weighted sum of the loss values ​​of the fundamental frequency prediction modules is taken as the final loss value.

4. The method according to claim 3, wherein: Each upsampling module includes an upsampling layer and a residual module, and each baseband prediction module includes a downsampling module and an output layer.

5. The method according to any one of claims 1 to 4, wherein: The calculation of the loss value based on the fundamental frequency feature, the clear and voiced feature, and the fundamental frequency prediction value in the selected training sample includes: Calculate the absolute error loss value and the square error loss value based on the fundamental frequency characteristics, the clear and voiced characteristics, and the fundamental frequency prediction value in the selected training samples; The weighted sum of the absolute error loss value and the square error loss value is taken as the final loss value.

6. A speech synthesis method, comprising: Acquiring acoustic features to be processed; The acoustic features are input into a vocoder trained according to the method of any one of claims 1 to 5 to output synthesized speech.

7. A device for training a vocoder, comprising: An acquisition unit is configured to acquire a training sample set, wherein the training sample includes a spectrogram, a fundamental frequency feature, and a voiced feature; The training unit is configured to select training samples from the training sample set; perform the following training steps: input the spectrogram in the selected training sample into the vocoder to obtain intermediate features; input the intermediate features into the fundamental frequency prediction network to obtain fundamental frequency prediction values; calculate the loss value based on the fundamental frequency features, voiced features, and fundamental frequency prediction values ​​in the selected training sample; if the loss value is less than a predetermined threshold, determine that the vocoder training is completed.

8. The device according to claim 7, wherein: The training unit is further configured to: If the loss value is greater than or equal to a predetermined threshold, relevant parameters of the vocoder and relevant parameters of the fundamental frequency prediction network are adjusted, and the training step is continued.

9. The device according to claim 7, wherein: The vocoder includes at least one up-sampling module, the pitch prediction network includes at least one pitch prediction module, and the at least one up-sampling module is connected to the pitch prediction module; and The training unit is further configured to: For at least one up-sampling module connected to the fundamental frequency prediction module, inputting the intermediate features output by the up-sampling module into the fundamental frequency prediction module connected thereto to obtain a fundamental frequency prediction value of the fundamental frequency prediction module; For at least one fundamental frequency prediction module, calculating a loss value of the fundamental frequency prediction module based on a fundamental frequency prediction value of the fundamental frequency prediction module, fundamental frequency features in the selected training samples, and clear and voiced features; The weighted sum of the loss values ​​of the fundamental frequency prediction modules is taken as the final loss value.

10. The device according to claim 9, wherein: Each upsampling module includes an upsampling layer and a residual module, and each baseband prediction module includes a downsampling module and an output layer.

11. The device according to any one of claims 7 to 10, wherein: The training unit is further configured to: Calculate the absolute error loss value and the square error loss value based on the fundamental frequency characteristics, the clear and voiced characteristics, and the fundamental frequency prediction value in the selected training samples; The weighted sum of the absolute error loss value and the square error loss value is taken as the final loss value.

12. A speech synthesis device, comprising: An acquisition unit, configured to acquire acoustic features to be processed; A synthesis unit is configured to input the acoustic features into a vocoder trained by the device according to any one of claims 7 to 11, and output synthesized speech.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

15. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images