Speech synthesis method and device, electronic equipment and storage medium

Abstract

The present application relates to the technical field of audio processing, and provides a speech synthesis method and device, electronic equipment and storage medium. The target acoustic feature of the to-be-processed text is obtained by extracting the acoustic feature from the to-be-processed text using a preset acoustic model; the target fundamental frequency feature is obtained by extracting the fundamental frequency feature from the target acoustic feature using a pre-trained fundamental frequency predictor, and the target energy feature is obtained by extracting the energy feature from the target acoustic feature using a pre-trained energy predictor; the target acoustic feature, the target fundamental frequency feature and the target energy feature are input into a pre-trained general vocoder to generate the speech audio of the to-be-processed text; the general vocoder is trained based on the speech audio of multiple speakers. By taking the acoustic feature, the fundamental frequency feature and the energy feature as the input of the vocoder for speech synthesis, and by training the vocoder using the speech of multiple speakers, the vocoder has general applicability, the training time of the vocoder is reduced, and the effect of speech synthesis is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, in particular to a speech synthesis method and device, electronic equipment and storage medium. BACKGROUND

[0002] Speech synthesis technology (Text-to-Speech) refers to converting text into speech audio, which is commonly used in human-computer interaction, audio book reading and other scenarios. A speech synthesis model usually includes an acoustic model and a vocoder. The acoustic model is used to convert text into acoustic features, and the vocoder is used to convert acoustic features into speech audio.

[0003] At present, when constructing a speech synthesis model for a certain timbre, a special vocoder is usually trained based on the speech of the speaker corresponding to the timbre. However, training a special vocoder requires a large amount of time and limited training samples, resulting in poor speech synthesis effect and low efficiency. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a speech synthesis method, device, electronic equipment and storage medium.

[0005] In order to achieve the above purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0006] In a first aspect, the present application provides a speech synthesis method, which comprises:

[0007] extracting acoustic features from the to-be-processed text using a preset acoustic model to obtain target acoustic features of the to-be-processed text;

[0008] extracting fundamental frequency features from the target acoustic features using a pre-trained fundamental frequency predictor to obtain target fundamental frequency features;

[0009] extracting energy features from the target acoustic features using a pre-trained energy predictor to obtain target energy features;

[0010] inputting the target acoustic features, the target fundamental frequency features and the target energy features into a pre-trained universal vocoder to generate speech audio of the to-be-processed text; the universal vocoder is trained based on speech audio of multiple speakers.

[0011] In an optional embodiment, the universal vocoder is trained in the following manner:

[0012] obtaining speech audio of multiple speakers to obtain multiple audio samples;

[0013] extracting an acoustic feature, a fundamental frequency feature and an energy feature from each of the audio samples to obtain an actual acoustic feature, an actual fundamental frequency feature and an actual energy feature of each of the audio samples;

[0014] inputting the actual acoustic feature, the actual fundamental frequency feature and the actual energy feature of each of the audio samples into an initial vocoder to generate a synthesized audio corresponding to each of the audio samples;

[0015] training the initial vocoder based on each of the audio samples and the synthesized audio corresponding thereto to obtain the general vocoder.

[0016] In an optional implementation, the step of training the initial vocoder based on each of the audio samples and the synthesized audio corresponding thereto to obtain the general vocoder comprises:

[0017] for each of the audio samples, inputting the audio sample and the synthesized audio corresponding thereto into a preset discriminator, and calculating a similarity between the audio sample and the synthesized audio corresponding thereto by using the preset discriminator to obtain a similarity corresponding to the audio sample, thereby obtaining the similarity corresponding to each of the audio samples;

[0018] training the initial vocoder based on the similarity corresponding to each of the audio samples until the similarity corresponding to each of the audio samples reaches a preset threshold, thereby obtaining the general vocoder.

[0019] In an optional implementation, the fundamental frequency predictor is trained in the following manner:

[0020] obtaining speech audios of multiple speakers to obtain multiple audio samples;

[0021] extracting an acoustic feature and a fundamental frequency feature from each of the audio samples to obtain an actual acoustic feature and an actual fundamental frequency feature of each of the audio samples;

[0022] for each of the audio samples, inputting the actual acoustic feature of the audio sample into an initial fundamental frequency predictor to obtain a predicted fundamental frequency feature of the audio sample, thereby obtaining the predicted fundamental frequency feature of each of the audio samples;

[0023] training the initial fundamental frequency predictor based on the actual fundamental frequency feature and the predicted fundamental frequency feature of each of the audio samples to obtain the fundamental frequency predictor.

[0024] In an optional implementation, the initial fundamental frequency predictor comprises two convolutional layers and one linear mapping layer.

[0025] The step of inputting the actual acoustic feature of the audio sample into the initial fundamental frequency predictor to obtain the predicted fundamental frequency feature of the audio sample comprises:

[0026] inputting the actual acoustic feature of the audio sample into a first convolutional layer in the initial fundamental frequency predictor, and performing convolutional processing on the actual acoustic feature of the audio sample by using the first convolutional layer to obtain a first feature vector;

[0027] inputting the first feature vector into a second convolutional layer in the initial fundamental frequency predictor, and performing convolutional processing on the first feature vector by using the second convolutional layer to obtain a second feature vector;

[0028] inputting the second feature vector into a linear mapping layer in the initial fundamental frequency predictor, and performing linear transformation on the second feature vector by using the linear mapping layer to obtain the predicted fundamental frequency feature of the audio sample.

[0029] In an optional implementation, the convolutional layer comprises a one-dimensional convolutional block, a preset activation function, a preset normalization function and a regularization block.

[0030] The step of performing convolutional processing on the actual acoustic feature of the audio sample by using the first convolutional layer to obtain a first feature vector comprises:

[0031] performing one-dimensional convolutional operation on the actual acoustic feature of the audio sample by using the one-dimensional convolutional block of the first convolutional layer, and then performing processing by using the preset activation function of the first convolutional layer to obtain a first vector;

[0032] performing processing on the first vector by using the preset normalization function of the first convolutional layer, and then performing regularization processing by using the regularization block of the first convolutional layer to obtain the first feature vector.

[0033] In an optional implementation, the energy predictor is obtained by training in the following manner:

[0034] obtaining speech audio of multiple speakers to obtain multiple audio samples;

[0035] extracting acoustic features and energy features from each of the audio samples to obtain actual acoustic features and actual energy features of each of the audio samples;

[0036] for each of the audio samples, inputting the actual acoustic feature of the audio sample into an initial energy predictor to obtain a predicted energy feature of the audio sample, thereby obtaining a predicted energy feature of each of the audio samples;

[0037] training the initial energy predictor based on the actual energy feature and the predicted energy feature of each of the audio samples to obtain the energy predictor.

[0038] In a second aspect, the present application provides a speech synthesis device, the device comprising:

[0039] an extraction module configured to extract acoustic features from the text to be processed using a preset acoustic model to obtain target acoustic features of the text to be processed;

[0040] extracting fundamental frequency features from the target acoustic features using a pre-trained fundamental frequency predictor to obtain target fundamental frequency features;

[0041] extracting energy features from the target acoustic features using a pre-trained energy predictor to obtain target energy features;

[0042] a generation module configured to input the target acoustic features, the target fundamental frequency features and the target energy features into a pre-trained general vocoder to generate speech audio of the text to be processed; the general vocoder is trained based on speech audio of multiple speakers.

[0043] In a third aspect, the present application provides an electronic device comprising a processor and a memory, the memory storing a computer program, and the processor implementing the method of any one of the preceding embodiments when executing the computer program.

[0044] In a fourth aspect, the present application provides a storage medium storing a computer program, and the computer program implementing the method of any one of the preceding embodiments when executed by a processor.

[0045] The speech synthesis method and device, the electronic device and the storage medium provided by the embodiments of the present application extract acoustic features from the text to be processed using a preset acoustic model to obtain target acoustic features of the text to be processed; then extract fundamental frequency features from the target acoustic features using a pre-trained fundamental frequency predictor to obtain target fundamental frequency features, and extract energy features from the target acoustic features using a pre-trained energy predictor to obtain target energy features; and then input the target acoustic features, the target fundamental frequency features and the target energy features into a pre-trained general vocoder to generate speech audio of the text to be processed; the general vocoder is trained based on speech audio of multiple speakers. Compared with the prior art, the present application extracts fundamental frequency features and energy features from acoustic features, and inputs the acoustic features, the fundamental frequency features and the energy features into a vocoder for speech synthesis, and the vocoder is trained by speech of multiple speakers, so that the vocoder has general applicability, the training time of the vocoder is reduced, and the effect of speech synthesis is ensured.

[0046] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be considered as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0048] Figure 1 A block diagram of an electronic device according to an embodiment of the present application is shown;

[0049] Figure 2 A flow diagram of a speech synthesis method according to an embodiment of the present application is shown;

[0050] Figure 3 Another flow diagram of a speech synthesis method according to an embodiment of the present application is shown;

[0051] Figure 4 Another flow diagram of a speech synthesis method according to an embodiment of the present application is shown;

[0052] Figure 5 Another flow diagram of a speech synthesis method according to an embodiment of the present application is shown;

[0053] Figure 6 An example diagram of a speech synthesis method according to an embodiment of the present application is shown;

[0054] Figure 7 Another flow diagram of a speech synthesis method according to an embodiment of the present application is shown;

[0055] Figure 8 Another example diagram of a speech synthesis method according to an embodiment of the present application is shown;

[0056] Figure 9 A functional module diagram of a speech synthesis device according to an embodiment of the present application is shown.

[0057] Icon: 110 - bus; 120 - processor; 130 - memory; 150 - I / O module; 170 - communication interface; 300 - speech synthesis device; 310 - extraction module; 330 - generation module; 350 - training module. DETAILED DESCRIPTION

[0058] The technical solutions of the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, and not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0059] Therefore, the following detailed description of the embodiments of the application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the application. Based on the embodiments of the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0060] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one from another entity or action, without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0061] Speech synthesis technology (Text-to-Speech) refers to converting text into speech audio, which is commonly used in human-computer interaction, audio book reading and other scenarios. The speech synthesis model usually includes an acoustic model and a vocoder, the acoustic model is used to convert text into acoustic features, and the vocoder is used to convert acoustic features into speech audio. At present, when constructing a speech synthesis model for a certain timbre, a special vocoder is usually trained based on the speech of the speaker corresponding to the timbre, but training a special vocoder requires a lot of time, and the training sample is limited, that is, only the speech of the speaker is used, so that the generalization ability of the vocoder is insufficient, resulting in poor speech synthesis effect and low efficiency. Further, the present application provides a speech synthesis method to solve the above problems.

[0062] Please refer to Figure 1 is a block schematic diagram of an electronic device provided by an embodiment of the present application. The electronic device includes a bus 110, a processor 120, a memory 130, an I / O module 150, and a communication interface 170.

[0063] The bus 110 can be a circuit that connects the above-mentioned elements to each other and transmits information between the above-mentioned elements.

[0064] The processor 120 can receive a command from the above-mentioned other elements (such as the memory 130, the I / O module 150, the communication interface 170, etc.) through the bus 110, can interpret the received command, and can perform calculation or data processing according to the interpreted command.

[0065] The processor 120 can be an integrated circuit chip having a signal processing capability. The processor 120 can be a general-purpose processor including a central processing unit (CPU), a network processor (NP), etc., and can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components.

[0066] The memory 130 can store commands or data received from the processor 120 or other elements (e.g., the I / O module 150, the communication interface 170, etc.) or generated by the processor 120 or other elements.

[0067] The memory 130 can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM).

[0068] The I / O module 150 can receive a command or data input from a user via an input-output means (e.g., a sensor, a keyboard, a touch screen, etc.), and can transmit the received command or data to the processor 120 or the memory 130 through the bus 110. Also, for displaying various information (e.g., multimedia data, text data) received, stored, or processed from the above elements, a video, an image, data, etc. can be displayed to the user.

[0069] The communication interface 170 can be used for signaling or data communication with other node devices.

[0070] It can be understood that, Figure 1 The structure shown is only a structural schematic diagram of the electronic device, and the electronic device can further include more or less components than those shown in the Figure 1 or have a different configuration from that shown in the Figure 1 The components shown in the Figure 1 may be implemented in hardware, software, or a combination thereof.

[0071] The electronic device can be a smart phone, a personal computer, a tablet computer, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and the like. The embodiments of the present application do not make any limitation in this regard.

[0072] The above electronic device is taken as an execution subject to execute each step in each method provided by the embodiments of the present application, and realize the corresponding technical effects.

[0073] It can be understood that the electronic device is installed with an audio application program, and the speech synthesis method provided by the embodiments of the present application is executed in the running process of the audio application program.

[0074] Please refer to Figure 2 , Figure 2 is a flowchart of a speech synthesis method provided by the embodiments of the present application.

[0075] In step S202, acoustic features are extracted from the to-be-processed text by using a preset acoustic model to obtain target acoustic features of the to-be-processed text.

[0076] In the embodiment, the preset acoustic model is stored in the electronic device. The to-be-processed text can be a text input by a user, for example, the user can input a text to be synthesized by an interactive operation in the process of using the audio application program. The interactive operation can be realized by voice, interactive keys, physical keys, external devices, and the like. The to-be-processed text is input into the preset acoustic model, acoustic features such as a mel spectrum are extracted from the to-be-processed text by using the preset acoustic model, and a mel spectrum of the to-be-processed text is obtained, that is, the target acoustic features are obtained.

[0077] In step S204, a fundamental frequency feature is extracted from the target acoustic features by using a pre-trained fundamental frequency predictor to obtain target fundamental frequency features.

[0078] In step S206, an energy feature is extracted from the target acoustic features by using a pre-trained energy predictor to obtain target energy features.

[0079] In the embodiment, the pre-trained fundamental frequency predictor and the pre-trained energy predictor are also stored in the electronic device. Based on the target acoustic features output by the preset acoustic model, the target acoustic features are respectively input into the fundamental frequency predictor and the energy predictor, the fundamental frequency feature is extracted from the target acoustic features by using the fundamental frequency predictor to obtain the target fundamental frequency features, and the energy feature is extracted from the target acoustic features by using the energy predictor to obtain the target energy features.

[0080] Step S208, input the target acoustic feature, the target fundamental frequency feature and the target energy feature into the pre-trained general vocoder to generate the speech audio of the to-be-processed text. The general vocoder is trained based on the speech audios of multiple speakers.

[0081] In the embodiment, the electronic device also stores a pre-trained general vocoder, which is different from the existing special vocoder in that it is trained based on the speech audios of multiple speakers. The target acoustic feature output by the preset acoustic model, the target fundamental frequency feature output by the fundamental frequency predictor and the target energy feature output by the energy predictor are input into the general vocoder, and speech synthesis is performed by using the general vocoder to obtain the speech audio of the to-be-processed text.

[0082] It can be understood that the existing vocoder generally synthesizes speech audio based on acoustic features, and the inventors have found that in order to make the vocoder have universality, i.e., can synthesize speech of multiple timbres, it is not enough to use only acoustic features, and more speech information is needed. The fundamental frequency feature and the energy feature are important features in speech, the fundamental frequency feature can reflect the tone of speaking, and the energy feature can reflect the strength of speaking. Further, the acoustic feature, the fundamental frequency feature and the energy feature are all used as inputs of the vocoder to perform speech synthesis, which not only makes the vocoder have universality, but also guarantees the effect of speech synthesis.

[0083] It can be seen that based on the above steps, the acoustic feature of the to-be-processed text is obtained by using the preset acoustic model to extract the acoustic feature from the to-be-processed text; then the target fundamental frequency feature is obtained by using the pre-trained fundamental frequency predictor to extract the fundamental frequency feature from the target acoustic feature, and the target energy feature is obtained by using the pre-trained energy predictor to extract the energy feature from the target acoustic feature; then the target acoustic feature, the target fundamental frequency feature and the target energy feature are input into the pre-trained general vocoder to generate the speech audio of the to-be-processed text; the general vocoder is trained based on the speech audios of multiple speakers. Compared with the prior art, the present application extracts the fundamental frequency feature and the energy feature from the acoustic feature, and uses the acoustic feature, the fundamental frequency feature and the energy feature as inputs of the vocoder to perform speech synthesis, and the vocoder is trained by the speech of multiple speakers, so that the vocoder has universality, the training time of the vocoder is reduced, and the effect of speech synthesis is guaranteed.

[0084] Optionally, for the above general vocoder, the embodiment of the present application provides an implementation manner of obtaining the general vocoder, please refer to Figure 3 .

[0085] Step S212, obtaining the speech audios of multiple speakers to obtain multiple audio samples;

[0086] Step S214, extracting the acoustic feature, the fundamental frequency feature and the energy feature from each audio sample to obtain the actual acoustic feature, the actual fundamental frequency feature and the actual energy feature of each audio sample;

[0087] In the embodiment, the speech audios of multiple speakers can be collected, for example, 5 minutes of speech audio of each speaker is collected to obtain multiple audio samples, that is, the training samples of the general vocoder.

[0088] For any one audio sample, feature extraction is performed on the audio sample, that is, the acoustic feature such as the mel spectrum, the fundamental frequency feature and the energy feature are extracted to obtain the actual acoustic feature, the actual fundamental frequency feature and the actual energy feature of the audio sample. In a similar manner, the actual acoustic feature, the actual fundamental frequency feature and the actual energy feature of each audio sample can be obtained.

[0089] Step S216, inputting the actual acoustic feature, the actual fundamental frequency feature and the actual energy feature of each audio sample into the initial vocoder to generate the corresponding synthesized audio of each audio sample;

[0090] Step S218, training the initial vocoder based on each audio sample and the corresponding synthesized audio to obtain the general vocoder;

[0091] In the embodiment, for any one audio sample, the actual acoustic feature, the actual fundamental frequency feature and the actual energy feature of the audio sample are input into the initial vocoder, and speech synthesis is performed by using the initial vocoder to obtain the corresponding synthesized audio of the audio sample. In a similar manner, the corresponding synthesized audio of each audio sample can be obtained, and the initial vocoder is trained based on each audio sample and the corresponding synthesized audio until a preset condition is reached, for example, a preset iteration number is reached or a reference value reaches a preset threshold, that is, the general vocoder is obtained.

[0092] Optionally, for the above step S218, the present embodiment provides a possible implementation manner, please refer to Figure 4 .

[0093] Step S218-1, for each audio sample, inputting the audio sample and the corresponding synthesized audio into a preset discriminator, and calculating the similarity of the audio sample and the corresponding synthesized audio by using the preset discriminator to obtain the similarity corresponding to the audio sample, to obtain the similarity corresponding to each audio sample;

[0094] Step S218-3, training the initial vocoder based on the similarity corresponding to each audio sample until the similarity corresponding to each audio sample reaches a preset threshold to obtain the general vocoder.

[0095] In this embodiment, for the synthesized audio of the initial vocoder output, the synthesized audio is input into the preset discriminator, and the synthesized audio and the corresponding audio sample are discriminated by using the preset discriminator to train the initial vocoder according to a discrimination result.

[0096] For any one audio sample, the audio sample and the corresponding synthesized audio are input into the preset discriminator, and the preset discriminator is used to calculate the similarity of the audio sample and the corresponding synthesized audio to obtain the similarity corresponding to the audio sample.

[0097] For example, the preset discriminator obtains a first audio segment from the audio sample and a second audio segment corresponding to the first audio segment from the synthesized audio; then preset audio features such as periodicity features and multi-spectrum features are extracted from the first audio segment and the second audio segment respectively, and the similarity is calculated based on the audio features of the two audio segments, that is, the similarity corresponding to the audio sample is obtained.

[0098] In a similar manner, for each audio sample, the similarity corresponding to each audio sample can be obtained, and the initial vocoder is trained based on the similarity corresponding to each audio sample until the similarity corresponding to each audio sample reaches a preset threshold, that is, a general vocoder is obtained.

[0099] Optionally, for the above-mentioned fundamental frequency predictor, the embodiment of the present application provides an implementation manner of obtaining the fundamental frequency predictor, please refer to Figure 5 .

[0100] Step S222, obtaining speech audios of multiple speakers to obtain multiple audio samples;

[0101] Step S224, extracting acoustic features and fundamental frequency features from each audio sample to obtain actual acoustic features and actual fundamental frequency features of each audio sample;

[0102] In this embodiment, speech audios of multiple speakers can be collected, for example, 5 minutes of speech audio of each speaker is collected to obtain multiple audio samples, that is, training samples of the fundamental frequency predictor. It can be understood that the training samples of the fundamental frequency predictor are the same as the training samples of the general vocoder to ensure the effect of speech synthesis.

[0103] For any one audio sample, feature extraction is performed on the audio sample, that is, acoustic features such as mel spectrum and fundamental frequency features are extracted to obtain actual acoustic features and actual fundamental frequency features of the audio sample. In a similar manner, for each audio sample, actual acoustic features and actual fundamental frequency features of each audio sample can be obtained.

[0104] Step S226, for each audio sample, input the actual acoustic feature of the audio sample into the initial fundamental frequency predictor to obtain the predicted fundamental frequency feature of the audio sample, to obtain the predicted fundamental frequency feature of each audio sample;

[0105] Step S228, based on the actual fundamental frequency feature and the predicted fundamental frequency feature of each audio sample, train the initial fundamental frequency predictor to obtain the fundamental frequency predictor;

[0106] In this embodiment, for any one audio sample, the actual acoustic feature of the audio sample is input into the initial fundamental frequency predictor, and the initial fundamental frequency predictor is used for fundamental frequency prediction to obtain the predicted fundamental frequency feature of the audio sample. In a similar manner for each audio sample, the predicted fundamental frequency feature of each audio sample can be obtained, and based on the actual fundamental frequency feature and the predicted fundamental frequency feature of each audio sample, the initial fundamental frequency predictor is trained until a preset condition is reached, such as reaching a preset iteration number or a reference value reaching a preset threshold, that is, the fundamental frequency predictor is obtained.

[0107] Optionally, for the step of inputting the actual acoustic feature of the audio sample into the initial fundamental frequency predictor to obtain the predicted fundamental frequency feature of the audio sample in the above step S226, the embodiment of the present application provides a possible implementation manner.

[0108] Step S226-1, input the actual acoustic feature of the audio sample into the first convolutional layer in the initial fundamental frequency predictor, and use the first convolutional layer to perform convolutional processing on the actual acoustic feature of the audio sample to obtain a first feature vector;

[0109] Step S226-3, input the first feature vector into the second convolutional layer in the initial fundamental frequency predictor, and use the second convolutional layer to perform convolutional processing on the first feature vector to obtain a second feature vector;

[0110] Step S226-5, input the second feature vector into the linear mapping layer in the initial fundamental frequency predictor, and use the linear mapping layer to perform linear transformation on the second feature vector to obtain the predicted fundamental frequency feature of the audio sample.

[0111] In this embodiment, the initial fundamental frequency predictor includes two convolutional layers and a linear mapping layer. For ease of understanding, the present embodiment provides a structural schematic diagram of the initial fundamental frequency predictor. Please refer to Figure 6 , which will be described below. Figure 6 The above steps S226-1 to S226-5 will be described.

[0112] The structures of the two convolutional layers are exactly the same, and each includes two sub-convolutional layers, i.e., a first sub-convolutional layer and a second sub-convolutional layer. The first sub-convolutional layer includes a one-dimensional convolutional block and a preset activation function, and the second sub-convolutional layer includes a preset normalization function and a regularization block.

[0113] The actual acoustic feature of the audio sample is input into the first convolutional layer in the initial fundamental frequency predictor, that is, the actual acoustic feature of the audio sample is input into a one-dimensional convolutional block, Conv1D, in the first convolutional layer, a one-dimensional convolutional operation is performed on the actual acoustic feature by using the one-dimensional convolutional block, and a vector output by the one-dimensional convolutional block is processed by using a preset activation function, ReLU function (Rectified Linear Unit), in the first convolutional layer, to obtain a first vector.

[0114] The first vector is processed by using a preset normalization function, LN function (LgayerNorm), in the first convolutional layer to obtain a processed first vector, and the processed first vector is regularized by using a regularization block, Dropout, in the first convolutional layer to obtain a first feature vector.

[0115] The ReLU function and the LN function are used to normalize the vector after convolution, and the Dropout is used to temporarily discard some neurons from the network for training according to a certain probability to prevent overfitting of the network.

[0116] It can be understood that the two convolutional layers in the initial fundamental frequency predictor have the same structure, that is, the first feature vector is input into the second convolutional layer in the initial fundamental frequency predictor, and the second feature vector can be obtained in the above similar manner.

[0117] Finally, the second feature vector is input into a linear mapping layer, Linear Layer, in the initial fundamental frequency predictor, and the second feature vector is linearly transformed by using the linear mapping layer to obtain the predicted fundamental frequency feature of the audio sample.

[0118] It can be understood that the initial fundamental frequency predictor has the same structure as the fundamental frequency predictor, and only the numerical values of the parameters are different, that is, the process of extracting the fundamental frequency feature from the acoustic feature by the fundamental frequency predictor is similar to the above implementation manner.

[0119] Optionally, for the above energy predictor, an implementation manner of obtaining the energy predictor is provided in an embodiment of the present application, please refer to Figure 7 .

[0120] In step S232, the speech audio of a plurality of speakers is obtained to obtain a plurality of audio samples.

[0121] In step S234, the acoustic feature and the energy feature are extracted from each audio sample to obtain the actual acoustic feature and the actual energy feature of each audio sample.

[0122] In the embodiment, the speech audios of multiple speakers can be collected, for example, 5 minutes of speech audio of each speaker is collected to obtain multiple audio samples, i.e., training samples of the energy predictor. It can be understood that the training samples of the general vocoder, the training samples of the fundamental frequency predictor and the training samples of the energy predictor are the same to ensure the effect of speech synthesis.

[0123] For any one audio sample, feature extraction is performed on the audio sample, i.e., acoustic features such as mel spectrum and energy features are extracted to obtain actual acoustic features and actual energy features of the audio sample. In a similar manner, actual acoustic features and actual energy features of each audio sample can be obtained for each audio sample.

[0124] In step S236, for each audio sample, the actual acoustic features of the audio sample are input into the initial energy predictor to obtain predicted energy features of the audio sample, and predicted energy features of each audio sample are obtained.

[0125] In step S238, the initial energy predictor is trained based on the actual energy features and the predicted energy features of each audio sample to obtain the energy predictor.

[0126] In the embodiment, for any one audio sample, the actual acoustic features of the audio sample are input into the initial energy predictor, and energy prediction is performed by using the initial energy predictor to obtain predicted energy features of the audio sample. In a similar manner, predicted energy features of each audio sample can be obtained, and the initial energy predictor is trained based on the actual energy features and the predicted energy features of each audio sample until a preset condition is reached, such as a preset number of iterations is reached or a reference value reaches a preset threshold, i.e., the energy predictor is obtained.

[0127] Optionally, the initial energy predictor can have the same structure as the initial fundamental frequency predictor described above, and thus the embodiment of the present application provides a possible implementation for the step of inputting the actual acoustic features of the audio sample into the initial energy predictor to obtain the predicted energy features of the audio sample in step S236 described above.

[0128] In step S236-1, the actual acoustic features of the audio sample are input into the first convolutional layer in the initial energy predictor, and the actual acoustic features of the audio sample are convoluted by using the first convolutional layer to obtain a third feature vector.

[0129] In step S236-3, the third feature vector is input into the second convolutional layer in the initial energy predictor, and the third feature vector is convoluted by using the second convolutional layer to obtain a fourth feature vector.

[0130] Step S236-5, input the fourth feature vector into the linear mapping layer in the initial energy predictor, and linearly transform the fourth feature vector by using the linear mapping layer to obtain the predicted energy feature of the audio sample.

[0131] In this embodiment, the initial energy predictor includes two convolutional layers and a linear mapping layer. For ease of understanding, the structural schematic diagram of the initial energy predictor is provided in this embodiment. Please refer to Figure 8 The above steps S236-1 to S236-5 will be described below in combination with Figure 8

[0132] The structures of the two convolutional layers are completely the same, and each of the two convolutional layers includes two sub-convolutional layers, i.e., a first sub-convolutional layer and a second sub-convolutional layer. The first sub-convolutional layer includes a one-dimensional convolution block and a preset activation function, and the second sub-convolutional layer includes a preset normalization function and a regularization block.

[0133] The actual acoustic feature of the audio sample is input into the first convolutional layer in the initial energy predictor, i.e., the actual acoustic feature of the audio sample is input into the one-dimensional convolution block, i.e., Conv1D, in the first convolutional layer. The one-dimensional convolution block is used to perform one-dimensional convolution operation, and the preset activation function, i.e., ReLU function (Rectified Linear Unit) in the first convolutional layer is used to process the vector output by the one-dimensional convolution block to obtain a third vector.

[0134] The preset normalization function, i.e., LN function (LgayerNorm) in the first convolutional layer is further used to process the third vector to obtain a processed third vector, and the regularization block, i.e., Dropout, of the first convolutional layer is used to perform regularization processing on the processed third vector to obtain a third feature vector.

[0135] The ReLU function and the LN function are used to perform normalization processing on the vector after convolution, and the Dropout is used to temporarily discard some neurons from the network for training according to a certain probability to prevent overfitting of the network.

[0136] It can be understood that the structures of the two convolutional layers in the initial energy predictor are completely the same, i.e., the second convolutional layer performs convolution processing on the third feature vector in a similar manner to the first convolutional layer performing convolution processing on the actual acoustic feature of the audio sample. Therefore, the third feature vector is input into the second convolutional layer in the initial energy predictor, and the fourth feature vector can be obtained in the above similar manner.

[0137] Finally, the fourth feature vector is input into the linear mapping layer, i.e., Linear Layer, in the initial energy predictor, and the linear mapping layer is used to linearly transform the fourth feature vector to obtain the predicted energy feature of the audio sample.​

[0138] It can be understood that the initial energy predictor is completely the same in structure as the energy predictor, and only differs in the numerical value of the parameters, that is, the process of extracting the energy feature from the acoustic feature by the energy predictor is similar to the above implementation manners.

[0139] In order to perform the corresponding steps in the above-mentioned embodiments and various possible manners, an implementation manner of a speech synthesis device is given below. Please refer to Figure 9 , Figure 9 A functional module diagram of a speech synthesis device provided by an embodiment of the present application is shown in the figure. It should be noted that the speech synthesis device 300 provided by the present embodiment has the same basic principles and technical effects as the above-mentioned embodiments, and for the sake of brief description, the part not mentioned in the present embodiment can be referred to the corresponding content in the above-mentioned embodiments. The speech synthesis device 300 comprises:

[0140] The extraction module 310 is configured to extract acoustic features from the to-be-processed text by using a preset acoustic model, so as to obtain target acoustic features of the to-be-processed text.

[0141] The target acoustic features are input into a pre-trained fundamental frequency predictor to extract fundamental frequency features from the target acoustic features, so as to obtain target fundamental frequency features.

[0142] The target acoustic features are input into a pre-trained energy predictor to extract energy features from the target acoustic features, so as to obtain target energy features.

[0143] The generation module 330 is configured to input the target acoustic features, the target fundamental frequency features and the target energy features into a pre-trained general vocoder to generate speech audio of the to-be-processed text. The general vocoder is trained based on speech audio of multiple speakers.

[0144] Optionally, the speech synthesis device 300 further comprises a training module 350 configured to: obtain speech audio of multiple speakers to obtain multiple audio samples; extract acoustic features, fundamental frequency features and energy features from each audio sample to obtain actual acoustic features, actual fundamental frequency features and actual energy features of each audio sample; input the actual acoustic features, the actual fundamental frequency features and the actual energy features of each audio sample into an initial vocoder to generate synthesized audio corresponding to each audio sample; and train the initial vocoder based on each audio sample and the synthesized audio corresponding thereto to obtain the general vocoder.

[0145] The training module 350 is further configured to: for each audio sample, input the audio sample and its corresponding synthesized audio into a preset discriminator, and obtain a similarity of the audio sample and its corresponding synthesized audio by using the preset discriminator to calculate the similarity, to obtain a corresponding similarity of the audio sample; and train the initial vocoder based on the corresponding similarity of each audio sample until the corresponding similarity of each audio sample reaches a preset threshold, to obtain the universal vocoder.

[0146] The training module 350 is further configured to: obtain speech audios of multiple speakers to obtain multiple audio samples; extract acoustic features and fundamental frequency features from each audio sample to obtain actual acoustic features and actual fundamental frequency features of each audio sample; for each audio sample, input the actual acoustic features of the audio sample into an initial fundamental frequency predictor to obtain predicted fundamental frequency features of the audio sample, to obtain predicted fundamental frequency features of each audio sample; and train the initial fundamental frequency predictor based on the actual fundamental frequency features and the predicted fundamental frequency features of each audio sample, to obtain the fundamental frequency predictor.

[0147] The training module 350 is further configured to: input the actual acoustic features of the audio sample into a first convolutional layer in the initial fundamental frequency predictor, and perform convolutional processing on the actual acoustic features of the audio sample by using the first convolutional layer to obtain a first feature vector; input the first feature vector into a second convolutional layer in the initial fundamental frequency predictor, and perform convolutional processing on the first feature vector by using the second convolutional layer to obtain a second feature vector; and input the second feature vector into a linear mapping layer in the initial fundamental frequency predictor, and perform linear transformation on the second feature vector by using the linear mapping layer to obtain the predicted fundamental frequency features of the audio sample.

[0148] The training module 350 is further configured to: after performing one-dimensional convolutional operation on the actual acoustic features of the audio sample by using a one-dimensional convolutional block of the first convolutional layer, perform processing by using a preset activation function of the first convolutional layer to obtain a first vector; after processing the first vector by using a preset normalization function of the first convolutional layer, perform regularization processing by using a regularization block of the first convolutional layer to obtain the first feature vector.

[0149] The training module 350 is further configured to: obtain speech audios of multiple speakers to obtain multiple audio samples; extract acoustic features and energy features from each audio sample to obtain actual acoustic features and actual energy features of each audio sample; for each audio sample, input the actual acoustic features of the audio sample into an initial energy predictor to obtain predicted energy features of the audio sample, to obtain predicted energy features of each audio sample; and train the initial energy predictor based on the actual energy features and the predicted energy features of each audio sample, to obtain the energy predictor.

[0150] The embodiment of the present application further provides an electronic device, comprising a processor and a memory, the memory 130 stores a computer program, and the processor implements the speech synthesis method disclosed by the embodiment of the present application when executing the computer program.

[0151] The embodiment of the present application further provides a storage medium, which stores a computer program, and the computer program is executed by a processor to implement the speech synthesis method disclosed by the embodiment of the present application.

[0152] In several embodiments provided by the present application, it should be understood that the disclosed device and method can also be implemented by other manners. The device embodiments described above are only schematic, for example, the flow charts and block diagrams in the drawings show the possible implementation architectures, functions and operations of the device, method and computer program product according to the embodiments of the present application. In this regard, each block in the flow charts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, or they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flow charts, and the combination of blocks in the block diagrams and / or flow charts, can be implemented by a dedicated hardware-based system, or by a combination of special-purpose hardware and computer instructions.

[0153] In addition, each functional module in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0154] If the functions are realized in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0155] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. The present application can have various changes and modifications for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A speech synthesis method characterized by, The method comprises: extracting acoustic features from the to-be-processed text by using a preset acoustic model to obtain target acoustic features of the to-be-processed text; extracting fundamental frequency features from the target acoustic features by using a pre-trained fundamental frequency predictor to obtain target fundamental frequency features; extracting energy features from the target acoustic features by using a pre-trained energy predictor to obtain target energy features; inputting the target acoustic features, the target fundamental frequency features and the target energy features into a pre-trained general vocoder to generate speech audio of the to-be-processed text; the general vocoder is trained based on speech audio of multiple speakers and comprises: obtaining speech audio of multiple speakers to obtain multiple audio samples; extracting acoustic features, fundamental frequency features and energy features from each of the audio samples to obtain actual acoustic features, actual fundamental frequency features and actual energy features of each of the audio samples; inputting the actual acoustic features, the actual fundamental frequency features and the actual energy features of each of the audio samples into an initial vocoder to generate synthesized audio corresponding to each of the audio samples; training the initial vocoder based on each of the audio samples and the corresponding synthesized audio to obtain the general vocoder.

2. The method of claim 1, wherein, The step of training the initial vocoder based on each of the audio samples and the corresponding synthesized audio to obtain the general vocoder comprises: for each of the audio samples, inputting the audio sample and the corresponding synthesized audio into a preset discriminator and calculating the similarity of the audio sample and the corresponding synthesized audio by using the preset discriminator to obtain the similarity corresponding to the audio sample, thereby obtaining the similarity corresponding to each of the audio samples; training the initial vocoder based on the similarity corresponding to each of the audio samples until the similarity corresponding to each of the audio samples reaches a preset threshold, thereby obtaining the general vocoder.

3. The method of claim 1, wherein, The fundamental frequency predictor is trained in the following manner: obtaining speech audio of multiple speakers to obtain multiple audio samples; extracting acoustic features and fundamental frequency features from each of the audio samples to obtain actual acoustic features and actual fundamental frequency features of each of the audio samples; for each of the audio samples, inputting the actual acoustic features of the audio sample into an initial fundamental frequency predictor to obtain predicted fundamental frequency features of the audio sample, thereby obtaining predicted fundamental frequency features of each of the audio samples; training the initial fundamental frequency predictor based on the actual fundamental frequency features and the predicted fundamental frequency features of each of the audio samples to obtain the fundamental frequency predictor.

4. The method of claim 3, wherein, The initial fundamental frequency predictor comprises two convolution layers and a linear mapping layer. The step of inputting the actual acoustic features of the audio sample into the initial fundamental frequency predictor to obtain predicted fundamental frequency features of the audio sample comprises: inputting the actual acoustic features of the audio sample into a first convolution layer in the initial fundamental frequency predictor and performing convolution processing on the actual acoustic features of the audio sample by using the first convolution layer to obtain a first feature vector; inputting the first feature vector into a second convolution layer in the initial fundamental frequency predictor and performing convolution processing on the first feature vector by using the second convolution layer to obtain a second feature vector; inputting the second feature vector into the linear mapping layer in the initial fundamental frequency predictor and performing linear mapping processing on the second feature vector by using the linear mapping layer to obtain the predicted fundamental frequency features of the audio sample. inputting the first feature vector into a second convolutional layer in the initial fundamental frequency predictor, and performing convolution processing on the first feature vector by using the second convolutional layer to obtain a second feature vector; inputting the second feature vector into a linear mapping layer in the initial fundamental frequency predictor, and performing linear transformation on the second feature vector by using the linear mapping layer to obtain a predicted fundamental frequency feature of the audio sample.

5. The method of claim 4, wherein, The convolutional layer comprises a one-dimensional convolution block, a preset activation function, a preset normalization function and a regularization block. The step of performing convolution processing on the actual acoustic feature of the audio sample by using the first convolutional layer to obtain a first feature vector comprises: performing one-dimensional convolution operation on the actual acoustic feature of the audio sample by using the one-dimensional convolution block of the first convolutional layer, performing processing by using the preset activation function of the first convolutional layer to obtain a first vector; performing processing on the first vector by using the preset normalization function of the first convolutional layer, and performing regularization processing by using the regularization block of the first convolutional layer to obtain the first feature vector.

6. The method of claim 1, wherein, The energy predictor is obtained by training in the following manner: obtaining speech audios of multiple speakers to obtain multiple audio samples; extracting acoustic features and energy features from each audio sample to obtain actual acoustic features and actual energy features of each audio sample; for each audio sample, inputting the actual acoustic feature of the audio sample into an initial energy predictor to obtain a predicted energy feature of the audio sample, thereby obtaining a predicted energy feature of each audio sample; training the initial energy predictor based on the actual energy feature and the predicted energy feature of each audio sample to obtain the energy predictor.

7. A speech synthesis apparatus characterized by comprising: The device comprises: an extraction module configured to extract acoustic features from the to-be-processed text by using a preset acoustic model to obtain target acoustic features of the to-be-processed text; extracting fundamental frequency features from the target acoustic features by using a pre-trained fundamental frequency predictor to obtain target fundamental frequency features; extracting energy features from the target acoustic features by using a pre-trained energy predictor to obtain target energy features; a generation module configured to input the target acoustic features, the target fundamental frequency features and the target energy features into a pre-trained general vocoder to generate speech audio of the to-be-processed text; the general vocoder is trained based on speech audios of multiple speakers, and the generation module is configured to: obtain speech audios of multiple speakers to obtain multiple audio samples; extract acoustic features, fundamental frequency features and energy features from each audio sample to obtain actual acoustic features, actual fundamental frequency features and actual energy features of each audio sample; input the actual acoustic features, the actual fundamental frequency features and the actual energy features of each audio sample into an initial vocoder to generate a corresponding synthesized audio of each audio sample; train the initial vocoder based on each audio sample and the corresponding synthesized audio to obtain the general vocoder.

8. An electronic device, comprising: A computer program product comprising a storage medium storing the computer program, which, when executed by a processor, implements the method of any one of claims 1 to 6.

9. A storage medium, characterized by The storage medium stores a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 6.

Images