Vocoder training method, speech synthesis method and related products

Abstract

An embodiment of the present invention provides a method for training a vocoder. The training method comprises: obtaining training data, wherein the training data is a Mel spectrum of speech data; adding a disturbance to the Mel spectrum to obtain a Mel spectrum after adding the disturbance; and training the vocoder using the Mel spectrum after adding the disturbance for synthesizing speech. By adding a disturbance to the Mel spectrum and using it for training the vocoder, the method of the present invention enables the vocoder to have the ability to resist disturbance and correct errors, thereby significantly reducing the adverse effects of the acoustic model prediction deviation on the speech synthesis effect, and bringing a better experience to the user. In addition, an embodiment of the present invention provides a method, an electronic device, and a computer-readable storage medium for performing speech synthesis using a vocoder.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology. More specifically, the embodiments of the present invention relate to a vocoder training method, a method for performing speech synthesis using a vocoder, an electronic device, and a computer-readable storage medium. Background Art

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The descriptions herein may include concepts that may be explored, but not necessarily concepts that have been previously thought of or explored. Therefore, unless otherwise noted herein, the content described in this section is not prior art with respect to the specification and claims of this application, and is not admitted to be prior art by inclusion in this section.

[0003] Speech synthesis is a technology that produces artificial speech through mechanical and electronic methods. TTS (text-to-speech, also known as text-to-speech conversion technology) belongs to speech synthesis. It converts text information generated by the computer itself or input externally into standard and fluent speech in real time, which is equivalent to installing an artificial mouth on the machine. It involves multiple disciplines such as acoustics, linguistics, digital signal processing, and computer science. It is a cutting-edge technology in the field of Chinese information processing. The main problem it solves is how to convert text information into audible sound information, that is, to make the machine speak like a human. At present, with the development of artificial intelligence technology, the use of speech synthesis technology is becoming more and more extensive.

[0004] The TTS structure mainly involves three aspects: language processing, prosody processing and acoustic processing. Among them, language processing plays an important role in text-to-speech conversion. It mainly simulates the process of human understanding of natural language - text regularization, word segmentation, grammatical analysis and semantic analysis, so that the computer can fully understand the input text and give various pronunciation prompts required for the latter two parts. Prosody processing plans the segmental features for the synthesized speech, such as high pitch, length, intensity and rhythm, so that the synthesized speech can express the speech correctly and sound more natural. Acoustic processing outputs the corresponding speech according to the content after the aforementioned language processing and prosody processing, that is, synthesized speech.

[0005] In the acoustic processing process, the vocoder is a key link in determining the final synthesized sound quality of the TTS system. Currently, there are two main vocoder training schemes: one is to directly extract the Mel spectrum from the real speech and participate in the training, and the other is to obtain the Mel spectrum by real speech alignment and participate in the training, that is, to use the duration of the real speech to force the acoustic model to predict the Mel spectrum of the same length as the real speech. The purpose of this is to reduce the error between the Mel spectrum output by the acoustic model and the Mel spectrum input by the vocoder. The advantage of the former is that it is simple and easy to use, but the disadvantage is that the quality of the synthesized speech is poor. The advantage of the latter is that the synthesis quality of the speech is high, but the disadvantage is that the operation is more complicated and cumbersome.

[0006] However, the above two methods also have a common defect. When the Mel spectrum predicted by the acoustic model is biased, the vocoder obtained in the above training process is difficult to adapt to the biased Mel spectrum feature, resulting in trembling synthesized speech, electric sound and noise. The reason is that the current vocoder training process uses the Mel spectrum corresponding to the nearly perfect fitting samples in the training set, resulting in the vocoder not having strong error correction and anti-interference capabilities. The Mel spectrum features in real scenarios are predicted by text outside the training set, which easily predicts poor Mel spectrum features, which results in the vocoder being unable to synthesize high-quality speech based on poor Mel spectrum features. Therefore, in the speech synthesis system, the anti-noise, error correction and anti-interference capabilities of the vocoder are particularly important.

[0007] In view of this, how to improve the anti-disturbance ability of the vocoder so that the vocoder can accurately correct and process when there are deviations in the Mel spectrum features predicted by the acoustic model is important for improving the speech conversion effect. Summary of the invention

[0008] In the prior art, conventional vocoders do not have anti-disturbance capabilities, resulting in poor quality of synthesized speech, which is a very annoying process.

[0009] Therefore, there is a great need for an improved vocoder training method so that the trained vocoder can synthesize high-quality speech based on poor-quality Mel spectrum features.

[0010] In this context, embodiments of the present invention are intended to provide a method for training a vocoder, a method for performing speech synthesis using a vocoder, an electronic device, and a computer-readable storage medium.

[0011] In a first aspect of an embodiment of the present invention, a method for training a vocoder is provided, comprising: acquiring training data, wherein the training data is a Mel spectrum of speech data; adding a disturbance to the Mel spectrum to obtain a Mel spectrum after the disturbance is added; and training the vocoder using the Mel spectrum after the disturbance is added for synthesizing speech.

[0012] In one embodiment of the present invention, the step of adding disturbance to the Mel spectrum to obtain the Mel spectrum after the disturbance is added includes: determining the disturbance category and frequency to be added; adding disturbance of corresponding disturbance category and frequency to the Mel spectrum, wherein the disturbance category includes one or more of mask window, harmonic disturbance, inter-harmonic noise and dotting sound.

[0013] In another embodiment of the present invention, adding a mask window in the Mel spectrum comprises: randomly selecting one or more positions in the Mel spectrum; and adding a mask window at the one or more positions.

[0014] In yet another embodiment of the present invention, the shape and size of the mask window are randomly generated.

[0015] In still another embodiment of the present invention, adding harmonic disturbance to the Mel spectrum includes: randomly selecting one or more time points in the Mel spectrum; and injecting harmonic disturbance at the one or more time points.

[0016] In one embodiment of the present invention, injecting harmonic disturbance at one or more time points includes: calculating harmonic characteristic quantities corresponding to one or more time points in the Mel spectrum; randomly shuffling the harmonic characteristic quantities and injecting them into the one or more time points to inject harmonic disturbance.

[0017] In another embodiment of the present invention, adding interharmonic noise to the Mel spectrum includes: acquiring interharmonic noise sample values; and superimposing the interharmonic noise sample values ​​with the harmonic part of the Mel spectrum to add the interharmonic noise.

[0018] In yet another embodiment of the present invention, the obtaining of inter-harmonic noise sample values ​​includes: constructing a distribution model of the harmonic part in the Mel spectrum; and performing sampling based on the distribution model to obtain corresponding inter-harmonic noise sample values.

[0019] In another embodiment of the present invention, the distribution model includes a Gaussian distribution model, and constructing the distribution model of the harmonic part in the Mel spectrum includes: calculating the mean and variance of the harmonic part in the Mel spectrum, and constructing a Gaussian distribution model based on the mean and variance.

[0020] In one embodiment of the present invention, adding punctuation sounds in the Mel spectrum includes: determining a starting position and / or an ending position in the Mel spectrum corresponding to a speech pause; and adding a pulse signal of a set size at the starting position and / or the ending position to add punctuation sounds.

[0021] In another embodiment of the present invention, adding disturbance to the Mel spectrum to obtain the Mel spectrum after adding disturbance further includes: adding disturbance of a set time percentage to the audio duration corresponding to the Mel spectrum of training data to obtain the Mel spectrum after adding disturbance.

[0022] In another embodiment of the present invention, the adding of disturbance further comprises: detecting the defect type and occurrence frequency of abnormal samples in the business scenario; and determining the disturbance category and frequency to be added based on the defect type and occurrence frequency of the abnormal samples.

[0023] In yet another embodiment of the present invention, detecting defect types and occurrence frequencies of abnormal samples in business scenarios includes: synthesizing speech data using text samples in the business scenario through a vocoder; determining abnormal samples based on the speech data; and determining defect types and occurrence frequencies of abnormal samples based on Mel spectra of the abnormal samples.

[0024] In one embodiment of the present invention, the defect types include tremolo, electronic music and dotted sounds, wherein based on the defect types and occurrence frequencies of the abnormal samples, determining the disturbance categories and frequencies to be added includes: debugging disturbances in the corresponding Mel spectrum according to the auditory performances corresponding to the tremolo, electronic music and dotted sounds to determine the disturbance categories to be added; and determining the frequency of adding disturbances to the Mel spectrum of the training data according to the occurrence frequencies of the defect types in the abnormal samples.

[0025] In another embodiment of the present invention, the Mel spectrum is obtained by any of the following methods: directly extracting the Mel spectrum from speech data; or inputting speech data and text data into an acoustic model to obtain a predicted Mel spectrum.

[0026] In another embodiment of the present invention, the acoustic model includes a duration prediction model and a Mel spectrum prediction model, wherein the speech data and text data are input into the acoustic model to obtain a predicted Mel spectrum, comprising: predicting the time length of the speech data using the duration prediction model to obtain a prediction result; and converting the text data according to the prediction result by the Mel spectrum prediction model to generate a Mel spectrum of corresponding time length.

[0027] In a second aspect of an embodiment of the present invention, a method for speech synthesis using a vocoder is provided, comprising: obtaining text data to be converted; generating a corresponding Mel spectrum based on the text data; and processing the Mel spectrum using a vocoder to synthesize a target speech, wherein the vocoder is trained according to the training method described in the foregoing text and in multiple embodiments below.

[0028] In a third aspect of the embodiments of the present invention, an electronic device is provided, comprising: a processor; and a memory storing computer instructions for training a vocoder or performing speech synthesis using a vocoder, wherein when the computer instructions are executed by the processor, the device executes the vocoder training method described in the foregoing and subsequent embodiments, or the device executes the speech synthesis method using a vocoder described in the foregoing and subsequent embodiments.

[0029] In a fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, comprising program instructions for training a vocoder or performing speech synthesis using a vocoder, wherein when the program instructions are executed by a processor, the vocoder training method described in the foregoing and following embodiments is implemented, or the method for performing speech synthesis using a vocoder described in the foregoing and following embodiments is implemented.

[0030] According to the training method of the vocoder of the embodiment of the present invention, the Mel spectrum with disturbance can be used during training to train the vocoder using the Mel spectrum after the disturbance, thereby significantly improving the error correction capability of the vocoder for disturbances, and reducing the interference of various disturbances on the speech synthesis results, bringing a better experience to the user. Furthermore, the present invention also adds one or more disturbances of different categories, frequencies, and durations to the Mel spectrum for the training of the vocoder, thereby effectively improving the ability of the vocoder to resist various disturbances, which helps to improve the robustness of the vocoder. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:

[0032] Figure 1 A block diagram schematically illustrates an exemplary computing system 100 suitable for implementing embodiments of the present invention;

[0033] Figure 2 A schematic diagram of a speech synthesis method according to an embodiment of the present invention is schematically shown;

[0034] Figure 3A flowchart of a vocoder training method according to an embodiment of the present invention is schematically shown;

[0035] Figure 4 The flowchart of the method for adding disturbance to the Mel spectrum according to an embodiment of the present invention is schematically shown;

[0036] Figure 5 A flow chart of a method for determining a disturbance type according to an embodiment of the present invention is schematically shown;

[0037] Figure 6 A flow chart of a method for adding different types of disturbances according to an embodiment of the present invention is schematically shown;

[0038] Figures 7a to 7d Schematically showing a visualization diagram of the Mel spectrum after adding different types of disturbances according to an embodiment of the present invention;

[0039] Figure 8 A flow chart of a method for performing speech synthesis using a vocoder according to an embodiment of the present invention is schematically shown;

[0040] Fig. 9 The diagram schematically shows a schematic diagram of an electronic device according to an embodiment of the present invention.

[0041] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0042] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0043] Figure 1 1 is a block diagram of an exemplary computing system 100 suitable for implementing embodiments of the present invention. Figure 1As shown, the computing system 100 may include: a central processing unit (CPU) 101, a random access memory (RAM) 102, a read-only memory (ROM) 103, a system bus 104, a hard disk controller 105, a keyboard controller 106, a serial interface controller 107, a parallel interface controller 108, a display controller 109, a hard disk 110, a keyboard 111, a serial external device 112, a parallel external device 113, and a display 114. Among these devices, the CPU 101, the RAM 102, the ROM 103, the hard disk controller 105, the keyboard controller 106, the serial controller 107, the parallel controller 108, and the display controller 109 are coupled to the system bus 104. The hard disk 110 is coupled to the hard disk controller 105, the keyboard 111 is coupled to the keyboard controller 106, the serial external device 112 is coupled to the serial interface controller 107, the parallel external device 113 is coupled to the parallel interface controller 108, and the display 114 is coupled to the display controller 109. It should be understood that Figure 1 The structural block diagram is only for the purpose of illustration, rather than limiting the scope of the present invention. In some cases, some equipment may be added or reduced according to specific circumstances.

[0044] It is known to those skilled in the art that the embodiments of the present invention may be implemented as a system, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit", "module" or "system". In addition, in some embodiments, the present invention may also be implemented in the form of a computer program product in one or more computer-readable media, which contains computer-readable program code.

[0045] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive examples) of computer-readable storage media may include, for example: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0046] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0047] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0048] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0049] The following will describe the implementation of the present invention with reference to the flowchart of the method of the embodiment of the present invention and the block diagram of the device (or system). It should be understood that each square frame of the flowchart and / or block diagram and the combination of square frames in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, thereby producing a machine, and these computer program instructions are executed by a computer or other programmable data processing device to produce a device that implements the function / operation specified in the square frame in the flowchart and / or block diagram.

[0050] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce a product that includes an instruction device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0051] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable device to provide a process for implementing the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0052] According to an embodiment of the present invention, a training method for a vocoder, a method for performing speech synthesis using a vocoder, an electronic device, and a computer-readable storage medium are proposed.

[0053] In this article, it is important to understand that the terms involved are:

[0054] The text-to-speech system (TTS) is generally composed of a front-end text analysis module, an acoustic model, and a vocoder. The text analysis process includes text regularization, prosody prediction, polyphone disambiguation, and word-to-phonetic translation (G2P).

[0055] Acoustic model (AM) can predict the corresponding Mel spectrum after inputting text. A duration prediction model is usually embedded in the acoustic model to estimate the pronunciation duration of the input text. Commonly used acoustic models include tactron2, fastspeech2, and bvae.

[0056] Mel spectrum: a common compression method for speech, and the mel spectrum cannot be directly restored to speech after compression.

[0057] Vocoder: A model that converts Mel spectra into speech. Commonly used neural network-based vocoders include wavenet, lpcnet, and hifigan.

[0058] In addition, any number of elements in the drawings is for illustration and not limitation, and any naming is only for distinction and does not have any limiting meaning.

[0059] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention. SUMMARY OF THE INVENTION

[0061] The inventors have found that it is difficult for the trained vocoder to synthesize high-quality speech by directly extracting the Mel spectrum from the real speech or training the vocoder using the Mel spectrum of the same length as the real speech predicted by the acoustic model. After analysis, it is found that when the acoustic model predicts the Mel spectrum using text outside the training set, it is easy to predict poor Mel spectrum features, and the existing vocoder uses the nearly perfect fitting samples in the training set during the training stage, and the corresponding Mel spectra are relatively normal, resulting in the trained vocoder having insufficient error correction and anti-interference capabilities for the poor quality Mel spectrum, and thus being unable to effectively synthesize high-quality speech.

[0062] Based on this, the inventors found that various types of disturbances can be added to the Mel spectrum of the training data, and the vocoder can be trained using the Mel spectrum with added disturbances, thereby effectively improving the error correction capability and anti-interference performance of the vocoder when synthesizing speech.

[0063] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention are described in detail below.

[0064] Application Scenario Overview

[0065] First reference Figure 2 Schematic diagram of the speech synthesis method shown in . Figure 2 As shown, in the speech synthesis system, the text analysis module 201, the acoustic model 202 and the vocoder 203 are mainly included. After the text to be synthesized is input into the speech synthesis system, the text analysis module 201 first performs text regularization, disambiguation and other processing processes on the text to extract text features. Then the acoustic model 202 will predict the corresponding Mel spectrum based on the text features, so as to obtain the time domain, frequency domain and other features of the speech to be restored. Finally, the obtained Mel spectrum is input into the vocoder 203, and the vocoder 203 will restore the speech content corresponding to the text based on the Mel spectrum.

[0066] The present invention mainly improves the training process of the vocoder in the speech synthesis system to enhance the anti-disturbance and error correction capabilities of the vocoder and improve the quality of speech synthesis.

[0067] Exemplary Methods

[0068] Combine the following Figure 2 For application scenarios, refer to Figure 3 The training method of the vocoder according to the exemplary embodiment of the present invention is described. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present invention, and the embodiments of the present invention are not limited in this respect. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.

[0069] Figure 3 The flowchart of the vocoder training method 300 according to the embodiment of the present invention is schematically shown.

[0070] like Figure 3 As shown, at step S301, training data is obtained. The training data may be a Mel spectrum of speech data. In some embodiments, the Mel spectrum may be obtained in any of the following ways: the first way is to directly extract the Mel spectrum from the speech data. The second way is to input the speech data and the text data into the acoustic model to obtain a predicted Mel spectrum. In the second way, the duration of the speech data may be obtained, and the acoustic model may be forced to predict a Mel spectrum of the same duration according to the text data, thereby reducing the deviation of the Mel spectrum predicted by the acoustic model.

[0071] In the second method described above, the acoustic model may include a duration prediction model and a Mel spectrum prediction model. When the acoustic model uses speech data and text data to obtain a predicted Mel spectrum, the duration prediction model may first be used to predict the duration of the speech data to obtain a prediction result. Then the Mel spectrum prediction model may convert the text data according to the prediction result to generate a Mel spectrum of the corresponding time length.

[0072] At step S302, a perturbation is added to the Mel spectrum to obtain a Mel spectrum after the perturbation is added. In some embodiments, when adding a perturbation to the Mel spectrum, multiple types of perturbations can be selected as needed. For example, one or more perturbations such as mask window, harmonic interference, harmonic noise, and dotting sound can be selected in any combination during the training process of the vocoder.

[0073] At step S303, the vocoder is trained using the Mel spectrum after adding the disturbance for synthesizing speech. In some embodiments, the vocoder is trained using the Mel spectrum after adding the disturbance so that the vocoder has a stronger error correction capability for the disturbance, so that when the vocoder is used to synthesize speech, higher quality speech can be synthesized.

[0074] Figure 4 The flowchart of the method 400 for adding disturbance to the Mel spectrum according to an embodiment of the present invention is schematically shown. It should be noted that the method 400 can be understood as Figure 3 A possible exemplary implementation of adding disturbances in the training method 300 of the vocoder. Figure 3 The relevant description in also applies to the following.

[0075] like Figure 4As shown, at step S401, the disturbance category and frequency to be added are determined. In some embodiments, by analyzing various abnormal samples, four different disturbance categories can be determined, including mask window, harmonic disturbance, inter-harmonic noise and dotting sound.

[0076] In step S402, disturbances corresponding to the disturbance category and frequency are added to the Mel spectrum.

[0077] In some embodiments, any one of mask window, harmonic disturbance, inter-harmonic noise and dotting sound can be added to the Mel spectrum. All disturbance categories can also be added. Further, multiple types of the above disturbance categories can also be added, such as "mask window + harmonic disturbance", "mask window + harmonic disturbance + harmonic noise", "mask window + harmonic noise + dotting sound" and other forms.

[0078] In step S403, a perturbation of a set time percentage is added to the audio duration corresponding to the Mel spectrum of the training data to obtain the Mel spectrum after the perturbation is added. In order to ensure the effectiveness of the vocoder training process, when adding perturbations, the audio duration corresponding to the selected one or more perturbations accounts for 15% of the total audio duration.

[0079] The above content explains how to add perturbations to the Mel spectrum corresponding to the training data. Before adding the perturbation, you need to determine the perturbation category that can be added based on the existing abnormal samples. Specifically, you can first detect the defect type and frequency of abnormal samples in the business scenario. Then, based on the defect type and frequency of abnormal samples, determine the perturbation category and frequency that need to be added. Next, we will combine Figure 5 The disturbance determination process is elaborated in detail.

[0080] Figure 5 A flow chart of a method 500 for determining a disturbance type according to an embodiment of the present invention is schematically shown.

[0081] like Figure 5 As shown, at step S501, the text sample in the business scenario is used to synthesize speech data through a vocoder. In practical applications, the vocoder in the speech synthesis system can synthesize speech according to the text in the actual business scenario, and the speech may contain various types of defects. Based on this, the vocoder to be trained in this embodiment can be used to synthesize speech data from the text sample in the business scenario, so as to obtain the corresponding disturbance type according to the speech data.

[0082] At step S502, abnormal samples are determined according to the voice data. In some embodiments, the voice data can be analyzed to determine samples containing various abnormalities. For example, when there is tremolo, electronic sound, etc. in the voice, the voice data can be used as an abnormal sample.

[0083] At step S503, the defect type and occurrence frequency of the abnormal sample are determined according to the Mel spectrum of the abnormal sample. In some embodiments, the acoustic model can determine the corresponding Mel spectrum according to the abnormal sample, and the Mel spectrum contains defect information. For example, the Mel spectrum of the abnormal sample can be compared with the normal Mel spectrum, and the defect types in the Mel spectrum can be classified and counted, so as to determine the defect type and occurrence frequency of the abnormal sample.

[0084] The above-mentioned defect types may include tremolo, electronic music and dotted sounds. At step S504, the disturbances in the corresponding Mel spectrum are debugged according to the auditory performances corresponding to the tremolo, electronic music and dotted sounds to determine the disturbance category that needs to be added. In some embodiments, technicians can summarize the rules based on abnormal samples and debug appropriate disturbance functions to fit such abnormal auditory performances, such as tremolo, electronic music, dotted sounds and other auditory perceptions. Further, the Mel spectrum of the abnormal sample can adopt any of the two methods of obtaining the Mel spectrum mentioned above, and those skilled in the art can choose according to actual needs.

[0085] At step S505, the frequency of adding disturbances to the Mel spectrum of the training data is determined according to the frequency of occurrence of the defect type in the abnormal sample. In some embodiments, disturbances of corresponding frequencies can be directly added to the Mel spectrum of the training data according to the frequency of occurrence of the defect type in the abnormal sample obtained by the above statistics. It is also possible to add disturbances of corresponding frequencies to the Mel spectrum according to the proportional relationship between multiple disturbance categories.

[0086] The above content explains how to determine the disturbance category and frequency. After determining the disturbance category and frequency to be added, the disturbance needs to be added to the Mel spectrum accordingly. Next, the process of adding four different types of disturbances will be explained separately.

[0087] Figure 6 The flowchart of a method 600 for adding different types of disturbances according to an embodiment of the present invention is schematically shown. Figures 7a to 7d The diagram schematically shows a visualization diagram of the Mel spectrum after adding different types of disturbances according to an embodiment of the present invention. It should be noted that the method 600 of adding different types of disturbances can be understood as Figure 3 A possible exemplary implementation of adding disturbances in the training method 300 of the vocoder. Figure 3 The relevant description in also applies to the following.

[0088] like Figure 6 As shown, at step S601, one or more positions are randomly selected in the Mel spectrum.

[0089] At step S602, a mask window is added at one or more locations. In some embodiments, the shape and size of the mask window are randomly generated. Figure 7a A schematic diagram of adding a rectangular mask window to the Mel spectrum is shown in FIG. Figure 7a As shown in FIG. 1 , the black rectangular window (the arrow in the figure) is the mask window added to the Mel spectrum. Using the Mel spectrum with the mask window added to train the vocoder can enhance the ability of the vocoder to restore the Mel spectrum according to the surrounding information, thereby ensuring the quality of the synthesized speech.

[0090] In step S603, one or more time points are randomly selected in the Mel spectrum.

[0091] At step S604, harmonic disturbances are injected at the one or more time points. In some embodiments, the harmonic disturbances injected at the one or more time points can be added by shuffling the harmonics. Specifically, the harmonic feature quantities at the corresponding one or more time points in the Mel spectrum are first calculated. Then the harmonic feature quantities are randomly shuffled and injected into the one or more time points in the Mel spectrum to achieve the process of injecting harmonic disturbances. Figure 7b As shown in the figure, the harmonic feature quantities are randomly shuffled and then injected into one or more time points in the Mel spectrum. The Mel spectrum corresponding to the position presents a messy structure. The box in the figure represents the structure of the visualized Mel spectrum after the harmonic perturbation is inserted.

[0092] In one application scenario, the inventor discovered that some voices trembled during speech synthesis, which was reflected in the spectrogram as abnormal jitter of harmonics. Based on this, adding a set proportion of harmonic disturbance to the Mel spectrum of the training data can enable the vocoder to synthesize smooth speech despite the interference of harmonic disturbance.

[0093] At step S605, the inter-harmonic noise sample value is obtained. In some embodiments, there may be electronic sounds and fuzzy sounds in the speech synthesis process, which are reflected in the spectrogram as unclear harmonic texture and more inter-harmonic noise. Based on this, by adding a set proportion of inter-harmonic noise to the Mel spectrum and training the vocoder, the anti-noise performance of the vocoder can be effectively improved. In some embodiments, the inter-harmonic noise sample value can be obtained by constructing a distribution model of the harmonic part in the Mel spectrum. Sampling is performed based on the distribution model to obtain the corresponding inter-harmonic noise sample value. The above distribution model takes the Gaussian distribution model as an example. First, the mean and variance of the harmonic part in the Mel spectrum are calculated, and then a Gaussian distribution model based on the mean and variance is constructed.

[0094] At step S606, the inter-harmonic noise sample values ​​are superimposed on the harmonic part of the Mel spectrum to add inter-harmonic noise. In some embodiments, by superimposing the inter-harmonic noise sample values ​​on the harmonic part of the Mel spectrum, noise can be inserted between harmonics, thereby implementing the process of adding inter-harmonic noise. Figure 7c The visual structure of the Mel spectrum with added inter-harmonic noise and harmonic disturbance is shown in FIG. 1 (the part in the box in the figure). It can be seen from the Mel spectrum that the harmonic texture is not clear and there is a lot of noise between the harmonics.

[0095] At step S607, the starting position and / or the ending position corresponding to the speech pause in the Mel spectrum is determined. During speech synthesis, a dotted sound is likely to appear at the starting or ending position of the speech pause. After measurement, the magnitude of the dotted sound is generally in the energy range of -6 to -2.

[0096] At step S608, a pulse signal of a set size is added at the start position and / or the end position to add a dotted sound. In some embodiments, by simulating the spectral representation of the dotted sound, a pulse signal of a set size can be used to represent the dotted sound. At the start position of the pause corresponding to the audio in the Mel spectrum, a pulse signal of a set size can be added to add a dotted sound disturbance. Figure 7d As shown in the visual graph of the Mel spectrum, when a pause occurs, a pulse signal structure appears at that position (the content in the box in the figure), that is, a dotted sound appears. By adding dotted sounds to the training data of the vocoder, the dotted sound phenomenon in speech synthesis can be effectively reduced.

[0097] The above steps S601 to S602 describe a method of adding a mask window in the Mel spectrum. Steps S603 to S604 describe a method of adding harmonic disturbances in the Mel spectrum. Steps S605 to S606 describe a method of adding interharmonic noise in the Mel spectrum. Steps S607 to S608 describe a method of adding dotting sounds in the Mel spectrum. It should be noted that the present invention does not limit the order and combination of adding various disturbances. Those skilled in the art can select any combination of disturbances as needed and add them to the Mel spectrum to train the vocoder. For example, the category and frequency of disturbances can be randomly added to the Mel spectrum. It can also be selected according to the frequency of abnormal speech in real scenes. Those skilled in the art can also choose to add disturbances of corresponding categories with emphasis according to actual application needs.

[0098] Figure 8 The flowchart of a method 800 for performing speech synthesis using a vocoder according to an embodiment of the present invention is schematically shown.

[0099] like Figure 8As shown, at step S801, text data to be converted is obtained. In some embodiments, any text data in a real scene can be used.

[0100] At step S802, a corresponding Mel spectrum is generated according to the text data. In some embodiments, the text analysis module in the speech synthesis system can perform regularization, prosody prediction and other processing on the text to obtain text features corresponding to the text data to be converted, and then the acoustic model predicts the corresponding Mel spectrum according to the text features.

[0101] At step S803, the Mel spectrum is processed by a vocoder to synthesize the target speech. Since the training method of the vocoder has been described in detail in the above content, it will not be repeated here. Since the vocoder uses training data containing disturbances during the training process, it can correct and process the defective Mel spectrum features, so that high-quality speech content can be restored based on the Mel spectrum with certain defects, effectively improving the quality of the synthesized speech of the speech synthesis system.

[0102] Exemplary Devices

[0103] After introducing the method of the exemplary embodiment of the present invention, next, refer to Fig. 9 Related products of exemplary embodiments of the present invention are described.

[0104] Fig. 9 Schematically shows a schematic diagram of an electronic device according to an embodiment of the present invention. Fig. 9 As shown, the device 900 includes: a processor 901 and a memory 902. The memory 902 stores computer instructions for training a vocoder or performing speech synthesis using a vocoder. When the computer instructions are executed by the processor 901, the device executes the vocoder training method described in the foregoing and multiple embodiments below, or the device executes the method for performing speech synthesis using a vocoder described in the foregoing and multiple embodiments below. The present invention does not limit the structural design that the device 900 may have.

[0105] Although the operations of the method of the present invention are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in the particular order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0106] The use of the verbs "comprise", "include" and their conjugations mentioned in the application documents does not exclude the presence of elements or steps other than those recorded in the application documents. The article "a" or "an" before an element does not exclude the presence of a plurality of such elements.

[0107] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and this division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims. The scope of the attached claims conforms to the broadest interpretation, thereby including all such modifications and equivalent structures and functions.

Claims

1. A method for training a vocoder, characterized in that: include: Acquire training data, where the training data is a Mel spectrum of speech data; Adding a disturbance to the Mel spectrum to obtain a Mel spectrum after the disturbance is added; as well as Using the perturbed Mel spectrum to train the vocoder for synthesizing speech; Adding a disturbance to the Mel spectrum to obtain the Mel spectrum after the disturbance is added includes: Detect defect types and frequencies of abnormal samples in business scenarios; Determining the type and frequency of disturbances to be added based on the defect type and frequency of the abnormal sample; and Adding disturbances of corresponding disturbance categories and frequencies to the Mel spectrum, wherein the disturbance categories include one or more of mask windows, harmonic disturbances, inter-harmonic noise, and dotting sounds; The defect types include tremolo, electric sound and tapping sound, and based on the defect type and occurrence frequency of the abnormal sample, determining the disturbance type and frequency to be added includes: According to the auditory performance of vibrato, electronic music and tapping sound, the corresponding perturbations in the Mel spectrum are debugged to determine the type of perturbation that needs to be added; The frequency of adding disturbance to the Mel spectrum of the training data is determined according to the frequency of occurrence of the defect type in the abnormal sample.

2. The training method according to claim 1, characterized in that: Wherein adding a mask window in the Mel spectrum comprises: Randomly selecting one or more positions in the Mel spectrum; A mask window is added at the one or more locations.

3. The training method according to claim 2, characterized in that: The shape and size of the mask window are randomly generated.

4. The training method according to claim 1, characterized in that: Wherein adding harmonic disturbance to the Mel spectrum comprises: Randomly select one or more time points in the Mel spectrum; A harmonic disturbance is injected at the one or more time points.

5. The training method according to claim 4, characterized in that: The injecting harmonic disturbance at one or more time points comprises: Calculating harmonic characteristic quantities corresponding to one or more time points in the Mel spectrum; The harmonic characteristic quantity is randomly disrupted and injected into the one or more time points to inject harmonic disturbance.

6. The training method according to claim 1, characterized in that: Wherein adding interharmonic noise to the Mel spectrum comprises: Get interharmonic noise sample value; The inter-harmonic noise sample value is superimposed on the harmonic part of the Mel spectrum to add the inter-harmonic noise.

7. The training method according to claim 6, characterized in that: The obtaining of the inter-harmonic noise sample value comprises: Construct a distribution model of the harmonic part of the Mel spectrum; Sampling is performed based on the distribution model to obtain corresponding inter-harmonic noise sample values.

8. The training method according to claim 7, characterized in that: The distribution model includes a Gaussian distribution model, and the distribution model for constructing the harmonic part in the Mel spectrum includes: The mean and variance of the harmonic part in the Mel spectrum are calculated, and a Gaussian distribution model based on the mean and variance is constructed.

9. The training method according to claim 1, characterized in that: Adding dotted notes to the Mel spectrum includes: Determine a starting position and / or an ending position in the Mel spectrum corresponding to a speech pause; A pulse signal of a set size is added at the starting position and / or the ending position to add a dot sound.

10. The training method according to claim 1, characterized in that: Adding perturbation to the Mel spectrum to obtain the Mel spectrum after adding perturbation also includes: Add a perturbation of a set time percentage to the audio duration corresponding to the Mel spectrum of the training data to obtain the Mel spectrum after the perturbation.

11. The training method according to claim 1, characterized in that: The defect types and frequencies of abnormal samples in the detection business scenarios include: Use text samples from business scenarios to synthesize speech data through a vocoder; Determining abnormal samples according to the speech data; The defect type and occurrence frequency of the abnormal sample are determined according to the Mel spectrum of the abnormal sample.

12. The training method according to claim 1, characterized in that: The Mel spectrum is obtained by any of the following methods: Extract Mel spectra directly from speech data; or The speech data and text data are input into the acoustic model to obtain the predicted Mel spectrum.

13. The training method according to claim 12, characterized in that: The acoustic model includes a duration prediction model and a Mel spectrum prediction model, wherein the speech data and the text data are input into the acoustic model to obtain a predicted Mel spectrum including: Predicting the time length of the speech data using the time length prediction model to obtain a prediction result; The Mel spectrum prediction model converts the text data according to the prediction result to generate a Mel spectrum of corresponding time length.

14. A method for speech synthesis using a vocoder, characterized in that: include: Get the text data to be converted; Generate a corresponding Mel spectrum according to the text data; The Mel spectrum is processed by a vocoder to synthesize a target speech, wherein the vocoder is trained according to the training method according to any one of claims 1 to 13.

15. An electronic device, characterized in that: include: processor; as well as A memory storing computer instructions for training a vocoder or performing speech synthesis using a vocoder, wherein when the computer instructions are executed by the processor, the device executes the vocoder training method according to any one of claims 1 to 13, or the device executes the method for performing speech synthesis using a vocoder according to claim 14.

16. A computer-readable storage medium, characterized in that: Contains program instructions for training a vocoder or performing speech synthesis using a vocoder. When the program instructions are executed by a processor, the vocoder training method according to any one of claims 1 to 13 is implemented, or the method for performing speech synthesis using a vocoder according to claim 14 is implemented.

Images