A speech synthesis method and system for abnormal speakers
By training multiple models, the pronunciation synthesis of abnormal pronunciation people is realized, solving the problem of difficulty in constructing a pronunciation synthesis system in the existing technology, reducing the requirements for normal pronunciation data, and realizing pronunciation synthesis and conversion of abnormal pronunciation people.
Patent Information
- Application Number
- CN202210224780.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-07
- Publication Date
- 2025-03-14
- Estimated Expiration
- 2042-03-07
AI Technical Summary
The prior art is difficult to build a speech synthesis system for abnormal pronunciation people, especially for people who cannot pronounce normally, such as infants and young children and people with pronunciation disorders.
By training the basic synthesis model, the first target pronunciation spectrum prediction model, the second target pronunciation neural network vocoder and the conversion model, the speech synthesis of the input text is realized, and a synthetic speech with the second target pronunciation timbre is generated without the need for an abnormal pronunciation library.
The pronunciation synthesis of abnormal pronunciation people is realized, the data volume requirements for normal pronunciation people are reduced, the pronunciation of abnormal pronunciation people can be imitated, and the pronunciation relationship between normal pronunciation people and abnormal pronunciation people is constructed.
Smart Images
Figure CN114464160B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of speech synthesis, and in particular relates to a speech synthesis method and system for an abnormal speaker. Background Art
[0002] Speech synthesis, also known as text-to-speech technology, can convert any text information into speech in real time, which is equivalent to installing an artificial mouth on the machine. It involves multiple disciplines such as acoustics, linguistics, digital signal processing, and computer science. Speech synthesis usually includes speech processing, prosody processing, and acoustic processing. Speech processing plays an important role in the text-to-speech system, mainly simulating the process of human understanding of natural language; prosody processing plans the characteristics of the sound segment for the synthesized speech, such as pitch, length, and intensity, so that the synthesized speech can correctly express the meaning and sound more natural; acoustic processing outputs speech according to the requirements of the processing results of the first two parts, that is, synthesized speech.
[0003] The speech synthesis process requires the creation of a corresponding sound library as training data. The speech library is a warehouse for storing sounds. The speech library is usually recorded by real people in the form of words or sentences, and then stored in a database. However, it is difficult to build a corresponding speech synthesis system for people who cannot pronounce normally, such as infants and young children, and people with pronunciation disorders. How to synthesize speech with abnormal pronunciation for input text is a technical problem worth solving. Summary of the invention
[0004] To this end, the present invention provides a speech synthesis method and system for an abnormal speaker, which enables a normal speaker to imitate the pronunciation of the abnormal speaker, without requiring a voice library of the abnormal speaker and reducing the data volume requirement for the normal speaker.
[0005] In order to achieve the above-mentioned object, the present invention provides the following technical solutions: a speech synthesis method for an abnormal speaker, comprising a model training stage and a speech synthesis stage;
[0006] The model training phase includes:
[0007] Step (11) training a basic synthesis model;
[0008] Step (12) training a first target speaker spectrum prediction model;
[0009] Step (13) training a second target speaker neural network vocoder;
[0010] Step (14) training a conversion model from the first target speaker to the second target speaker;
[0011] The speech synthesis stage includes:
[0012] Step (21) parses the input text, and sends the obtained phoneme representation as input to the first target speaker spectrum prediction model to obtain the spectrum parameters of the input text;
[0013] Step (22) sends the obtained spectrum parameters of the input text as input to the conversion model to obtain the converted spectrum parameters;
[0014] Step (23) sends the converted spectral parameters as input to a neural network vocoder to obtain synthesized speech with the timbre of the second target speaker.
[0015] As a preferred solution for the speech synthesis method for abnormal speakers, step (11) is a process of training a basic synthesis model, obtaining a preset amount of synthesis sound library data, wherein the synthesis sound library data includes text and speech pairs, and the basic synthesis model is trained using the synthesis sound library data.
[0016] As a preferred solution for the speech synthesis method for a non-normal speaker, step (12) is a process of training a spectrum prediction model for the first target speaker, using the normal pronunciation data of the first target speaker to retrain the basic synthesis model to obtain the spectrum prediction model for the first target speaker.
[0017] As a preferred solution for the speech synthesis method for a non-normal speaker, step (13) trains the neural network vocoder process of the second target speaker, uses the spectrum parameters of the second target speaker as input and the speech as output, retrains the basic neural network vocoder model, and obtains the neural network vocoder model.
[0018] As a preferred embodiment of the speech synthesis method for a non-normal speaker, step (14) trains a conversion model process from the first target speaker to the second target speaker, extracts spectral parameters from the abnormal speech of the first target speaker and the speech of the second target speaker, takes the spectral parameters corresponding to the speech of the first target speaker as input, and takes the spectral parameters corresponding to the speech of the second target speaker as output, and trains the conversion model based on a neural network model.
[0019] As a preferred solution for the speech synthesis method for a non-normal speaker, step (14) also includes: aligning the spectral parameters of the first target speaker and the second target speaker, and then using the spectral parameters corresponding to the speech of the first target speaker as input and the spectral parameters corresponding to the speech of the second target speaker as output, and training the conversion model based on the neural network model.
[0020] The present invention also provides a speech synthesis system for an abnormal speaker, based on the above-mentioned speech synthesis method for an abnormal speaker, comprising a model training unit and a speech synthesis unit;
[0021] The model training unit comprises:
[0022] A first training subunit, used for training a basic synthesis model;
[0023] A second training subunit is used to train a first target speaker spectrum prediction model;
[0024] A third training subunit is used to train a neural network vocoder for a second target speaker;
[0025] A fourth training subunit, for training a conversion model from a first target speaker to a second target speaker;
[0026] The speech synthesis unit comprises:
[0027] The text parsing subunit is used to parse the input text and send the obtained phoneme representation as input to the first target speaker spectrum prediction model to obtain the spectrum parameters of the input text;
[0028] The spectrum conversion subunit is used to send the obtained spectrum parameters of the input text as input to the conversion model to obtain the converted spectrum parameters;
[0029] The speech generation subunit is used to send the converted spectrum parameters as input to the neural network vocoder to obtain the synthesized speech with the timbre of the second target speaker.
[0030] As a preferred solution of the speech synthesis system for abnormal speakers, in the first training subunit, a preset amount of synthesized sound library data is obtained, the synthesized sound library data includes text and speech pairs, and the basic synthesis model is trained by the synthesized sound library data;
[0031] In the second training subunit, the basic synthesis model is retrained using normal pronunciation data of the first target speaker to obtain a spectrum prediction model of the first target speaker;
[0032] In the third training subunit, the spectrum parameters of the second target speaker are used as input and the speech is used as output to retrain the basic neural network vocoder model to obtain a neural network vocoder model.
[0033] As a preferred solution for the speech synthesis system for abnormal speakers, in the fourth training subunit, spectral parameters are extracted from the abnormal speech of the first target speaker and the speech of the second target speaker, and the spectral parameters corresponding to the speech of the first target speaker are used as input, and the spectral parameters corresponding to the speech of the second target speaker are used as output, and the conversion model is trained based on the neural network model.
[0034] As a preferred solution for the speech synthesis system for non-normal speakers, it also includes a spectrum alignment subunit, which is used to align the spectrum parameters of the first target speaker and the second target speaker, and then use the spectrum parameters corresponding to the speech of the first target speaker as input and the spectrum parameters corresponding to the speech of the second target speaker as output to train the conversion model based on the neural network model.
[0035] The present invention has the following advantages: through the model training stage and the speech synthesis stage; the model training stage includes training the basic synthesis model; training the spectrum prediction model of the first target speaker; training the neural network vocoder of the second target speaker; training the conversion model from the first target speaker to the second target speaker; the speech synthesis stage includes parsing the input text, sending the obtained phoneme representation as input to the spectrum prediction model of the first target speaker, and obtaining the spectrum parameters of the input text; sending the obtained spectrum parameters of the input text as input to the conversion model to obtain the converted spectrum parameters; sending the converted spectrum parameters as input to the neural network vocoder to obtain the synthesized speech with the timbre of the second target speaker. The present invention collects the speech of people who cannot pronounce normally (such as infants who cannot speak normally or people with pronunciation disorders), and imitates them through normal people to build a sound conversion system between the timbre of normal people and people who cannot pronounce normally. People who can pronounce normally can imitate the pronunciation of abnormal speakers, and build a conversion relationship between the two speakers. The sound library of abnormal speakers is not required, and the model of normal speakers is built by retraining method, so that the data volume requirement for normal speakers is reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.
[0037] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.
[0038] Figure 1 A schematic flow chart of a method for synthesizing speech for an abnormal speaker provided in Example 1 of the present invention;
[0039] Figure 2 Schematic diagram of a speech synthesis system for an abnormal speaker provided in Example 2 of the present invention. DETAILED DESCRIPTION
[0040] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0041] Example 1
[0042] See also Figure 1 , Embodiment 1 of the present invention provides a speech synthesis method for a non-normal speaker, including an S1 model training stage and an S2 speech synthesis stage;
[0043] The S1 model training phase includes:
[0044] S11, training basic synthesis model;
[0045] S12, training a first target speaker spectrum prediction model;
[0046] S13, training a neural network vocoder for a second target speaker;
[0047] S14, training a conversion model from the first target speaker to the second target speaker;
[0048] The S2 speech synthesis stage includes:
[0049] S21, parsing the input text, and sending the obtained phoneme representation as input to the first target speaker spectrum prediction model to obtain spectrum parameters of the input text;
[0050] S22, sending the obtained spectrum parameters of the input text as input to the conversion model to obtain the converted spectrum parameters;
[0051] S23, sending the converted spectrum parameters as input to a neural network vocoder to obtain synthesized speech with the timbre of the second target speaker.
[0052] In this embodiment, the first target speaker is a person who can speak normally, and the second target speaker is a person who does not speak normally, such as an infant who cannot speak normally, or a person with a speech disorder.
[0053] In this embodiment, step S11 is a process of training a basic synthesis model, obtaining a preset amount of synthesis sound library data, wherein the synthesis sound library data includes text and speech pairs, and the basic synthesis model is trained by using the synthesis sound library data.
[0054] Specifically, a basic synthesis model is trained to obtain large-scale synthetic sound library data, which includes text and speech pairs. The basic synthesis model is trained using the synthetic sound library data. The training steps are as follows:
[0055] 1) Spectrum prediction model: obtain the phoneme representation corresponding to the text, take the phoneme representation of the text as input, and extract the spectrum parameters from the speech as output to train the end-to-end basic synthesis model e2e_base;
[0056] 2) Neural network vocoder: With spectrum parameters as input and speech as output, the neural network vocoder is trained using the WaveNet model proposed by Google. The trained neural network vocoder model is denoted as nnvocoder_base.
[0057] In this embodiment, step S12 is a process of training the spectrum prediction model of the first target speaker, using the normal pronunciation data of the first target speaker to retrain the basic synthesis model to obtain the spectrum prediction model of the first target speaker.
[0058] Specifically, the spectrum prediction model of the first target speaker (normal speaker) is trained: a small amount of normal pronunciation data of the first target speaker is used to retrain the basic synthesis model e2e_base. During the retraining process, the phoneme representation corresponding to the text of the first target speaker data is obtained, the phoneme representation of the text is used as input, and the spectrum parameters extracted from the speech are used as output. The end-to-end basic synthesis model e2e_base is retrained to obtain the spectrum prediction model e2e_target1 of the first target speaker.
[0059] In this embodiment, step S13 is a process of training the neural network vocoder of the second target speaker, using the spectrum parameters of the second target speaker as input and the speech as output, to retrain the basic neural network vocoder model and obtain a neural network vocoder model.
[0060] Specifically, the neural network vocoder of the second target speaker (non-normal speaker) is trained: the spectrum parameters of the second target speaker are used as input and the speech is used as output, and the basic neural network vocoder model nnvocoder_base is retrained to obtain the neural network vocoder model nnvocoder_target2.
[0061] In this embodiment, step S14 trains the conversion model process from the first target speaker to the second target speaker, extracts spectral parameters from the abnormal speech of the first target speaker and the speech of the second target speaker, takes the spectral parameters corresponding to the speech of the first target speaker as input, and the spectral parameters corresponding to the speech of the second target speaker as output, and trains the conversion model based on the neural network model. Step S14 also includes: aligning the spectral parameters of the first target speaker and the second target speaker, and then taking the spectral parameters corresponding to the speech of the first target speaker as input, and the spectral parameters corresponding to the speech of the second target speaker as output, and training the conversion model based on the neural network model.
[0062] Specifically, by training a conversion model from the first target speaker (normal speaker) to the second target speaker (abnormal speaker), spectral parameters are extracted from the abnormal speech of the first target speaker and the speech of the second target speaker, respectively. Since the speech lengths of the first target speaker and the second target speaker may be inconsistent, the spectral parameters of the first target speaker and the second target speaker are aligned based on Dynamic Time Warping (DTW). Then, the spectral parameters corresponding to the speech of the first target speaker are used as input, and the spectral parameters corresponding to the speech of the second target speaker are used as output, and the conversion model convermodel is trained based on a neural network.
[0063] In this embodiment, in the speech synthesis stage, text is input to obtain synthesized speech with the timbre of the second target speaker (non-normal speaker). The input text is parsed, and the obtained phoneme representation is sent as input to the first target speaker spectrum prediction model e2e_target1 to obtain spectrum parameters; the obtained spectrum parameters are sent as input to the conversion model convermodel to obtain converted spectrum parameters; the converted spectrum parameters are sent as input to the neural network vocoder model nnvocoder_target2 to obtain synthesized speech with the timbre of the second target speaker.
[0064] In summary, the present invention includes a model training stage and a speech synthesis stage; the model training stage includes training a basic synthesis model, obtaining a preset amount of synthesis sound library data, the synthesis sound library data includes text and speech pairs, and the basic synthesis model is trained through the synthesis sound library data; training a first target speaker's spectrum prediction model, using the first target speaker's normal pronunciation data, retraining the basic synthesis model to obtain the first target speaker's spectrum prediction model; training a second target speaker's neural network vocoder, using the second target speaker's spectrum parameters as input and speech as output, retraining the basic neural network vocoder model to obtain a neural network vocoder model; training a conversion model from the first target speaker to the second target speaker, extracting spectrum parameters from the first target speaker's abnormal speech and the second target speaker's speech number, taking the spectral parameters corresponding to the voice of the first target speaker as input and the spectral parameters corresponding to the voice of the second target speaker as output, training a conversion model based on a neural network model, aligning the spectral parameters of the first target speaker and the second target speaker, and then taking the spectral parameters corresponding to the voice of the first target speaker as input and the spectral parameters corresponding to the voice of the second target speaker as output, training a conversion model based on a neural network model; the speech synthesis stage includes parsing the input text, sending the obtained phoneme representation as input to the spectral prediction model of the first target speaker to obtain the spectral parameters of the input text; sending the obtained spectral parameters of the input text as input to the conversion model to obtain the converted spectral parameters; sending the converted spectral parameters as input to the neural network vocoder to obtain synthesized speech with the timbre of the second target speaker. The present invention collects the speech of people who cannot pronounce normally (such as infants who cannot speak normally or people with pronunciation disorders), and has normal people imitate them to build a sound conversion system between the timbre of normal people and that of people who cannot pronounce normally. Normal-pronouncing people can imitate the pronunciation of abnormal-pronouncing people, and build a conversion relationship between the two speakers. The present invention does not require a sound library of abnormal-pronouncing people, and adopts a retraining method to build a model of normal-pronouncing people, thereby reducing the data requirement for normal-pronouncing people.
[0065] Example 2
[0066] See also Figure 2 , Embodiment 2 of the present invention further provides a speech synthesis system for an abnormal speaker, a speech synthesis method for an abnormal speaker based on Embodiment 1 or any possible implementation thereof, comprising a model training unit 1 and a speech synthesis unit 2;
[0067] The model training unit 1 comprises:
[0068] A first training subunit 11 is used to train a basic synthesis model;
[0069] A second training subunit 12, used for training a first target speaker spectrum prediction model;
[0070] A third training subunit 13, used for training a second target speaker neural network vocoder;
[0071] A fourth training subunit 14, for training a conversion model from a first target speaker to a second target speaker;
[0072] The speech synthesis unit 2 comprises:
[0073] The text parsing subunit 21 is used to parse the input text and send the obtained phoneme representation as input to the first target speaker spectrum prediction model to obtain the spectrum parameters of the input text;
[0074] The spectrum conversion subunit 22 is used to send the obtained spectrum parameters of the input text as input to the conversion model to obtain the converted spectrum parameters;
[0075] The speech generation subunit 23 is used to send the converted spectrum parameters as input to the neural network vocoder to obtain synthesized speech with the timbre of the second target speaker.
[0076] In this embodiment, in the first training subunit 11, a preset amount of synthetic sound library data is obtained, the synthetic sound library data includes text and speech pairs, and the basic synthesis model is trained by the synthetic sound library data;
[0077] In the second training subunit 12, the basic synthesis model is retrained using the normal pronunciation data of the first target speaker to obtain a spectrum prediction model of the first target speaker;
[0078] In the third training subunit 13, the spectrum parameters of the second target speaker are used as input and the speech is used as output to retrain the basic neural network vocoder model to obtain a neural network vocoder model.
[0079] In this embodiment, in the fourth training subunit 14, spectral parameters are extracted from the abnormal speech of the first target speaker and the speech of the second target speaker, and the spectral parameters corresponding to the speech of the first target speaker are used as input, and the spectral parameters corresponding to the speech of the second target speaker are used as output, and the conversion model is trained based on the neural network model.
[0080] In this embodiment, a spectrum alignment subunit 15 is also included, which is used to align the spectrum parameters of the first target speaker and the second target speaker, and then use the spectrum parameters corresponding to the voice of the first target speaker as input and the spectrum parameters corresponding to the voice of the second target speaker as output to train the conversion model based on the neural network model.
[0081] It should be noted that the information interaction, execution process, etc. between the modules / units of the above-mentioned system are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown in the previous part of the present application, and will not be repeated here.
[0082] Example 3
[0083] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which a program code of a speech synthesis method for a person with a non-normal speaker is stored. The program code includes instructions for executing the speech synthesis method for a person with a non-normal speaker of Embodiment 1 or any possible implementation thereof.
[0084] The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).
[0085] Example 4
[0086] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;
[0087] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the speech synthesis method for a person with an abnormal speaker in Example 1 or any possible implementation thereof.
[0088] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor implemented by reading software codes stored in a memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.
[0089] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.
[0090] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0091] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.
Claims
1. A method for synthesizing speech of an abnormal speaker, characterized in that: Includes model training stage and speech synthesis stage; The model training phase includes: Step (11) training a basic synthesis model; Step (12) training a first target speaker spectrum prediction model; Step (13) training a second target speaker neural network vocoder; Step (14) training a conversion model from the first target speaker to the second target speaker; The speech synthesis stage includes: Step (21) parses the input text, and sends the obtained phoneme representation as input to the first target speaker spectrum prediction model to obtain the spectrum parameters of the input text; Step (22) sends the obtained spectrum parameters of the input text as input to the conversion model to obtain the converted spectrum parameters; Step (23) sends the converted spectrum parameters as input to a neural network vocoder to obtain a synthesized speech with the timbre of the second target speaker; Step (12) is a process of training the spectrum prediction model of the first target speaker, using the normal pronunciation data of the first target speaker to retrain the basic synthesis model to obtain the spectrum prediction model of the first target speaker; Step (13) training the neural network vocoder process of the second target speaker, using the spectrum parameters of the second target speaker as input and the speech as output, retraining the basic neural network vocoder model to obtain the neural network vocoder model; Step (14) is a process of training a conversion model from the first target speaker to the second target speaker, extracting spectrum parameters from the abnormal speech of the first target speaker and the speech of the second target speaker, taking the spectrum parameters corresponding to the speech of the first target speaker as input and the spectrum parameters corresponding to the speech of the second target speaker as output, and training the conversion model based on the neural network model; Step (14) also includes: aligning the spectral parameters of the first target speaker and the second target speaker, and then using the spectral parameters corresponding to the speech of the first target speaker as input and the spectral parameters corresponding to the speech of the second target speaker as output, and training the conversion model based on the neural network model.
2. The method for synthesizing speech of an abnormal speaker according to claim 1, characterized in that: Step (11) is a process of training a basic synthesis model, obtaining a preset amount of synthesis sound library data, wherein the synthesis sound library data includes text and speech pairs, and the basic synthesis model is trained using the synthesis sound library data.
3. A speech synthesis system for an abnormal speaker, based on the speech synthesis method for an abnormal speaker according to any one of claims 1 to 2, characterized in that: Includes a model training unit and a speech synthesis unit; The model training unit comprises: A first training subunit, used for training a basic synthesis model; A second training subunit is used to train a first target speaker spectrum prediction model; A third training subunit is used to train a neural network vocoder for a second target speaker; A fourth training subunit, for training a conversion model from a first target speaker to a second target speaker; The speech synthesis unit comprises: The text parsing subunit is used to parse the input text and send the obtained phoneme representation as input to the first target speaker spectrum prediction model to obtain the spectrum parameters of the input text; The spectrum conversion subunit is used to send the obtained spectrum parameters of the input text as input to the conversion model to obtain the converted spectrum parameters; A speech generation subunit, used for sending the converted spectrum parameters as input to a neural network vocoder to obtain a synthesized speech with a timbre of a second target speaker; In the second training subunit, the basic synthesis model is retrained using normal pronunciation data of the first target speaker to obtain a spectrum prediction model of the first target speaker; In the third training subunit, the spectrum parameters of the second target speaker are used as input and the speech is used as output to retrain the basic neural network vocoder model to obtain a neural network vocoder model; In the fourth training subunit, spectral parameters are extracted from the abnormal speech of the first target speaker and the speech of the second target speaker, and the spectral parameters corresponding to the speech of the first target speaker are used as input and the spectral parameters corresponding to the speech of the second target speaker are used as output to train the conversion model based on the neural network model; It also includes a spectrum alignment subunit, which is used to align the spectrum parameters of the first target speaker and the second target speaker, and then use the spectrum parameters corresponding to the voice of the first target speaker as input and the spectrum parameters corresponding to the voice of the second target speaker as output to train the conversion model based on the neural network model.
4. A speech synthesis system for an abnormal speaker according to claim 3, characterized in that: In the first training subunit, a preset amount of synthetic sound library data is obtained, the synthetic sound library data includes text and speech pairs, and a basic synthesis model is trained by using the synthetic sound library data.
Citation Information
Patent Citations
Voice conversion method, device and equipment and storage medium
CN111247585A
End-to-end accent conversion method
CN111462769A