Method and apparatus for synthesizing unified voice based on self-supervised learning

Abstract

To provide a method and an apparatus for synthesizing voices similar to actual voices using an artificial neural network self-learned by self-supervised learning without the need to train the artificial neural network using a large amount of voices and text datasets.SOLUTION: Provided is a self-supervised learning-based voice synthesis method including: training a voice analysis module to output voice features for training voice signals by using the training voice signals representing training voices, and outputting voice features for the training voices; training a voice synthesis module to synthesize voice signals from the voice features for the training voices by using the output voice features, and synthesizing synthesized voice signals, representing synthesized voices, from the output voice features; and calculating reconstruction loss between the training voice signals and the synthesized voice signals, and training the voice analysis module and the voice synthesis module based on the calculated reconstruction loss and the training voices.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an integrated speech synthesis method and device based on self-supervised learning, and more particularly to an integrated speech synthesis method and device based on self-supervised learning that enables speech to be synthesized using a machine learning model trained based on self-supervised learning. [Background technology]

[0002] Text to Speech (TTS) technology is a technology that uses a computer to generate human voice, i.e., voice signals, from any sentence entered as text. Conventional voice synthesis technologies are divided into concatenative TTS, which generates a voice signal for the entire sentence by combining pre-recorded voice signals of one syllable, and parametric TTS, which generates a voice signal using a vocoder from high-dimensional parameters that represent voice characteristics.

[0003] Conventional concatenative speech synthesis methods combine pre-recorded speech signals of words, syllables, and phonemes to generate a complete speech signal for a sentence according to an input text. However, because the speech signal for a sentence generated in this way is a synthesis of pre-recorded speech signals, the intonation and prosody of the sentence are not expressed in the speech signal, resulting in a problem that the connection between speech sounds is not smooth and a strange feeling different from a human voice is felt.

[0004] Artificial intelligence (AI) technology has made great strides recently. It is also being used in a variety of ways in the field of speech synthesis. Machine learning algorithms can be used as AI algorithms, and machine learning can be broadly categorized into supervised learning and self-supervised learning. Supervised learning trains an AI-based machine learning model using data to be learned and a correct label for that data. Self-supervised learning trains an AI-based machine learning model using only the data to be learned, without a correct label for the data to be learned.

[0005] Furthermore, conventional parametric speech synthesis methods have evolved into methods that use machine learning to improve the naturalness of speech signals. This involves training an artificial neural network using a huge amount of text and speech data, and then using the trained artificial neural network to generate a speech signal for an input sentence of text. Since machine learning-based parametric speech synthesis methods generate a speech signal for an input text using an artificial neural network, they can generate a speech signal that expresses the intonation, prosody, etc., of the speech of the trained speech signal. Therefore, they can generate a more natural speech signal than concatenative speech synthesis methods. However, such machine learning-based parametric speech synthesis methods have the disadvantage of requiring a huge amount of speech and text data set to train the artificial neural network.

[0006] The drawbacks of the conventional voice synthesis technology described above are also present in singing voice synthesis (SVS) technology. Singing voice synthesis technology generates singing voice signals using lyric text and musical score data. Among conventional voice synthesis technologies, concatenative voice synthesis methods can only generate pre-recorded phoneme utterances and cannot generate singing voice signals that can be freely modified in pitch, duration, beat, etc. Therefore, parametric voice synthesis methods using artificial neural networks are mainly used in the field of singing voice synthesis.

[0007] In such a parametric speech synthesis method based on an artificial neural network, an artificial neural network is first trained with the singing voice of a given singer, the musical score of the song, and the lyrics text of the song. The trained artificial neural network can generate a singing voice signal that is similar in timbre and singing style to the singing voice (i.e., the song) of the trained singer based on the input musical score and lyrics text.

[0008] Such problems are not limited to parametric voice synthesis methods, but also apply to other existing voice synthesis or singing voice synthesis methods.

[0009] Therefore, there is an emerging need for a speech or singing voice synthesis method based on self-supervised learning to solve such problems. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Korean Patent Publication No. 10-2020-0015418 [Patent Document 2] Korean Patent Registration No. 10-1991733 [Patent Document 3] Korean Patent Registration No. 10-2057926 [Non-patent literature]

[0011] [Non-Patent Document 1] Hyeong-Seok Choi, Jinhyeok Yang, Juheon Lee, and Hyeongju Kim, “NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis,”arXiv:2211.09407v1 [csSD], 17 Nov 2022. Summary of the Invention [Problem to be solved by the invention]

[0012] The present invention aims to provide a method and apparatus for integrated speech synthesis based on self-supervised learning. Furthermore, the present invention is not limited to the technical problems described above, and other technical problems can be derived from the following description. [Means for solving the problem]

[0013] A speech synthesis method based on self-supervised learning according to one embodiment of the present invention includes the steps of: using a training speech signal representing a training speech, training a speech analysis module to output speech features for the training speech signal, and outputting the speech features for the training speech; and using the output speech features, training a speech synthesis module to synthesize a speech signal from the speech features of the training speech, and synthesizing a synthetic speech signal representing a synthetic speech from the output speech features.

[0014] A speech synthesis method based on self-supervised learning according to one embodiment of the present invention further includes calculating reconstruction losses of the training speech signal and the synthesized speech signal based on the training speech signal and the synthesized speech signal, and performing training on the speech analysis module and the speech synthesis module based on the calculated reconstruction losses.

[0015] The speech characteristics of the training speech are the fundamental frequency F0, the periodic amplitude A p [n], non-periodic amplitude A ap[n], pronunciation features and timbre features, and the step of outputting speech features for the training speech includes converting the training speech signal into a probability distribution spectrum of a plurality of frequency bins, and extracting the fundamental frequency F0, periodic amplitude A of the training speech from the converted probability distribution spectrum. p [n] and aperiodic amplitude A ap [n], and outputting the training speech signal and converting the training speech signal into a mel spectrogram and outputting timbre features of the training speech from the converted mel spectrogram.

[0016] The step of synthesizing a synthetic speech signal representing a synthetic speech from the output speech features includes: synthesizing a fundamental frequency F0, a periodic amplitude A of the training speech, p [n] and aperiodic amplitude A ap generating an input excitation signal based on [n]; generating a time-varying tonal embedding based on timbre features of the training speech; generating frame-level conditions for the synthetic speech based on pronunciation features of the training speech and the generated time-varying tonal embedding; and synthesizing a synthetic speech signal representing the synthetic speech based on the input excitation signal and the frame-level conditions.

[0017] The input excitation signal is according to the following mathematical formula 1:

[0018]

number

[0019] (N s is the sampling rate and n[t] is the sampled noise)

[0020] A speech synthesis device based on self-supervised learning according to another embodiment of the present invention includes a speech analysis module that uses a training speech signal representing a training speech to train a speech analysis module to output speech features for the training speech signal and outputs the speech features for the training speech, and a speech synthesis module that uses the output speech features to train a speech synthesis module to synthesize a speech signal from the speech features of the training speech and synthesizes a synthetic speech signal representing a synthetic speech from the output speech features.

[0021] According to yet another embodiment of the present invention, a singing voice synthesis method based on self-supervised learning executed by a speech synthesis device including: a speech analysis module that uses a training speech signal indicating a training speech to train a speech analysis module to output speech features for the training speech signal, and outputs the speech features for the training speech; and a speech synthesis module that uses the output speech features to train a speech synthesis module to synthesize a speech signal from the speech features of the training speech, and synthesizes a synthetic speech signal indicating a synthetic speech from the output speech features. The method includes the steps of: training a speech analysis module using a training speech signal indicating a training speech to output speech features for the training speech signal, and outputting the speech features for the training speech; training a speech synthesis module using the output speech features to synthesize a speech signal from the speech features of the training speech, and synthesizing a synthetic speech signal indicating a synthetic speech from the output speech features; obtaining a singing voice synthesis request including a song to be synthesized and a singer to be synthesized; obtaining a voice signal related to the singer to be synthesized based on the singing voice synthesis request; A synthesis module calculates a fundamental frequency F0, a periodic amplitude A for the target song and the target singer based on the request for singing voice synthesis and the voice signal related to the target singer. p [n], non-periodic amplitude A apgenerating singing voice features including [n] and pronunciation features, in the voice analysis module generating timbre features of the synthesis target singer based on the acquired voice signal related to the synthesis target singer, and in the voice synthesis module synthesizing a singing voice signal indicative of the voice of the synthesis target singer singing the synthesis target song based on the singing voice features and the timbre features.

[0022] The SVS module is an artificial neural network that is pre-trained using a training dataset containing training songs, training singer voices, and training vocal features to output vocal features for an input target song and target singer to be synthesized.

[0023] According to yet another embodiment of the present invention, a method for synthesizing altered speech based on self-supervised learning executed by a speech synthesis device including a speech analysis module that uses a training speech signal indicative of a training speech to train a speech analysis module to output speech features for the training speech signal and output the speech features for the training speech, and a speech synthesis module that uses the output speech features to train a speech synthesis module to synthesize a speech signal from the speech features of the training speech and synthesize a synthetic speech signal indicative of a synthetic speech from the output speech features, includes the steps of: using a training speech signal indicative of the training speech to train a speech analysis module to output speech features for the training speech signal and outputting the speech features for the training speech; p [n], non-periodic amplitude A apThe method includes a step of outputting pre-conversion speech features including [n] and pronunciation features, a step of acquiring speech attributes for the converted speech, a step of outputting converted speech features including a fundamental frequency F0 and timbre features for the converted speech in a VOD (Voice Design) module based on the speech attributes for the converted speech, and a step of synthesizing converted speech based on the pre-conversion speech features and the converted speech features in the speech synthesis module.

[0024] The VOD module is an artificial neural network that is pre-trained with a training dataset including training voice attributes, training fundamental frequency F0, and training timbre features to output the fundamental frequency F0 and timbre features of the converted voice based on the input voice attributes.

[0025] According to yet another embodiment of the present invention, there is provided a Text To Speech (TTS) system based on self-supervised learning, which is executed by a speech synthesis device including: a speech analysis module that uses a training speech signal representing a training speech to train a speech analysis module to output speech features for the training speech signal, and outputs the speech features for the training speech; and a speech synthesis module that uses the output speech features to train a speech synthesis module to synthesize a speech signal from the speech features of the training speech, and synthesizes a synthetic speech signal representing a synthetic speech from the output speech features. The speech synthesis method includes the steps of: using a training speech signal representing a training speech, training a speech analysis module to output speech features for the training speech signal, and outputting the speech features for the training speech; training a speech synthesis module to synthesize a speech signal from the speech features of the training speech using the output speech features, and synthesizing a synthetic speech signal representing a synthetic speech from the output speech features; acquiring a synthesis target text and a synthesis target speech subject for TTS synthesis; acquiring a speech related to the synthesis target speech subject based on the synthesis target speech subject; outputting speech features of the synthesis target speech subject, including timbre features of the synthesis target speech subject, based on the speech related to the synthesis target speech subject, in the speech analysis module; and outputting speech features of the synthesis target speech subject, including timbre features of the synthesis target speech subject, in the TTS module, based on the synthesis target text and the speech related to the synthesis target speech subject. p [n] and aperiodic amplitude A ap [n], and outputting the speech characteristics of the text speech including the fundamental frequency F0, the periodic amplitude A p [n] and aperiodic amplitude A ap [n], and synthesizing text speech based on the timbre features of the subject speech to be synthesized.

[0026] The TTS module calculates the fundamental frequency F0 and periodic amplitude A of the text speech based on the input text and speech using a training dataset including a training synthetic text, a training speech, and training speech features. p [n] and aperiodic amplitude A ap It is an artificial neural network pre-trained to output [n].

[0027] A computer-readable recording medium according to yet another embodiment of the present invention has recorded thereon a program for executing a speech synthesis method based on self-supervised learning according to an embodiment of the present invention. [Effects of the Invention]

[0028] A speech synthesis device and speech synthesis method according to the present invention use artificial intelligence based on self-supervised learning to analyze a speech signal, extract speech features from the speech signal, and resynthesize speech based on the extracted speech features. The artificial intelligence repeatedly analyzes and synthesizes the speech signal, outputs speech features of the input speech, and resynthesizes speech based on the output speech features. According to an embodiment of the present invention, an artificial neural network learns to extract speech features from the input speech and to output synthetic speech based on the extracted speech features. By using an artificial intelligence model based on self-supervised learning, compared to artificial intelligence models based on supervised learning in the field of speech synthesis, a massive amount of training audio data is not required to train the artificial neural network model, and the artificial neural network for speech synthesis can be trained quickly and easily.

[0029] In addition, the speech analysis module and speech synthesis module, which are artificial neural networks of the speech synthesizer, calculate a reconstruction loss between the input speech and the synthesized speech during self-supervised learning, and train the speech analysis module and speech synthesis module based on the calculated reconstruction loss, thereby minimizing the difference between the speech synthesized by the speech synthesizer and the actual speech. By utilizing the loss function, the speech synthesizer can synthesize speech that is very similar to the actual speech more naturally.

[0030] The voice synthesis method utilizes an artificial neural network trained in a self-supervised learning manner to convert any singing voice into the voice of a singer of the user's choice. Therefore, even if the singing voice is not actually sung by the target singer, a natural singing voice that is identical or very similar to the singing voice of the target singer can be synthesized.

[0031] In addition, the speech synthesis method can analyze speech using only predetermined training speech and train an artificial neural network that synthesizes speech based on the analysis results. For example, a small amount of speech of less than 10 minutes can be used to train an artificial neural network for speech synthesis. Therefore, even if you do not have a large amount or length of speech data, you can restore the voice of a deceased singer or great figure from a short recording of their voice when they were alive. [Brief explanation of the drawings]

[0032] [Figure 1] 1 is a configuration diagram of a speech synthesis device according to an embodiment of the present invention. [Figure 2] 2 is an exemplary diagram illustrating a process of training a voice analysis module and a voice synthesis module in the voice synthesis device shown in FIG. 1. [Figure 3] FIG. 1 is a detailed structural diagram of a pitch encoder. [Figure 4] FIG. 1 is a detailed structural diagram of a pronunciation encoder. [Figure 5] FIG. 1 is a detailed structural diagram of a tone encoder. [Figure 6] FIG. 1 is a detailed structural diagram of a frame-level synthetic artificial neural network. [Figure 7] FIG. 1 is a detailed structural diagram of a time-varying timbre artificial neural network. [Figure 8] 1 is a flowchart of a speech synthesis method based on self-supervised learning according to an embodiment of the present invention. [Figure 9] 1 is a flowchart of a singing voice synthesis method based on self-supervised learning according to an embodiment of the present invention. [Figure 10] 1 is a flowchart of a method for synthesizing distorted speech based on self-supervised learning according to another embodiment of the present invention. [Figure 11] 10 is a flowchart of a TTS synthesis method based on self-supervised learning according to yet another embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0033] The advantages and features of the present invention, as well as methods for achieving them, will become more apparent from the following detailed description of the embodiments with reference to the accompanying drawings. However, the technical concept of the present invention is not limited to the following embodiments, and may be embodied in various different forms. However, the following embodiments are provided to fully convey the technical concept of the present invention and to fully inform those skilled in the art of the present invention of the scope of the present invention, and the technical concept of the present invention is defined only by the scope of the claims.

[0034] When adding reference numerals to components in each drawing, care should be taken to use the same numerals as much as possible for the same components, even if they are displayed in different drawings. Furthermore, in the description of the present invention, if it is determined that a detailed description of related publicly known structures or functions may obscure the gist of the present invention, the detailed description will be omitted.

[0035] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in a manner commonly understood by those skilled in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries should not be interpreted ideally or excessively unless expressly defined otherwise. The terms used herein are intended to describe the embodiments and are not intended to limit the present invention. In this specification, the singular form includes the plural form unless otherwise specified in the text.

[0036] Furthermore, in describing components of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. Such terms are merely used to distinguish the component from other components, and do not limit the essence, order, procedure, etc. of the components. When a component is described as being "coupled," "coupled," or "connected" to another component, it should be understood that the component can be directly coupled or connected to the other component, but that other components can also be "coupled," "coupled," or "connected" between the components.

[0037] As used herein, "comprises" and / or "comprising" does not exclude the presence or addition of one or more other components, steps, operations and / or elements to the stated components, steps, operations and / or elements.

[0038] Terms used in the following detailed description of the embodiments of the present invention have the following meanings: "Voice" refers to the human voice, a sound produced through the human vocal organs, and includes not only human speech but also singing voice, which is a song expressed in a human voice. Sound is a type of wave, and refers to waves transmitted by vibrations generated by the vibration of the vocal cords in the larynx in the neck. "Audio signal" refers to a signal that represents a voice-based sound.

[0039] "Voice subject" refers to the speaker who produced the sound. "Timbre" is a vocal characteristic unique to each voice subject that is physically determined by the structure of the vocal organs, and has characteristics that are distinguished by the harmonic structure of the voice.

[0040] Components having a common function with components included in one embodiment may be described with the same name in other embodiments. Unless otherwise specified, a description of one embodiment may be applied to other embodiments as well, and detailed description may be omitted to the extent that it overlaps or is obvious to a person skilled in the art.

[0041] Hereinafter, several embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0042] While the present invention can be modified in various ways and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description, but it should be understood that this is not intended to limit the present invention to the specific embodiments, and that the present invention includes all modifications, equivalents, and alternatives that fall within the spirit and technical scope of the present invention.

[0043] 1 is a block diagram of a speech synthesizer according to an embodiment of the present invention. Referring to FIG. 1, the speech synthesizer 10 includes a processor 101, an input module 102, a speech analysis module 103, a speech synthesis module 104, a singing voice synthesis (SVS) module 105, a voice design (VOD) module 106, a text to speech (TTS) module 107, an output module 108, and a storage 109.

[0044] The processor 101 of the speech synthesizer 10 handles the general tasks of the speech synthesizer 10 .

[0045] The input module 102 of the speech synthesizer 10 acquires a speech synthesis request for a target speech to be synthesized and learning data for training the artificial neural network included in the speech synthesizer 10. The input module 102 receives from the user the speech attributes of the speech to be converted, musical score, text of the synthesized speech data, the target speech, and the singer to be synthesized. Here, the speech attributes include the gender of the voice subject, the age of the voice subject, and pitch. Examples of the input module 102 include a keyboard, a mouse, and a touch panel.

[0046] The speech analysis module 103 of the speech synthesizer 10 outputs the characteristics of the input speech. The speech analysis module 103 outputs the characteristics of the speech input via the input module 102. More specifically, the speech analysis module 103 outputs the characteristics of the input speech, such as the fundamental frequency F0, the periodic amplitude A p [n], aperiodic amplitude A ap The speech analysis module 103 outputs the fundamental frequency F0, periodic amplitude A, and linguistic feature [n]. Here, the speech analysis module 103 analyzes the speech based on self-supervised learning. The process of outputting the features of the input speech by the speech analysis module 103 will be described in detail below. The speech analysis module 103 outputs the fundamental frequency F0, periodic amplitude A, and linguistic feature [n]. p [n], non-periodic amplitude A ap [n], pronunciation features, and timbre features are input to the speech synthesis module 104.

[0047] The speech synthesis module 104 of the speech synthesizer 10 synthesizes speech based on the input from the speech analysis module 103. More specifically, the speech synthesis module 104 synthesizes speech based on the fundamental frequency F0 and periodic amplitude A p [n], non-periodic amplitude A ap[n], and synthesizes a speech signal based on the pronunciation features and timbre features. The process of synthesizing a speech signal based on the input speech features in the speech synthesis module 104 will be described in detail below.

[0048] The SVS module 105 of the voice synthesizer 10 is a module configured to output singing voice characteristics in order to synthesize a singing voice signal. The SVS module 105 outputs singing voice characteristics based on the input song and singer voice to be synthesized. More specifically, the SVS module 105 analyzes the input song and singer voice, and outputs the fundamental frequency F0, periodic amplitude A, and p [n], non-periodic amplitude A ap The SVS module 105 outputs the vocal characteristics [n] and pronunciation features. The target song for synthesis includes the musical score and lyrics for the song. The vocal characteristics output from the SVS module 105 are vocal characteristics for the voice of the input singer singing the target song for synthesis.

[0049] Here, the SVS module 105 is an artificial neural network that is pre-trained to output vocal features of an input song and singer from an input song and singer audio. The SVS module 105 is an artificial neural network that is pre-trained with a training dataset that includes training songs, training singer audio, and training vocal features.

[0050] The SVS module 105 outputs the fundamental frequency F0 and periodic amplitude A p [n], non-periodic amplitude A ap [n] and the pronunciation features are input to the voice synthesis module 104. The voice synthesis module 104 synthesizes a voice in which the input musical score is sung in the input singer's voice, based on the singing voice features input from the SVS module 105 and the timbre features input from the voice analysis module 103.

[0051] The VOD module 106 of the speech synthesizer 10 is a module configured to output characteristics of converted speech in order to convert any speech into speech with desired characteristics. The VOD module 106 outputs the characteristics of converted speech based on speech attributes for the converted speech input by the user. More specifically, the VOD module 106 outputs the fundamental frequency F0 and timbre characteristics for the converted speech input from the input module 102. Here, the converted speech refers to speech in which voice characteristics such as voice, tone, timbre, and pitch have been converted from the original speech. The voice attributes for the converted speech are commands input by the user via the input module 102.

[0052] The voice attributes of the converted voice include the attributes of the basic voice, the gender of the voice subject, the age of the voice subject, and the voice pitch as the attributes of the voice to be converted. For example, if a user wishes to convert a voice whose current voice subject is male into a voice whose current voice subject is female, the user inputs the gender of the voice subject as female when inputting the voice attributes for the converted voice. The user inputs the voice attributes of the converted voice through the input module 102 according to the characteristics of the voice subject that the user wishes to convert. The user can input the gender and age, which are the voice attributes of the voice to be converted, into the input module 102.

[0053] The VOD module 106 is an artificial neural network pre-trained to output features of converted speech from speech attributes of the input converted speech. The VOD module 106 is an artificial neural network pre-trained with a training dataset including a training speech signal, training converted speech attributes, the fundamental frequency F0 of the training converted speech, and timbre features of the training converted speech.

[0054] According to yet another embodiment of the present invention, the VOD module 106 is an artificial neural network that is pre-trained to output fundamental frequency F0 and timbre features for the converted speech based on speech attributes for the original and converted speech.

[0055] The VOD module 106 inputs the fundamental frequency F0 and timbre features, which are the features of the output converted speech, to the speech synthesis module 104. The speech synthesis module 104 inputs the timbre features and fundamental frequency F0 input from the VOD module 106 and the periodic amplitude A p [n], non-periodic amplitude A ap Based on [n] and the pronunciation characteristics, a converted speech having the converted characteristics input by the user is synthesized.

[0056] The TTS module 107 of the speech synthesizer 10 is a module configured to output characteristics of input text in order to convert text into speech. The TTS module 107 outputs dialogue (speech) voice characteristics based on the input text and the speaker. More specifically, the TTS module 107 analyzes the voice of the input text and the speaker and outputs dialogue voice characteristics of the input text and the speaker. The TTS module 107 calculates the fundamental frequency F0, periodic amplitude A, and so on of the input text and the speaker. p [n], non-periodic amplitude A ap [n] and pronunciation features are output.

[0057] Here, the TTS module 107 is an artificial neural network that is pre-trained to output input text and speech-based dialogue speech features from input text and speech-based speech. The TTS module 107 is an artificial neural network that is pre-trained with a training dataset that includes training text, training speech-based speech, and training speech features.

[0058] The output module 108 of the speech synthesizer 10 converts speech into an auditory signal that can be heard by the user and outputs the converted audio. The output module 108 converts the speech synthesized by the speech synthesis module 104 into an auditory signal and outputs the audio. An example of the output module 108 is a speaker.

[0059] The storage 109 of the speech synthesizer 10 stores data necessary for speech synthesis. For example, the storage 109 stores a training speech dataset for training the artificial neural network that constitutes the speech synthesizer 10. Here, the training speech dataset includes not only speech data representing speech but also singing data representing singing.

[0060] In the speech synthesis device 10 according to an embodiment of the present invention, the input module 102, the speech analysis module 103 and the speech synthesis module 104 may be implemented by a separate dedicated processor different from the processor 101, or may be implemented by the execution of a computer program executed by the processor 101.

[0061] The speech synthesizer 10 may further include additional components in addition to the components described above. For example, as shown in FIG. 1, the speech synthesizer 10 may include a bus for transmitting data between multiple components. Although not shown in FIG. 1, the speech synthesizer 10 may further include components such as a power supply unit for supplying driving power to each component, a training unit for training the artificial neural network, and a communication module for exchanging data and signals with an external terminal. Thus, detailed descriptions of components that are obvious to those skilled in the art to which this embodiment pertains will be omitted to avoid obscuring the features of this embodiment. Below, each component of the speech synthesizer 10 will be described in detail in the course of describing a speech synthesis method according to one embodiment of the present invention.

[0062] Figure 2 is an example diagram showing a process of training a voice analysis module and a voice synthesis module in the voice synthesizer shown in Figure 1. Referring to Figure 2, the voice analysis module 103 of the voice synthesizer 10 includes a pitch encoder 1031, a linguistic encoder 1032, and a timbre encoder 1033. The voice synthesis module 104 of the voice synthesizer 10 includes a sinusoid noise generator 1041, a frame-level synthesis neural network 1042, a time-varying timbre neural network 1043, and a sample-level synthesis neural network 1044.

[0063] The speech input to the speech analysis module 103 of the speech synthesizer 10 is input to a pitch encoder 1031 , a pronunciation encoder 1032 , and a timbre encoder 1033 .

[0064] The pitch encoder 1031 is an encoder for analyzing the pitch of the voice. The pitch encoder 1031 calculates the fundamental frequency F0 and periodic amplitude A of the input voice based on the voice input to the voice analysis module 103. p [n] and aperiodic amplitude A ap Outputs [n]. Pitch refers to the highness or lowness of a sound, and the highness or lowness of a sound is determined by the frequency of the sound. High notes have high frequencies, and low notes have low frequencies.

[0065] The pitch encoder 1031 consists of a Constant-Q Transform (CQT) and a pitch analysis artificial neural network. The CQT generates a pitch (i.e., a musical scale) that falls at a constant rate as a spectrum in a specific frequency band. The CQT divides the frequency range from 50 Hz to 1,000 Hz into 64 frequency bins and outputs a probability distribution for each frequency bin. The CQT inputs the output probability distribution spectrum for each frequency bin into the pitch analysis artificial neural network.

[0066] The pitch analysis artificial neural network analyzes the probability distribution spectrum of each frequency bin input from the CQT and derives the fundamental frequency F0 and periodic amplitude A of the voice input to the pitch encoder 1031. p [n] and aperiodic amplitude A ap The pitch analysis artificial neural network analyzes the probability distribution spectrum of each frequency bin based on self-supervised learning, and outputs the fundamental frequency F0, periodic amplitude A p [n] and aperiodic amplitude A ap This is an artificial neural network model that outputs [n].

[0067] In this regard, Figure 3 is a detailed structural diagram of the pitch encoder. More specifically, Figure 3 shows the neural architecture of the pitch encoder 1031. Referring to Figure 3, the pitch analysis artificial neural network of the pitch encoder 1031 is composed of nine layers. The pitch encoder 1031 is composed of Conv1d, ResBlock, Reshape, GRU, Linear, ReLU, F0Head, Pamp Head, Ap amp Head, Softmax, and Exp.Sigmoid layers.

[0068] A description of each layer will be omitted to avoid obscuring the features of the present invention.

[0069] The pronunciation encoder 1032 analyzes the pronunciation features of the text included in the speech. The pronunciation encoder 1032 outputs the pronunciation features of the input speech based on the speech input to the speech analysis module 103. The pronunciation encoder 1032 extracts a feature vector of phonetic symbols representing the text included in the input speech.

[0070] The pronunciation encoder 1032 is composed of wav2vec (wave form to vector) and a linguistic analysis artificial neural network. wav2vec recognizes input speech and converts it into text in the form of a symbolic sequence. wav2vec inputs the converted text into the pronunciation analysis artificial neural network. wav2vec is an artificial neural network model that recognizes text from speech based on self-supervised learning.

[0071] The pronunciation analysis artificial neural network analyzes input text from wav2vec and outputs the pronunciation features of the text contained in the audio. The pronunciation analysis artificial neural network is an artificial neural network model that analyzes input text based on self-supervised learning and outputs the pronunciation features of the text.

[0072] In this regard, Figure 4 is a detailed structural diagram of the pronunciation encoder. More specifically, Figure 4 shows the neural architecture of the pronunciation encoder 1032. Referring to Figure 4, the pronunciation encoder 1032 is composed of six layers: two PreConvs, two ConvGLUs, a Conv1d, and an L2Normalization layer.

[0073] A description of each layer will be omitted to avoid obscuring the features of the present invention.

[0074] The timbre encoder 1033 analyzes timbre features, which are the inherent timbre of the voice subject contained in the voice, and outputs timbre features of the input voice based on the voice input to the voice analysis module 103.

[0075] The timbre encoder 1033 consists of a Mel-spectrogram conversion unit and a timbre analysis artificial neural network. The Mel-spectrogram conversion unit converts the input audio into a Mel-spectrogram. The Mel-spectrogram converts the audio into a Mel scale and inputs it into the timbre analysis artificial neural network.

[0076] The timbre analysis artificial neural network analyzes the input speech from the mel-spectrogram conversion unit and outputs the timbre features of the subject who spoke the speech.The timbre analysis artificial neural network is an artificial neural network model that analyzes the input mel-scale speech based on self-supervised learning and outputs the timbre features of the speech.

[0077] In this regard, FIG. 5 is a detailed structural diagram of the timbre encoder. More specifically, FIG. 5 shows the neural structure of the timbre encoder 1033. Referring to FIG. 5, the timbre encoder 1033 is composed of five layers. The timbre encoder 1033 is composed of ECAPA-TDNN blocks, MFA (Multilayer Feature Aggregation), TTB (Timber Token Block), ASP (Attentive Statistical Pooling), Linear, and L2Normalization layers. The timbre features output by the timbre encoder 1033 include global timbre embedding and timbre tokens. The global timbre embedding represents the timbre information of the entire speech in vector form. A timbre token is a basic unit representing a speech waveform and represents the timbre features corresponding to each phoneme in speech, and is expressed in vector form.

[0078] A description of each layer will be omitted to avoid obscuring the features of the present invention.

[0079] The speech synthesis module 104 includes a sinusoidal noise generator 1041, a frame-level synthesis artificial neural network 1042, a time-varying timbre artificial neural network 1043, and a sample-level synthesis artificial neural network 1044.

[0080] The sinusoidal noise generator 1041 generates a sinusoidal wave and noise. The sinusoidal noise generator 1041 generates a sinusoidal wave based on the voice features output by the voice analysis module 103. More specifically, the sinusoidal noise generator 1041 generates a sinusoidal wave based on a fundamental frequency F0, a periodic amplitude A p [n] and aperiodic amplitude A ap The sine wave and noise generated by the sine wave noise generator 1041 are expressed by Equation 1 and Equation 2, respectively.

[0081]

number

[0082]

number

[0083] where N s is the sampling rate, and A p [t] is the frame level A p [n] is the upsampled value to the sample level, and A ap [t] is the frame level A ap[n] is upsampled to the sample level. n[t] is the sampled noise value, more specifically, a noise chosen by uniformly sampling between -1 and 1.

[0084] The sinusoidal noise generator 1041 adds the generated sinusoidal wave and noise to generate an input excitation signal z[t]=x[t]+y[t] for the speech synthesis module 104. The sinusoidal noise generator 1041 inputs the generated input excitation signal z[t] to a sample-level synthesis artificial neural network 1044.

[0085] The frame-level synthesis artificial neural network 1042 generates frame-level conditions for the sample-level synthesizer based on the pronunciation features input from the speech analysis module 103 and the time-varying timbre embedding input from the time-varying timbre artificial neural network 1043. The frame-level conditions indicate detailed features such as timbre, pronunciation, and emotion in each frame of speech.

[0086] In this regard, Figure 6 is a detailed structural diagram of a frame-level synthesis artificial neural network. More specifically, Figure 6 shows the neural structure of the frame-level synthesis artificial neural network. Referring to Figure 6, the frame-level synthesis artificial neural network 1042 is composed of a PreConv layer, multiple ConvGLU layers, and a Conv1d layer. The frame-level synthesis artificial neural network 1042 is a neural network based on self-supervised learning, and outputs frame-level conditions through the input value processing between the layers described above.

[0087] The frame-level synthesis artificial neural network 1042 inputs the generated frame-level conditions into the sample-level synthesis artificial neural network 1044 .

[0088] The time-varying timbre artificial neural network 1043 generates a time-varying timbre embedding based on the timbre features input from the speech analysis module 103. The time-varying timbre embedding represents the timbre as a vector for each time step of the speech. In this regard, FIG. 7 is a detailed structural diagram of the time-varying timbre artificial neural network. More specifically, FIG. 7 shows the neural structure of the time-varying timbre artificial neural network. Referring to FIG. 7, the time-varying timbre artificial neural network 1043 is composed of multi-head attention, linear, L2 normalization, slerp, and tile layers. The time-varying timbre artificial neural network 1043 is a neural network based on self-supervised learning, and outputs a time-varying timbre embedding by processing the input values ​​between the layers as described above.

[0089] The time-varying timbre artificial neural network 1043 inputs the generated time-varying timbre embeddings into the frame-level synthesis artificial neural network 1042 .

[0090] The sample-level synthesis artificial neural network 1044 synthesizes speech based on sine waves, noise, and frame-level conditions. The sample-level synthesis artificial neural network 1044 according to one embodiment of the present invention is based on the parallel wave generative adversarial network (PWGAN) model. The sample-level synthesis artificial neural network 1044 is a neural network based on self-supervised learning, and synthesizes and outputs speech by processing input values ​​between the layers as described above.

[0091] Figure 8 is a flowchart of a speech synthesis method based on self-supervised learning according to one embodiment of the present invention. The speech synthesis device that executes the speech synthesis method based on self-supervised learning shown in Figure 8 assumes that, prior to executing the speech synthesis method according to one embodiment of the present invention, each of the SVS module 105, VOD module 106, and text-to-speech module 107, which are supervised learning models in the artificial neural network of the speech synthesis device 10, has been trained in advance using a learning dataset.

[0092] Referring to FIG. 8 , in step 801, the speech synthesizer 10 uses a training speech signal representing a training speech to train the speech analysis module 103 of the speech synthesizer 10 to output speech features for the training speech, and outputs the speech features for the training speech. More specifically, the processor 101 of the speech synthesizer 10 inputs the training speech signal stored in the storage 109 to the speech analysis module 103. The speech analysis module 103 trains the speech analysis module 103 to output speech features for the training speech represented by the input training speech signal. The speech synthesizer 10 trains the speech analysis module 103 through self-supervised learning. The training speech signal is a signal representing speech recorded by an arbitrary speech subject. The training speech is not limited to an arbitrary signal, and a recorded human voice can be used for training.

[0093] The processor 101 of the speech synthesizer 10 inputs a training speech signal stored in the storage 109 to the speech analysis module 103. The speech synthesizer 10 inputs training speech to the speech analysis module 103 to train the speech analysis module 103 to output speech features indicated by the training speech signal input to the speech analysis module 103. The speech analysis module 103 includes multiple encoders. The speech analysis module 103 performs self-supervised learning to output speech features from the input training speech.

[0094] More specifically, the speech analysis module 103 analyzes the fundamental frequency F0 and periodic amplitude A of the input training speech. p [n], non-periodic amplitude A ap The speech analysis module 103 outputs the training speech signal to a pitch encoder 1031, a pronunciation encoder 1032, and a timbre encoder 1033.

[0095] The pitch encoder 1031 outputs the pitch features of the input training speech. The pitch encoder 1031 includes a CQT that converts speech into a probability distribution spectrum of frequency bins, and a pitch analysis artificial neural network that outputs the pitch features of the speech. The pitch analysis artificial neural network is a neural network based on self-supervised learning.

[0096] The CQT of the pitch encoder 1031 converts the input training speech into a probability distribution spectrum for each frequency bin through the CQT, and the pitch analysis artificial neural network calculates the fundamental frequency F0 and periodic amplitude A of the training speech based on the probability distribution spectrum for each frequency bin. p [n] and aperiodic amplitude A ap Generate and output [n].

[0097] The pronunciation encoder 1032 outputs pronunciation features of the text included in the speech represented by the training speech signal from the input training speech signal. The pronunciation encoder 1032 analyzes the training speech signal using an artificial neural network and outputs a feature vector of phonetic symbols representing the text included in the training speech. According to an embodiment of the present invention, the pronunciation encoder 1032 converts the training speech included in the input training speech signal into a symbolic sequence, converts the symbolic sequence into pronunciation features of the training speech, and outputs the symbolic sequence.

[0098] According to another embodiment of the present invention, the pronunciation encoder 1032 recognizes and extracts text from an input training speech signal, and generates and outputs pronunciation features of the extracted text. The pronunciation encoder 1032 includes a wav2vec that recognizes text from a speech signal and a pronunciation analysis artificial neural network that outputs pronunciation features from the text. The pronunciation analysis artificial neural network is a neural network based on self-supervised learning. The wav2vec of the pronunciation encoder 1032 converts the input training speech into a symbolic sequence, and the pronunciation analysis artificial neural network converts the symbolic sequence into pronunciation features of the training speech and outputs them.

[0099] The timbre encoder 1033 outputs the timbre features of the input training speech. The timbre encoder 1033 converts the input training speech into a mel spectrogram, analyzes the converted mel spectrogram, and outputs the timbre features. The timbre encoder 1033 includes a mel spectrogram conversion unit that converts the input speech into a mel spectrogram, and a timbre analysis artificial neural network that outputs the timbre features from the mel spectrogram. The timbre analysis artificial neural network is a neural network based on self-supervised learning.

[0100] The timbre encoder 1033 converts the waveform of the audio signal into a mel spectrogram and inputs the mel spectrogram into a timbre analysis artificial neural network. The timbre analysis artificial neural network converts the input mel spectrogram into timbre features through each layer and outputs the mel spectrogram.

[0101] The voice analysis module 103 analyzes the fundamental frequency F0 and periodic amplitude A p [n], non-periodic amplitude A ap [n], speech features of the training speech, including pronunciation features and timbre features, are input to the speech synthesis module 104.

[0102] In step 802, the speech synthesis device 10 uses speech features of the training speech to train the speech synthesis module 104 to synthesize a speech signal from the speech features of the training speech, and synthesizes a synthetic speech signal from the speech features of the training speech. More specifically, the speech synthesis module 104 of the speech synthesis device 10 trains the speech synthesis module 104 to synthesize a synthetic speech signal based on the speech features of the training speech input from the speech analysis module 103. The speech synthesis device 10 trains the speech synthesis module 104 by self-supervised learning. Here, the synthetic speech signal is a signal indicating the synthetic speech synthesized by the speech synthesis module 104.

[0103] The voice synthesis module 104 receives the fundamental frequency F0 and periodic amplitude A p [n], non-periodic amplitude A ap [n], synthesizes speech based on pronunciation and timbre features.

[0104] More specifically, the voice analysis module 103 calculates the fundamental frequency F0, the periodic amplitude A p [n] and aperiodic amplitude A ap [n] is input to a sinusoidal noise generator 1041, the phonetic features are input to a frame-level synthesis artificial neural network 1042, and the timbre features are input to a time-varying timbre artificial neural network 1043.

[0105] The sinusoidal noise generator 1041 generates a fundamental frequency F0 and a periodic amplitude A p [n] and aperiodic amplitude A ap The sinusoidal noise generator 1041 performs learning to generate a sine wave and noise based on [n]. Here, the sinusoidal noise generator 1041 is an artificial neural network model based on self-supervised learning. The sinusoidal noise generator 1041 generates a sine wave and noise according to the above-mentioned Equation 1 and Equation 2. The sinusoidal noise generator 1041 sums the generated sine wave and noise to generate an input excitation signal z[t], which is input to the sample-level synthesis artificial neural network 1044.

[0106] The time-varying timbre artificial neural network 1043 performs training to generate time-varying timbre embeddings based on the timbre features input from the speech analysis module 103. Here, the time-varying timbre artificial neural network 1043 is an artificial neural network model based on self-supervised learning. The time-varying timbre artificial neural network 1043 generates time-varying timbre embeddings based on multiple layers of timbre features, including global timbre embeddings and timbre tokens. The time-varying timbre artificial neural network 1043 inputs the generated time-varying timbre embeddings to the frame-level synthesis artificial neural network 1042.

[0107] The frame-level synthesis artificial neural network 1042 performs learning to generate frame-level conditions based on the pronunciation features input from the speech analysis module 103 and the time-varying timbre embeddings input from the time-varying timbre artificial neural network 1043. Here, the frame-level synthesis artificial neural network 1042 is an artificial neural network model based on self-supervised learning. The frame-level synthesis artificial neural network 1042 generates frame-level conditions based on multiple layers of pronunciation features and time-varying timbre embeddings. The frame-level synthesis artificial neural network 1042 inputs the generated frame-level conditions to the sample-level synthesis artificial neural network 1044.

[0108] The sample-level synthesis artificial neural network 1044 performs training to synthesize a speech signal from an input excitation signal and frame-level conditions, where the sample-level synthesis artificial neural network 1044 is an artificial neural network model based on self-supervised learning. The sample-level synthesis artificial neural network 1044 synthesizes a synthetic speech signal based on the input excitation signal and frame-level conditions.

[0109] In operation 803, the speech synthesizer 10 calculates a reconstruction loss between the input training speech and the synthesized speech, and performs training on the speech analysis module 103 and the speech synthesis module 104 using the calculated reconstruction loss and the training speech. The processor 101 of the speech synthesizer 10 calculates the reconstruction loss between the input training speech and the output speech. Reconstruction losses include multi-scale spectrum (MSS) loss, mel-spectrogram loss, adversarial loss, and feature matching loss. The MSS loss uses a linear frequency scale spectrogram rather than a log-scale spectrogram.

[0110] In addition to the reconstruction loss, the speech synthesizer 10 according to another embodiment of the present invention can calculate the difference between the input training speech and the output synthetic speech using other loss functions such as Kullback-Leibler divergence loss (KLD), mean squared error (MSE), root mean squared error (RMSE), binary crossentropy, etc.

[0111] The processor 101 inputs the calculated reconstruction loss and training speech to the speech analysis module 103. The speech analysis module 103 and the speech synthesis module 104 perform training based on the input reconstruction loss and training speech.

[0112] Fig. 9 is a flowchart of a singing voice synthesis method based on self-supervised learning according to one embodiment of the present invention. The speech synthesis device that executes the singing voice synthesis method based on self-supervised learning shown in Fig. 9 assumes that, prior to executing the speech synthesis method according to one embodiment of the present invention, the SVS module 105, VOD module 106, and text-to-speech module 107, which are supervised learning models in the artificial neural network of the speech synthesis device 10, have each been trained in advance using a training dataset.

[0113] 9, in step 901, the speech synthesis device 10 uses a training speech signal indicating a training speech to perform training on the speech analysis module 103 of the speech synthesis device 10 so as to output speech features for the training speech. The speech synthesis device 10 trains the speech analysis module 103 through self-supervised learning. For a detailed description of the operation and training method of the speech analysis module 103 in step 901, please refer to the description in step 801.

[0114] In step 902, the speech synthesis device 10 uses the speech features of the training speech to train the speech synthesis module 104 so as to synthesize a speech signal from the speech features of the training speech, and synthesizes a synthetic speech signal from the speech features of the training speech. The speech synthesis device 10 trains the speech synthesis module 104 by self-supervised learning. For a detailed description of the operation and training method of the speech synthesis module 104 in step 902, please refer to the description in step 802.

[0115] In step 903, the speech synthesizer 10 calculates a reconstruction loss between the input training speech and the synthesized speech, and performs training on the speech analysis module 103 and the speech synthesis module 104 using the calculated reconstruction loss and the training speech. The processor 101 of the speech synthesizer 10 calculates the reconstruction loss between the input training speech and the output speech. The processor 101 inputs the calculated reconstruction loss and the training speech to the speech analysis module 103. The speech analysis module 103 and the speech synthesis module 104 perform training based on the input reconstruction loss and the training speech. For a detailed description of the reconstruction loss calculation and training method of the speech synthesizer 10 in step 903, please refer to the description of step 803.

[0116] In operation 904, the voice synthesizer 10 receives a singing voice synthesis request including a reference singer and a target song to be synthesized. The input module 102 of the voice synthesizer 10 receives the reference singer and target song that the user wishes to hear from the user. The input module 102 inputs the reference singer and target song to the SVS module 105. Here, the reference song includes the musical score and lyrics for the song that the user wishes to synthesize. Data indicating the musical score and lyrics for the reference song may be stored in the storage 109 of the voice synthesizer 10 or may be input by the user.

[0117] In step 905, the voice synthesizer 10 acquires a voice signal associated with the target singer based on the singing synthesis request. The voice synthesizer 10 acquires the voice signal associated with the target singer included in the singing synthesis request. For example, the processor 101 of the voice synthesizer 10 searches the storage 109 for a voice associated with the target singer and acquires, from the storage 109, a voice signal representing the voice associated with the target singer. According to another embodiment, the input module 102 of the voice synthesizer 10 can acquire a voice associated with the target singer from a user. The voice associated with the target singer is a recording of a song sung by the target singer. The voice synthesizer 10 inputs the acquired voice signal associated with the target singer to the voice analysis module 103 and the SVS module 105.

[0118] In step 906, the voice synthesizer 10 calculates the fundamental frequency F0, periodic amplitude A of the singing voice sung by the singing voice of the singing voice target singer based on the singing voice synthesis request and the voice associated with the singing voice target singer. p [n], non-periodic amplitude A ap The SVS module 105 of the speech synthesizer 10 generates and outputs singing voice features including the fundamental frequency F0, periodic amplitude A, and pronunciation features. More specifically, the SVS module 105 of the speech synthesizer 10 generates and outputs singing voice features for the singing voice based on the input song to be synthesized and the voice associated with the singer to be synthesized. The singing voice features include the fundamental frequency F0, periodic amplitude A, and pronunciation features. p [n], non-periodic amplitude A ap The SVS module 105 calculates the fundamental frequency F0, periodic amplitude A of the singing voice of the target singer based on the input musical score and text of the target song and the voice of the target singer. p [n], non-periodic amplitude A ap Generate and output [n] and pronunciation features.

[0119] Here, the SVS module 105 is an artificial neural network pre-trained by a training dataset. When a song and a singer are input using an SVS training dataset including training songs, training singers, and training vocal features, the SVS module 105 is pre-trained to output vocal features for the input song and singer. The SVS module 105 is an artificial neural network trained using a supervised learning method.

[0120] In step 907, the voice synthesizer 10 generates timbre features of the target singer based on the voice signal associated with the target singer. The voice analysis module 103 of the voice synthesizer 10 analyzes the voice associated with the target singer to generate timbre features of the target singer. The voice analysis module 103 is an artificial neural network trained by self-supervised learning in step 901. The voice analysis module 103 analyzes the input voice associated with the target singer and generates the fundamental frequency F0, periodic amplitude A of the voice associated with the target singer. p [n], non-periodic amplitude A ap [n], pronunciation features and timbre features are output. Here, the voice analysis module 103 inputs only the timbre features of the singer to be synthesized to the voice synthesis module 104.

[0121] In step 908, the voice synthesizer 10 synthesizes a singing voice signal representing the voice of the synthesis target singer based on the singing voice characteristics and the timbre characteristics of the synthesis target singer. The voice synthesis module 104 of the voice synthesizer 10 calculates the fundamental frequency F0, periodic amplitude A of the singing voice of the synthesis target singer input from the SVS module 105. p [n], non-periodic amplitude A ap A singing voice is synthesized based on [n] and pronunciation features, as well as voice timbre features associated with the singer to be synthesized, input from the voice analysis module 103. The voice synthesis module 104 is an artificial neural network trained by self-supervised learning in step 902. The voice synthesis module 104 inputs the synthesized singing voice signal to the output module 108.

[0122] In step 909, the output module 108 of the voice synthesizer 10 outputs the singing voice signal. The output module 108 converts the singing voice signal into sound waves and outputs the sound waves.

[0123] The singing voice synthesis method based on self-supervised learning shown in FIG. 9 can use a voice analysis module 103 and a voice synthesis module 104 that have been trained in advance using a self-learning method. In such a case, steps 901, 902, and 903 can be omitted.

[0124] The singing voice synthesis method according to the above-described embodiment of the present invention can synthesize a song desired by the user with a voice sung by a singer desired by the user in the above-described manner.

[0125] Fig. 10 is a flowchart of a method for synthesizing altered speech based on self-supervised learning according to another embodiment of the present invention. The speech synthesis device that executes the method for synthesizing altered speech based on self-supervised learning shown in Fig. 10 assumes that, prior to executing the speech synthesis method according to this embodiment of the present invention, each of the SVS module 105, VOD module 106, and text-to-speech module 107, which are supervised learning models in the artificial neural network of the speech synthesis device 10, has been trained in advance using a training dataset.

[0126] 10, in step 1001, the speech synthesis device 10 uses a training speech signal indicating a training speech to train the speech analysis module 103 of the speech synthesis device 10 to output speech features for the training speech. The speech synthesis device 10 trains the speech analysis module 103 through self-supervised learning. For a detailed description of the operation and training method of the speech analysis module 103 in step 1001, please refer to the description in step 801.

[0127] In step 1002, the speech synthesis device 10 uses the speech features of the training speech to train the speech synthesis module 104 so as to synthesize a speech signal from the speech features of the training speech, and synthesizes a synthetic speech signal from the speech features of the training speech. The speech synthesis device 10 trains the speech synthesis module 104 by self-supervised learning. For a detailed description of the operation and training method of the speech synthesis module 104 in step 1002, please refer to the description in step 802.

[0128] In step 1003, the speech synthesizer 10 calculates a reconstruction loss between the input training speech and the synthesized speech, and performs training on the speech analysis module 103 and the speech synthesis module 104 using the calculated reconstruction loss and the training speech. The processor 101 of the speech synthesizer 10 calculates the reconstruction loss between the input training speech and the output speech. The processor 101 inputs the calculated reconstruction loss and the training speech to the speech analysis module 103. The speech analysis module 103 and the speech synthesis module 104 perform training based on the input reconstruction loss and the training speech. For a detailed description of the reconstruction loss calculation and training method of the speech synthesizer 10 in step 1003, please refer to the description of step 803.

[0129] In step 1004, the speech synthesizer 10 acquires pre-conversion speech to be subjected to speech conversion. The input module 102 of the speech synthesizer 10 acquires the pre-conversion speech to be subjected to speech conversion from the user. The speech synthesizer 10 can receive the pre-conversion speech directly from the user via the input module 102. Alternatively, the speech synthesizer 10 can receive a command for the pre-conversion speech to be subjected to speech conversion from the user via the input module 102, and retrieve and acquire the pre-conversion speech from the storage 109. The input module 102 inputs the acquired pre-conversion speech to the speech analysis module 103.

[0130] In step 1005, the speech synthesizer 10 generates and outputs pre-conversion speech features indicating the characteristics of the pre-conversion speech based on the acquired pre-conversion speech. More specifically, the speech analysis module 103 of the speech synthesizer 10 generates and outputs speech features of the pre-conversion speech from the pre-conversion speech. The speech analysis module 103 trained in step 1001 analyzes the input pre-conversion speech and determines the fundamental frequency F0, periodic amplitude A of the pre-conversion speech. p [n], non-periodic amplitude A ap The speech analysis module 103 outputs the periodic amplitude A [n], pronunciation feature, and timbre feature from among the speech features of the output pre-conversion speech. p [n], non-periodic amplitude A ap The [n] and pronunciation features are input to the speech synthesis module 104 .

[0131] In operation 1006, the speech synthesizer 10 acquires speech attributes for the converted speech. The input module 102 of the speech synthesizer 10 receives, from the user, speech attributes for the converted speech that the user wishes to convert. The input module 102 inputs the input speech attributes to the VOD module 106. Here, the speech attributes for the converted speech are attributes of the speech to be converted among the attributes of the basic speech, and include the age, gender, and pitch of the speaker. The user inputs the speech attributes to be converted. For example, if the user wishes to convert a current speech that is currently male to a female, the user inputs a command to convert the speech attributes from male to female. The input module 102 inputs the speech attributes for the converted speech to the VOD module 106.

[0132] In operation 1007, the VOD module 106 of the speech synthesizer 10 outputs converted speech features based on the speech attributes of the input converted speech. The converted speech features include a fundamental frequency F0 and a timbre feature for the converted speech. The VOD module 106 generates and outputs the fundamental frequency F0 and the timbre feature for the converted speech converted into the speech that the user wants to convert based on the speech attributes of the input converted speech.

[0133] The VOD module 106 is an artificial neural network pre-trained with a training dataset. The VOD module 106 is pre-trained to input speech attributes for converted speech using a VOD training dataset including training speech, training speech attributes, training fundamental frequency F0, and training timbre features, and to output the fundamental frequency F0 and timbre features for converted speech based on the input speech attributes. The VOD module 106 is an artificial neural network trained using a supervised learning method.

[0134] The VOD module 106 inputs the fundamental frequency F0 and timbre characteristics of the output converted voice to the voice synthesis module 104.

[0135] In operation 1008, the speech synthesizer 10 synthesizes a converted speech based on the converted speech features and the pre-conversion speech features. More specifically, the speech synthesis module 104 of the speech synthesizer 10 synthesizes a periodic amplitude A p [n], non-periodic amplitude A ap The converted speech signal is synthesized based on the [n] and pronunciation features, as well as the fundamental frequency F0 and timbre features of the converted speech input from the VOD module 106. The speech synthesis module 104 is an artificial neural network trained by self-supervised learning in step 1002. The speech synthesis module 104 inputs a converted speech signal representing the synthesized converted speech to the output module 108.

[0136] In step 1009, the output module 108 of the speech synthesizer 10 outputs the converted speech signal. The output module 108 converts the converted speech signal into sound waves and outputs the sound waves.

[0137] The modified voice synthesis method according to the above-described embodiment of the present invention can convert a voice to have voice attributes desired by a user in the above-described manner.

[0138] The method for synthesizing modified speech based on self-supervised learning shown in Fig. 10 can use the speech analysis module 103 and speech synthesis module 104 that have been trained in advance using a self-training method. In this case, steps 1001, 1002, and 1003 can be omitted.

[0139] Figure 11 is a flowchart of a TTS synthesis method based on self-supervised learning according to another embodiment of the present invention. The speech synthesizer that executes the TTS synthesis method based on self-supervised learning shown in Figure 11 assumes that, prior to executing the speech synthesis method according to this embodiment of the present invention, each of the SVS module 105, VOD module 106, and text-to-speech module 107, which are supervised learning models in the artificial neural network of the speech synthesizer 10, has been trained in advance using a training dataset.

[0140] 11, in step 1101, the speech synthesizer 10 uses a training speech signal indicating a training speech to train the speech analysis module 103 of the speech synthesizer 10 so as to output speech features for the training speech. The speech synthesizer 10 trains the speech analysis module 103 through self-supervised learning. For a detailed description of the operation and training method of the speech analysis module 103 in step 1101, please refer to the description in step 801.

[0141] In step 1102, the speech synthesis device 10 uses the speech features of the training speech to train the speech synthesis module 104 so as to synthesize a speech signal from the speech features of the training speech, and synthesizes a synthetic speech signal from the speech features of the training speech. The speech synthesis device 10 trains the speech synthesis module 104 by self-supervised learning. For a detailed description of the operation and training method of the speech synthesis module 104 in step 1102, please refer to the description in step 802.

[0142] In step 1103, the speech synthesizer 10 calculates a reconstruction loss between the input training speech and the synthesized speech, and performs training on the speech analysis module 103 and the speech synthesis module 104 using the calculated reconstruction loss and the training speech. The processor 101 of the speech synthesizer 10 calculates the reconstruction loss between the input training speech and the output speech. The processor 101 inputs the calculated reconstruction loss and the training speech to the speech analysis module 103. The speech analysis module 103 and the speech synthesis module 104 perform training based on the input reconstruction loss and the training speech. For a detailed description of the reconstruction loss calculation and training method of the speech synthesizer 10 in step 1103, please refer to the description of step 803.

[0143] In step 1104, the speech synthesizer 10 acquires a synthesis target text and a synthesis target voice subject for TTS synthesis. The input module 102 of the speech synthesizer 10 acquires a synthesis target text and a synthesis target voice subject for TTS synthesis from the user. The synthesis target text is the text for which TTS synthesis is to be performed, and the synthesis target voice subject refers to the voice subject reading the synthesis target text. The input module 102 inputs the acquired synthesis target text to the TTS module 107 and inputs the synthesis target voice subject to the processor 101.

[0144] In operation 1105, the speech synthesizer 10 acquires a voice related to the subject of speech to be synthesized based on the subject of speech to be synthesized. More specifically, the processor 101 of the speech synthesizer 10 searches for and acquires a voice related to the subject of speech to be synthesized from the storage 109. The processor 101 inputs the voice related to the subject of speech to the speech analysis module 103 and the TTS module 107.

[0145] In step 1106, the speech synthesizer 10 generates and outputs speech features of the subject of speech to be synthesized based on speech related to the subject of speech to be synthesized. The speech analysis module 103 of the speech synthesizer 10 generates and outputs speech features of the subject of speech to be synthesized from speech related to the subject of speech to be synthesized. The speech analysis module 103 trained in step 1101 analyzes the input speech related to the subject of speech to be synthesized and calculates the fundamental frequency F0, periodic amplitude A of the speech related to the subject of speech to be synthesized. p [n], non-periodic amplitude A ap The speech analysis module 103 outputs the pronunciation feature and the timbre feature from among the speech features related to the subject of the speech to be synthesized that have been output, to the speech synthesis module 104.

[0146] In operation 1107, the speech synthesizer 10 outputs speech features for text-to-speech in which the text to be synthesized is read in the voice of the subject of the speech to be synthesized, based on the speech related to the text to be synthesized and the subject of the speech to be synthesized. Here, the text-to-speech means a speech in which the text to be synthesized is read in the voice of the subject of the speech to be synthesized. The speech features for the text-to-speech include the fundamental frequency F0, periodic amplitude A, and the like of the text-to-speech. p [n] and aperiodic amplitude A ap The TTS module 107 of the speech synthesizer 10 determines the fundamental frequency F0, periodic amplitude A, and other parameters for the text speech that the user wishes to synthesize based on the input text to be synthesized and the speech related to the subject of the speech to be synthesized. p [n] and aperiodic amplitude A ap Generate and output [n].

[0147] The TTS module 107 is an artificial neural network pre-trained by a training dataset. When the TTS module 107 receives a synthetic text and a voice related to the subject of the subject of the speech to be synthesized, the TTS module 107 calculates the fundamental frequency F0, periodic amplitude A of the text speech, which is the speech of the subject of the speech to be synthesized reading the input synthetic text, using the TTS training dataset including the training synthetic text, the training voice, and the training voice features. p [n] and aperiodic amplitude A apIt is an artificial neural network pre-trained to output [n].

[0148] The TTS module 107 determines the fundamental frequency F0 and periodic amplitude A for the output text speech. p [n] and aperiodic amplitude A ap [n] is input to the speech synthesis module 104.

[0149] In operation 1108, the speech synthesizer 10 synthesizes text speech based on the speech features of the text speech and the speech features of the subject speech to be synthesized. More specifically, the speech synthesis module 104 of the speech synthesizer 10 synthesizes a fundamental frequency F0, a periodic amplitude A1, and a periodic amplitude B2 of the text speech input from the TTS module 107. p [n] and aperiodic amplitude A ap Based on [n] and the timbre features input from the speech analysis module 103, the speech synthesis module 104 synthesizes a text-to-speech signal representing the text speech. The speech synthesis module 104 is an artificial neural network trained by self-supervised learning in step 1102. The speech synthesis module 104 inputs the synthesized text-to-speech signal to the output module 108.

[0150] The output module 108 of the speech synthesizer 10 outputs the text-to-speech signal in step 1109. The output module 108 converts the text-to-speech signal into sound waves and outputs the sound waves.

[0151] The text-to-speech synthesis method according to the above-described embodiment of the present invention can synthesize speech by reading the text for which the user wishes to perform speech synthesis in the voice of a desired voice subject (speaker) in the above-described manner.

[0152] The TTS synthesis method based on self-supervised learning shown in Fig. 11 can use the voice analysis module 103 and the voice synthesis module 104 that have been trained in advance by a self-training method. In this case, steps 1101, 1102, and 1103 can be omitted.

[0153] According to the above-described embodiments of the present invention, the speech synthesis device and speech synthesis method use artificial intelligence based on self-supervised learning to analyze a speech signal, extract speech features from the speech signal, and resynthesize speech based on the extracted speech features. The artificial intelligence repeatedly analyzes and synthesizes the speech signal, outputs speech features of the input speech, and resynthesizes speech based on the output speech features. According to the embodiments of the present invention, the artificial neural network learns to extract speech features from the input speech and to output synthetic speech based on the extracted speech features. By using an artificial intelligence model based on self-supervised learning, compared to artificial intelligence models based on supervised learning in the field of speech synthesis, a massive amount of training audio data is not required to train the artificial neural network model, and the artificial neural network for speech synthesis can be trained quickly and easily.

[0154] Furthermore, according to an embodiment of the present invention, the speech analysis module and speech synthesis module, which are artificial neural networks of the speech synthesizer, calculate a reconstruction loss between an input speech and a synthesized speech during self-supervised learning, and train the speech analysis module and speech synthesis module based on the calculated reconstruction loss, thereby minimizing the difference between the speech synthesized by the speech synthesizer and the actual speech. By utilizing the loss function, the speech synthesizer can synthesize a more natural speech that closely resembles the actual speech.

[0155] A voice synthesis method according to an embodiment of the present invention utilizes an artificial neural network trained in a self-supervised learning manner to convert any singing voice into the voice of a singer desired by a user. Therefore, it is possible to synthesize a natural singing voice that is identical to or very similar to the actual singing voice of the singer to be synthesized, even if the singing voice is not actually sung by the singer to be synthesized.

[0156] In addition, the voice synthesis method according to the embodiment of the present invention can analyze a voice using only a predetermined training voice and train an artificial neural network that synthesizes a voice based on the analysis results. For example, a small amount of voice of less than 10 minutes can be used to train the artificial neural network for voice synthesis. Therefore, even if a large amount or length of voice data is not available, it is possible to restore the voice of a deceased singer or great figure using only short data recorded during their lifetime.

[0157] Furthermore, the speech synthesis method according to an embodiment of the present invention utilizes an artificial neural network trained in a self-supervised learning manner to convert a recorded speech of any voice into a speech having other speech characteristics. For example, a male voice can be modulated into a female voice, or a young voice into an elderly voice, and the speech can be freely modulated into a speech having speech characteristics desired by the user.

[0158] Additionally, a speech synthesis method according to an embodiment of the present invention can synthesize speech recorded from any text in the voice of any subject by utilizing an artificial neural network trained in a self-supervised learning manner. By synthesizing text to speech using an artificial neural network trained in a self-supervised learning manner, it is possible to synthesize a very natural speech, as if the speech input by the user was recorded reading directly from the subject.

[0159] Meanwhile, the above-described embodiments of the present invention can be created as a computer-executable program and can be embodied in a general-purpose digital computer that runs the program using a computer-readable recording medium. Furthermore, the data structures used in the above-described embodiments of the present invention can be recorded in a computer-readable recording medium by various means. The computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM (Read Only Memory), floppy disks, hard disks, etc.) and optical storage media (e.g., CD-ROMs, DVDs, etc.). A program for executing the singing voice synthesis method according to the embodiments of the present invention is recorded in the computer-readable recording medium.

[0160] The present invention has been described above based on preferred embodiments. Those skilled in the art will understand that the present invention can be embodied in various modified forms without departing from the essential characteristics of the present invention. Therefore, the embodiments of the present invention should be considered from an illustrative rather than a restrictive perspective. The scope of the present invention is defined by the claims, not the foregoing description, and all variations within the scope of the claims should be construed as being within the scope of the present invention. [Explanation of symbols]

[0161] 10. Speech synthesizer 101 processors 102 Input Module 103 Voice Analysis Module 1031 Pitch Encoder 1032 Phonetic Encoder 1033 Timbre Encoder 104 Speech Synthesis Module 1041 Sine Wave Noise Generator 1042 Frame-Level Synthetic Artificial Neural Networks 1043 Time-Varying Tone Artificial Neural Network 1044 Sample-Level Synthetic Artificial Neural Networks 105 Singing Voice Synthesis Module 106 Audio Design Module 107 Text-to-Speech Module 108 Output Module 109 Storage

Claims

1. 1. A speech synthesis method based on self-supervised learning, comprising: using a training speech signal representing a training speech, training a speech analysis module to output speech features for the training speech signal, and outputting speech features for the training speech; and executing training on a speech synthesis module using the output speech features to synthesize a speech signal from the speech features of the training speech, and synthesizing a synthetic speech signal representing a synthetic speech from the output speech features, The speech characteristics of the training speech are the fundamental frequency F ０ , including amplitude, pronunciation features and timbre features; The step of outputting speech features for the training speech includes: The training speech signal is converted into a probability distribution spectrum of a plurality of frequency bins, and a fundamental frequency F of the training speech is obtained from the converted probability distribution spectrum. ０ and outputting the amplitude; outputting pronunciation features of the text included in the training speech from the training speech signal; converting the training speech signal into a spectrogram and outputting the timbre features of the training speech from the converted spectrogram; The step of synthesizing a synthetic speech signal representing a synthetic speech from the output speech features includes: The fundamental frequency F ０ and generating an input excitation signal based on the amplitude; generating a time-varying embedding based on the timbre features of the training speech; generating frame-level conditions for the synthesized speech based on the pronunciation features of the training speech and the generated time-varying timbre embedding; synthesizing a synthetic speech signal indicative of the synthetic speech based on the input excitation signal and the frame-level conditions.

2. 2. The method of claim 1, further comprising: calculating a reconstruction loss of the training speech signal and the synthesized speech signal based on the training speech signal and the synthesized speech signal; and performing training on the speech analysis module and the speech synthesis module based on the calculated reconstruction loss.

3. A speech synthesizer based on self-supervised learning, comprising: a speech analysis module (103) that uses a training speech signal indicating a training speech to perform training on a speech analysis module so as to output speech features for the training speech signal, and outputs speech features for the training speech; a speech synthesis module (104) that uses the output speech features to train a speech synthesis module so as to synthesize a speech signal from the speech features of the training speech, and synthesizes a synthetic speech signal representing a synthetic speech from the output speech features, The speech characteristics of the training speech are the fundamental frequency F ０ , including amplitude, pronunciation features and timbre features; The voice analysis module (103) The training speech signal is converted into a probability distribution spectrum of a plurality of frequency bins, and a fundamental frequency F of the training speech is obtained from the converted probability distribution spectrum. ０ and a pitch encoder (1031) that outputs the amplitude; a pronunciation encoder (1032) for outputting pronunciation features of the text included in the training speech from the training speech signal; a timbre encoder (1033) that converts the training speech signal into a spectrogram and outputs timbre features of the training speech from the converted spectrogram; The speech synthesis module (104) The fundamental frequency F ０ a sinusoidal noise generator (1041) for generating an input excitation signal based on the amplitude and a time-varying timbre artificial neural network (1043) for generating a time-varying timbre embedding based on the timbre features of the training speech; a frame-level synthesis artificial neural network (1042) for generating frame-level conditions for the synthesized speech based on the pronunciation features of the training speech and the generated time-varying timbre embeddings; a sample-level synthesis artificial neural network (1044) for synthesizing a synthetic speech signal indicative of said synthetic speech based on said input excitation signal and said frame-level conditions.

4. A singing voice synthesis method based on self-supervised learning executed by a speech synthesis device including: a speech analysis module that uses a training speech signal representing a training speech, performs training on a speech analysis module to output speech features for the training speech signal, and outputs the speech features for the training speech; and a speech synthesis module that uses the output speech features, performs training on a speech synthesis module to synthesize a speech signal from the speech features of the training speech, and synthesizes a synthetic speech signal representing a synthetic speech from the output speech features, receiving a singing voice synthesis request including a synthesis target song and a synthesis target singer; acquiring a voice signal associated with the singer to be synthesized based on the singing voice synthesis request; A SVS (Singing Voice Synthesis) module calculates the fundamental frequency F for the target song and the target singer based on the singing voice synthesis request and the voice signal related to the target singer. ０ generating vocal features including amplitude and pronunciation features; generating, in the voice analysis module, timbre characteristics of the target singer based on the acquired voice signal associated with the target singer; and synthesizing, in the voice synthesis module, a singing voice signal representing a voice of the synthesis target singer singing the synthesis target song based on the singing voice features and the timbre features, The speech characteristics of the training speech are the fundamental frequency F ０ , including amplitude, pronunciation features and timbre features; The voice analysis module (103) The training speech signal is converted into a probability distribution spectrum of a plurality of frequency bins, and a fundamental frequency F of the training speech is obtained from the converted probability distribution spectrum. ０ and a pitch encoder (1031) that outputs the amplitude; a pronunciation encoder (1032) for outputting pronunciation features of the text included in the training speech from the training speech signal; a timbre encoder (1033) that converts the training speech signal into a spectrogram and outputs timbre features of the training speech from the converted spectrogram; The speech synthesis module (104) The fundamental frequency F ０ a sinusoidal noise generator (1041) for generating an input excitation signal based on the amplitude and a time-varying timbre artificial neural network (1043) for generating a time-varying timbre embedding based on the timbre features of the training speech; a frame-level synthesis artificial neural network (1042) for generating frame-level conditions for the synthesized speech based on the pronunciation features of the training speech and the generated time-varying timbre embeddings; a sample-level synthesis artificial neural network (1044) for synthesizing a synthetic speech signal indicative of said synthetic speech based on said input excitation signal and said frame-level conditions.

5. 5. The singing voice synthesis method based on self-supervised learning according to claim 4, wherein the SVS module is an artificial neural network pre-trained to output singing voice features for an input target song and target singer using a training dataset including a training song, a training singer's voice, and training singing voice features.

6. A method for synthesizing altered speech based on self-supervised learning, which is executed by a speech synthesis device including: a speech analysis module that uses a training speech signal representing a training speech to train a speech analysis module to output speech features for the training speech signal, and outputs the speech features for the training speech; and a speech synthesis module that uses the output speech features to train a speech synthesis module to synthesize a speech signal from the speech features of the training speech, and synthesizes a synthetic speech signal representing a synthetic speech from the output speech features, obtaining a pre-conversion voice to be subjected to voice conversion; The speech analysis module calculates a fundamental frequency F for the pre-conversion speech based on the acquired pre-conversion speech. ０ outputting pre-conversion speech features including amplitude and pronunciation features; acquiring speech attributes for the converted speech; A VOD (Voice Design) module determines a fundamental frequency F for the converted voice based on the voice attributes of the converted voice. ０ and outputting transformed speech features including timbre features. and synthesizing, in the speech synthesis module, a converted speech based on the pre-conversion speech features and the converted speech features; The speech characteristics of the training speech are the fundamental frequency F ０ , including amplitude, pronunciation features and timbre features; The voice analysis module (103) The training speech signal is converted into a probability distribution spectrum of a plurality of frequency bins, and a fundamental frequency F of the training speech is obtained from the converted probability distribution spectrum. ０ and a pitch encoder (1031) that outputs the amplitude; a pronunciation encoder (1032) for outputting pronunciation features of the text included in the training speech from the training speech signal; a timbre encoder (1033) that converts the training speech signal into a spectrogram and outputs timbre features of the training speech from the converted spectrogram; The speech synthesis module (104) The fundamental frequency F ０ a sinusoidal noise generator (1041) for generating an input excitation signal based on the amplitude and a time-varying timbre artificial neural network (1043) for generating a time-varying timbre embedding based on the timbre features of the training speech; a frame-level synthesis artificial neural network (1042) for generating frame-level conditions for the synthesized speech based on the pronunciation features of the training speech and the generated time-varying timbre embeddings; a sample-level synthesis artificial neural network (1044) for synthesizing a synthetic speech signal indicative of the synthetic speech based on the input excitation signal and the frame-level conditions.

7. The VOD module includes a training voice attribute and a training fundamental frequency F ０ and a training dataset including training timbre features, and a fundamental frequency F of the converted voice is calculated based on the input voice attributes. ０ 7. The method for synthesizing modified speech based on self-supervised learning according to claim 6, characterized in that the input signal is an artificial neural network pre-trained to output the timbre features.

8. A TTS (Text To Speech) synthesis method based on self-supervised learning that is executed by a speech synthesis device including: a speech analysis module that uses a training speech signal representing a training speech to train a speech analysis module to output speech features for the training speech signal, and outputs the speech features for the training speech; and a speech synthesis module that uses the output speech features to train a speech synthesis module to synthesize a speech signal from the speech features of the training speech, and synthesizes a synthetic speech signal representing a synthetic speech from the output speech features, obtaining a text to be synthesized and a subject of speech to be synthesized for TTS synthesis; obtaining a voice related to the subject of the voice to be synthesized based on the subject of the voice to be synthesized; outputting, in the voice analysis module, voice features of the subject voice to be synthesized, including timbre features of the subject voice to be synthesized, based on a voice related to the subject voice to be synthesized; The TTS module calculates a fundamental frequency F for a text-to-speech read by the subject voice of the synthesis target voice based on the text-to-speech and the voice related to the subject voice of the synthesis target voice. ０ and outputting speech features of the text-to-speech, including amplitude; The fundamental frequency F ０ and synthesizing a text-to-speech based on the amplitude and timbre characteristics of the subject speech to be synthesized, The speech characteristics of the training speech are the fundamental frequency F ０ , including amplitude, pronunciation features and timbre features; The voice analysis module (103) The training speech signal is converted into a probability distribution spectrum of a plurality of frequency bins, and a fundamental frequency F of the training speech is obtained from the converted probability distribution spectrum. ０ and a pitch encoder (1031) that outputs the amplitude; a pronunciation encoder (1032) for outputting pronunciation features of the text included in the training speech from the training speech signal; a timbre encoder (1033) that converts the training speech signal into a spectrogram and outputs timbre features of the training speech from the converted spectrogram; The speech synthesis module (104) The fundamental frequency F ０ a sinusoidal noise generator (1041) for generating an input excitation signal based on the amplitude and a time-varying timbre artificial neural network (1043) for generating a time-varying timbre embedding based on the timbre features of the training speech; a frame-level synthesis artificial neural network (1042) for generating frame-level conditions for the synthesized speech based on the pronunciation features of the training speech and the generated time-varying timbre embeddings; a sample-level synthesis artificial neural network (1044) that synthesizes a synthetic speech signal indicative of the synthetic speech based on the input excitation signal and the frame-level conditions.

9. The TTS module calculates a fundamental frequency F of the text speech based on the input text and speech using a training dataset including a training synthetic text, a training speech, and training speech features. ０ 9. The method of claim 8, wherein the method is an artificial neural network pre-trained to output the amplitude and the amplitude of the sigma.

10. A computer-readable recording medium having a program recorded thereon for executing the method according to any one of claims 1 and 2.

Images