Method and apparatus for multi-language and multi-speaker speech synthesis

Abstract

A method for multi-language and multi-speaker speech synthesis includes receiving training text and training audio signals obtained from a speaker speaking the training text. The method further includes identifying a language identifier corresponding to the training text, and speaking a training reference audio signal obtained from a reference text different from the training text from the speaker. The method further includes training a speech synthesis model using training samples including the training text, the training audio signal, the language identifier, and the training reference audio signal.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an apparatus and method for multi-lingual and multi-speaker speech synthesis. BACKGROUND

[0002] The contents described in this section are merely provided to assist in understanding the background of the present disclosure and can not constitute the prior art.

[0003] Recent advances in speech synthesis have enabled its widespread use in various fields such as voice guidance and education. Speech synthesis is a technology that generates a sound similar to human speech, and is commonly referred to as a text-to-speech (TTS) system. Speech synthesis technology delivers information to users through a speech signal rather than text or images, and thus is particularly useful when users cannot see the screen of a machine in operation (for example, when the user is driving a car or when the user is blind). In recent years, the development and popularization of smart home devices such as artificial intelligence speakers, smart TVs, and smart refrigerators, and personal portable devices such as smartphones, e-book readers, and car navigation systems have been actively promoted, leading to a rapid increase in demand for speech synthesis technology and voice output devices.

[0004] The conventional speech synthesis method includes various methods such as unit selection synthesis (USS) and statistical parameter synthesis (HMM-based speech synthesis, HTS). The USS method divides speech data into phoneme units and stores them, identifies and connects sound segments suitable for speech synthesis. The HTS method extracts parameters corresponding to speech characteristics, generates a statistical model, and converts text into speech based on the statistical model.

[0005] The conventional speech synthesis method includes generating a spectrogram based on input text and generating a sound wave based on the spectrogram. Here, the spectrogram is a tool for visualizing and understanding sound or a waveform. The spectrogram is obtained by converting an audio signal in the time domain into a frequency component with respect to the time domain axis. Based on the spectrogram, the characteristics of the waveform and its spectrum can be visualized.

[0006] In addition, the speech synthesis method can generate a sound wave reflecting the speech characteristics of a speaker. The speech synthesis method can generate a speech signal corresponding to the input text based on the speaker's voice, prosody, pitch, speech rate, and the like.

[0007] Recently, a speech synthesis method using an artificial neural network to generate speech from text has received attention.

[0008] However, the conventional speech synthesis model is difficult to synthesize speech for a speaker-language combination that has not been seen. Specifically, training data used to train the conventional speech synthesis model consists of [text, speaker, language]. Since most speakers speak in one language, the speech synthesis model has difficulty in naturally generating speech for another language for the same speaker. For example, a speech synthesis model trained based on speech data of a male speaking English has limitations in synthesizing speech data of the same male speaking Korean.

[0009] However, the conventional speech synthesis method described above has many limitations in synthesizing natural speech reflecting the speech style or emotional expression of a speaker.

[0010] In addition, in the field to which the speech synthesis system is applied, low-quality synthesized speech (e.g., speech with incorrect intonation or tone) is often used without correction. Since a single-speaker speech synthesis model can only generate speech for one speaker, its application is limited to specific purposes. SUMMARY

[0011] The present disclosure provides an apparatus and a method for synthesizing natural speech, giving a feeling that a target speaker speaks a text in a target language fluently even if audio data for the target speaker in the target language is sparse or missing.

[0012] The present disclosure also provides a training technique for a speech synthesis model and a speech synthesis model using the technique, which can control the speech synthesis model to separate information between input factors without overfitting to the input factors.

[0013] An object of the present disclosure is to provide a scalable model architecture that can separate various sound features from speech.

[0014] Technical objects of the present disclosure are not limited to the above-described content. Those of ordinary skill in the art can more clearly understand other technical objects from the above description. The technical objects of the present disclosure can be more clearly understood by the following description.

[0015] According to one aspect of this disclosure, a method for training a speech synthesis model is provided. The method includes: receiving training text and a training audio signal obtained from a speaker speaking the training text. The method further includes: identifying a language identifier corresponding to the training text and a training reference audio signal obtained from a speaker speaking reference text different from the training text. The method further includes: training the speech synthesis model using training samples including the training text, the training audio signal, the language identifier, and the training reference audio signal. Training the speech synthesis model may include: transforming the language identifier into a language embedding through a language embedding module of the speech synthesis model. Training the speech synthesis model may include: transforming the training audio signal and the training reference audio signal into speaker embeddings through a speaker encoder of the speech synthesis model. Training the speech synthesis model may include: determining a loss based on a first speaker embedding transformed from the training reference audio signal, a second speaker embedding transformed from the training audio signal, and the language embedding. Training the speech synthesis model may include: using the loss to update the parameters of the speaker encoder.

[0016] According to another aspect of this disclosure, an apparatus is provided, comprising at least one processor and a memory configured to store instructions. The at least one processor is configured by executing instructions to: receive training text and a training audio signal obtained from a predetermined speaker uttering the training text. The at least one processor is further configured to: identify a language identifier corresponding to the training text and a training reference audio signal obtained from a speaker uttering reference text different from the training text. The at least one processor is further configured to: train a speech synthesis model using training samples including the training text, the training audio signal, the language identifier, and the training reference audio signal. The speech synthesis model may include a language embedding module configured to: transform the language identifier into a language embedding. The speech synthesis model may also include a speaker encoder configured to: transform the training audio signal and the training reference audio signal into speaker embeddings. The at least one processor is further configured to: determine a loss based on a first speaker embedding transformed from the training reference audio signal, a second speaker embedding transformed from the training audio signal, and the language embeddings. The at least one processor is further configured to: use the loss to update parameters of the speaker encoder.

[0017] According to another aspect of this disclosure, a speech synthesis method is provided. The method includes: receiving a speech synthesis request including input text and target speaker information. The method further includes: identifying a language identifier corresponding to the input text, and a training reference audio signal obtained from a reference text spoken by a speaker that differs from training text. The method further includes: generating a synthesized audio signal that mimics speech of the input text by applying the input text, the language identifier, and the reference audio signal to a speech synthesis model. Generating the synthesized audio signal may include: transforming the language identifier into a target language embedding using a language embedding module of the speech synthesis model. Generating the synthesized audio signal may include: transforming the reference audio signal into a target speaker embedding using a speaker encoder of the speech synthesis model. The speaker encoder may have been trained using speaker embeddings transformed from multiple training audio signals from a speaker, and language embeddings transformed from training text corresponding to one of the multiple training audio signals.

[0018] According to another aspect of this disclosure, an apparatus is provided, comprising at least one processor and a memory configured to store instructions. The at least one processor is configured to execute instructions that cause the apparatus to: receive a speech synthesis request including input text and target speaker information. The at least one processor is further configured to: identify a language identifier corresponding to the input text, and a reference audio signal obtained from a speaker uttering reference text different from the input text. The at least one processor is further configured to: generate a synthesized audio signal that mimics the speech of the target speaker uttering the input text by applying the input text, the language identifier, and the reference audio signal to a speech synthesis model. The speech synthesis model may include a language embedding module configured to: transform the language identifier into a target language embedding. The speech synthesis model may also include a speaker encoder configured to: transform the reference audio signal into a target speaker embedding. The speaker encoder may have been trained using speaker embeddings transformed from multiple training audio signals from a speaker, and language embeddings transformed from training text corresponding to one of the multiple training audio signals, based on a loss determined by the speaker. Attached Figure Description

[0019] Figure 1 The structure of a vehicle according to an embodiment of the present disclosure is shown.

[0020] Figure 2 This illustrates speech synthesis according to one embodiment of the present disclosure.

[0021] Figure 3 The operation of a speech synthesis apparatus according to an embodiment of the present disclosure is shown.

[0022] Figure 4The training of a speech synthesis model according to one embodiment of the present disclosure is shown.

[0023] Figure 5 This illustration shows a metric learning loss between speaker embedding and language embedding according to one embodiment of the present disclosure.

[0024] Figure 6 The operation of a speech synthesis model according to an embodiment of the present disclosure is shown.

[0025] Figure 7 An example of a language normalization affine coupling layer according to an embodiment of the present disclosure is shown.

[0026] Figure 8 An example of a language denormalization affine coupling layer according to an embodiment of the present disclosure is shown.

[0027] Figure 9 This is a flowchart illustrating a method for training a speech synthesis model according to an embodiment of the present disclosure.

[0028] Figure 10 This is a flowchart illustrating a speech synthesis method according to an embodiment of the present disclosure.

[0029] Figure 11 This is a block diagram that briefly illustrates a computing device that can be used to implement the apparatus and methods according to this disclosure. Detailed Implementation

[0030] In the following, embodiments of the present disclosure are described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals denote the same elements, even if these elements are shown in different drawings. Furthermore, for clarity and brevity, detailed descriptions of related known functions and components have been omitted in this disclosure.

[0031] Various terms such as first, second, A, B, (a), (b), etc., are used only to distinguish one component from another and do not imply or suggest the composition, order, or sequence of components. In this disclosure, when a component “comprises” or “includes” a component, the component is intended to further include other components and not to exclude other components, unless otherwise specifically stated to the contrary.

[0032] Terms such as "unit," "module," etc., refer to one or more components for processing at least one function or operation, and can be implemented by hardware, software, or a combination thereof. When components, devices, modules, elements, etc., of this disclosure are described as having a purpose or performing an operation, function, etc., the component, device, or element shall be considered herein as "configured to" satisfy that purpose or perform that operation or function. Individual controllers, units, modules, components, devices, elements, etc., may be embodied separately or included in a processor and memory (e.g., a non-transitory computer-readable medium) as part of a device.

[0033] The following detailed description and accompanying drawings are intended to illustrate exemplary embodiments of the present disclosure and are not intended to represent the only embodiments in which the present disclosure may be practiced.

[0034] Figure 1 The structure of a vehicle according to an embodiment of the present disclosure is shown.

[0035] Reference Figure 1 The vehicle 10 may include all or some of the following: a microphone 110 for inputting user voice; an input module 120 for receiving vehicle information; a speaker 130 for outputting sound required to provide the user's desired service; a display 140 for displaying images required to provide the user's desired service; a communication module 150 for communicating with external devices; and a controller 160 for controlling the aforementioned components and other components of the vehicle.

[0036] The microphone 110 can be located inside the vehicle 10 at a position for inputting user voice. The user inputting voice into the microphone 110 located in the vehicle 10 can be the driver. The microphone 110 can be installed in, for example, the steering wheel, center panel, headliner, or rearview mirror to receive the driver's voice.

[0037] In addition to the user's voice, various audio sounds generated around the microphone 110 can also be input to the microphone 110. The microphone 110 can output an audio signal corresponding to the input audio signal. The output audio signal can be processed by the controller 160 or sent to an external server device via the communication module 150.

[0038] In addition to microphone 110, vehicle 10 may also include input module 120 for receiving user commands. Input module 120 may be located in the dashboard area, the audio-video-navigation (AVN) area of ​​the center panel, the gearbox area, or the steering wheel in the form of buttons or shuttle buttons.

[0039] In addition, in order to receive control commands related to passenger seats, the input module 120 may include interface devices provided on the door of each seat and interface devices provided on the armrests of the front seats or the armrests of the rear seats.

[0040] In addition, the input module 120 may include a touchpad integrated with the display 140 to implement a touchscreen.

[0041] In addition, the input module 120 may include a camera. The camera can acquire at least one of the interior or exterior images of the vehicle 10. The camera can be installed inside, outside, or both inside and outside the vehicle 10. The images collected by the camera are processed by the controller 160 or an external server device. Based on the collected images, the occupants' gaze, mouth shape, facial features, behavior, or state in the video can be analyzed.

[0042] The speaker 130 can output electrical signals in the form of sound waves. The speaker 130 can be positioned near each door, roof, front window, or rear window facing the interior of the vehicle 10. The speaker 130 can refer to various types of speakers, such as loudspeakers and array speakers.

[0043] The display 140 may include an AVN display, an instrument panel display, or a head-up display (HUD) disposed on the center panel of the vehicle 10. Alternatively, the display 140 may include a rear seat display disposed on the rear of the headrest of the front seat for use by passengers in the rear seats. Alternatively, when the vehicle 10 is a multi-passenger vehicle, the display 140 may include a display mounted on the headliner.

[0044] The display 140 can be positioned in a location where the occupants of the vehicle 10 can see it, and there are no other restrictions on the number or location of the display 140.

[0045] The communication module 150 can exchange signals with other devices by employing at least one of various wireless communication methods, such as Bluetooth, 4G communication, 5G communication, or Wi-Fi. Alternatively, the communication module 150 can exchange information with other devices via a cable connected to a Universal Serial Bus (USB) port, an AUX port, or the like.

[0046] In addition, the communication module 150 can exchange information signals with two or more other devices by being equipped with two or more communication interfaces that support different communication methods.

[0047] For example, the communication module 150 can: communicate with a mobile device located inside the vehicle 10 via Bluetooth to receive information (user's video, voice, contact information, schedule, etc.) obtained or stored therein by the mobile device; transmit the user's voice via 4G or 5G communication with a server; and receive signals necessary to provide the user's desired services. Furthermore, the communication module 150 can exchange necessary signals with the server through a mobile device connected to the vehicle 10.

[0048] In addition to the above, vehicle 10 may include navigation equipment for providing route guidance, air conditioning equipment for controlling interior temperature, window control equipment for controlling the opening / closing of windows, seat heating equipment for heating seats, seat positioning equipment for adjusting seat position, height or angle, and lighting equipment for adjusting interior lighting.

[0049] The aforementioned devices provide convenient functions related to vehicle 10, and some devices may be omitted depending on the vehicle model and options. Furthermore, it should be noted that other devices may be included in addition to those described above. The drive of vehicle 10 uses a known configuration, and its description is omitted in this disclosure.

[0050] The controller 160 can turn the microphone 110 on / off, and can process or store the voice input to the microphone 110, and / or can send the input voice to another device via the communication module 150.

[0051] In addition, the controller 160 can control the image to be displayed on the display 140 and the sound to be output to the speaker 130.

[0052] In addition, the controller 160 can perform various control operations related to the vehicle 10. For example, based on user commands input via the microphone 110 or the input module 120, the controller 160 can control at least one of the following: navigation equipment, air conditioning equipment, window control equipment, seat heating equipment, seat positioning equipment, or lighting equipment.

[0053] The controller 160 may include at least one memory storing a program for performing the operations described above and below. The controller 160 may also include at least one processor executing the stored program.

[0054] The controller 160 can operate as a speech synthesis device. For example, a user can request audio output so that the text displayed on the display 140 is spoken by a selected speaker in a specified language. The user's desired language and speaker can be preset.

[0055] In some embodiments, controller 160 can synthesize speech corresponding to text by converting text into an audio signal based on a selected language and a selected speaker. For example, a user might want to hear the English text "Directions to home will be provided" spoken in Korean (e.g., the accent or intonation of a Korean speaker speaking English). Controller 160 can acquire pre-stored audio samples of Korean speech and apply a speech synthesis model to the English text and the audio samples. The speech synthesis model can generate an audio signal that makes the English text sound like it's being spoken naturally in Korean. The audio signal can be output through speaker 130. The user can then hear the English text being spoken naturally by a selected Korean speaker.

[0056] In another example, controller 160 can perform speech-based questioning and answering by synthesizing and transforming responses to user questions into audio signals based on a preset speaker. For example, if a user inputs the question "What is 'Encantado de conocerlo' in English?" through microphone 110, controller 160 can generate "It's 'Nice to meet you'" using the voice of a single target speaker. Controller 160 can obtain pre-stored audio samples for the target speaker and can apply a speech synthesis model to the multilingual text and audio samples. The speech synthesis model can generate audio signals that make each word in the multilingual text sound as if spoken with the correct intonation in the corresponding language. In other words, the user can hear the multilingual text sounding like it was spoken naturally by the selected target speaker.

[0057] In another example, controller 160 can synthesize and convert a user's voice or text based on different speakers and languages.

[0058] According to another embodiment, the controller 160 and the communication module 150 can be combined with electronic devices located outside the vehicle 10 to provide voice synthesis functionality.

[0059] Figure 2 This illustrates speech synthesis according to one embodiment of the present disclosure.

[0060] Reference Figure 2 The speech synthesis system may include a vehicle 210 and an electronic device 220. The speech synthesis method may be implemented by the vehicle 210 and / or the electronic device 220. The speech synthesis model may be implemented on the electronic device 220, and the speech synthesis method may be executed by the electronic device 220.

[0061] Electronic device 220 can perform speech synthesis. Electronic device 220 can be implemented by at least one of server device 221 and mobile terminal 223.

[0062] Vehicle 210 can send a speech synthesis request to electronic device 220, which can respond to vehicle 210 with an audio signal as the result of speech synthesis. The speech synthesis request may include the text to be synthesized into speech, the language identifier of the text, and speaker information.

[0063] Specifically, vehicle 210 can send a speech synthesis request to electronic device 220, the speech synthesis request containing a set consisting of [text, speaker] or [text, speaker, language]. Electronic device 220 can generate an audio signal representing the requested text spoken by the desired speaker. Because the speaker does not actually read the text aloud, the audio signal can represent generated data rather than a recorded signal. However, the audio signal can reproduce natural pronunciation and speech, as if it were a recording of an actual speaker fluently speaking the requested text. Electronic device 220 can send the generated audio signal to vehicle 210 as the result of speech synthesis. Vehicle 210 can output audio sounds to the user based on the requested text, the requested speaker, and the requested language by playing the received audio signal.

[0064] In addition, electronic device 220 may include a processor and memory for speech synthesis.

[0065] Figure 3 The operation of a speech synthesis apparatus according to an embodiment of the present disclosure is shown.

[0066] Reference Figure 3 The speech synthesis device can receive speech synthesis requests 30 that include text and speaker information.

[0067] A speech synthesis device can obtain a language identifier corresponding to text. The language identifier may include a number that uniquely identifies various languages. For example, a language identifier may have a value of 0 for English and a value of 1 for Korean. The length of the language identifier may be the same as the length of the string corresponding to the utterance content in the text. In some embodiments, the language identifier may be included separately from the text in the speech synthesis request. In other embodiments, information for language identification (e.g., language codes) may be combined in the text. The speech synthesis device can extract language codes from the text and can generate a language identifier based on the extracted language codes. In yet another embodiment, the speech synthesis device can generate a language identifier by automatically detecting (or recognizing) the language of the text.

[0068] Speech synthesis requests can be broadly categorized into three types: intra-language synthesis, cross-language synthesis, and code-mixed synthesis. Intra-language synthesis refers to synthesizing the speech of a speaker in a specific language. For example, intra-language synthesis could include synthesizing Korean speech from a Korean speaker. Cross-language synthesis refers to synthesizing the speech of a speaker in a specific language in different languages. For example, cross-language synthesis could include synthesizing Korean speech from an American speaker. Code-mixed synthesis refers to synthesizing the speech of a speaker in a specific language in multiple languages. For example, code-mixed synthesis could include synthesizing speech from a multilingual text containing Korean and English, using the voice of a Korean speaker.

[0069] Table 1 shows examples of intra-language synthesis, cross-language synthesis, and code hybrid synthesis.

[0070] [Table 1]

[0071]

[0072] The speech synthesis device can identify the type of speech synthesis request based on speaker information and / or language identifiers, and can select a speech synthesis model 32 (S300) to synthesize speech corresponding to the requested text based on the type of speech synthesis request. For example, the speech synthesis device may include all or part of a multi-speaker-multilingual speech synthesis model, a multi-speaker-monolingual speech synthesis model, and a monolingual speech synthesis model. If the requested task involves cross-language synthesis or code-mixed synthesis, the speech synthesis device can select a multi-speaker-multilingual speech synthesis model as the model to be used for inference.

[0073] The speech synthesis device can select a reference audio signal based on speaker information (S320). For example, the speech synthesis device can have one or more audio signals prepared in advance for each speaker. The speech synthesis device can randomly select a reference audio signal from one or more audio signals for the target speaker indicated by the speaker information. As another example, the reference audio signal can be included as speaker information in the speech synthesis request 30.

[0074] The speech synthesis device can apply text, language identifiers and reference audio signals to a selected speech synthesis model 32, thereby synthesizing speech of text spoken by a target speaker indicated by speaker information (S340).

[0075] The speech synthesis device can output a synthesized audio signal 34 as a response to the speech synthesis request 30.

[0076] Figure 4 The training of a speech synthesis model according to an embodiment of the present disclosure is illustrated.

[0077] Reference Figure 4 The diagram illustrates a model architecture 40 detailing the training phase of a speech synthesis model. During the training phase, model architecture 40 may include all or some of the following: a language embedding model 400, a character embedding module 410, an encoder 420, a random duration predictor 430, a speaker encoder 440, a projection module 450, an aligned data estimator 460, a decoder 470, a posterior encoder 480, and an audio generator 490. In another embodiment, some of the constituent elements included in model architecture 40 may be omitted, or the order of the constituent elements may be changed. Model architecture 40 may also include a discriminator. However, the discriminator is not shown in the figure. The posterior encoder 480 and the discriminator may be used solely for training the speech synthesis model.

[0078] exist Figure 4 In the diagram, dashed arrows can indicate global adjustment. In this disclosure, embedding adjustment can refer to adding, multiplying, or subtracting the embedding from the input or its interior. To ensure dimensionality matching, convolutional layers can adjust the dimensions of the embedding. For example, within decoder 470, the speaker embedding can be incorporated into the latent variables.

[0079] To train a speech synthesis model, a training dataset can be prepared in advance. The training dataset can include text data, corresponding audio data, and language data. The audio data can be recordings of the text data actually spoken by one or more speakers.

[0080] Training data may include pairs of [training text, training audio signal]. Training text may include character sequences of natural language. For example, character sequences may include alphanumeric characters, numbers, punctuation marks, or other special characters. Training audio signals may represent a speaker's speech data. Training audio signals may include the speaker's voice characteristics and / or speech features. Speaker speech features may include at least one of various elements, such as speech rate, pause intervals, pitch, tone, prosody, intonation, articulation, and emotion. In one embodiment, the training dataset may include audio signals from multiple speakers.

[0081] Furthermore, linear or Mel spectrograms converted from the training audio signal can be prepared in advance and used as training data. Linear spectrograms can be generated by applying Short-Time Fourier Transform (STFT), Discrete Fourier Transform (DFT), or Fast Fourier Transform (FFT) to the audio signal. Mel spectrograms can be obtained by adjusting the frequency intervals of the linear spectrogram to a Mel scale. Mel spectrograms can also be obtained by applying a Mel filter bank to the linear spectrogram.

[0082] The training data may also include speaker information for identifying the speaker of the audio signal. Additionally or alternatively, the training data may also include language identifiers for identifying the language corresponding to the training text and the training audio signal. The language identifier may be represented as a number. For example, a language identifier may be a number that can uniquely identify one of various languages ​​(e.g., Korean, English, German, Japanese, and Chinese). For example, English may have a value of 0, Korean may have a value of 1, and German may have a value of 2. In some embodiments, language identifiers may be assigned to individual words or characters in the training text. For example, if the training text consists of a single language, language identifiers may be generated by repeatedly padding individual language identifiers to match the length of the training text.

[0083] Furthermore, since most speakers may only speak a few languages, audio signals containing speech from multiple speakers in various languages ​​may be scarce. In other words, multi-speaker, multilingual datasets may be rare.

[0084] To enable zero-shot speech synthesis of unseen speaker-language speech, the training data can also include training reference audio signals obtained from a speaker uttering a reference text different from the training text. The training reference audio signals can be randomly selected from multiple audio signals prepared in advance for the speaker.

[0085] In some embodiments, during the preprocessing stage of training, all or part of the training reference audio signal, training spectrogram, and / or language identifier sequence may be added to the training dataset, but this disclosure is not limited to this specific embodiment.

[0086] The following is for reference Figure 4 This describes the steps involved in processing training text, training audio signals, training reference audio signals, training spectrograms, and language identifiers within the training dataset for training purposes.

[0087] The language embedding module 400 can transform linguistic identifiers corresponding to training text into language embeddings. For example, a language embedding can correspond to a trainable embedding. In another example, the language embedding module 400 can use one-hot encoding to map linguistic identifiers to language embeddings. Language embeddings can be in vector form. Because one-hot encoding is a well-known technique in the field of speech synthesis, a detailed description of it is omitted.

[0088] The speaker encoder 440 can transform or map the speaker's training audio signal and training reference audio signal into their respective speaker embeddings. The speaker embedding represents the speaker's speech characteristics and can be expressed in vector form. Furthermore, the speaker embedding can include speaker identification information. The speaker embedding can have the same dimensionality as the language embedding.

[0089] The speaker encoder 440 can represent discontinuous data values ​​contained in speaker information as vectors composed of continuous digits. For example, the speaker encoder 440 can generate speaker embedding vectors based on at least one or more combinations of various artificial neural network models, including Pre-Net, CBHG modules, deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and bidirectional recurrent deep neural networks (BRDNNs).

[0090] The character embedding module 410 can transform or map the training text into character embeddings. For example, the training text can be composed of sentences or characters. The character embedding module 410 can segment the training text into character units and transform each segment into a character embedding. Alternatively, the character embedding module 410 can segment the training text into alphabetic units or phoneme units, and then transform them into character embeddings. For example, the text embedding module 410 can use an artificial neural network model to perform text embedding. Character embeddings can be represented as learnable vectors.

[0091] The encoder 420 can extract text feature vectors from character embeddings. The text feature vectors extracted by the encoder 420 can include character embeddings, i.e., features of the training text.

[0092] In one embodiment, encoder 420 may perform encoding on a phoneme-by-phoneme basis. To do this, encoder 420 may segment character embeddings into phoneme units of the training text. In another embodiment, encoder 420 may perform encoding on the entire set of character embeddings.

[0093] Encoder 420 can be an artificial neural network. For example, encoder 420 can be a transform-based encoder 420. A transform-based encoder 420 includes multiple transform blocks, and each transform block includes at least one encoder 420, at least one decoder, and an attention module. For example, a transform-based encoder 420 can include 10 transform blocks. The transform blocks use encoder 420 to extract context vectors from character embeddings, use the attention module to identify important character embeddings, and use decoder 470 to generate text feature vectors from the context vectors and the output of the attention module.

[0094] The projection module 450 can output a distribution of text feature vectors to match the dimension. Here, the distribution of text feature vectors can be a prior distribution including the mean and standard deviation of the text feature vectors. The distribution can include the mean and standard deviation of each text feature vector corresponding to each phoneme. The projection module 450 can be a linear projection layer.

[0095] The posterior encoder 480 can encode the training spectrogram and output latent variables. Encoding can mean extracting features from existing data and transforming them into data of reduced size or dimensionality compared to the existing data. In other words, the output of the encoding can be the result of compressing the input data. Latent variables can be latent vectors. Latent variables can include the speaker's voice and / or speech characteristics.

[0096] The training spectrogram can be a linear-scale spectrogram or a Mel spectrogram transformed from the speaker's training audio signal. In another embodiment, an audio file format (e.g., wav or mp4) is input to the posterior encoder 480, and the posterior encoder 480 can encode the audio signal to extract a latent vector.

[0097] The posterior encoder 480 can also employ speaker embeddings to output latent variables. In other words, the posterior encoder 480 can receive a training spectrogram and speaker embeddings, and can extract latent variables from both. Speaker embeddings can be used for global conditioning. For example, speaker embeddings can be added to the training spectrogram or latent variables. The conditioned latent variables include features from both the training spectrogram and speaker embeddings.

[0098] The posterior encoder 480 can be a deep neural network. For example, the posterior encoder 480 can be a variational autoencoder (VAE) encoder. The posterior encoder 480 can include non-causal WaveNet residual blocks used in the WaveGlow and Glow-TTS models. For example, the posterior encoder 480 can include 12 WaveNet residual blocks. The non-causal WaveNet residual blocks can consist of extended convolutional layers with gated activation units and skip connections. A linear projection layer above this block generates the mean and variance of a normal posterior distribution.

[0099] Decoder 470 can output transformed latent variables based on latent variables, language embeddings, and speaker embeddings. Decoder 470 can generate latent variables with distributions different from the prior distributions of the latent variables. Here, the different distribution can be a normal distribution.

[0100] Decoder 470 can use speaker embeddings as conditioning information. For example, decoder 470 can condition the speaker embedding as input or output by adding or multiplying it with latent variables or transformed latent variables.

[0101] Furthermore, decoder 470 can remove linguistic information from latent variables. In one embodiment, decoder 470 receives linguistic embeddings corresponding to training text. Decoder 470 can remove linguistic features within latent variables by normalizing the linguistic information of the latent variables.

[0102] As described above, decoder 470 can modulate latent variables and speaker embeddings, normalize language-related features within the language-embedded latent variables, and generate transformed latent variables by sampling variables from distributions that are simpler or more complex than the distribution of the preprocessed latent variables. Here, preprocessing refers to the modulation of speaker embeddings and the normalization of language embeddings. Decoder 470 can remove language-specific features of the training text from the modulated latent variables by normalizing the language-embedded latent variables. Language-specific features can also be referred to as language features or language information. The transformed latent variables include features of the training audio signal but may exclude language-specific features of the training text.

[0103] Decoder 470 can be a normalized stream function. Decoder 470 obtains transformed latent variables by applying function f to the preprocessed latent variables. Because the distribution transformation of decoder 470 is invertible, there exists an inverse function of decoder f. Compared to the original latent variables, the transformed latent variables can have the same, different, or more complex distributions. Here, a complex distribution refers to a distribution that differs from a simple normal distribution and has multiple local minima and maxima.

[0104] Decoder 470 may be a deep neural network. Specifically, decoder 470 may be a stream-based decoder. Decoder 470 may include multiple affine coupling layers. For example, decoder 470 may include four affine coupling layers. At least a portion of the multiple affine coupling layers may be used to exclude language-specific features. The affine coupling layer used to exclude language-specific features may be called a Language Normalization Affine Coupling Layer (LNAC). The following will refer to... Figure 7 Describe the transformation of the LNAC layer.

[0105] In some embodiments, speaker embeddings may be additionally considered during the normalization process of the language embeddings. For example, the speaker embeddings may be input into a neural network to generate scaling parameters and / or bias parameters. In another example, at least a portion of the affine coupling layer of the inverting decoder 680 may be used independently for speaker embedding applications.

[0106] By using multiple affine coupling layers, decoder 470 can generate transformed latent variables that are modulated by speaker embeddings and normalized by language embeddings.

[0107] The Alignment Data Estimator 460 can output aligned data based on the distribution of text feature vectors and transformed latent variables.

[0108] In one embodiment, the alignment data estimator 460 may estimate a matrix as alignment data based on the mean, standard deviation, and transformed latent variables of the text feature vectors. The dimensionality of the alignment data may depend on the length of the latent variables and the length of the character embeddings. For example, rows may represent phonemes, and columns may represent time intervals. In the alignment data, the duration of each phoneme may be represented as a path; elements along the path may have a value of 1, while other elements may have a value of 0. In other words, the alignment data may refer to the alignment information between the phonemes of the training text and their corresponding latent variables.

[0109] To estimate matrix A (which is the alignment data between phonemes included in the training text), monotonic alignment search (MAS) can be used. MAS is a method that searches for alignments that maximize the likelihood of the data parameterized by the flow normalization function. The alignment data estimator 460 estimates the alignment data by applying the MAS method to the distribution of the text feature vectors and the transformed latent variables. Because the MAS method is well-known, a detailed description of it is omitted.

[0110] Aligned data can be used to train a random duration predictor 430. Aligned data can refer to the similarity between text feature vectors and transformed latent variables.

[0111] The random duration predictor 430 can receive text feature vectors, alignment data, and speaker embeddings, and based on the received input, can predict the duration of each phoneme in the training text. In other words, the random duration predictor 430 can predict phoneme duration data.

[0112] The random duration predictor 430 can use speaker embeddings as conditioning information. The random duration predictor 430 can conditionally incorporate speaker embeddings during the computation process. For example, speaker embeddings can be added to or multiplied by text feature vectors or alignment data.

[0113] The random duration predictor 430 can be a stream-based generative model trained via maximum likelihood estimation. Furthermore, noise can be computed during the prediction of phoneme length data.

[0114] The audio generator 490 can generate synthesized audio signals in the time domain based on latent variables. In other words, the audio generator 490 can generate speech waveforms based on the prior distribution of latent variables.

[0115] Audio generator 490 can be a deep neural network. Audio generator 490 can be a vocoder. For example, audio generator 490 can be a HiFi-GAN generator. Audio generator 490 can include transposed convolutional stacks, each convolution possibly followed by a multi-receptive field fusion (MRF) module. The output of the MRF is the sum of the outputs of "residual blocks" with corresponding receptive field sizes. Audio generator 490 can include a linear layer responsible for transforming the speaker embedding, which can add the speaker embedding to the latent variable z, and can generate an audio signal from the combination of the latent variable and the speaker embedding.

[0116] The architecture 40 of the speech synthesis model can be trained by a computer-implemented training device. In some embodiments, end-to-end training can be applied to the architecture 40 of the speech synthesis model, but this disclosure is not limited to specific training techniques.

[0117] To reduce the dependency between the speaker embedding and the language embedding generated by the speech synthesis model, a cross-training technique can be applied. This technique utilizes a training reference audio signal obtained by having a speaker speak a reference text different from the training text. The cross-training technique aims to remove language-related information from the speaker embedding and clearly separate the information contained in the speaker embedding from the information in the language embedding.

[0118] Specifically, the learning loss can be quantified using the distance measured between embeddings as the loss function for training the speech synthesis model.

[0119] Figure 5 This illustration shows a metric learning loss between speaker embedding and language embedding according to one embodiment of the present disclosure.

[0120] As described above, the speaker encoder 440 can transform the training reference audio signal and the training audio signal into corresponding speaker embeddings 500 or 520, and the language embedding module 400 can transform the linguistic identifiers of the training text into a language embedding 540. In this case, speaker embeddings 500 and 520, as well as language embedding 540, can have the same dimension. This means that embeddings 500, 520, and 540 can be embedded into the same feature space, and therefore the distance between any two embeddings among 500, 520, and 540 can be calculated.

[0121] To ensure that the information in each speaker embedding 500 or 520 and the language embedding 540 does not overlap, a metric learning loss can be designed to maximize the distance between each speaker embedding 500 or 520 and the language embedding 540. Alternatively, a metric learning loss can be designed to minimize the distance between speaker embeddings 500 and 520, thus forcing speaker embeddings 500 and 520 to contain only speaker identity information and not textual information.

[0122] In other words, contrastive learning can be applied to minimize the distance between speaker embeddings 500 and 520, while maximizing the distance between each speaker embedding 500 and 520 and the language embedding 540.

[0123] Furthermore, cosine distance or Euclidean distance can be used as the distance metric for measuring the learning loss. Here, in the case of cosine distance, even if the absolute distance between embeddings is large, if the features are similar, the measured distance may still be close. Therefore, Euclidean distance may be more suitable than cosine distance for independently processing speaker information and language information. Furthermore, the learning loss based on Euclidean distance can include contrastive loss and triplet loss. In contrastive loss, the absolute distance between positive pairs and the absolute distance between negative pairs are measured separately. However, excessively increasing the distance between negative pairs may degrade speech quality. On the other hand, because triplet loss measures the relative distance between the anchor, positive sample, and negative sample, triplet loss can prevent the obtained distance from being too large. Therefore, in one embodiment, triplet loss can be used as the learning loss metric, which uses speaker embeddings 500 and 520 and language embedding 540 as the anchor, positive sample, and negative sample, respectively. When speaker embeddings 500 and 520 and language embedding 540 are represented as... , and When the triplet loss is equal to 1, the triplet loss can be expressed by Equation 1.

[0124] [Formula 1]

[0125]

[0126] In Formula 1, This represents a hyperparameter that determines the lower bound of the distance between negative pairs.

[0127] Furthermore, during the inference step of the speech synthesis model, the model can generate a synthesized audio signal based on a reference audio signal that records the speech of the target speaker as different from the training text (i.e., the speech content of the synthesized audio signal to be generated). In contrast, when the training audio signal is used to generate the synthesized audio signal during the training step, the amount of information referenced by the model in the training and inference steps may differ. When a model trained in this way synthesizes speech of an unseen speaker-language, the quality of the generated speech may degrade, resulting in poor sound quality or unclear pronunciation.

[0128] Considering the above issues, the speech synthesis model can be designed to generate synthesized audio signals based on the training reference audio signal, rather than the training audio signal itself, during both the training and inference steps. For example, the speaker embedding 500 derived from the training reference audio signal can be input into the decoder 470 and / or the posterior encoder 480. Furthermore, the speaker embedding 520 derived from the training audio signal can be used only in the loss function and does not need to be directly used to generate the synthesized audio signal.

[0129] Furthermore, it will be apparent to those skilled in the art that other loss functions can also be used to train the speech synthesis model.

[0130] For example, the loss function of a speech synthesis model can also use at least one of the following: reconstruction loss, Kullback-Leibler divergence loss, duration loss, adversarial loss, and feature matching loss.

[0131] The reconstruction loss can be calculated based on the difference between the generated synthetic audio signal (or its spectrogram) and the training audio signal (or its spectrogram). A converter can be additionally used to convert the generated synthetic audio signal into a spectrogram.

[0132] The KL divergence loss can be calculated based on the difference between the latent variables and the text feature vectors. Alternatively, it can be calculated based on the difference between the posterior probability of the latent variables and the conditional prior probability of the text feature vectors. In other words, the KL divergence loss can be considered the similarity between the distribution of the latent variables and the distribution of the text feature vectors.

[0133] The duration loss can be calculated based on the difference between the phoneme duration data predicted by the random duration predictor 430 and the phoneme durations generated by the aligned data estimator 460. In another embodiment, the pronunciation durations of each phoneme in an actual audio sample recorded by the speaker can be prepared in advance as labels for calculating the duration loss. The duration loss can be calculated using mean squared error (MSE). The duration loss is designed to enable the random duration predictor 430 to predict the duration of each phoneme spoken by the adjusted speaker. The duration loss can be referred to as the lower bound of the variation of the log-likelihood of the phoneme sequence.

[0134] Adversarial loss can be calculated based on whether the synthesized audio signal generated by the audio generator 490 is real, determined by a discriminator. For this purpose, a discriminator trained to distinguish between real and fake input audio signals can be used. The discriminator could, for example, be a HiFi discriminator. To reduce adversarial loss, the discriminator needs to determine that the generated synthesized audio signal is real data. The adversarial loss causes the discriminator to output a value of 1 in response to real data input and a value of 0 in response to fake data input.

[0135] Furthermore, the feature matching loss is calculated based on the difference between the features extracted by the discriminator from the generated audio signal and the features extracted by the discriminator from the real audio signal.

[0136] By training based on adversarial loss and feature matching loss, the audio generator 490 can generate audio signals that are almost identical to real data.

[0137] Optionally, the loss function of model architecture 40 may also include speaker consistency loss (SCL). Speaker consistency loss can be calculated based on the difference between the output of speaker encoder 440 and the true baseline.

[0138] Model architecture 40 can be updated in the direction of reducing the above loss function. By iteratively training based on the overall loss function, the components of model architecture 40 are refined, enabling the speech synthesis model to generate natural speech signals from the speaker.

[0139] Through the training process described above, the speech synthesis model becomes robust to linguistic diversity. Speaker dependence on a specific language is reduced. In other words, the speech synthesis model is trained based on text and speaker rather than a specific language. However, during the inference phase, the speech synthesis model utilizes linguistic information. Even when the speech synthesis model receives text in an unseen language, it can use information from that unseen language to generate natural speaker speech from the text. For example, even though the training dataset primarily consists of [Korean text, Korean speech] pairs with a small number of [Korean text, American speech] pairs, the speech synthesis model learns the meaning of the Korean text and captures the speech characteristics of the American speaker without linguistic information. Then, during the inference phase, the speech synthesis model can synthesize natural speech by incorporating Korean embeddings into the [Korean text, American speech] data.

[0140] Figure 6 The operation of a speech synthesis model according to an embodiment of the present disclosure is shown.

[0141] Reference Figure 6 The diagram illustrates the configuration of a speech synthesis model 60. This model generates audio signals as if the input text in a given language were spoken by a specific speaker. For example, the speech synthesis device stores user-defined language identifiers and pre-recorded audio samples of selected speakers. The content of the audio samples may differ from the content of the input text. The speech synthesis device synthesizes audio signals by applying the speech synthesis model to the language identifiers, the speaker's audio signal, and the target text.

[0142] During the inference phase, the speech synthesis model 60 includes a language embedding module 610, a character embedding module 620, an encoder 630, a random duration predictor 640, a speaker encoder 650, a projection module 660, an alignment module 670, an inversion decoder 680, and an audio generator 690.

[0143] Speech synthesis model 60 has been passed Figure 4 The method was used for training. For example, Figure 6 The language embedding module 610, character embedding module 620, encoder 630, random duration predictor 640, speaker encoder 650, projection module 660, phase inversion decoder 680, and audio generator 690 correspond to... Figure 4 The system includes a language embedding module 400, a character embedding module 410, an encoder 420, a random duration predictor 430, a speaker encoder 440, a projection module 450, a decoder 470, and an audio generator 490. An inverse decoder 680 can represent the inverse function of decoder 470.

[0144] The language embedding module 610 can convert the language identifiers of the input text into language embeddings. In another embodiment, the language embedding module 610 can be omitted, and language embeddings corresponding to various languages ​​can be pre-stored. In other words, language embeddings corresponding to the language identifiers of the input text are pre-stored, and the inverting decoder 680 can receive the language embeddings. In some examples, the input text may include words or characters corresponding to multiple languages. The language embedding module 610 can generate language embeddings for each word or character.

[0145] The character embedding module 620 can convert a given input text into a character embedding. The input text can be mapped to a variable space used for character embedding.

[0146] The encoder 630 can output a text feature vector for the input text by encoding the character embeddings. The text feature vector can include the features of each phoneme of the input text.

[0147] Speaker encoder 650 receives a reference audio signal recording the speech of a selected target speaker and outputs a speaker embedding by encoding the reference audio signal. The speaker embedding may include the speaker's speech and / or phonological characteristics. Speaker encoder 650 can generate speaker embeddings with the same dimensions as language embeddings.

[0148] The random duration predictor 640 can predict the duration of individual phonemes in the input text based on text feature vectors and speaker embeddings, and can output phoneme duration data containing the durations of the phonemes. The phoneme duration data includes the predicted duration of each phoneme based on the speaker's speech and / or phonological characteristics. The phoneme duration data can be converted to integers and input into the alignment module 670. For example, a rounding function can be applied to the durations of the individual phonemes predicted by the random duration predictor 640, but it is not limited to this.

[0149] Projection module 660 can generate a distribution of text feature vectors. This distribution may include the mean and standard deviation of the text feature vectors. During this process, the text feature vectors can be transformed to match the dimensions of the alignment data from alignment module 670. The dimensions of the data representing the distribution may correspond to one of the dimensions of the alignment data.

[0150] Alignment module 670 can generate latent variables based on the distribution of text feature vectors and phoneme duration data. Latent variables can be generated from text feature vectors based on phoneme duration data. For example, alignment module 670 can use alignment data to calculate the mean and standard deviation of the text feature vectors corresponding to each phoneme, and output latent variables as the calculation results. Latent variables can include features of each phoneme of the input text and features related to the duration of each phoneme.

[0151] The inverse decoder 680 can generate transformed latent variables based on latent variables, language embeddings, and speaker embeddings. The inverse decoder 680 can condition latent variables with language embeddings and speaker embeddings, thereby transforming latent variables into variables with a prior distribution different from the original latent variables. During moderation, language embeddings and speaker embeddings can be added to or multiplied by the latent variables.

[0152] Because the inverse decoder 680 is trained with language normalization during the training phase to exclude language-related features, the language embeddings must be incorporated into the latent variables during the inference phase. To this end, at least some affine coupling layers in the inverse decoder 680 are used to denormalize the language embeddings. Therefore, the affine coupling layers used for denormalization can be called language denormalization affine coupling layers. By denormalizing the language embeddings, the features of the language embeddings can be reflected in the latent variables or in the latent variables adjusted according to the speaker embeddings. The following will refer to... Figure 8 The description language denormalizes the transformation of the affine coupling layer, i.e., the inverse transformation of the normalized affine coupling layer.

[0153] In some embodiments, speaker embeddings may be additionally considered during the denormalization of language embeddings. For example, speaker embeddings may be input into a neural network to generate scaling parameters and / or bias parameters. In another embodiment, at least a portion of the affine coupling layer of the inverse decoder 680 may be used independently for the application of speaker embeddings. The affine coupling layer may apply speaker embeddings as conditioning information for latent variables or intermediate computations of latent variables. For example, speaker embeddings may be added to or multiplied by latent variables.

[0154] The inverse decoder 680 can transform latent variables modulated by language embeddings and speaker embeddings. The inverse decoder 680 can transform the inverse function f of the normalized stream function used during the training phase. -1 The transformation is applied to latent variables to obtain transformed latent variables. Transformed latent variables can have simpler or more complex distributions than the adjusted latent variables. The inverse decoder 680 can transform the distribution of latent variables based on speaker embeddings and language embeddings. Transformed latent variables include features of the input text, features of language identifiers, features of the target speaker's reference audio signal, and duration features.

[0155] Audio generator 690 can generate an audio signal representing sound waves from transformed latent variables and speaker embeddings. The speaker embeddings can be incorporated into the transformed latent variables through adjustment. Audio generator 690 can generate the audio signal from the latent variables adjusted by the speaker embeddings. Specifically, audio generator 690 can predict the audio signal from the distribution of the adjusted latent variables. The generated audio signal can be identical or similar to an audio recording of the target speaker speaking the input text. Even if the target speaker is unfamiliar with the language of the input text, a result can be generated as if the target speaker had already spoken the input text in that language. Furthermore, even if the input text includes multiple languages, natural speech can be generated.

[0156] Figure 7 An example of a language normalization affine coupling layer according to an embodiment of the present disclosure is shown. Figure 8 An example of a language denormalization affine coupling layer according to an embodiment of the present disclosure is shown.

[0157] Language normalization (LN), used to remove linguistic information from latent variables, and language denormalization (LDN), used to reflect linguistic information in latent variables, can be defined by Equation 2 and Equation 3, respectively.

[0158] [Formula 2]

[0159]

[0160] [Formula 3]

[0161]

[0162] here, It can represent latent variables (or latent variables moderated by speaker embedding) that serve as moderation targets. Represents language embedding, and This represents the linear projection of the language embedding used to generate the mean and variance parameters.

[0163] Language normalization removes linguistic information from latent variables by subtracting the mean of the language embeddings and dividing the result by the variance of the language embeddings. On the other hand, language denormalization reflects linguistic information back into the latent variables by adding the mean of the language embeddings to the result of multiplying the latent variables by the variance of the language embeddings.

[0164] Language normalization and denormalization can be applied to certain dimensions of the input latent variables. For example, referencing... Figure 7 The language normalization affine coupling layer 70 generates output latent variables by applying language normalization to a portion of the input latent variables, applying affine transformations to the normalization results based on scaling and bias parameters, and combining the transformation results with the language normalization results for the remaining dimensions of the input latent variables. The forward transformation of the language normalization affine coupling layer 70 can be represented by Equation 4.

[0165] [Formula 4]

[0166]

[0167] In formula 4, and This represents the D-dimensional input and output latent variables. and This represents the function used to generate the scaling and bias parameters. For example, and This can be achieved using neural networks.

[0168] Affine coupling layers are readily invertible and possess a triangular Jacobian matrix. The determinant can be computed based on the Jacobian expression, from which the model density q can be readily calculated. For example, the inverse transformation of affine coupling layer 70, i.e., the language denormalization affine coupling layer 80, can be... Figure 8 As shown in Formula 5.

[0169] [Formula 5]

[0170]

[0171] Finally, the logarithmic determinant of the conditional flow can be expressed by formula 6.

[0172] [Formula 6]

[0173]

[0174] here, This represents a bijective function used to transform latent variables.

[0175] Next, the experimental results of implementing the speech synthesis model are described with reference to Tables 2 to 4.

[0176] Table 2 shows a performance comparison of speech synthesis models according to different embodiments, as well as speech synthesis models (MSVITS and SANE-TTS) according to comparative examples.

[0177] [Table 2]

[0178]

[0179] In this experiment, for intralingual synthesis, Korean speech was synthesized using Korean reference speech, and English speech was synthesized using English reference speech. For cross-lingual synthesis, English speech was synthesized using Korean reference speech, and Korean speech was synthesized using English reference speech. Furthermore, Mean Opinion Score (MOS) and Speaker Encoder Cosine Similarity (SECS) were used as performance metrics.

[0180] In Table 2, (i) shows the performance when both contrastive learning and language normalization affine coupling layers are applied; (ii) shows the performance when contrastive learning is omitted; and (iii) shows the performance when both contrastive learning and language normalization affine coupling layers are omitted. Table 2 confirms that the model employing both contrastive learning and language normalization affine coupling layers scores the highest in cross-linguistic synthesis environments.

[0181] Table 3 shows a comparison of the speech synthesis performance of models (i) to (iii). In this experiment, word error rate (WER) and character error rate (CER) were used as performance evaluation metrics.

[0182] [Table 3]

[0183]

[0184] As shown in Table 4, it can be confirmed that the model employing both contrastive learning and language normalization affine coupling layers exhibits the lowest error rate in cross-linguistic synthesis environments (KO2EN and EN2KO).

[0185] Table 4 shows the comparison results of cross-lingual speech synthesis performance obtained due to the type of metric learning used for contrastive learning. In this experiment, cosine distance, contrastive loss, and triplet loss were used as metric learning losses, and WER and CER were used as performance evaluation metrics.

[0186] [Table 4]

[0187]

[0188] As shown in Table 4, it can be confirmed that the lowest error rate was obtained when triplet loss was used as the metric for learning loss.

[0189] Figure 9 This is a flowchart illustrating a method for training a speech synthesis model according to an embodiment of the present disclosure.

[0190] The training device receives training text and training audio signals obtained from a predetermined speaker reciting the training text (S900). The training text may include a sequence of one or more characters.

[0191] The training device identifies language identifiers corresponding to the training text and obtains a training reference audio signal from a reference text spoken by a speaker that differs from the training text (S920). In some embodiments, the language identifiers may include a sequence of identifiers corresponding to each character in the training text. The language identifiers may be prepared in advance for each training text or generated in real time by the training device. In some embodiments, multiple audio signals may be prepared in advance for the speaker. The training device may identify one or more audio signals other than the training audio signal from the multiple audio signals and randomly select a training reference audio signal from the identified one or more audio signals.

[0192] The training device uses training samples, including training text, training audio signals, language identifiers, and training reference audio signals, to train the speech synthesis model (S940).

[0193] The speech synthesis model may include a language embedding module and / or a speaker encoder. The language embedding module may receive language identifiers as input and transform the received language identifiers into language embeddings. The speaker encoder may separately receive training audio signals and training reference audio signals and transform them into speaker embeddings. The training device may determine a loss based on a first speaker embedding transformed from the training reference audio signal, a second speaker embedding transformed from the training audio signal, and the language embeddings. The training device may use the determined loss to update the parameters of the speaker encoder.

[0194] In some embodiments, a speaker encoder can be trained to generate speaker embeddings that are independent of language-specific features of the training audio signal and the training reference audio signal. For example, the speaker encoder can be trained to embed the training audio signal and the training reference audio signal into a feature space in which the language embeddings are embedded. Additionally or alternatively, the speaker encoder can be trained to place the first speaker embedding and the language embedding close to each other in the corresponding feature space. Additionally or alternatively, the speaker encoder can be trained to place the first speaker embedding and the language embedding far apart. The loss for this is called the contrastive cross loss or metric learning loss. This loss can include a triplet loss, which uses, for example, the first speaker embedding, the second speaker embedding, and the language embedding as anchors, positive samples, and negative samples, respectively. In some embodiments, the contrastive cross loss or metric learning loss can constitute a portion of the total loss used to train the speech synthesis model. The training apparatus can use the total loss to update at least a portion of the speech synthesis model.

[0195] During the training phase, the speech synthesis model may also include a posterior encoder, decoder, character embedding module, encoder, projection module, aligned data estimator, random duration predictor and / or audio generator.

[0196] A posterior encoder can encode the spectrogram of a training audio signal into latent variables.

[0197] The decoder can output transformed latent variables based on latent variables, language embeddings, and first speaker embeddings. The decoder can be trained to remove language-specific features from the training audio signal by normalizing the latent variables based on language embeddings.

[0198] The character embedding module can transform training text into character embeddings.

[0199] An encoder can embed characters into text feature vectors.

[0200] The projection module can output the distribution of text feature vectors, where the distribution may include the mean and / or standard deviation.

[0201] Alignment data estimators can align data based on the distribution of text feature vectors and transformed latent variables.

[0202] A random duration predictor can be trained to predict the duration associated with speaker speech characteristics for each phoneme in the training text. The random duration predictor can be trained based on first speaker embeddings, text feature vectors, and alignment data.

[0203] An audio generator can generate synthetic audio signals corresponding to training text from latent variables.

[0204] The total loss used to train the speech synthesis model may also include the reconstruction loss between the training audio signal and the synthesized audio signal, the KL divergence loss calculated based on the text feature vectors and latent variables (or transformed latent variables), the duration loss calculated based on the phoneme duration predicted by the random duration predictor and the phoneme duration generated by the aligned data estimator, the adversarial loss of the audio generator, and / or the feature matching loss between the real audio signal and the synthesized audio signal.

[0205] As described above, the language embedding and the first speaker embedding can be provided as input to the sub-networks following the language embedding module and the speaker encoder (e.g., a posteriori encoder, decoder, and / or random duration predictor) to generate a synthetic audio signal corresponding to the training text. On the other hand, the second speaker embedding can be used only for loss calculation and may not be used directly to generate the synthetic audio signal.

[0206] Figure 10 This is a flowchart illustrating a speech synthesis method according to an embodiment of the present disclosure.

[0207] The speech synthesis device can receive a speech synthesis request containing input text and target speaker information (S1000). The input text may include a sequence of one or more characters.

[0208] The speech synthesis device can recognize language identifiers corresponding to the input text, as well as reference audio signals obtained from a target speaker uttering reference text different from the input text (S1020). Language identifiers may include sequences of language identifiers corresponding to individual characters within the training text. The speech synthesis request may also include language identifiers. For example, the speech synthesis device can recognize language identifiers from the speech synthesis request. Alternatively, the speech synthesis device can directly recognize (or detect) language identifiers from the received input text. The reference audio signal can be included in the speech synthesis request as target speaker information. For example, the speech synthesis device can recognize the reference audio signal from the speech synthesis request. Alternatively, audio signals obtained from multiple speakers can be pre-stored in the speech synthesis device and / or can be linked to an external data storage device. For example, the speech synthesis device receives an identifier uniquely assigned to the target speaker as target speaker information, and can use the received identifier to recognize the target speaker's reference audio signal from the pre-stored audio signals.

[0209] The speech synthesis device generates a synthesized audio signal that mimics the speech of a target speaker uttering the input text by applying the input text, language identifiers, and reference audio signals to a speech synthesis model (S1040).

[0210] A speech synthesis model may include a language embedding module and / or a speaker encoder. The language embedding module receives language identifiers and transforms them into a target language embedding. The speaker encoder receives a reference audio signal and transforms it into a target speaker embedding.

[0211] Speech synthesis models can already be passed Figure 4 or Figure 9 The process (or operation) described herein has been trained. For example, a speaker encoder can be trained using a loss determined based on speaker embeddings transformed from multiple training audio signals and language embeddings transformed from training text corresponding to one of the multiple training audio signals. The multiple training audio signals can be obtained from a common speaker. The speaker encoder can have been trained to generate speaker embeddings independent of the language-specific features of the training audio signals. The multiple training audio signals may include a first training audio signal obtained from a particular speaker speaking text different from the training text, and a second training audio signal obtained from the same speaker speaking the training text. The speaker encoder can have been trained to embed the first and second training audio signals into the same feature space as the language embeddings transformed from the training text. The speaker encoder can have been trained to place the first speaker embedding transformed from the first training audio signal and the second speaker embedding transformed from the second training audio signal close to each other in the feature space. Additionally or alternatively, the speaker encoder can have been trained to place the first speaker embedding and the language embedding far apart.

[0212] The speech synthesis model may also include a character embedding module, an encoder, a random duration predictor, a projection module, an alignment unit, an inversion decoder, and / or an audio generator.

[0213] The character embedding module can transform input text into character embeddings.

[0214] An encoder can embed characters into text feature vectors.

[0215] A random duration predictor can predict the duration of each phoneme in an input text. It can predict durations associated with the speech characteristics of a target speaker based on the text feature vector and the target speaker embedding. For example, even for the same phoneme, different durations can be predicted depending on the speaker.

[0216] The projection module can output the distribution of text feature vectors. This distribution can include the mean and standard deviation.

[0217] Alignment units can generate latent variables based on the distribution of text feature vectors and the predicted duration for each phoneme.

[0218] The inverse decoder can output transformed latent variables based on latent variables, target speaker embeddings, and target language embeddings. Specifically, the inverse decoder can adjust the latent variables according to the target speaker embeddings and target language embeddings, and output transformed latent variables based on the adjusted latent variables.

[0219] An audio generator can generate an audio signal from transformed latent variables. The audio generator can adjust the transformed latent variables based on the target speaker embedding and generate an audio signal from the adjusted transformed latent variables.

[0220] Figure 11 This is a schematic diagram of an exemplary configuration of a computing device that can be used to implement the apparatus and methods described herein.

[0221] The computing device 11 may include some or all of the following: memory 1100, processor 1120, storage device 1140, input / output (I / O) interface 1160, and communication interface 1180. The computing device 11 may structurally and / or functionally include at least a portion of a speech synthesis device or training device. The computing device 11 may be a fixed computing device, such as a desktop computer, server, or AI accelerator, or a mobile computing device, such as a laptop or smartphone. The computing device 11 may include any dedicated hardware accelerator capable of efficiently processing AI model computations. For example, the computing device 11 may include a graphics processing unit (GPU), a tensor processing unit (TPU), or a neural processing unit (NPU).

[0222] Memory 1100 may store programs that allow processor 1120 to perform methods or operations according to various embodiments of the present disclosure. For example, the program may include a plurality of instructions executable by processor 1120, and these instructions may be executed by processor 1120 to perform the methods or operations described above. Memory 1100 may be a single memory or multiple memories. In this case, information required to perform the methods or operations according to various embodiments of the present disclosure may be stored in a single memory or partitioned across multiple memories. When memory 1100 includes multiple memories, the multiple memories may be physically separate. Memory 1100 may include at least one of volatile memory or non-volatile memory. Volatile memory includes static random access memory (SRAM), dynamic random access memory (DRAM), etc., and non-volatile memory includes flash memory.

[0223] Processor 1120 may include at least one core capable of executing at least one instruction. Processor 1120 may execute instructions stored in memory 1100. Processor 1120 may be a single processor or multiple processors.

[0224] Even when the power to the computing device 11 is cut off, the storage device 1140 can retain the stored data. For example, the storage device 1140 may include non-volatile memory, or may include storage media such as magnetic tape, optical disc, or magnetic disk. Programs stored in the storage device 1140 can be loaded into the memory 1100 before being executed by the processor 1120. The storage device 1140 can store files created using a programming language, and programs created from files using a compiler or similar means can also be loaded into the memory 1100. The storage device 1140 can store data to be processed by the processor 1120 and / or data already processed by the processor 1120.

[0225] I / O interface 1160 can provide an interface for input devices (e.g., a keyboard or mouse) and / or output devices (e.g., a display device or printer). Users can trigger the execution of programs in processor 1120 through input devices and / or check the processing results of processor 1120 through output devices.

[0226] The communication interface 1180 can provide access to external networks. For example, the computing device 11 can communicate with another device (e.g., a vehicle or a speech synthesis device) through the communication interface 1180.

[0227] Various embodiments of the systems and techniques described herein can be implemented using digital electronic circuits, integrated circuits, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. Various embodiments may include implementation by one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a dedicated or general-purpose processor) coupled to receive and transmit data and instructions from a storage system, at least one input device, and at least one output device. The computer program (also referred to as a program, software, software application, or code) includes instructions for the programmable processor and is stored in a computer-readable recording medium.

[0228] Computer-readable recording media can include all types of storage devices capable of storing computer-readable data. Computer-readable recording media can be non-volatile or non-transitory media, such as read-only memory (ROM), random access memory (RAM), optical disc ROM (CD-ROM), magnetic tape, floppy disk, or optical data storage devices. Furthermore, computer-readable recording media can also include transient media, such as data transmission media. Moreover, computer-readable recording media can be distributed across computer systems connected via a network, and computer-readable program code can be distributed for storage and execution.

[0229] Although the operations shown in the flowcharts / sequence diagrams of this disclosure are presented as being performed sequentially, this is merely an illustrative description of the technical concept of this disclosure. In other words, those skilled in the art to which this disclosure pertains should understand that various modifications and alterations can be made without departing from the essential characteristics of this disclosure. For example, the order shown in the flowcharts / sequence diagrams can be changed, and one or more operations can be performed in parallel. Therefore, the flowcharts / sequence diagrams are not limited to a chronological order.

[0230] According to embodiments of this disclosure, even if the audio data of the target speaker in the target language is scarce or missing, natural speech that gives the impression that the target speaker is fluently speaking the text in the target language can be synthesized.

[0231] According to one embodiment of this disclosure, by using cross-training techniques and contrastive learning during the training phase of the speech synthesis model, the speech synthesis model can be controlled to separate information between input factors without overfitting the input factors.

[0232] According to one embodiment of this disclosure, by separating and explicitly controlling speaker-related features and language-related features present in a speech signal, the pronunciation clarity and sound quality of synthesized speech can be improved in the case of cross-language synthesis or code-mixed synthesis. For example, even if the text to be synthesized includes words from multiple languages, speech can be generated in which each word is pronounced with the correct intonation in the corresponding language.

[0233] According to one embodiment of this disclosure, language features can be effectively learned by utilizing language embedding-based normalization and denormalization during the training and inference phases of a speech synthesis model.

[0234] According to one embodiment of this disclosure, multilingual and multi-speaker speech synthesis can be achieved without complex fine-tuning. In other words, because no additional fine-tuning process is required after training, the overall learning time and cost can be reduced. Furthermore, because no separate module is needed, the size of the speech synthesis model does not increase. Therefore, the speech synthesis according to this disclosure can be widely used in various application environments that utilize limited memory resources, such as vehicle environments.

[0235] According to one embodiment of this disclosure, vehicle passengers can be provided with voice guidance composed of the speaker's voice and language synthesis, as desired by the passengers.

[0236] The features of this disclosure are not limited to those described above. Those skilled in the art to which this disclosure pertains may understand other features not mentioned herein based on the above description.

[0237] Although embodiments of this disclosure have been described for illustrative purposes, those skilled in the art will understand that various modifications, additions, and substitutions are possible without departing from the concept and scope of this disclosure. Therefore, embodiments of this disclosure have been described for the sake of brevity and clarity. The scope of the technical concept of this disclosure is not limited by the illustrations. Therefore, those skilled in the art will understand that the scope of this disclosure should not be limited by the embodiments explicitly described above, but rather by the appended claims and their equivalents.

Claims

1. A method comprising the steps of: receiving a training text and a training audio signal obtained from a speaker speaking the training text; identifying a language identifier corresponding to the training text, and a training reference audio signal obtained from the speaker speaking a reference text different from the training text; and training a speech synthesis model using training samples comprising the training text, the training audio signal, the language identifier, and the training reference audio signal, wherein the step of training the speech synthesis model comprises: transforming, by a language embedding module of the speech synthesis model, the language identifier into a language embedding; transforming, by a speaker encoder of the speech synthesis model, the training audio signal and the training reference audio signal into speaker embeddings; determining a loss based on a first speaker embedding transformed from the training reference audio signal, a second speaker embedding transformed from the training audio signal, and the language embedding; and updating parameters of the speaker encoder using the loss. The speaker encoder is trained to generate speaker embeddings that are independent of language-specific characteristics of the training audio signal and the training reference audio signal.

2. The method of claim 1, wherein, The speaker encoder is trained to:

3. The method of claim 1, wherein, embed the training audio signal and the training reference audio signal into a feature space in which the language embedding is embedded; place the first speaker embedding and the second speaker embedding close to each other in the feature space; and place the first speaker embedding and the language embedding far apart. The loss comprises a triplet loss using the first speaker embedding, the second speaker embedding, and the language embedding as an anchor, a positive sample, and a negative sample, respectively. The training text comprises a sequence of one or more characters, and 4. The method of claim 1, wherein, wherein the language identifier comprises a sequence of identifiers corresponding to individual characters within the training text.

5. The method of claim 1, wherein, The step of training the speech synthesis model further comprises: encoding, by a posterior encoder of the speech synthesis model, a spectrogram of the training audio signal into a latent variable; 6. The method of claim 1, wherein, outputting, by a decoder of the speech synthesis model, a transformed latent variable based on the latent variable, the language embedding, and the first speaker embedding; and wherein the decoder is trained to remove language-specific characteristics of the training audio signal by normalizing the latent variable based on the language embedding. The step of training the speech synthesis model further comprises: transforming, by a character embedding module of the speech synthesis model, the training text into a character embedding; 7. The method of claim 6, wherein, encoding, by an encoder of the speech synthesis model, the character embedding into a text feature vector; generating, by a projection module of the speech synthesis model, a distribution of the text feature vector; estimating, by an alignment data estimator of the speech synthesis model, alignment data based on the transformed latent variable and the distribution of the text feature vector; and ​ ​ predicting, by a random duration predictor of the speech synthesis model, a duration associated with speech characteristics of a speaker for each phoneme of the training text based on the first speaker embedding, the text feature vector, and the alignment data.

8. The method of claim 1, wherein, The step of identifying the language identifier comprises: identifying one or more audio signals from a plurality of audio signals obtained from the speaker other than the training audio signal; and randomly selecting the training reference audio signal from the one or more audio signals.

9. The method of claim 1, wherein, The step of training the speech synthesis model further comprises: providing the language embedding and the first speaker embedding as inputs to a subnetwork following the language embedding module and the speaker encoder to generate a synthesized audio signal corresponding to the training text, wherein the second speaker embedding is not used to generate the synthesized audio signal.

10. An apparatus comprising: a memory configured to store instructions; at least one processor configured to, by executing the instructions: receive a training text and a training audio signal obtained from a predetermined speaker speaking the training text; identify a language identifier corresponding to the training text, and a training reference audio signal obtained from the speaker speaking a reference text different from the training text; and train a speech synthesis model using training samples including the training text, the training audio signal, the language identifier, and the training reference audio signal, wherein the speech synthesis model comprises: a language embedding module configured to transform the language identifier into a language embedding; and a speaker encoder configured to transform the training audio signal and the training reference audio signal into speaker embeddings, and wherein the at least one processor is further configured to: determine a loss based on a first speaker embedding transformed from the training reference audio signal, a second speaker embedding transformed from the training audio signal, and the language embedding; and update parameters of the speaker encoder using the loss.

11. A method comprising the steps of: receiving a speech synthesis request including an input text and target speaker information; identifying a language identifier corresponding to the input text, and a reference audio signal obtained from the target speaker speaking a reference text different from the input text; and generating a synthesized audio signal mimicking speech of the target speaker speaking the input text by applying the input text, the language identifier, and the reference audio signal to a speech synthesis model; wherein the step of generating the synthesized audio signal comprises: transforming, by a language embedding module of the speech synthesis model, the language identifier into a target language embedding; and transforming, by a speaker encoder of the speech synthesis model, the reference audio signal into a target speaker embedding, and wherein the speaker encoder has been trained using a loss determined based on speaker embeddings transformed from a plurality of training audio signals from a speaker, and a language embedding transformed from a training text corresponding to one of the plurality of training audio signals.

12. The method of claim 11, wherein, The speaker encoder has been trained to generate speaker embeddings that are independent of language-specific characteristics of the training audio signals.

13. The method of claim 11, wherein, The plurality of training audio signals includes a first training audio signal corresponding to a reference text that is different from the training text and a second training audio signal corresponding to the training text, wherein the speaker encoder has been trained to: embed the first training audio signal and the second training audio signal into a feature space in which a language embedding transformed from the training text is embedded; in the feature space, place a first speaker embedding transformed from the first training audio signal and a second speaker embedding transformed from the second training audio signal close to each other; and place the first speaker embedding and the language embedding far apart.

14. The method of claim 13, wherein, The loss includes a triplet loss that uses the first speaker embedding, the second speaker embedding, and the language embedding of the training text as an anchor, a positive sample, and a negative sample, respectively.

15. The method of claim 11, wherein, The input text includes a sequence of one or more characters, and the language identifier includes a sequence of identifiers corresponding to individual characters within the input text.

16. The method of claim 11, wherein, The step of generating the synthesized audio signal further includes: outputting, by a reverse decoder of the speech synthesis model, a transformed latent variable based on a latent variable derived from the input text, the reference audio signal, the target language embedding, and the target speaker embedding; and generating, by an audio generator of the speech synthesis model, the synthesized audio signal from the transformed latent variable.

17. The method of claim 16, wherein, The step of outputting the transformed latent variable includes: adjusting, by the reverse decoder, the latent variable according to the target speaker embedding and the target language embedding; and outputting, by the reverse decoder, the transformed latent variable based on the adjusted latent variable.

18. The method of claim 16, wherein, The step of generating the synthesized audio signal from the transformed latent variable includes: adjusting, by the audio generator, the transformed latent variable according to the target speaker embedding; and generating, by the audio generator, the synthesized audio signal from the adjusted transformed latent variable.

19. The method of claim 16, wherein, The step of generating the synthesized audio signal further includes: transforming, by a character embedding module of the speech synthesis model, the input text into a character embedding; encoding, by an encoder of the speech synthesis model, the character embedding into a text feature vector; generating, by a projection module of the speech synthesis model, a distribution of the text feature vector; predicting, by a random duration predictor of the speech synthesis model, a duration associated with a speech characteristic of the target speaker for each phoneme of the input text based on the target speaker embedding and the text feature vector; and generating, by an alignment unit of the speech synthesis model, the latent variable based on the distribution of the text feature vector and the predicted duration for each phoneme.

20. An apparatus comprising: a memory configured to store instructions; and at least one processor configured to, upon execution of the instructions, perform the method of any of claims 11-19.

Images