Method for speech synthesis and apparatus therefor

Abstract

A method for speech synthesis and an apparatus therefor are disclosed. The method for speech synthesis, according to one embodiment, may comprise the steps of: generating, on the basis of pitch information associated with the input text, an excitation representation of target speech corresponding to an input text; refining the excitation representation through a diffusion model; generating a formant representation of the target speech on the basis of the input text regardless of the diffusion model; obtaining a spectrogram by aggregating the refined excitation representation and the formant representation; and synthesizing the target speech on the basis of the spectrogram. According to the method, both prosody expression and pronunciation accuracy of a target speech can be improved.

Description

Voice synthesis method and device thereof

[0001] The disclosed embodiments relate to speech synthesis technology.

[0002] Speech synthesis (or Text-to-Speech (TTS)) technology converts input text into spoken language. Currently, speech synthesis technology is being applied in diverse fields such as virtual assistants, audiobooks, automatic translation, and virtual voice actors. For natural-sounding speech synthesis, neural network-based methods are primarily utilized.

[0003] Recently, a method has been proposed to apply the diffusion model, which has demonstrated excellent performance in image synthesis, to speech synthesis. The diffusion model is a generative model that utilizes a forward process that gradually adds noise to data and a reverse process that gradually removes noise to generate data. The proposed method applies the diffusion model to generate a spectrogram.

[0004] However, the proposed method has a problem in that the diffusion model inaccurately generates or distorts (deforms) pronunciation-related parts within the spectrogram, which frequently causes serious pronunciation errors in the synthesized speech.

[0005] The technical problem to be solved through the disclosed embodiments is to provide a method and device capable of synthesizing high-quality speech from input text.

[0006] Specifically, the technical problem to be solved through the disclosed embodiments is to provide a speech synthesis method and device capable of improving the pronunciation accuracy of synthesized speech.

[0007] In addition, another technical problem to be solved through the disclosed embodiments is to provide a speech synthesis method and device capable of improving the prosodic expressiveness of synthesized speech.

[0008] In addition, another technical problem to be solved through the disclosed embodiments is to provide a method and device capable of accurately synthesizing the voice of a target speaker in a multi-speaker environment.

[0009] In addition, another technical challenge to be solved through the disclosed embodiments is to provide a method and device capable of accurately synthesizing even the voice of a new speaker that the model has never seen before during the learning (training) process.

[0010] The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art of the present disclosure from the description below.

[0011] A speech synthesis method according to one embodiment may include a step of generating an excitation representation of a target voice corresponding to an input text based on pitch information associated with the input text; a step of refining the excitation representation through a diffusion model; a step of generating a formant representation of the target voice based on the input text regardless of the diffusion model; a step of obtaining a spectrogram by aggregating the refined excitation representation and the formant representation; and a step of synthesizing the target voice based on the spectrogram.

[0012] The input text may include a phoneme sequence, and the step of generating an excitation representation of the target speech may include: encoding the input text to generate a phoneme embedding sequence; predicting pitch information based on the phoneme embedding sequence, wherein the pitch information includes a pitch value for each phoneme; predicting duration information based on the phoneme embedding sequence, wherein the duration information includes a duration value for each phoneme; and generating the excitation representation based on the pitch information and the duration information.

[0013] The step of generating the phoneme embedding sequence may include the step of encoding information of a target speaker to generate a speaker embedding; and the step of encoding the input text and the speaker embedding together to generate the phoneme embedding sequence.

[0014] The step of generating the excitation representation of the target speech may include the step of predicting energy information associated with the input text based on the input text; and the step of generating the excitation representation further based on the energy information.

[0015] The input text may include a phoneme sequence, and the step of generating a formant representation of the target speech may include: encoding the input text to generate a phoneme embedding sequence; predicting duration information based on the phoneme embedding sequence, wherein the duration information includes a duration value for each phoneme; and generating the formant representation based on the phoneme embedding sequence and the duration information.

[0016] The pitch information may not be used during the process of generating the above formant representation.

[0017] The step of generating the excitation representation of the target speech may include the step of encoding information of the target speaker to generate a speaker embedding; and the step of generating the excitation representation further based on the speaker embedding.

[0018] The information of the target speaker may include a voice sample of the target speaker or a spectrogram extracted from the voice sample.

[0019] The step of generating a formant representation of the target speech may include a step of encoding information of the target speaker to generate a speaker embedding; and a step of generating the formant representation further based on the speaker embedding.

[0020] According to one embodiment, a speech synthesis device includes one or more processors; and a memory storing a computer program executed by the one or more processors, wherein the computer program may include instructions for: generating an excitation representation of a target voice corresponding to an input text based on pitch information associated with the input text; refining the excitation representation through a diffusion model; generating a formant representation of the target voice based on the input text regardless of the diffusion model; obtaining a spectrogram by aggregating the refined excitation representation and the formant representation; and synthesizing the target voice based on the spectrogram.

[0021] The input text may include a phoneme sequence, and the operation of generating an excitation representation of the target speech may include: encoding the input text to generate a phoneme embedding sequence; predicting pitch information based on the phoneme embedding sequence, wherein the pitch information includes a pitch value for each phoneme; predicting duration information based on the phoneme embedding sequence, wherein the duration information includes a duration value for each phoneme; and generating the excitation representation based on the pitch information and the duration information.

[0022] The operation of generating the excitation representation of the target speech may include an operation of predicting energy information associated with the input text based on the input text; and an operation of generating the excitation representation further based on the energy information.

[0023] The input text may include a phoneme sequence, and the operation of generating a formant representation of the target speech may include: an operation of encoding the input text to generate a phoneme embedding sequence; an operation of predicting duration information based on the phoneme embedding sequence, the duration information including a duration value for each phoneme; and an operation of generating the formant representation based on the phoneme embedding sequence and the duration information.

[0024] The operation of generating the excitation representation of the target speech may include an operation of generating a speaker embedding by encoding information of the target speaker; and an operation of generating the excitation representation further based on the speaker embedding.

[0025] The operation of generating a formant representation of the target speech may include an operation of generating a speaker embedding by encoding information of the target speaker; and an operation of generating the formant representation further based on the speaker embedding.

[0026] According to the disclosed embodiments, a spectrogram of a target speech can be generated through a refinement (restoration) process based on a diffusion model. In this case, the prosodic (or pitch) information of the spectrogram is enhanced, and as a result, the prosodic expressiveness of the synthesized target speech can be significantly improved.

[0027] In addition, the first decoding pass related to prosody (or pitch) information and the second decoding pass related to formant (or pronunciation) information can be separated, and the diffusion model can be applied only to the first decoding pass. That is, the spectrogram of the target speech can be generated by aggregating the computational results of the first decoding pass (i.e., the refined excitation representation) and the computational results of the second decoding pass (i.e., the formant representation). In this case, pronunciation errors in the synthesized target speech due to the diffusion model can be effectively prevented. Furthermore, the advantages of the diffusion model (e.g., strong prosody expression ability) can be maintained.

[0028] Additionally, variance information, such as pitch, energy, and duration information, can be used as conditioning information for spectrogram generation. In this case, oversmoothing in the generated spectrogram can be easily prevented.

[0029] Furthermore, speaker embeddings generated by encoding information about the target speaker can be used as conditioning information for the decoder (or other modules). In this case, a target speech corresponding to the target speaker's voice can be accurately synthesized in a multi-speaker environment. Furthermore, the inventors' experimental results confirmed that when a spectrogram is generated using a decoder with a diffusion model and separate decoding passes, even speech from a target speaker unseen during the model training phase can be accurately synthesized.

[0030] The effects according to the embodiments of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0031] FIG. 1 is an exemplary diagram for explaining the operation of a speech synthesis device according to one embodiment at a system level.

[0032] FIG. 2 is an exemplary drawing for further explaining the operation of a speech synthesis device according to one embodiment.

[0033] FIGS. 3 and 4 are exemplary drawings for explaining the structure and operating principle of a voice synthesis model according to one embodiment.

[0034] Figure 5 is an exemplary diagram illustrating a problem when a diffusion model is applied to a decoder having a single decoding pass.

[0035] FIG. 6 is an exemplary diagram for explaining the operating principle of a diffusion model according to one embodiment.

[0036] Fig. 7 is an exemplary flowchart illustrating a voice synthesis method according to one embodiment.

[0037] Fig. 8 is an exemplary drawing for explaining the structure and operating principle of a voice synthesis model according to another embodiment.

[0038] Fig. 9 is an exemplary flowchart illustrating a voice synthesis method according to another embodiment.

[0039] FIG. 10 is an exemplary block diagram illustrating a computing device capable of implementing a voice synthesis device according to one embodiment and a computing environment including the same.

[0040] The following detailed description is provided to facilitate a comprehensive understanding of the methods, devices, and / or systems described herein. However, these are merely examples and the technical scope of the present disclosure is not limited thereto.

[0041] In describing embodiments of the present disclosure, if a detailed description of a known technology related to the present disclosure is judged to unnecessarily obscure the gist of the present disclosure, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of their functions in the present disclosure, and these may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification. The terminology used in the detailed description is only for the purpose of describing embodiments of the present disclosure and should not be limited in any way. Unless clearly used otherwise, the singular form includes the plural form. In this specification, expressions such as "comprises" or "having" are intended to indicate certain features, numbers, steps, operations, elements, parts, or combinations thereof, and should not be construed to exclude the presence or possibility of one or more other features, numbers, steps, operations, elements, parts, or combinations thereof other than those described.

[0042] Hereinafter, specific embodiments of the present disclosure will be described with reference to the attached drawings.

[0043] FIG. 1 is an exemplary drawing for explaining the operation of a voice synthesis device (100) according to one embodiment at a system level.

[0044] As illustrated in FIG. 1, a speech synthesis device (100) is a computing device capable of synthesizing and outputting (providing) a voice (130, e.g., a voice signal) corresponding to an input text (120). For example, the speech synthesis device (100) can synthesize a voice (130) corresponding to an input text (120) using a speech synthesis model (110).

[0045] The text (120) may be a text composed of a phoneme sequence or may be general text. Here, a phoneme sequence means one or more phonemes arranged in order. If the text (120) is general text, the speech synthesis device (100) may perform preprocessing to convert the text (120) into a phoneme sequence.

[0046] The speech synthesis model (110) is based on a diffusion model and is designed to have a structure that can simultaneously guarantee strong prosodic expressivity and high pronunciation accuracy. Specifically, the speech synthesis model (110) has a structure in which the prosodic (or pitch) related decoding pass and the formant (or pronunciation) related decoding pass are separated from each other, and the diffusion model is applied only to the prosodic (or pitch) related decoding pass. Through this structure, strong prosodic expressivity and high pronunciation accuracy can be simultaneously guaranteed, which will be described in more detail later with reference to FIGS. 3 and 4.

[0047] Additionally, the speech synthesis model (110) may be a model for a single speaker or a model for multiple speakers. Depending on the intended use of the model, the input, structure, and / or operation of the speech synthesis model (110) may vary, which will be described in detail later with reference to the drawings in FIG. 3 and below.

[0048] Additionally, the voice synthesis model (110) may be located inside or outside the voice synthesis device (100). If the voice synthesis model (110) is located inside the voice synthesis device (100), the components of the voice synthesis model (110) may be considered as components of the voice synthesis device (100).

[0049] The speech synthesis device (100) can provide speech synthesis functionality in a single speaker and / or multi-speaker environment.

[0050] In one embodiment, the speech synthesis device (100) may provide speech synthesis functionality in a single-speaker environment. For example, the speech synthesis device (100) may convert text (e.g., 120) into the speech (e.g., 130) of a specific speaker using a speech synthesis model (110) specialized for that speaker. For further details on this embodiment, please refer to the descriptions of FIGS. 3 to 7.

[0051] In another embodiment, the speech synthesis device (100) may provide a speech synthesis function in a multi-speaker environment. For example, as illustrated in FIG. 2, the speech synthesis device (100) may input text (220) and information (210) of a speaker (A) into a speech synthesis model (110) to synthesize a speech (230) corresponding to the text (220) using the voice of the speaker (A). In addition, the speech synthesis device (100) may input text (220) and information (240) of a speaker (B) into a speech synthesis model (110) to synthesize a speech (250) corresponding to the text (220) using the voice of the speaker (B). For these embodiments, further reference is made to the descriptions of FIGS. 8 and 9.

[0052] In the preceding embodiments, speaker information (e.g., 210) may include various information related to the speaker's characteristics / style without limitation. Examples of such information may include a speaker's voice sample, a spectrogram extracted from the voice sample (e.g., a mel-spectrogram), a speaker identifier (e.g., one-hot codes, etc.), and other information (e.g., descriptions of language, tone, speech rate, stress, intonation, emotion, speech style, etc.), but the scope of the present disclosure is not limited thereto.

[0053] In another embodiment, the speech synthesis device (100) may provide speech synthesis functions based on various combinations of the above-described embodiments. For example, the speech synthesis device (100) may provide speech synthesis functions for a specific speaker using a first speech synthesis model specialized for that speaker, and simultaneously provide speech synthesis functions for multiple speakers using a second speech synthesis model.

[0054] Meanwhile, in some cases, the voice synthesis device (100) may provide a voice synthesis service to multiple users. For example, the voice synthesis device (100) may receive a service request including a user's voice sample (or other speaker information) and text from a user terminal, and may synthesize and provide the user's voice corresponding to the text using the received voice sample.

[0055] The above-described speech synthesis device (100) may be implemented by at least one computing device. For example, all functions of the speech synthesis device (100) may be implemented by a single computing device, or a first function of the speech synthesis device (100) may be implemented by a first computing device and a second function may be implemented by a second computing device. Alternatively, specific functions of the speech synthesis device (100) may be implemented by multiple computing devices.

[0056] The computing device may include, without limitation, any device having computing (processing) capabilities, and for an example of such a device, see FIG. 10.

[0057] Note that a computing device is a collection of interacting components (e.g., memory, processor, etc.), and thus may sometimes be referred to as a "computing system." Of course, the term "computing system" can also encompass the concept of a collection of interacting computing devices.

[0058] So far, the operation of a speech synthesis device (100) according to one embodiment has been roughly described with reference to FIGS. 1 and 2. Hereinafter, embodiments related to a single speaker environment will be described with reference to FIGS. 3 to 7, and for the clarity of the present disclosure, the reference number '110' of the speech synthesis model will be changed and indicated according to the embodiment.

[0059] First, an exemplary speech synthesis model (110-1) that can be used in a single speaker environment will be described with reference to FIGS. 3 to 6.

[0060] Figures 3 and 4 are exemplary drawings for explaining the structure (architecture) and operating principles of a speech synthesis model (110-1) according to one embodiment. Figure 4 illustrates the structure of the speech synthesis model (110-1) in more detail (however, the illustration of the vocoder (340) is omitted in Figure 4).

[0061] As illustrated in FIG. 3, a speech synthesis model (110-1) according to an embodiment may be configured to include an encoder (310), a distributed information predictor (320), a decoder (330), and a vocoder (340). However, in some cases, the vocoder (340) may not be considered a component of the speech synthesis model (110-1).

[0062] The encoder (310) is a module that encodes the input text (350) to generate a text embedding (e.g., an embedding vector). For example, the encoder (310) may be configured to input a text (350) composed of a phoneme sequence and output a phoneme embedding sequence (i.e., an embedding sequence in phoneme units). The phoneme sequence (350) may be input to the encoder (310) in, for example, a numerical form (i.e., an embedded form). In addition, as illustrated in FIG. 4, positional encoding information may be further input to the encoder (310) to transmit positional information of each phoneme constituting the phoneme sequence (350) to the encoder (310). Those skilled in the art will likely be familiar with the positional encoding technique, and therefore, a description thereof will be omitted. For the convenience of understanding, Figure 3 also shows a phoneme sequence (360) corresponding to “hello, world!”

[0063] The encoder (310) may be implemented (configured) based on various types / forms of neural networks. For example, the encoder (310) may be implemented based on a neural network such as a transformer, but the scope of the present disclosure is not limited thereto.

[0064] Since the encoder (310) is a module that encodes / embeds text, it may be named as ‘text encoder’ or ‘text embedder’ in some cases.

[0065] Next, the variance information predictor (320) is a module that predicts various variance information associated with the input text (350) (or target speech). Here, variance information refers to information about elements that represent the diversity and variability of speech, and examples of such elements include pitch, energy, and duration. However, the scope of the present disclosure is not limited thereto. The predicted variance information can be used by the decoder (330) to generate a spectrogram (i.e., the variance information is used as conditioning information for generating the spectrogram). In this case, the over-smoothing phenomenon in the generated spectrogram can be easily prevented.

[0066] As illustrated, the distributed information predictor (320) may be configured to include a pitch predictor (321), an energy predictor (322), and a duration predictor (323). However, in some cases, the distributed information predictor (320) may be configured to include other modules or may be configured with some modules omitted. For example, the distributed information predictor (320) may be configured with the energy predictor (322) omitted. Hereinafter, each component of the distributed information predictor (320) will be described.

[0067] The pitch predictor (321) is a module that predicts pitch information based on text embedding. For example, the pitch predictor (321) can be configured to input a phoneme embedding sequence and output (predict) a pitch value for each phoneme.

[0068] The pitch predictor (321) can be implemented (configured) based on, for example, various types / forms of neural networks, and can be trained based on pitch loss, which indicates the difference between predicted pitch information and correct pitch information. For more information, please refer to the explanation related to Mathematical Formula 1 below.

[0069] Next, the energy predictor (322) is a module that predicts energy information based on text embedding. For example, the energy predictor (322) may be configured to input a phoneme embedding sequence and output (predict) an energy value for each phoneme.

[0070] The energy predictor (322) can also be implemented (configured) based on, for example, various types / forms of neural networks, and can be trained based on energy loss, which represents the difference between predicted energy information and correct energy information. Further reference is made to the explanation related to Mathematical Expression 1 below.

[0071] Next, the duration predictor (323) is a module that predicts duration information based on text embedding. For example, the duration predictor (323) can be configured to input a phoneme embedding sequence and output (predict) a duration value for each phoneme. And, as illustrated in FIG. 4, the predicted duration information (e.g., duration value for each phoneme) can be used by the length regulators (410, 420) to adjust the temporal length of the spectrogram (or speech). For example, the length regulator (410) can adjust the temporal length of the spectrogram corresponding to each phoneme using the duration value of each phoneme. For reference, the length regulators (410, 420) may refer to the same module or may refer to two different modules.

[0072] The duration predictor (323) can also be implemented (configured) based on, for example, various types / forms of neural networks, and can be trained based on duration loss, which represents the difference between predicted duration information and correct duration information. Further reference is made to the explanation related to Mathematical Expression 1 below.

[0073] Next, the decoder (330) is a module that decodes the input text embedding and variance information to generate a spectrogram (e.g., mel-spectrogram). For example, the decoder (330) may be configured to receive a phoneme embedding sequence and variance information and output (generate) a spectrogram (450 in FIG. 4) of a target speech. At this time, the decoder (330) may generate a spectrogram (450) through a refinement (restoration) process based on a diffusion model (332). By doing so, the quality of the spectrogram (450) may be improved and the prosodic expressiveness of the target speech may be strengthened. The diffusion model (332) is a model that utilizes a forward process and a reverse process, which will be described in more detail later with reference to FIG. 6.

[0074] Since the decoder (330) is a module that generates a spectrogram, it may be named as a ‘spectrogram decoder’ or a ‘spectrogram generator’ in some cases.

[0075] As illustrated in FIG. 3, the decoder (330) may be configured to perform a decoding operation according to two separate decoding passes (334, 335) and aggregate (e.g., add) the operation results of each decoding pass (334, 335) to generate a spectrogram (450 of FIG. 4) of a target voice. Here, the first decoding pass (334) is a pass that performs a decoding operation related to prosody (or pitch) information with the support of the diffusion model (332), and the second decoding pass (335) may be understood as a pass that performs a decoding operation related to formant (or pronunciation) information regardless of the diffusion model (332).

[0076] For ease of understanding, the reason why the decoder (330) is designed to have separate decoding passes (334, 335) will be briefly explained with reference to FIG. 5.

[0077] Fig. 5 is an exemplary diagram illustrating the problems encountered when applying a diffusion model to a decoder with a single decoding pass. The mel-spectrogram (530) depicted at the bottom of Fig. 5 is the result of performing a reverse process of approximately '100 times' on the mel-spectrogram (510) at the top using the diffusion model.

[0078] Referring to Fig. 5, it can be confirmed that the magnitude is high only up to a frequency band of about 1000 Hz in the part (520) related to the pronunciation of 'ㅓ' in the upper mel-spectrogram (510) (see 'band shape'). On the other hand, it can be confirmed that the magnitude is high up to a frequency band of about 1600 Hz in the same time zone part (540) of the lower mel-spectrogram (530). This means that the formant information of the part (520) related to the pronunciation of 'ㅓ' has been transformed through the diffusion model (refer to the fact that the formant frequency F2 of the pronunciation of 'ㅏ' is about 1610 Hz), and shows that the diffusion model can cause pronunciation errors in synthesized speech. For reference, if an oversmoothing phenomenon exists in the mel-spectrogram (510), this pronunciation error problem can become more serious.

[0079] It can be understood that the decoder (330) is designed to have separate decoding passes (334, 335) to solve the pronunciation error problem described above. In other words, by separating the first decoding pass (334) related to prosody (or pitch) information and the second decoding pass (335) related to formant (or pronunciation) information, and applying the diffusion model (332) only to the first decoding pass (334), the diffusion model (332) can be prevented from modifying the formant information of the target speech. As a result, the pronunciation accuracy of the synthesized speech is improved, and the advantages of the diffusion model (332) (e.g., strong prosody expressiveness) can be maintained.

[0080] Referring again to FIG. 3, etc., the explanation of the decoder (330) will continue.

[0081] The decoder (330) may be configured to include a representation generator (331), a diffusion model (332), and a formant representation generator (333). Hereinafter, each component of the decoder (330) will be described.

[0082] Here, the excitation representation generator (331) is a module located in the first decoding pass (334) that generates an excitation representation (420, FIG. 4) including prosody (or pitch) information. For example, the excitation representation generator (331) may be configured to receive text embedding and variance information and output (generate) an excitation representation (430) in the form of a spectrogram. As a more specific example, as illustrated in FIG. 4, the excitation representation generator (331) may comprehensively analyze a phoneme embedding sequence, its positional encoding information, pitch information (e.g., pitch value per phoneme), energy information (e.g., energy value per phoneme), and duration information (e.g., duration value per phoneme) to generate an excitation representation (430) including prosody (or pitch) information. As described above, variance information such as pitch information and energy information may play a role in preventing an oversmoothing phenomenon from occurring in the excitation representation (430) in the form of a spectrogram.

[0083] For reference, the reason why the operation result (e.g., 430 in FIG. 4) of the first decoding pass (334) is named as 'here expression' can be understood as because the operation result (e.g., 430) acts as a source in the source-filter theory, and the formant representation (450) generated in the second decoding pass (335) acts as a filter in the source-filter theory. Those skilled in the art will already be familiar with the source-filter theory, so an explanation thereof will be omitted.

[0084] The expression here (430 in FIG. 4) is a spectrogram-type expression that includes meter (or pitch) information and is intermediate information used to generate the final spectrogram (460 in FIG. 4), and therefore may be named as such, such as "meter (or pitch) expression," "intermediate spectrogram," or "intermediate expression," depending on the case. Furthermore, since the expression here (430) serves as a source in the source-filter theory, it may also be named as such, such as "source expression."

[0085] Next, the diffusion model (332) is a module that refines the excitation representation (430 in FIG. 4) in the form of a spectrogram. For example, as illustrated in FIG. 4, the diffusion model (332) can generate a refined excitation representation (440) by repeatedly performing a reverse process on the excitation representation (430) (e.g., the excitation representation (430) with added noise), and through this refinement process, a spectrogram-form excitation representation (440) with enhanced prosody (or pitch) information can be generated. For easier understanding, a further explanation is provided with reference to FIG. 6.

[0086] As illustrated in Fig. 6, the diffusion model (332) is a model that utilizes a forward process (610) that gradually adds noise to a spectrogram (refer to 'X0') and a reverse process (620) that gradually removes noise to refine (restore) the spectrogram. For example, the diffusion model (332) removes noise ('X') of a prior distribution (e.g., Gaussian distribution) through the forward process (610). T 'Reference) and the spectrogram ('X0' reference) can be learned, and the spectrogram can be refined (restored) by performing a reverse process (620, i.e., denoising process) based on the learning result.

[0087] The diffusion model (332) may be, for example, a score-based diffusion model utilizing a neural network-based score predictor, but the scope of the present disclosure is not limited thereto. Those skilled in the art will likely already be familiar with score-based diffusion models and their operating principles, and therefore a detailed description thereof will be omitted.

[0088] This is explained again with reference to Figure 3, etc.

[0089] The formant representation generator (333) is a module located in the second decoding pass (335) that generates a formant representation (440, formant representation in FIG. 4) that includes formant (or pronunciation) information. For example, the formant representation generator (333) may be configured to receive a text embedding as input and output (generate) a formant representation (440) in the form of a spectrogram. As a more specific example, as illustrated in FIG. 4, the formant representation generator (333) may analyze a phoneme embedding sequence, its positional encoding information, and duration information (e.g., duration value per phoneme) to generate a formant representation (440) that includes formant information.

[0090] The formant representation (440 in Fig. 4) is a spectrogram-type representation that includes formant (or pronunciation) information and is intermediate information used to generate the final spectrogram (450 in Fig. 4), and therefore may be named as a "pronunciation representation," "intermediate spectrogram," or "intermediate representation" depending on the case. In addition, the formant representation (420) plays a filter role in the source-filter theory, and therefore may be named as a "filter representation."

[0091] Here, the expression generator (331) and the formant expression generator (333) can be implemented (configured) based on, for example, various types / forms of neural networks and can be trained based on reconstruction loss related to spectrograms, etc. For this purpose, please refer to the explanation related to mathematical expression 1 below.

[0092] For reference, it can be understood that the expression generator (331) here naturally generates a spectrogram expression (420 in FIG. 4) that mainly includes prosody (or pitch) information, as it is trained to generate a spectrogram expression based on input pitch information (e.g., trained based on a restoration loss). In addition, it can be understood that the formant expression generator (333) naturally generates a spectrogram expression (440 in FIG. 4) that mainly includes formant (or pronunciation) information, as it is trained to generate a spectrogram expression without using pitch information.

[0093] Finally, the vocoder (340) is a module that analyzes the input spectrogram (450 in FIG. 4) and synthesizes (generates) a target speech (370, e.g., a speech signal). The vocoder (340) may be implemented based on a neural network or may be implemented in another manner. As described above, the vocoder (340) may or may not be considered a component of the speech synthesis model (110-1).

[0094] The above-described speech synthesis model (110-1) can be trained, for example, using a single speaker's dataset (e.g., a dataset consisting of text and correct speech / spectrogram). Furthermore, the above-described speech synthesis model (110-1) can be trained, for example, based on a loss according to the following mathematical expression 1.

[0095] [Mathematical Formula 1]

[0096]

[0097] In mathematical expression 1, Ltotal means total loss, L d stands for duration loss, L p means pitch loss. Also, L e means energy loss, L align refers to the alignment loss between the input text and the spectrogram (or speech) generated by the decoder (330), and L prior means the restoration loss for the spectrogram. Finally, L diff refers to the loss (i.e., diffusion loss) for the diffusion model (332).

[0098] For reference, the duration loss can be calculated based on the difference between the predicted duration information and the correct duration information (e.g., the duration information extracted from the correct speech / spectrogram), and the pitch loss and energy loss can be calculated in a similar manner. In addition, the restoration loss can be calculated based on the difference between the generated spectrogram (e.g., 450) and the correct spectrogram, or can be calculated based on the difference between the spectrogram generated by aggregating (e.g., adding) the excitation representation (e.g., 420) and the formant representation (e.g., 440) before refinement and the correct spectrogram. In addition, the calculation method of the alignment loss can be defined in various ways, and the calculation method of the diffusion loss can also be defined in various ways depending on the type of the diffusion model (332).

[0099] So far, the structure and operating principles of a voice synthesis model (110-1) according to one embodiment have been described with reference to FIGS. 3 to 6. Hereinafter, a voice synthesis method according to one embodiment will be described with reference to FIG. 7.

[0100] Hereinafter, for the convenience of understanding, the explanation will continue assuming that all steps / operations of the speech synthesis method to be described later are performed on the speech synthesis device (100) described above. Therefore, if the subject of a specific step / operation is omitted, it can be understood that the step / operation is performed on the speech synthesis device (100). However, in an actual environment, some steps / operations of the speech synthesis method to be described later may be performed on another computing device. For example, learning (training) of the speech synthesis model (110) may be performed on another computing device.

[0101] In addition, for convenience of explanation, the voice synthesis device (100) is abbreviated as 'device (100)' below.

[0102] Figure 7 is an exemplary flowchart illustrating a speech synthesis method according to one embodiment. However, this is merely a preferred embodiment for achieving the objectives of the present disclosure, and it is to be understood that some steps may be added or deleted as necessary. Figure 7 assumes that the speech synthesis model (110-1) illustrated in Figure 3 and others is used, and illustrates the process of synthesizing the speech of a specific speaker using the speech synthesis model (110-1).

[0103] As illustrated in FIG. 7, the present embodiment may begin at step S710, where text (i.e., text corresponding to a target speech) is input. For example, the device (100) may input text composed of a phoneme sequence. If general text is input, the device (100) may perform preprocessing to convert the input general text into a phoneme sequence (i.e., text in phoneme units).

[0104] In step S720, a text embedding is generated by encoding the input text. For example, the device (100) can generate a phoneme embedding sequence by encoding an input text composed of a phoneme sequence through an encoder (310).

[0105] In step S730, variance information including pitch information is predicted based on text embedding. For example, the device (100) can predict a pitch value for each phoneme from a phoneme embedding sequence through a pitch predictor (321), and can predict a duration value for each phoneme from a phoneme embedding sequence through a duration predictor (323). This variance information can be used as conditioning information for the decoder (330), thereby effectively preventing oversmoothing from occurring in the spectrogram (or expression here) generated by the decoder (330).

[0106] In one embodiment, the device (100) can further predict energy information based on text embedding. For example, the device (100) can predict phoneme-specific energy values ​​from a phoneme embedding sequence via an energy predictor (322). According to this embodiment, by further utilizing the energy information as conditioning information for the decoder (330), the spectrogram (or excitation representation) of the target speech can be generated more accurately.

[0107] In step S740, an excitation representation of the target speech is generated based on text embedding and pitch information. For example, the device (100) can generate an excitation representation in the form of a spectrogram by comprehensively analyzing a phoneme embedding sequence, pitch information (e.g., pitch value per phoneme), and duration information (e.g., duration value per phoneme) through an excitation representation generator (331).

[0108] In one embodiment, the device (100) can generate an excitation representation of the target voice based more on energy information. For example, the device (100) can generate an excitation representation of the target voice by further analyzing energy information (e.g., phoneme-specific energy values) through the excitation representation generator (331). In this case, the excitation representation of the target voice can be generated more accurately.

[0109] In step S750, the excitation expression is refined through a diffusion model (332). For example, the device (100) can refine the excitation expression by adding noise to the spectrogram-shaped excitation expression and repeatedly performing a reverse process through the diffusion model (332). In this case, the prosodic information within the excitation expression is further strengthened, ultimately significantly improving the prosodic expressiveness of the target speech.

[0110] In step S760, a formant representation of the target speech is generated based on text embedding. For example, the device (100) can analyze a phoneme embedding sequence and duration information (e.g., duration value for each phoneme) through a formant representation generator (333) to generate a formant representation in the form of a spectrogram. Since this formant representation is generated independently of the diffusion model (332), the formant (or pronunciation) information can be prevented from being modified by the diffusion model (332), and as a result, the formant information can be accurately reflected in the spectrogram of the target speech.

[0111] In step S770, the refined excitation representation and the formant representation are aggregated to obtain (generate) a spectrogram (e.g., a mel spectrogram) of the target speech. This aggregation process can be understood as generating the spectrogram of the target speech according to the source-filter theory (i.e., the refined excitation representation can be understood as acting as a source and the formant representation as acting as a filter).

[0112] For example, the device (10) can generate a spectrogram (e.g., a Mel spectrogram) of a target voice by aggregating the refined excitation representation and the formant representation in the form of a spectrogram through a sum operation. However, the scope of the present disclosure is not limited thereto, and the aggregation method may vary depending on the embodiment.

[0113] In step S780, a target voice (e.g., a voice signal) is synthesized based on the spectrogram. For example, the device (100) can analyze the spectrogram through a vocoder (340) to synthesize a target voice (i.e., a specific speaker's voice corresponding to the input text).

[0114] The above-described step S780 may be omitted in the step of learning (training) the speech synthesis model (110-1). For example, in the model learning step, the device (100) may postpone the execution of step S780 and calculate a loss (e.g., refer to mathematical expression 1) based on the spectrogram generated in step S770 and the correct spectrogram. Then, the device (100) may update the parameters of the speech synthesis model (110-1) (e.g., weight parameters of modules implemented as a neural network) based on the calculated loss. With regard to the training of the speech synthesis model (110-1), further reference may be made to the description of FIGS. 3 to 6 and mathematical expression 1.

[0115] So far, a speech synthesis method according to one embodiment has been described with reference to FIG. 7. As described above, a spectrogram of a target speech can be generated through a refinement (restoration) process based on a diffusion model (332). In this case, the prosodic (or pitch) information of the spectrogram is enhanced, and as a result, the prosodic expressiveness of the synthesized target speech can be significantly improved.

[0116] Additionally, a spectrogram of a target speech can be generated by aggregating the refined excitation representation through the diffusion model (332) and the formant representation generated independently of the diffusion model (332). In this case, pronunciation errors in the synthesized target speech due to the diffusion model (332) can be effectively prevented. Furthermore, the advantages of the diffusion model (332) (e.g., strong prosodic expressiveness) can be maintained.

[0117] Additionally, distributed information, such as pitch information, energy information, and duration information, can be used as conditioning information for generating spectrograms. In this case, oversmoothing in the generated spectrogram can be easily prevented.

[0118] So far, embodiments related to a single-speaker environment have been described with reference to FIGS. 3 through 7. Below, embodiments related to a multi-speaker environment will be described with reference to FIGS. 8 and 9. However, for the clarity of the present disclosure, descriptions of content that overlaps with previous embodiments will be omitted.

[0119] First, with reference to Fig. 8, an exemplary speech synthesis model (110-2) that can be used in a multi-speaker environment will be described.

[0120] Fig. 8 is an exemplary diagram illustrating the structure (architecture) and operating principles of a speech synthesis model (110-2) according to another embodiment. For ease of understanding, Fig. 8 also illustrates a phoneme sequence (830) corresponding to "hello, world!"

[0121] As illustrated in FIG. 8, the speech synthesis model (110-2) according to the embodiment may be configured to further include a speaker encoder (810) in addition to the encoder (310), the distributed information predictor (320), the decoder (330), and the vocoder (340).

[0122] The speaker encoder (810) is a module that encodes the input target speaker information (840) to generate a speaker embedding (850, e.g., embedding vector). FIG. 8 illustrates an example in which the target speaker information (840) is a spectrogram (e.g., mel-spectrogram) extracted from a speech sample of the target speaker, but the scope of the present disclosure is not limited thereto. As described above, the target speaker information (840) may include various information related to the characteristics / style of the target speaker, such as a speech sample, an identifier (e.g., one-hot code, etc.), a speaker identifier (e.g., one-hot code, etc.), and other information (e.g., descriptions of language, tone, speech rate, stress, intonation, emotion, speech style, etc.) without limitation.

[0123] The speaker encoder (810) can be implemented (configured) based on various types / forms of neural networks, and its specific form can vary depending on the input speaker information.

[0124] The speaker encoder (810) can be viewed as a module that extracts the target speaker's speech style information from the target speaker's information (e.g., when generating a speaker embedding from a spectrogram or voice sample), and thus may be named as a "style encoder" or a "style extractor" depending on the case. Furthermore, the speaker embedding (850) may be named as a "style embedding", a "style vector", a "speaker feature", or a "style feature" depending on the case.

[0125] As illustrated, the speaker embedding (850) is used as conditioning information for other modules (e.g., 330). By doing so, the decoder (330) generates a spectrogram reflecting the characteristics / style of the target speaker, and as a result, the vocoder (340) can synthesize a voice (860) corresponding to the input text (820) in the voice of the target speaker. However, the types and number of modules that use the speaker embedding (850) as conditioning information can be designed in various ways.

[0126] For example, the speaker embedding (850) can be used as conditioning information for the decoder (330). For example, the speaker embedding (850) can be input to the excitation representation generator (331), the formant representation generator (333), and / or the diffusion model (332) to provide information about the target speaker. In this case, the excitation representation generator (331), the formant representation generator (333), and / or the diffusion model (332) perform related operations by further considering the information about the target speaker.

[0127] As another example, speaker embedding (850) can be used as conditioning information for the encoder (310). In this case, the encoder (310) encodes the input text (820) and speaker embedding (850) together to generate a text embedding (e.g., phoneme embedding sequence).

[0128] As another example, the speaker embedding (850) can be used as conditioning information for the variance information predictor (310). For example, the speaker embedding (850) can be input to the pitch predictor (321), the energy predictor (322), and / or the duration predictor (332). In this case, the pitch predictor (321), the energy predictor (322), and / or the duration predictor (332) take into account more information about the target speaker to predict the variance information.

[0129] As another example, the type and number of modules that use speaker embeddings (850) as conditioning information may be determined based on various combinations of the examples described above.

[0130] For other components (310 to 340) of the voice synthesis model (110-2), refer to the descriptions in FIGS. 3 and 4.

[0131] The above-described speech synthesis model (110-2) can be trained, for example, using a multi-speaker dataset (e.g., a dataset consisting of texts and correct speech / spectrograms from multiple speakers). Furthermore, the speech synthesis model (110-2) can be trained, for example, based on a loss according to mathematical equation 1; for this, please refer to the description of the speech synthesis model (110-1).

[0132] So far, the structure and operating principles of a voice synthesis model (110-2) according to another embodiment have been described with reference to FIG. 8. Hereinafter, a voice synthesis method according to another embodiment will be described with reference to FIG. 9.

[0133] Figure 9 is an exemplary flowchart illustrating a speech synthesis method according to another embodiment. However, this is merely a preferred embodiment for achieving the purpose of the present disclosure, and it is to be understood that some steps may be added or deleted as needed. Figure 9 assumes that the speech synthesis model (110-2) illustrated in Figure 8 is used, and illustrates the process of synthesizing speech into the voice of a target speaker among multiple speakers through the speech synthesis model (110-2).

[0134] As illustrated in FIG. 9 , the present embodiment may begin with step S910, which inputs text (i.e., text corresponding to a target speech) and information about a target speaker. For example, the device (100) may input text composed of a phoneme sequence and information about the target speaker, such as a spectrogram or a speech sample. For further details on step S910 , refer to the description of step S710 described above.

[0135] In one embodiment, a spectrogram (or speech sample) of a target speaker that the speech synthesis model (110-2) has never seen before during the training phase can be input to the speaker encoder (810). According to the experimental results of the present inventors, when generating a spectrogram using the decoder (330) of the structure illustrated in FIG. 8, it was confirmed that even the speech of such a target speaker can be accurately synthesized. In other words, it was experimentally confirmed that the zero-shot speech synthesis performance is also significantly superior.

[0136] In step S920, text embedding and speaker embedding are generated by encoding the input text and target speaker information, respectively. For example, the device (100) can generate a phoneme embedding sequence by encoding an input text composed of a phoneme sequence through an encoder (310). In addition, the device (100) can generate a speaker embedding by encoding the target speaker information through a speaker encoder (810). For this step S920, further reference is made to the description of step S720 described above.

[0137] In one embodiment, the device (100) can generate a text embedding by encoding input text and speaker embedding together through the encoder (310). That is, the speaker embedding can be used as conditioning information for the encoder (310).

[0138] In step S930, variance information including pitch information is predicted based on text embedding. For example, the device (100) can predict variance information (e.g., pitch, energy, duration information) from text embedding via a variance information predictor (320). For further details on step S930, refer to the description of step S730 described above.

[0139] In one embodiment, the device (100) can predict distributed information (e.g., pitch, energy, duration information) by analyzing text embedding and speaker embedding together through a distributed information predictor (320). That is, the speaker embedding can be used as conditioning information for the distributed information predictor (320).

[0140] In step S940, an excitation representation of the target speech is generated based on text embedding, speaker embedding, and variance information. For example, the device (100) can generate an excitation representation in the form of a spectrogram that reflects the characteristics / style of the target speaker by comprehensively analyzing a phoneme embedding sequence, speaker embedding, pitch information (e.g., pitch value per phoneme), and duration information (e.g., duration value per phoneme) through an excitation representation generator (331). For this step S940, further reference is made to the description of the above-described step S740.

[0141] In step S950, the expression here is refined through the diffusion model (332). For this step S950, refer to the description of step S750 described above.

[0142] In one embodiment, the device (100) can refine the excitation representation in spectrogram form by using the speaker embedding as conditioning information for the diffusion model (332). In this case, the effect of refining the excitation representation by taking into account the characteristics / style of the target speaker can be achieved.

[0143] In step S960, a formant representation of the target speech is generated based on the text embedding and speaker embedding. For example, the device (100) can analyze the phoneme embedding sequence, speaker embedding, and duration information (e.g., duration value per phoneme) through the formant representation generator (333) to generate a formant representation in the form of a spectrogram that reflects the characteristics / style of the target speaker. For this step S960, please refer to the description of the above-described step S760.

[0144] In step S970, the refined excitation and formant representations are aggregated to obtain a spectrogram of the target speech. For example, the device (10) can generate a spectrogram (e.g., a mel spectrogram) that reflects the characteristics / style of the target speaker by aggregating the refined excitation and formant representations in spectrogram form through a sum operation. For this step S970, please refer to the description of the above-described step S770.

[0145] In step S980, a target voice is synthesized based on the spectrogram. For example, the device (100) can analyze the spectrogram through a vocoder (340) to synthesize a target voice (i.e., a voice corresponding to the input text and the target speaker's voice). For further details on step S980, refer to the description of step S780 described above.

[0146] The above-described step S980 may be omitted in the step of learning (training) the speech synthesis model (110-2). For example, in the model learning step, the device (100) may postpone the execution of step S980 and calculate a loss (e.g., refer to mathematical expression 1) based on the spectrogram generated in step S970 and the correct spectrogram. Then, the device (100) may update the parameters of the speech synthesis model (110-2) (e.g., weight parameters of modules implemented as a neural network) based on the calculated loss. For this, refer further to the description of FIGS. 3 to 6, FIG. 8, and mathematical expression 1.

[0147] Heretofore, a speech synthesis method according to another embodiment has been described with reference to FIG. 9. As described above, a speaker embedding generated by encoding information about a target speaker can be used as conditioning information for a decoder (330) (or another module). In this case, a target speech corresponding to the voice of the target speaker can be accurately synthesized in a multi-speaker environment. Furthermore, according to the experimental results of the present inventors, when a spectrogram is generated through a decoder (330) having separate decoding passes (334, 335) and a diffusion model (332), it has been confirmed that even the speech of a target speaker that has not been seen in the model training stage can be accurately synthesized.

[0148] So far, embodiments related to a multi-speaker environment have been described with reference to FIGS. 8 and 9. Hereinafter, an exemplary computing device (12) capable of implementing the above-described speech synthesis device (100) will be described with reference to FIG. 10.

[0149] FIG. 10 is an exemplary block diagram illustrating an exemplary computing device (12) and a computing environment (10) including the same. Components (modules) within the computing device (12) and the computing environment (10) may have different functions and capabilities other than those described below, and in addition to the illustrated modules, additional components may be included in the computing device (12) and the computing environment (10).

[0150] As illustrated in FIG. 10, a computing environment (10) may include a computing device (12). In addition, the computing device (12) may include one or more processors (14), a computer-readable storage medium (16), and a communication bus (18).

[0151] The processor (14) can execute one or more programs (20) stored in a computer-readable storage medium (16). The one or more processors (14) may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit (NPU), but are not necessarily limited thereto.

[0152] One or more programs (20, i.e., computer programs) may include one or more computer-executable instructions, which, when executed by the processor (14), may be configured to cause the computing device (12) to perform specific operations.

[0153] For example, the program (20) may include instructions that cause the computing device (12) to perform the following operations: generating an excitation representation of a target voice corresponding to an input text based on pitch information associated with the input text, refining the excitation representation through a diffusion model (332), generating a formant representation of the target voice based on the input text regardless of the diffusion model (332), obtaining a spectrogram by aggregating the refined excitation representation and the formant representation, and synthesizing the target voice based on the spectrogram.

[0154] As another example, the program (20) may include instructions that cause the computing device (12) to perform at least some of the operations described with reference to FIGS. 1 through 9.

[0155] In the case as exemplified, a voice synthesis device (100) according to one embodiment can be implemented through a computing device (12).

[0156] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data, and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by the processor (14). In one embodiment, the computer-readable storage medium (16) may be a memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, any other form of storage medium that can be accessed by the computing device (12) and store desired information, or a suitable combination thereof.

[0157] A communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and computer-readable storage media (16).

[0158] The computing device (12) may also include one or more input / output interfaces (22) that provide interfaces for one or more input / output devices (24) and one or more network communication interfaces (26). The input / output interfaces (22) and the network communication interfaces (26) are connected to the communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) via the input / output interfaces (22). Exemplary input / output devices (24) may include input devices such as pointing devices (such as a mouse or a trackpad), a keyboard, a touch input device (such as a touchpad or a touchscreen), a voice or sound input device, various types of sensor devices and / or photographing devices, and / or output devices such as display devices, printers, speakers and / or network cards. The exemplary input / output devices (24) may be included within the computing device (12) as a component constituting the computing device (12), or may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0159] Meanwhile, embodiments of the present disclosure may include a program for performing the methods described herein on a computer, and a computer-readable recording medium including the program. The computer-readable recording medium may include program commands, local data files, local data structures, etc., alone or in combination. The medium may be specially designed and configured for the present disclosure, or may be one commonly used in the field of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of the program may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

[0160] While the embodiments of the present disclosure have been described in detail above, those skilled in the art will appreciate that various modifications to the above-described embodiments are possible without departing from the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments, but should be determined not only by the claims set forth below but also by equivalents thereof.

Claims

1. A method performed by at least one processor, A step of generating an excitation representation of a target voice corresponding to an input text based on pitch information associated with the input text; A step of refining the above expression through a diffusion model; A step of generating a formant representation of the target speech based on the input text, regardless of the diffusion model; A step of obtaining a spectrogram by aggregating the above refined here representation and the above formant representation; and Comprising a step of synthesizing the target voice based on the spectrogram, Voice synthesis method.

2. In paragraph 1, The above input text contains a sequence of phonemes, The step of generating the expression of the target voice above is: A step of encoding the above input text to generate a phoneme embedding sequence; A step of predicting the pitch information based on the phoneme embedding sequence, wherein the pitch information includes a pitch value for each phoneme; A step of predicting duration information based on the above phoneme embedding sequence, wherein the duration information includes a duration value for each phoneme; and A step of generating the above expression based on the above pitch information and the above duration information, Voice synthesis method.

3. In paragraph 2, The step of generating the above phoneme embedding sequence is: A step of generating a speaker embedding by encoding information of the target speaker; and A step of generating the phoneme embedding sequence by encoding the input text and the speaker embedding together, Voice synthesis method.

4. In paragraph 1, The step of generating the expression of the target voice above is: A step of predicting energy information associated with the input text based on the input text; and comprising a step of generating the above expression based further on the above energy information; Voice synthesis method.

5. In paragraph 1, The above input text contains a sequence of phonemes, The step of generating a formant representation of the target voice is as follows: A step of encoding the above input text to generate a phoneme embedding sequence; A step of predicting duration information based on the above phoneme embedding sequence, wherein the duration information includes a duration value for each phoneme; and A step of generating the formant representation based on the phoneme embedding sequence and the duration information, Voice synthesis method.

6. In paragraph 5, The pitch information is not used during the process of generating the above formant representation. Voice synthesis method.

7. In paragraph 1, The step of generating the expression of the target voice above is: A step of generating a speaker embedding by encoding information of the target speaker; and comprising a step of generating the above representation based further on the above speaker embedding; Voice synthesis method.

8. In paragraph 7, The information of the target speaker includes a voice sample of the target speaker or a spectrogram extracted from the voice sample. Voice synthesis method.

9. In paragraph 1, The step of generating a formant representation of the target voice is as follows: A step of generating a speaker embedding by encoding information of the target speaker; and comprising a step of generating the formant representation further based on the speaker embedding; Voice synthesis method.

10. One or more processors; and A memory storing a computer program executed by one or more processors, The above computer program: An operation of generating an excitation representation of a target voice corresponding to an input text based on pitch information associated with the input text; An operation to refine the above expression through a diffusion model; An operation of generating a formant representation of the target speech based on the input text, regardless of the diffusion model; An operation of obtaining a spectrogram by aggregating the above refined here representation and the above formant representation; and Including commands for synthesizing the target voice based on the spectrogram, Speech synthesis device.

11. In paragraph 10, The above input text contains a sequence of phonemes, The action of generating a representation of the above target voice is: An operation of encoding the above input text to generate a phoneme embedding sequence; An operation of predicting the pitch information based on the phoneme embedding sequence, wherein the pitch information includes a pitch value for each phoneme; An operation of predicting duration information based on the above phoneme embedding sequence, wherein the duration information includes a duration value for each phoneme; and Including an operation of generating the above expression based on the above pitch information and the above duration information, Speech synthesis device.

12. In paragraph 10, The action of generating a representation of the above target voice is: An operation of predicting energy information associated with the input text based on the input text; and Including an operation of generating the above expression based on the above energy information, Speech synthesis device.

13. In paragraph 10, The above input text contains a sequence of phonemes, The operation of generating a formant representation of the above target voice is as follows: An operation of encoding the above input text to generate a phoneme embedding sequence; An operation of predicting duration information based on the above phoneme embedding sequence, wherein the duration information includes a duration value for each phoneme; and An operation for generating the formant representation based on the phoneme embedding sequence and the duration information, Speech synthesis device.

14. In paragraph 10, The action of generating a representation of the above target voice is: The act of generating a speaker embedding by encoding information of the target speaker; and comprising an operation of generating the above representation based further on the above speaker embedding; Speech synthesis device.

15. In paragraph 10, The operation of generating a formant representation of the above target voice is as follows: The act of generating a speaker embedding by encoding information of the target speaker; and comprising an operation of generating the formant representation further based on the speaker embedding; Speech synthesis device.

Images