Electronic device, method, and non-transient computer-readable storage medium for generating a voice signal corresponding to a specified text

Abstract

An electronic device may be configured to include at least one processor, wherein the at least one processor receives an input text, a first voice signal representing a first user’s utterance regarding text different from the input text, and a second voice signal representing a second user’s utterance regarding text different from the input text, and, using a first neural network, obtain first information representing the prosody of the input text based on the prosody of the first voice signal, and using a second neural network, obtain second information representing the pronunciation of the input text by the second user corresponding to the second voice signal, and, using the first information and the second information, generate a third voice signal representing the utterance of the second user regarding the input text according to the prosody expressed by the first information.

Description

Technology Field

[0001] The following descriptions relate to an electronic device, a method, and a non-transient computer-readable storage medium for generating a voice signal corresponding to a specified text. Background Technology

[0002] Electronic devices can convert text into speech signals. For example, the function of converting text into speech signals can be referred to as TTS (text to speech) functionality.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure. means of solving the problem

[0004] An electronic device is provided. The electronic device may include a memory that stores instructions and includes one or more storage media. The electronic device may include at least one processor that includes a processing circuit. When executed individually or collectively by the at least one processor, the instructions may cause receiving an input text, a first voice signal representing a first user’s utterance regarding text different from the input text, and a second voice signal representing a second user’s utterance regarding text different from the input text. When executed individually or collectively by the at least one processor, the instructions may cause obtaining fundamental frequency sequences and energy sequences of the input text according to the prosody of the first voice signal using a first neural network. When executed individually or collectively by the at least one processor, the above instructions may cause to obtain first information expressing the prosody of the input text according to the prosody of the first speech signal, based on the fundamental frequency sequences and the energy sequences, using a second neural network. When executed individually or collectively by the at least one processor, the above instructions may cause to obtain second information expressing the pronunciation of the input text by the second user corresponding to the second speech signal, based on the energy sequences, using a third neural network. When executed individually or collectively by the at least one processor, the above instructions may cause to generate a third speech signal representing the utterance of the second user regarding the input text according to the prosody expressed by the first information, using the first information and the second information.

[0005] A method performed by an electronic device is provided. The method may include the operation of receiving an input text, a first voice signal representing a first user’s utterance regarding text different from the input text, and a second voice signal representing a second user’s utterance regarding another text different from the input text. The method may include the operation of obtaining fundamental frequency sequences and energy sequences of the input text according to the prosody of the first voice signal using a first neural network. The method may include the operation of obtaining first information expressing the prosody of the input text according to the prosody of the first voice signal based on the fundamental frequency sequences and the energy sequences using a second neural network. The method may include the operation of obtaining second information expressing the pronunciation of the input text by the second user corresponding to the second voice signal based on the energy sequences using a third neural network. The above method may include an operation of generating a third voice signal representing the second user's speech regarding the input text according to the prosody expressed by the first information, using the first information and the second information.

[0006] A non-transient computer-readable storage medium is provided for storing one or more programs. The one or more programs may include instructions that cause the electronic device to receive an input text, a first voice signal representing a first user’s utterance regarding text different from the input text, and a second voice signal representing a second user’s utterance regarding text different from the input text, when executed individually or collectively by at least one processor of the electronic device. The one or more programs may include instructions that cause the electronic device to obtain fundamental frequency sequences and energy sequences of the input text according to the prosody of the first voice signal using a first neural network when executed individually or collectively by at least one processor of the electronic device. The above one or more programs may include instructions that cause the electronic device to obtain first information expressing the prosody of the input text according to the prosody of the first speech signal, based on the fundamental frequency sequences and the energy sequences, using a second neural network when executed individually or collectively by at least one processor of the electronic device. The above one or more programs may include instructions that cause the electronic device to obtain second information expressing the pronunciation of the input text by the second user corresponding to the second speech signal, based on the energy sequences, using a third neural network when executed individually or collectively by at least one processor of the electronic device.The above one or more programs may include instructions that cause the electronic device to generate a third voice signal representing the second user’s utterance regarding the input text according to the prosody expressed by the first information, using the first information and the second information, when executed individually or collectively by at least one processor of the electronic device.

[0007] An electronic device is provided. The electronic device may include a memory that stores instructions and includes one or more storage media. The electronic device may include at least one processor that includes a processing circuit. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to acquire fundamental frequency sequences and energy sequences of an input text according to the prosodic nature of a first voice signal representing a first user's utterance using an autoregressive (AR) model. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to acquire first information expressing the prosodic nature of the input text according to the first voice signal based on the fundamental frequency sequences and the energy sequences. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to acquire second information representing the pronunciation of the input text by the second user, corresponding to a second voice signal representing the second user's utterance, based on the energy sequences. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to generate a spectrogram representing the second user's utterance for the input text according to the prosody, based on the first information and the second information.When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to generate a third voice signal based on the spectrogram using an acoustic decoder that utilizes adaptive kernel selection.

[0008] A method performed by an electronic device is provided. The method may include an operation of acquiring fundamental frequency sequences and energy sequences of an input text according to the prosodic nature of a first voice signal representing a first user's utterance using an autoregressive (AR) model. The method may include an operation of acquiring first information representing the prosodic nature of the input text according to the first voice signal based on the fundamental frequency sequences and the energy sequences. The method may include an operation of acquiring second information representing the pronunciation of the input text by the second user corresponding to a second voice signal representing a second user's utterance based on the energy sequences. The method may include an operation of generating a spectrogram representing the utterance of the second user for the input text according to the prosodic nature based on the first information and the second information. The method may include an operation of generating a third voice signal based on the spectrogram using an acoustic decoder utilizing adaptive kernel selection.

[0009] A non-transient computer-readable storage medium comprising one or more programs is provided. The one or more programs may include instructions that cause the electronic device to obtain fundamental frequency sequences and energy sequences of an input text according to the prosodic nature of a first voice signal representing a first user’s utterance, using an autoregressive (AR) model, when executed individually or collectively by at least one processor of the electronic device. The one or more programs may include instructions that cause the electronic device to obtain first information expressing the prosodic nature of the input text according to the first voice signal, based on the fundamental frequency sequences and the energy sequences, when executed individually or collectively by at least one processor of the electronic device. The one or more programs may include instructions that cause the electronic device to obtain second information expressing the pronunciation of the input text by the second user, corresponding to a second voice signal representing a second user’s utterance, based on the energy sequences, when executed individually or collectively by at least one processor of the electronic device. The above one or more programs may include instructions that cause the electronic device to generate a spectrogram representing the second user’s utterance for the input text according to the prosody, based on the first information and the second information, when executed individually or collectively by at least one processor of the electronic device. The above one or more programs may include instructions that cause the electronic device to generate a third speech signal based on the spectrogram, using an acoustic decoder utilizing adaptive kernel selection, when executed individually or collectively by at least one processor of the electronic device. Brief explanation of the drawing

[0010] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. FIG. 1 illustrates a simplified block diagram of an electronic device according to one embodiment. FIG. 2 illustrates a block diagram of an electronic device for generating a voice signal according to one embodiment. FIG. 3 illustrates an example for explaining a neural network for acquiring acoustic features according to one embodiment. FIG. 4 illustrates a simplified block diagram of an acoustic decoder according to one embodiment. FIG. 5 is a flowchart illustrating the operation of an electronic device for generating a voice signal according to one embodiment. FIG. 6 is a flowchart illustrating the operation of an electronic device for controlling acoustic features according to one embodiment. FIG. 7 illustrates an example for controlling acoustic features according to one embodiment. FIG. 8 illustrates an example for controlling acoustic features according to one embodiment. FIG. 9 is a flowchart illustrating the operation of an electronic device for generating a voice signal according to one embodiment. Specific details for implementing the invention

[0011] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, a server, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0012] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another corresponding component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0013] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, in one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0014] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium readable by a machine (e.g., an electronic device (105)). For example, a processor of the machine (e.g., an electronic device (105)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0015] In one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™ or App Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0016] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the components of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to the integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0017] FIG. 1 illustrates a simplified block diagram of an electronic device according to one embodiment. Referring to FIG. 1, the electronic device (100) may include a processor (102), memory (104), storage device (106), a high-speed controller (108) (e.g., northbridge, MCH (main controller hub)), and a low-speed controller (110) (e.g., southbridge, ICH (input / output controller hub)). Within the electronic device (100), each of the processor (102), memory (104), storage device (106), high-speed controller (108), and low-speed controller (110) may be interconnected using various buses.

[0018] For example, the processor (102) may process instructions for execution within the electronic device (100) to display graphic information for a graphical user interface (GUI) on an external input / output device, such as a display (112) connected to a high-speed controller (108). The instructions may be contained in memory (104) or storage device (106). The instructions may cause the electronic device (100) to perform one or more operations described below when executed by the processor (102). According to embodiments, the processor (102) may be composed of a plurality of processors including a communication processor and a graphical processing unit (GPU).

[0019] For example, the memory (104) may store information within the electronic device (100). For example, the memory (104) may include volatile memory. For example, the memory (104) may include non-volatile memory. For example, the memory (104) may include both volatile memory and non-volatile memory. For example, the memory (104) may include other forms of computer-readable media, such as magnetic or optical disks.

[0020] For example, the memory (104) of the electronic device (100) may store a set of parameters associated with multiple neural networks. The multiple neural networks may include a recognition model implemented in software or hardware that mimics the computational capabilities of a biological system using multiple artificial neurons (or nodes). For example, the parameters associated with the multiple neural networks may represent weights assigned to multiple nodes included in the multiple neural networks and / or to connections between said multiple nodes.

[0021] For example, the storage device (106) can provide mass storage space to the electronic device (100). For example, the storage device (106) may be a computer-readable medium, such as a hard disk device, an optical disk device, a flash memory, a solid-state memory device, or an array of devices within a storage area network (SAN).

[0022] For example, the high-speed controller (108) may manage bandwidth-intensive operations for the electronic device (100), while the low-speed controller (110) may manage low bandwidth-intensive operations for the electronic device (100). For example, the high-speed controller (108) may be coupled to the memory (104) and coupled to the display (112) via a GPU or accelerator, while the low-speed controller (110) may be coupled to the storage device (106) and coupled to various communication ports (e.g., USB (universal serial bus), Bluetooth, Ethernet, wireless Ethernet) or communication circuits for communication with external electronic devices (e.g., keyboard, transducer, scanner, or network device (e.g., switch or router)).

[0023] FIG. 2 illustrates a block diagram of an electronic device for generating a voice signal according to one embodiment.

[0024] According to source-filter theory, speech can be generated based on source representation and filter representation. For example, the source representation can be generated based on prosody according to the fundamental frequency generated by the vibration of the vocal cords. For example, the filter representation can be generated based on pronunciation according to the shape and size of the vocal tract (e.g., oral cavity, pharynx, nasal cavity). In Figure 2, an artificial intelligence model that utilizes source representation and filter representation for speech synthesis is described.

[0025] In one embodiment, the processor (102) may receive (or acquire) a first voice signal (201), input text (202), a second voice signal (203), or a combination thereof. For example, the processor (102) may receive (or acquire) a first voice signal (201) representing a first user's speech. In one example, the first voice signal (201) may be referred to as a style voice prompt. For example, the processor (102) may receive (or acquire) input text (202) for converting into a third voice signal (201). For example, the processor (102) may receive (or acquire) a second voice signal (203) representing a second user's speech. In one example, the second voice signal (203) may be referred to as a speaker voice prompt.

[0026] In one embodiment, text representing the content of the first voice signal (201) may be different from the input text (202). For example, text representing the content of the second voice signal (203) may be different from the input text (202). For example, text representing the content of the first voice signal (201) may be different from the text representing the content of the second voice signal (203).

[0027] In one embodiment, the processor (102) may perform speech synthesis of one of a plurality of speech syntheses based on input text (202) and speech signals (201, 203). For example, the plurality of speech syntheses may include a first speech synthesis (e.g., zero-shot TTS (text to speech)), a second speech synthesis (e.g., style transfer TTS), a third speech synthesis (e.g., cross-lingual TTS), and a fourth speech synthesis (e.g., intra-lingual TTS).

[0028] In one embodiment, when a first voice synthesis (e.g., zero-shot TTS) is performed, the first user of the first voice signal (201) may be the same as the second user of the second voice signal (203). For example, when a second voice synthesis (e.g., style transfer TTS) is performed, the first user of the first voice signal (201) may be different from the second user of the second voice signal (203). For example, when a third voice synthesis (e.g., cross-language TTS) is performed, the language of the first voice signal (201) (e.g., Korean) may be different from the language of the input text (202) (e.g., English). For example, in the case of the third voice synthesis, the language of the second voice signal (203) (e.g., Korean) may be different from the language of the input text (202) (e.g., English). For example, when a fourth speech synthesis (e.g., internal language TTS) is performed, the language of the first speech signal (201) may be the same as the language of the input text (202). For example, in the case of the fourth speech synthesis, the language of the second speech signal (203) may be the same as the language of the input text (202).

[0029] In one embodiment, the processor (102) can identify (or obtain) a hidden representation of the first speech signal (201) using a first SPE (speech prompt encoder). For example, the hidden representation of the first speech signal (201) may include vectors representing features of the first speech signal (201). For example, the processor (102) can identify (or obtain) a hidden representation of the second speech signal (203) using a second SPE. For example, the hidden representation of the second speech signal (203) may include vectors representing features of the second speech signal (203).

[0030] In one embodiment, when a first speech synthesis (e.g., zero-shot TTS) is performed, the first SPE may be the same as the second SPE. For example, when a second speech synthesis (e.g., style transfer TTS) is performed, the first SPE may be different from the second SPE.

[0031] In one embodiment, the processor (102) uses a text encoder (TE) (205) to obtain phoneme sequences (e.g., x1, x2, …, x) for the input text (202). T ) can be identified (or obtained). For example, phoneme sequences can be based on phoneme units. For example, the first phoneme sequence (x1) of the phoneme sequences can correspond to the first phoneme of the input text (202).

[0032] In one embodiment, the processor (102) can identify (or acquire) acoustic features using the first neural network (204). For example, the acoustic features may represent acoustic features of the speech signal of the input text (202) based on the first speech signal (201). For example, the acoustic features may represent the duration, pitch (or fundamental frequency), energy, or a combination thereof of the speech signal of the input text (202) based on the first speech signal (201). For example, the processor (102) may use the neural network (204) to identify acoustic sequences (e.g., c1, c2, …, c T ) can be obtained. For example, acoustic sequences, duration sequences (e.g., d1, d2, …, d T ), pitch sequences (e.g., p1, p2, …, p T )(or, fundamental frequency sequences), energy sequences (e.g., e1, e2, …, e TIt may include ), or a combination thereof. For example, the first acoustic sequence (c1) may include a first duration sequence (d1), a first pitch sequence (p1), and a first energy sequence (e1). For example, the first duration sequence (d1) may represent the duration of speech at the first phoneme (x1) of the input text (202). For example, the first pitch sequence (p1) may represent the pitch of speech at the first phoneme (x1) of the input text (202). For example, the first energy sequence (e1) may represent the energy of speech at the first phoneme (x1) of the input text (202). For example, the duration sequences, pitch sequences, and energy sequences may be based on phoneme units. For example, the duration sequences may correspond to phoneme sequences, respectively. For example, the pitch sequences may correspond to phoneme sequences, respectively. For example, energy sequences can correspond to phoneme sequences, respectively.

[0033] In one embodiment, the first neural network (206) may include an autoregressive (AR) model. For example, since acoustic features have a time-varying distribution, an AR model may be used to identify (or acquire) acoustic features. For example, acoustic sequences according to the AR model may be defined (or identified) according to Equation 1 below.

[0034]

[0035] c represents sound sequences. x represents input text (202). r represents the first voice signal (201). represents the parameters of the prosody prediction model.

[0036] In one embodiment, the first neural network (206) may be referred to as a speech feature extraction module, a speech feature extraction network, a speech feature extraction model, a prosody prediction module, a prosody prediction network, a prosody prediction model, or other terms having an equivalent technical meaning.

[0037] In one embodiment, the processor (102) may acquire (or identify) first information using the second neural network (206). For example, the first information may include information expressing the prosody of the input text (202) based on the first speech signal (201). For example, the first information may be referred to as source representation, source information, source spectrogram, or other terms having an equivalent technical meaning. For example, pitch sequences (or fundamental frequency sequences) and energy sequences identified (or acquired) by the first neural network (204) may be provided to the input of the second neural network (206). For example, the processor (102) may acquire (or identify) the first information using the second neural network (206) based on the pitch sequences and energy sequences. The second neural network (206) may be referred to as a source generation module, a source generation network, a source generation model, or other terms having an equivalent technical meaning. For example, the second neural network (206) may include a non-autoregressive (NAR) model.

[0038] In one embodiment, the processor (102) may acquire (or identify) second information using the third neural network (207). For example, the second information may represent the content of the input text (202). For example, the second information may include information representing the speech (or pronunciation, tone) of the input text (202) by the second user of the second voice signal (203). For example, the second information may be referred to as a filter representation, filter information, filter spectrogram, or other terms having an equivalent technical meaning. For example, energy sequences acquired (or identified) by the first neural network (204), phoneme sequences for the input text (202), and hidden representations of the second voice signal (203) (e.g., vectors representing features of the second voice signal (203)) may be provided as inputs to the third neural network (207). For example, the processor (102) may acquire (or identify) second information based on energy sequences, phoneme sequences, and feature vectors of the third speech signal (203). For example, the third neural network (207) may be referred to as a filter generation module, a filter generation network, a filter generation model, or other terms having an equivalent technical meaning. For example, the third neural network (207) may include a NAR model.

[0039] In one embodiment, the processor (102) may obtain a first mel-spectogram based on first information and second information. For example, the first mel-spectogram may be referred to as a coarse mel-spectogram. For example, the mel-spectogram may represent the second user's speech in the input text (202) according to the prosody of the first voice signal (201) represented by the first information.

[0040] In one embodiment, the processor (102) can acquire (or generate) a third voice signal (or output voice signal) (210) using an acoustic decoder (209). For example, the processor (102) can acquire (or generate) the third voice signal (210) based on a first Mel spectrogram and a second voice signal (203).

[0041] In one embodiment, the processor (102) can identify (or obtain) a global style embedding for a second voice signal (203) using a speaker encoder (208). For example, the global style embedding may represent the identity of the second user of the second voice signal (203) (e.g., the second user's intonation, speech pattern). For example, the speaker encoder may be pre-trained to represent the identity of the second user. For example, the processor (102) may use a mapping network to map the style vector of the global style embedding to the sample-adaptive comfort layer of the acoustic decoder (209). For example, the filters of the sample-adaptive convolution layer may be aggregated based on weights predicted by the style vector.

[0042] In one embodiment, the processor (102) may acquire (or generate) a third speech signal (203) based on a first Mel-spectogram by using an acoustic decoder (209) that includes aggregated filters. For example, the processor (102) may acquire (or generate) a second Mel-spectogram for the output speech signal. For example, since sample-adaptive kernel selection is used, the filter capacity of the acoustic decoder (209) may be enhanced. For example, since the filter capacity of the acoustic decoder (209) is enhanced, information of the first Mel-spectogram (e.g., speech content, style) may be preserved.

[0043] FIG. 3 illustrates an example for explaining a neural network for acquiring acoustic features according to one embodiment. The neural network (305) exemplified in FIG. 3 may correspond to the first neural network (204) of FIG. 2.

[0044] Referring to FIG. 3, in one embodiment, a processor (102) uses a text encoder (TE) (303) to obtain phoneme sequences (e.g., x1, x2, …, x) for an input text (301). T ) can be identified. In one embodiment, the processor (102) can identify (or obtain) a hidden representation of the speech signal (302) using a speech prompt encoder (SPE). For example, the hidden representation of the speech signal (302) may include vectors representing features of the speech signal (302). In one embodiment, phoneme sequences and vectors representing features of the speech signal (302) may be provided as inputs to a neural network (305).

[0045] In one embodiment, the neural network (305) may include an autoregressive (AR) model. For example, an AR model can efficiently predict dynamic characteristics of time-series data that change over time. For example, since acoustic features have a time-varying distribution, an AR model may be used.

[0046] In one embodiment, acoustic sequences can be identified (or acquired) based on the AR model of the neural network (305). For example, an acoustic sequence (c T-1 To identify (or acquire) )(308) acoustic sequences (c1 to c T-2 ) can be used. For example, an acoustic sequence (c T To identify (or acquire) (309) acoustic sequences (c1 to c T-1 ) can be used.

[0047] In one embodiment, the processor (102) can identify (or acquire) acoustic features using a neural network (305). For example, the processor (102) can use the neural network (305) to identify (or acquire) acoustic sequences (e.g., c1, c2, …, c T ) can be obtained. For example, acoustic sequences, duration sequences (e.g., d1, d2, …, d T ), pitch sequences (e.g., p1, p2, …, p T )(or, fundamental frequency sequences), energy sequences (e.g., e1, e2, …, e T It may include ), or a combination thereof. For example, the acoustic sequence (306) may include a duration sequence (d1), a pitch sequence (p1), and an energy sequence (e1). For example, the acoustic sequence (307) may include a duration sequence (d2), a pitch sequence (p2), and an energy sequence (e2).

[0048] FIG. 4 illustrates a simplified block diagram of an acoustic decoder according to one embodiment. The acoustic decoder (410) exemplified in FIG. 4 may correspond to the acoustic decoder (209) of FIG. 2.

[0049] In one embodiment, the acoustic decoder (410) may include a self attention block (411), a residual connection and layer normalization block (Add & Norm) (412), a sample adaptive convolution layer (413), a sample adaptive convolution layer (414), and a residual connection and layer normalization block (415). FIG. 4 illustrates two sample adaptive convolution layers, but this is merely an example. For example, the acoustic decoder (410) may include one sample adaptive convolution layer.

[0050] In one embodiment, the processor (102) can identify (or obtain) a global style embedding for a speech signal (402) using a speaker encoder (420). For example, the global style embedding may represent the user identity of the speech signal (402) (e.g., the accent, manner of speech of the user of the speech signal (402). For example, the speaker encoder (420) may be pre-trained to represent the user identity of the speech signal (402). For example, the processor (102) may identify a style vector of the global style embedding. For example, the processor (102) may map the style vector and random noise to the sample-adaptive convolution layer (413) and sample-adaptive convolution layer (414) of the acoustic decoder (410). For example, the processor (102) can identify weights predicted by style vectors. For example, the processor (102) can aggregate filters of the sample adaptive convolution layer (413) based on the weights. For example, the processor (102) can aggregate filters of the sample adaptive convolution layer (414) based on the weights. For example, fixed filters in the sample adaptive convolution may not be used. For example, filters reflecting the user's identity of the voice signal (402) may be used in the sample adaptive convolution. For example, since filters reflecting the user's identity are used, the filter capacity of the speaker encoder (420) may be improved. For example, the processor (102) can generate (or obtain) an output voice signal (403) from a first Mel-spectogram (401) based on aggregated filters. For example, the processor (102) can identify (or obtain) a second Mel-spectogram for the output voice signal (403).

[0051] FIG. 5 is a flowchart illustrating the operation of an electronic device for generating a voice signal according to one embodiment. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed. For example, at least two operations may be performed in parallel.

[0052] Referring to FIG. 5, in operation 501, the processor (102) can acquire input text and voice signals.

[0053] In one embodiment, the voice signals may include a first voice signal (201) and a second voice signal (203). For example, the first voice signal (201) may correspond to a target voice signal for style modulation. In one example, the first voice signal may be referred to as a style voice prompt. For example, the second voice signal (203) may correspond to a speaker's voice signal. In one example, the second voice signal (203) may be referred to as a speaker voice prompt. For example, the input text (202) may correspond to text for conversion into an output voice signal (210).

[0054] In one embodiment, text representing the content of the first voice signal (201) may be different from the input text (202). For example, text representing the content of the second voice signal (203) may be different from the input text (202). For example, text representing the content of the first voice signal (201) may be different from the text representing the content of the second voice signal (203).

[0055] In one embodiment, the processor (102) may perform speech synthesis of one of a plurality of speech syntheses based on input text (202) and speech signals (201, 203). For example, the plurality of speech syntheses may include a first speech synthesis (e.g., zero-shot TTS (text to speech)), a second speech synthesis (e.g., style transfer TTS), a third speech synthesis (e.g., cross-lingual TTS), and a fourth speech synthesis (e.g., intra-lingual TTS).

[0056] In one embodiment, when a first voice synthesis (e.g., zero-shot TTS) is performed, the first user of the first voice signal (201) may be the same as the second user of the second voice signal (203). For example, when a second voice synthesis (e.g., style transfer TTS) is performed, the first user of the first voice signal (201) may be different from the second user of the second voice signal (203). For example, when a third voice synthesis (e.g., cross-language TTS) is performed, the language of the first voice signal (201) may be different from the language of the input text (202). For example, when a third voice synthesis is performed, the language of the second voice signal (203) may be different from the language of the input text (202). For example, when a fourth voice synthesis (e.g., internal language TTS) is performed, the language of the first voice signal (201) may be the same as the language of the input text (202). For example, when a fourth voice synthesis is performed, the language of the second voice signal (203) may be the same as the language of the input text (202).

[0057] In one embodiment, the processor (102) can identify (or obtain) a hidden representation of the first speech signal (201) using a first SPE (speech prompt encoder). For example, the hidden representation of the first speech signal (201) may include vectors representing features of the first speech signal (201).

[0058] In one embodiment, the processor (102) can identify (or obtain) a hidden representation of the second voice signal (203) using the second SPE. For example, the hidden representation of the second voice signal (203) may include vectors representing features of the second voice signal (203).

[0059] In one embodiment, when a first speech synthesis (e.g., zero-shot TTS) is performed, the first SPE may be the same as the second SPE. For example, when a second speech synthesis (e.g., style transfer TTS) is performed, the first SPE may be different from the second SPE.

[0060] In one embodiment, the processor (102) uses a text encoder (TE) (205) to obtain phoneme sequences (e.g., x1, x2, …, x) for the input text (202). T ) can be identified (or obtained). For example, phoneme sequences can be based on phoneme units. For example, the first phoneme sequence (x1) of the phoneme sequences can correspond to the first phoneme of the input text (202).

[0061] In one embodiment, the processor (102) may identify (or acquire) acoustic features using the first neural network (204). For example, the acoustic features may represent acoustic features of the speech signal of the input text (202) based on the first speech signal (201). For example, the acoustic features may represent the duration, pitch (or fundamental frequency), energy, or a combination thereof of the speech signal of the input text (202) based on the first speech signal (201). For example, the acoustic features may represent sequences of durations of the speech signal for the input text (202) according to (or to which the prosody of) the first speech signal (201) is applied (e.g., d1, d2, …, d T ), pitch sequences (e.g., p1, p2, …, p T )(or, fundamental frequency sequences), energy sequences (e.g., e1, e2, …, e T It may include ), or a combination thereof. For example, duration sequences may be based on phoneme units. For example, duration sequences may be phoneme sequences (e.g., x1, x2, …, x TIt can correspond to ). For example, among the duration sequences, the first duration sequence (e.g., d1) may represent the duration of speech at the first phoneme (e.g., x1) of the input text (202). For example, pitch sequences may be based on phoneme units. For example, pitch sequences may correspond to phoneme sequences. For example, among the pitch sequences, the first pitch sequence (e.g., p1) may represent the pitch of speech at the first phoneme of the input text (202). For example, energy sequences may be based on phoneme units. For example, energy sequences may correspond to phoneme sequences. For example, among the energy sequences, the first energy sequence (e.g., e1) may represent the energy of speech at the first phoneme of the input text (202).

[0062] In one example, the first neural network (204) may be referred to as a speech feature extraction module, a speech feature extraction network, a speech feature extraction model, a prosody prediction module, a prosody prediction network, a prosody prediction model, or other terms having an equivalent technical meaning. For example, the first neural network (204) may include an autoregressive (AR) model. For example, since acoustic features have a time-varying distribution, an AR model may be used to identify (or acquire) acoustic features.

[0063] In operation 502, the processor (102) can obtain (or identify) first information expressing the prosody of the input text (202).

[0064] In one embodiment, the first information may include information expressing the prosody of the input text (202) based on the first voice signal (201). For example, the first information may be referred to as a source representation, source information, source spectrogram, or other terms having an equivalent technical meaning that express the prosody of the input text (202) based on the first voice signal (201).

[0065] In one embodiment, the processor (102) may obtain (or identify) first information expressing the prosody of the input text (202) using a second neural network (206). For example, pitch sequences (or fundamental frequency sequences) and energy sequences obtained (or identified) by the first neural network (204) may be provided as inputs to the second neural network (206). For example, the processor (102) may obtain (or identify) the first information using the second neural network (206) based on the pitch sequences and energy sequences. For example, the second neural network (206) may be referred to as a source generation module, a source generation network, a source generation model, or other terms having an equivalent technical meaning. For example, the second neural network (206) may include a non-autoregressive (NAR) model.

[0066] In operation 503, the processor (102) can identify (or obtain) second information representing the contents of the input text (202).

[0067] In one embodiment, the second information may include information representing the speech (or pronunciation, tone) of the input text (202) by the second user of the second voice signal (203). For example, the second information may be referred to as a filter representation, filter information, filter spectrogram, or other terms having an equivalent technical meaning.

[0068] In one embodiment, the processor (102) may use a third neural network (207) to obtain (or identify) second information representing the content of the input text (202). For example, energy sequences obtained (or identified) by the first neural network (204), phoneme sequences for the input text (202), and hidden representations of the second speech signal (203) (e.g., vectors representing features of the second speech signal (203)) may be provided to the input of the third neural network (207). For example, the processor (102) may obtain (or identify) the second information based on the energy sequences, phoneme sequences, and feature vectors of the second speech signal (203). For example, the third neural network (207) may be referred to as a filter generation module, a filter generation network, a filter generation model, or other terms having an equivalent technical meaning. For example, the third neural network (207) may include a NAR model.

[0069] In operation 504, the processor (102) can obtain an output voice signal (210) based on the first information and the second information.

[0070] In one embodiment, the processor (102) may obtain a first mel-spectogram based on first information and second information. For example, the first mel-spectogram may be referred to as a coarse mel-spectogram. For example, the mel-spectogram may represent the speech of a second user for an input text (202) according to the prosody of a first voice signal (201) represented by the first information.

[0071] In one embodiment, the processor (102) can obtain an output voice signal based on a first Mel-spectogram. For example, the processor (102) can obtain an output voice signal based on the first Mel-spectogram using an acoustic decoder (209).

[0072] In one embodiment, the acoustic decoder (209) may use sample-adaptive kernel selection to learn a convolutional filter for a speech prompt. For example, since sample-adaptive kernel selection is used, the filter capacity of the acoustic decoder (209) may be enhanced. For example, since the filter capacity of the acoustic decoder (209) is enhanced, information (e.g., speech content, style) of the first Mel-spectogram (e.g., coarse Mel-spectogram) may be preserved. For example, the processor (102) may use a speaker encoder to identify (or obtain) a global style embedding for the second speech signal (203). For example, the global style embedding can represent the identity of the second user of the second voice signal (203) (e.g., the second user's intonation, speech pattern). For example, the speaker encoder can be pre-trained to represent the identity of the second user. For example, the processor (102) can use a mapping network to map the style vector of the global style embedding and random noise sampled from a normal Gaussian distribution to the sample-adaptive convolution layer of the acoustic decoder (209). For example, the filters of the sample-adaptive convolution layer can be aggregated based on weights predicted by the style vector. For example, the processor (102) can obtain an output voice signal by using an acoustic decoder (209) including aggregated filters based on a first Mel-spectrogram and a second voice signal (203).For example, the processor (102) can obtain a second Mel-spectogram for an output voice signal by using an acoustic decoder (209) that includes aggregated filters based on a first Mel-spectogram and a second voice signal (203).

[0073] FIG. 6 is a flowchart illustrating the operation of an electronic device for controlling acoustic features according to one embodiment. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed. For example, at least two operations may be performed in parallel.

[0074] Referring to FIG. 6, in operation 601, the processor (102) can acquire acoustic features.

[0075] In one embodiment, acoustic features may represent acoustic features of a voice signal for an input text (202) based on a first voice signal (201). For example, the first voice signal (201) may correspond to a target voice signal for style modulation. For example, acoustic features may represent the duration, pitch (or fundamental frequency), energy, or a combination thereof of the voice signal of the input text (202) based on the first voice signal (201).

[0076] In operation 602, the processor (102) may display parameters associated with acoustic features using a display (112). In an example that is not limited, the processor (102) may display parameters associated with acoustic features through a display of another electronic device outside the electronic device (100) using a communication circuit.

[0077] In one embodiment, parameters associated with acoustic features may include first to eighth parameters. For example, the first parameter may represent the playback speed of the output voice signal (210). For example, the first parameter may be referred to as a global rate. For example, the second parameter may represent the pitch of the output voice signal (210). For example, the second parameter may be referred to as a global pitch. For example, the third parameter may represent the total playback time of the output voice signal (210). For example, the third parameter may be referred to as a total duration. For example, the fourth parameter may represent the pitch of the portion of the output voice signal (210) corresponding to a word in the input text (202). For example, the fifth parameter may represent the playback time of the portion of the output voice signal (210) corresponding to a word in the input text (202). For example, the sixth parameter may represent the pitch of a portion of the output voice signal (210) corresponding to a character of the input text (202). For example, the seventh parameter may represent the playback time of a portion of the output voice signal (210) corresponding to a character of the input text (202). For example, the eighth parameter may represent the pitch of a portion of the output voice signal (210) corresponding to a phoneme of the input text (202). For example, the ninth parameter may represent the playback time of a portion of the output voice signal (210) corresponding to a phoneme of the input text (202).

[0078] In operation 603, the processor (102) can obtain user input for parameters associated with acoustic features. For example, the processor (102) can obtain user input for at least one of the parameters associated with acoustic features.

[0079] In operation 604, the processor (102) can change acoustic features based on user input. For example, the processor (102) can change at least one acoustic feature among the acoustic features based on user input.

[0080] In one embodiment, acoustic features may include duration sequences, pitch sequences (or fundamental frequency sequences), energy sequences, or a combination thereof of a speech signal for an input text (202) that corresponds to (or has prosody applied to) the prosody of the first speech signal (201).

[0081] In one example, when user input for the first parameter is obtained, the processor (102) can control the playback speed of the output voice signal (210) corresponding to the input text (202). For example, the processor (102) can change the values ​​of the duration sequences, the values ​​of the pitch sequences, the values ​​of the energy sequences, or a combination thereof to control the playback speed of the output voice signal (210).

[0082] In one example, when user input for the second parameter is obtained, the processor (102) can control the pitch of the output voice signal (210) corresponding to the input text (202). For example, the processor (102) can change the values ​​of the pitch sequences to control the pitch of the output voice signal (210).

[0083] In one example, when user input for a third parameter is obtained, the processor (102) can control the total duration of the output voice signal (210) corresponding to the input text (202). For example, the processor (102) can change the values ​​of the duration sequences to control the total duration of the output voice signal (210).

[0084] In one example, when user input for the fourth parameter is obtained, the processor (102) can control the pitch of a portion of the output voice signal (210) corresponding to a word of the input text (202). For example, the processor (102) can identify at least one pitch sequence among pitch sequences that corresponds to a portion of the output voice signal (210). For example, the processor (102) can change the value of the identified at least one pitch sequence.

[0085] In one example, when user input for the fifth parameter is obtained, the processor (102) can control the playback time for a portion of the output voice signal (210) corresponding to a word of the input text (202). For example, the processor (102) can identify at least one duration sequence among duration sequences that corresponds to a portion of the output voice signal (210). For example, the processor (102) can change the value of the identified at least one duration sequence.

[0086] In one example, when user input for the sixth parameter is obtained, the processor (102) can control the pitch of a portion of the output voice signal (210) corresponding to a character of the input text (202). For example, the processor (102) can identify at least one pitch sequence among pitch sequences that corresponds to a portion of the output voice signal (210). For example, the processor (102) can change the value of the identified at least one pitch sequence.

[0087] In one example, when user input for the seventh parameter is obtained, the processor (102) can control the playback time for a portion of the output voice signal (210) corresponding to a character of the input text (202). For example, the processor (102) can identify at least one duration sequence among duration sequences that corresponds to a portion of the output voice signal (210). For example, the processor (102) can change the value of the identified at least one duration sequence.

[0088] In one example, when user input for the eighth parameter is obtained, the processor (102) can control the pitch of a portion of the output voice signal (210) corresponding to a phoneme of the input text (202). For example, the processor (102) can identify a pitch sequence among pitch sequences that corresponds to a portion of the output voice signal (210). For example, the processor (102) can change the value of the identified pitch sequence.

[0089] In one example, when user input for the ninth parameter is obtained, the processor (102) can control the playback time for a portion of the output voice signal (210) corresponding to a phoneme of the input text (202). For example, the processor (102) can identify a duration sequence among duration sequences that corresponds to a portion of the output voice signal (210). For example, the processor (102) can change the value of the identified duration sequence.

[0090] FIG. 7 illustrates an example for controlling acoustic features according to one embodiment. In FIG. 7, an example for controlling the overall acoustic features of an output voice signal (210) is described.

[0091] Referring to FIG. 7, in one embodiment, the processor (102) may display a user interface (UI) (701) associated with a first parameter using a display (112). In an example that is not limited, the processor (102) may display the UI (701) associated with the first parameter through a display of another electronic device outside the electronic device (100) using a communication circuit. For example, the UI (701) may include an object (702) for controlling the first parameter. For example, when user input is provided to the object (702), the processor (102) may change the values ​​of duration sequences, values ​​of pitch sequences, values ​​of energy sequences, or a combination thereof to control the playback speed of the output voice signal (210).

[0092] In one embodiment, the processor (102) may display a UI (703) associated with the second parameter using a display (112). In an example that is not limited, the processor (102) may display the UI (703) associated with the second parameter through a display of another electronic device outside the electronic device (100) using a communication circuit. For example, the UI (703) may include an object (704) for controlling the second parameter. For example, when user input for the object (704) is provided, the processor (102) may change the values ​​of pitch sequences to control the pitch of the output voice signal (210).

[0093] In one embodiment, the processor (102) may display a UI (705) associated with a third parameter using a display (112). In an example that is not limited, the processor (102) may display the UI (705) associated with the third parameter through a display of another electronic device outside the electronic device (100) using a communication circuit. For example, the UI (705) may include an object (706) for controlling the third parameter. For example, when user input for the object (706) is provided, the processor (102) may change the values ​​of the duration sequences to control the playback time of the output voice signal (210).

[0094] FIG. 8 illustrates an example for controlling acoustic features according to one embodiment. In FIG. 8, an example for controlling the acoustic features of an output voice signal (210) in the range of words, letters, or phonemes is described.

[0095] Referring to FIG. 8, in one embodiment, the processor (102) may display a user interface (UI) (810) associated with a fourth parameter and a fifth parameter using a display (112). For example, the UI (810) may include an object (811) for displaying a unit (e.g., a word) for which acoustic features are controlled, an object (812) for displaying the pitch of a word of the output voice signal (210), an object (813) for displaying the length of a word of the output voice signal (210), an object (814) for controlling the pitch of a word of the output voice signal (210), an object (815) for controlling the length of a word of the output voice signal (210), or a combination thereof.

[0096] In one example, when user input (e.g., pitch value input) is provided for an object (812), the processor (102) can identify at least one pitch sequence among pitch sequences that corresponds to a word of the output speech signal (210). For example, the processor (102) can change the value of the identified at least one pitch sequence.

[0097] In one example, when user input (e.g., length value input) for an object (813) is provided, the processor (102) can identify at least one duration sequence among the duration sequences that corresponds to a word of the output speech signal (210). For example, the processor (102) can change the value of the identified at least one duration sequence.

[0098] In one example, when user input (e.g., drag input) is provided for an object (814), the processor (102) can identify at least one pitch sequence among the pitch sequences that corresponds to a word of the output voice signal (210). For example, the processor (102) can change the value of the identified at least one pitch sequence.

[0099] In one example, when user input (e.g., drag input) is provided for an object (815), the processor (102) can identify at least one duration sequence among the duration sequences that corresponds to a word of the output voice signal (210). For example, the processor (102) can change the value of the identified at least one duration sequence.

[0100] In one embodiment, when input (e.g., click) for an object (811) is provided, the processor (102) can change the UI (810) displayed through the display (112) into a UI (820).

[0101] In one embodiment, the processor (102) may display a UI (820) associated with the sixth parameter and the seventh parameter using a display (112). For example, the UI (820) may include an object (821) for displaying a unit (e.g., a character) for which acoustic features are controlled, an object (822) for displaying the pitch of a character in the output voice signal (210), an object (823) for displaying the length of a character in the output voice signal (210), an object (824) for controlling the pitch of a character in the output voice signal (210), an object (825) for controlling the length of a character in the output voice signal (210), or a combination thereof.

[0102] In one example, when user input (e.g., pitch value input) is provided for an object (822), the processor (102) can identify at least one pitch sequence among pitch sequences that corresponds to a character of the output voice signal (210). For example, the processor (102) can change the value of at least one identified duration sequence.

[0103] In one example, when user input (e.g., length value input) for an object (823) is provided, the processor (102) can identify at least one duration sequence among the duration sequences that corresponds to a character of the output voice signal (210). For example, the processor (102) can change the value of the identified at least one duration sequence.

[0104] In one example, when user input (e.g., drag input) is provided for an object (824), the processor (102) can identify at least one pitch sequence among pitch sequences that corresponds to a character of the output voice signal (210). For example, the processor (102) can change the value of at least one identified duration sequence.

[0105] In one example, when user input (e.g., drag input) is provided for an object (825), the processor (102) can identify at least one pitch sequence among pitch sequences that corresponds to a character of the output voice signal (210). For example, the processor (102) can change the value of at least one identified duration sequence.

[0106] In one embodiment, when user input (e.g., click) is provided for an object (821), the processor (102) can change the UI (820) displayed through the display (112) to a UI (830).

[0107] In one embodiment, the processor (102) may display a UI (830) associated with the eighth parameter and the ninth parameter using a display (112). For example, the UI (830) may include an object (831) for displaying a unit (e.g., a phoneme) for controlling acoustic features, an object (832) for displaying the pitch of a phoneme of the output voice signal (210), an object (833) for displaying the length of a phoneme of the output voice signal (210), an object (834) for controlling the pitch of a phoneme of the output voice signal (210), and an object (835) for controlling the length of a phoneme of the output voice signal (210).

[0108] In one example, when user input (e.g., pitch value input) is provided for an object (832), the processor (102) can identify a pitch sequence among pitch sequences that corresponds to a phoneme of the output speech signal (210). For example, the processor (102) can change the value of the identified pitch sequence.

[0109] In one example, when user input (e.g., length value input) is provided for an object (833), the processor (102) can identify a duration sequence among duration sequences that corresponds to a phoneme of the output speech signal (210). For example, the processor (102) can change the value of the identified duration sequence.

[0110] In one example, when user input (e.g., drag input) is provided for an object (834), the processor (102) can identify a pitch sequence among pitch sequences that corresponds to a phoneme of the output speech signal (210). For example, the processor (102) can change the value of the identified pitch sequence.

[0111] In one example, when user input (e.g., drag input) is provided for an object (835), the processor (102) can identify a duration sequence among duration sequences that corresponds to a phoneme of the output speech signal (210). For example, the processor (102) can change the value of the identified duration sequence.

[0112] In one embodiment, when user input (e.g., click) is provided for an object (831), the processor (102) can change the UI (820) displayed through the display (112) to the UI (810).

[0113] FIG. 9 is a flowchart illustrating the operation of an electronic device for generating a voice signal according to one embodiment. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed. For example, at least two operations may be performed in parallel.

[0114] Referring to FIG. 9, in operation 901, the processor (102) can obtain a speaker voice prompt. For example, the speaker voice prompt may correspond to a voice signal of the speaker.

[0115] In operation 902, the processor (102) can obtain a style voice prompt. For example, the style voice prompt may correspond to a target voice signal for style modulation.

[0116] In operation 903, the processor (102) can identify whether the speaker of the speaker voice prompt and the speaker of the style voice prompt are the same.

[0117] In operation 904, the processor (102) may decide to use zero-shot speech synthesis for speech synthesis. For example, the processor (102) may decide to use zero-shot speech synthesis for speech synthesis based on the identification that the speaker of the speaker voice prompt and the speaker of the style voice prompt are the same.

[0118] In operation 905, the processor (102) may decide to use style-transfer speech synthesis for speech synthesis. For example, the processor (102) may decide to use style-transfer speech synthesis for speech synthesis based on the identification that the speaker of the speaker voice prompt and the speaker of the style voice prompt are different.

[0119] In operation 906, the processor (102) can obtain input text.

[0120] In operation 907, the processor (102) can identify whether the language of the speaker voice prompt and the language of the input text are the same.

[0121] In operation 908, the processor (102) may decide to use internal language speech synthesis for speech synthesis. The processor (102) may decide to use internal language speech synthesis for speech synthesis based on the identification that the language of the speaker voice prompt and the language of the input text are the same.

[0122] In operation 909, the processor (102) may decide to use cross-language synthesis for speech synthesis. The processor (102) may decide to use cross-language speech synthesis for speech synthesis based on the identification that the language of the speaker voice prompt and the language of the input text are different.

[0123] In operation 910, the processor (102) can perform speech synthesis.

[0124] In one embodiment, the processor (102) uses a text encoder (TE) (205) to obtain phoneme sequences (e.g., x1, x2, …, x) for the input text (202). T ) can be identified (or obtained). For example, phoneme sequences can be based on phoneme units. For example, the first phoneme sequence (x1) of the phoneme sequences can correspond to the first phoneme of the input text (202).

[0125] In one embodiment, the processor (102) may identify (or acquire) acoustic features using an autoregressive (AR) model (e.g., the first neural network (204)). For example, the acoustic features may represent acoustic features of the speech signal of the input text (202) based on the first speech signal (201). For example, the acoustic features may represent the duration, pitch (or fundamental frequency), energy, or a combination thereof of the speech signal of the input text (202) based on the first speech signal (201). For example, the acoustic features may represent sequences of durations of the speech signal for the input text (202) according to (or to which the prosody of) the first speech signal (201) (e.g., d1, d2, …, d T ), pitch sequences (e.g., p1, p2, …, p T)(or, fundamental frequency sequences), energy sequences (e.g., e1, e2, …, e T It may include ), or a combination thereof. For example, duration sequences may be based on phoneme units. For example, duration sequences may be phoneme sequences (e.g., x1, x2, …, x T It can correspond to ). For example, among the duration sequences, the first duration sequence (e.g., d1) may represent the duration of speech at the first phoneme (e.g., x1) of the input text (202). For example, pitch sequences may be based on phoneme units. For example, pitch sequences may correspond to phoneme sequences. For example, among the pitch sequences, the first pitch sequence (e.g., p1) may represent the pitch of speech at the first phoneme of the input text (202). For example, energy sequences may be based on phoneme units. For example, energy sequences may correspond to phoneme sequences. For example, among the energy sequences, the first energy sequence (e.g., e1) may represent the energy of speech at the first phoneme of the input text (202).

[0126] In one embodiment, the processor (102) may obtain (or identify) first information representing the prosody of the input text (202) using a source generation module (e.g., a second neural network (206)). For example, the first information may be referred to as a source representation, source information, source spectrogram, or other terms having an equivalent technical meaning that represent the prosody of the input text (202) based on the first speech signal (201). For example, pitch sequences (or fundamental frequency sequences) and energy sequences obtained (or identified) by an AR model may be provided to the input of the source generation module. For example, the processor (102) may obtain (or identify) the first information using the source generation module based on the pitch sequences and energy sequences. In one example, the source generation module may correspond to a non-autoregressive (NAR) model.

[0127] In one embodiment, the processor (102) may obtain (or identify) second information representing the content of the input text (202) using a filter generation module (e.g., a third neural network (207)). For example, the second information may include information representing the speech (or pronunciation, tone) of the input text (202) by the second user of the second voice signal (203). In one example, the second information may be referred to as a filter representation, filter information, filter spectrogram, or other terms having an equivalent technical meaning. For example, energy sequences obtained (or identified) by an AR model, phoneme sequences for the input text (202), and a hidden representation of the second voice signal (203) (e.g., vectors representing features of the second voice signal (203)) may be provided to the input of the filter generation module. For example, the processor (102) can acquire (or identify) second information based on energy sequences, phoneme sequences, and feature vectors of the second speech signal (203). In one example, the filter generation module may correspond to a NAR model.

[0128] In one embodiment, the processor (102) may obtain a first mel-spectogram based on first information and second information. For example, the first mel-spectogram may be referred to as a coarse mel-spectogram. For example, the mel-spectogram may represent the second user's speech (or pronunciation, timbre, oral structure) for the input text (202) according to the prosody of the first voice signal (201) represented by the first information.

[0129] In one embodiment, the processor (102) can obtain an output speech signal using an acoustic decoder (209) based on a first Mel-spectogram. The acoustic decoder (209) may use sample-adaptive kernel selection to learn a convolutional filter for a speech prompt. For example, since sample-adaptive kernel selection is used, the filter capacity of the acoustic decoder (209) can be improved. For example, since the filter capacity of the acoustic decoder (209) is improved, information (e.g., speech content, style) of the first Mel-spectogram (e.g., coarse Mel-spectogram) can be preserved. For example, the processor (102) can identify (or obtain) a global style embedding for the second voice signal (203) using a speaker encoder. For example, the global style embedding may represent the identity of the second user of the second voice signal (203) (e.g., the second user's intonation, manner of speech). For example, the speaker encoder may be pre-trained to represent the identity of the second user. For example, the processor (102) may use a mapping network to map the style vector of the global style embedding and random noise sampled from a normal Gaussian distribution to the sample-adaptive convolution layer of the acoustic decoder (209). For example, the filters of the sample-adaptive convolution layer may be aggregated based on weights predicted by the style vector. For example, the processor (102) can obtain an output voice signal by using an acoustic decoder (209) including aggregated filters based on a first Mel-spectrogram and a second voice signal (203).For example, the processor (102) can obtain a second Mel-spectogram for an output voice signal by using an acoustic decoder (209) that includes aggregated filters based on a first Mel-spectogram and a second voice signal (203).

[0130] In operation 911, the processor (102) can identify whether a control input for controlling acoustic features is obtained.

[0131] In one embodiment, acoustic features may represent acoustic features of a voice signal for an input text (202) according to a first voice signal (201). For example, acoustic features may represent the duration, pitch (or fundamental frequency), energy, or a combination thereof of the voice signal for an input text (202) according to the first voice signal (201).

[0132] In one embodiment, the processor (102) may display parameters associated with acoustic features using a display (112). For example, parameters associated with acoustic features may include first to eighth parameters. For example, the first parameter may represent the playback speed of the output voice signal (210). In one example, the first parameter may be referred to as a global rate. For example, the second parameter may represent the pitch of the output voice signal (210). In one example, the second parameter may be referred to as a global pitch. For example, the third parameter may represent the total playback time of the output voice signal (210). In one example, the third parameter may be referred to as a total duration. For example, the fourth parameter may represent the pitch of the portion of the output voice signal (210) corresponding to a word of the input text (202). For example, the fifth parameter may represent the playback time for a portion of the output voice signal (210) corresponding to a word of the input text (202). For example, the sixth parameter may represent the pitch for a portion of the output voice signal (210) corresponding to a character of the input text (202). For example, the seventh parameter may represent the playback time for a portion of the output voice signal (210) corresponding to a character of the input text (202). For example, the eighth parameter may represent the pitch for a portion of the output voice signal (210) corresponding to a phoneme of the input text (202). For example, the ninth parameter may represent the playback time for a portion of the output voice signal (210) corresponding to a phoneme of the input text (202).

[0133] In one embodiment, the processor (102) can identify whether a control input (or user input) for parameters associated with acoustic features displayed through the display (112) is obtained.

[0134] In operation 912, the processor (102) can adjust the output of the AR model according to the identification that a control input is obtained. For example, the output of the AR model may include duration sequences, pitch sequences (or fundamental frequency sequences), energy sequences, or a combination thereof of the voice signal for the input text (202) according to (or prosody applied to) the prosody of the first voice signal (201).

[0135] In one example, the processor (102) can control the playback speed of an output voice signal (210) corresponding to an input text (202) according to a control input for a first parameter. For example, the processor (102) can change the values ​​of duration sequences, values ​​of pitch sequences, values ​​of energy sequences, or a combination thereof to control the playback speed of the output voice signal (210).

[0136] In one example, the processor (102) can control the pitch of the output voice signal (210) corresponding to the input text (202) according to a control input for the second parameter. For example, the processor (102) can change the values ​​of the pitch sequences to control the pitch of the output voice signal (210).

[0137] In one example, the processor (102) can control the total duration of the output voice signal (210) corresponding to the input text (202) according to a control input for a third parameter. For example, the processor (102) can change the values ​​of the duration sequences to control the total duration of the output voice signal (210).

[0138] In one example, the processor (102) can control the pitch of a portion of the output voice signal (210) corresponding to a word of the input text (202) according to a control input for a fourth parameter. For example, the processor (102) can identify at least one pitch sequence among pitch sequences that corresponds to a portion of the output voice signal (210). The processor (102) can change the value of the at least one pitch sequence to control the pitch of the portion of the output voice signal (210).

[0139] In one example, the processor (102) can control the playback time of a portion of the output voice signal (210) corresponding to a word of the input text (202) according to a control input for the fifth parameter. For example, the processor (102) can identify at least one duration sequence among duration sequences that corresponds to a portion of the output voice signal (210). The processor (102) can change the value of the at least one duration sequence to control the playback time of the portion of the output voice signal (210).

[0140] In one example, the processor (102) can control the pitch of a portion of the output voice signal (210) corresponding to a character of the input text (202) according to a control input for the sixth parameter. For example, the processor (102) can identify at least one pitch sequence among pitch sequences that corresponds to a portion of the output voice signal (210). The processor (102) can change the value of the at least one pitch sequence to control the pitch of the portion of the output voice signal (210).

[0141] In one example, the processor (102) can control the playback time of a portion of the output voice signal (210) corresponding to a character of the input text (202) according to a control input for the seventh parameter. For example, the processor (102) can identify at least one duration sequence among duration sequences that corresponds to a portion of the output voice signal (210). The processor (102) can change the value of the at least one duration sequence to control the playback time of the portion of the output voice signal (210).

[0142] In one example, the processor (102) can control the pitch of a portion of the output voice signal (210) corresponding to a phoneme of the input text (202) according to a control input for the eighth parameter. For example, the processor (102) can identify a pitch sequence among pitch sequences that corresponds to a portion of the output voice signal (210). The processor (102) can change the value of the pitch sequence to control the pitch of the portion of the output voice signal (210).

[0143] In one example, the processor (102) can control the playback time of a portion of the output voice signal (210) corresponding to a phoneme of the input text (202) according to a control input for the ninth parameter. For example, the processor (102) can identify a duration sequence corresponding to a portion of the output voice signal (210) among duration sequences. The processor (102) can change the value of the duration sequence to control the playback time of the portion of the output voice signal (210).

[0144] The electronic device according to the present disclosure can train a plurality of speech synthesizers (e.g., zero-shot text-to-speech (TTS), style transfer TTS, cross-lingual TTS, and internal lingual TTS) in a single learning session. Additionally, the electronic device according to the present disclosure can enable speech synthesis with relatively little data because explicit speech features such as duration, pitch (or fundamental frequency), energy, or combinations thereof are utilized. Furthermore, since the electronic device according to the present disclosure utilizes explicit speech features, speech features can be controlled on a unit basis (e.g., word, letter, phoneme).

[0145] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description above.

[0146] An electronic device as described above may include a memory that stores instructions and includes one or more storage media. The electronic device may include at least one processor that includes a processing circuit. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause to receive an input text, a first voice signal representing a first user’s utterance regarding text different from the input text, and a second voice signal representing a second user’s utterance regarding another text different from the input text. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause to obtain fundamental frequency sequences and energy sequences of the input text according to the prosody of the first voice signal using a first neural network. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to obtain first information expressing the prosody of the input text according to the prosody of the first speech signal, based on the fundamental frequency sequences and the energy sequences, using a second neural network. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to obtain second information expressing the pronunciation of the input text by the second user corresponding to the second speech signal, based on the energy sequences, using a third neural network.When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the second user to generate a third voice signal representing the second user’s speech in the input text according to the prosody expressed by the first information, using the first information and the second information.

[0147] For example, the electronic device may further include a display. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause parameters associated with the prosody of the input text to be displayed through the display. The parameters may include a parameter associated with the fundamental frequency of a word in the input text, a parameter associated with the length of the word, a parameter associated with the fundamental frequency of a phoneme of the word, a parameter associated with the length of the phoneme, or a combination thereof.

[0148] For example, when the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to obtain user input for at least one parameter associated with the prosody of the input text displayed through the display. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to change at least one fundamental frequency sequence among the fundamental frequency sequences, at least one energy sequence among the energy sequences, or a combination thereof, based on the user input.

[0149] For example, when the instructions are executed individually or collectively by the at least one processor to generate the third voice signal, the electronic device may cause the electronic device to acquire a spectrogram representing the second user’s utterance for the input text according to the prosody expressed by the first information, based on a combination of the first information and the second information. When the instructions are executed individually or collectively by the at least one processor to generate the third voice signal, the electronic device may cause the electronic device to generate the third voice signal representing the characteristics of the second user based on the spectrogram and the second voice signal, using a fourth neural network that uses sample-adaptive kernel selection.

[0150] For example, the first information may be associated with the intonation of the first user, the stress pattern of the first voice signal, or a combination thereof. The second information may be associated with the pronunciation of the second user, the content of the input text, or a combination thereof.

[0151] For example, when the first speech synthesis is performed, the first user and the second user may be the same. When the second speech synthesis is performed, the first user and the second user may be different. When the third speech synthesis is performed, the language of the input text and the language of the first speech signal and the second speech signal may be different.

[0152] For example, the first neural network can correspond to an AR (autoregressive) model. The second neural network and the third neural network can correspond to a NAR (non-autoregressive) model.

[0153] A method performed by an electronic device as described above may include receiving an input text, a first voice signal representing a first user’s utterance regarding text different from the input text, and a second voice signal representing a second user’s utterance regarding another text different from the input text. The method may include using a first neural network to obtain fundamental frequency sequences and energy sequences of the input text according to the prosody of the first voice signal. The method may include using a second neural network to obtain first information expressing the prosody of the input text according to the prosody of the first voice signal based on the fundamental frequency sequences and the energy sequences. The method may include using a third neural network to obtain second information expressing the pronunciation of the input text by the second user corresponding to the second voice signal based on the energy sequences. The above method may include an operation of generating a third voice signal representing the second user's speech regarding the input text according to the prosody expressed by the first information, using the first information and the second information.

[0154] For example, the above method may include the operation of displaying parameters associated with the prosody of the input text through a display. The parameters may include a parameter associated with the fundamental frequency of a word in the input text, a parameter associated with the length of the word, a parameter associated with the fundamental frequency of a phoneme of the word, a parameter associated with the length of the phoneme, or a combination thereof.

[0155] For example, the above method may include an operation of obtaining user input for at least one parameter among parameters associated with the prosody of the input text displayed through a display. The above method may include an operation of changing at least one fundamental frequency sequence among the fundamental frequency sequences, at least one energy sequence among the energy sequences, or a combination thereof, based on the user input.

[0156] For example, the operation of generating the third voice signal may include an operation of acquiring a spectrogram representing the second user’s utterance regarding the input text according to the prosody expressed by the first information, based on a combination of the first information and the second information. The operation of generating the third voice signal may include an operation of generating the third voice signal representing the characteristics of the second user based on the spectrogram and the second voice signal, using a fourth neural network that uses sample-adaptive kernel selection.

[0157] For example, the first information may be associated with the intonation of the first user, the stress pattern of the first voice signal, or a combination thereof. The second information may be associated with the pronunciation of the second user, the content of the input text, or a combination thereof.

[0158] For example, when the first speech synthesis is performed, the first user and the second user may be the same. When the second speech synthesis is performed, the first user and the second user may be different. When the third speech synthesis is performed, the language of the input text and the language of the first speech signal and the second speech signal may be different.

[0159] For example, the first neural network can correspond to an AR (autoregressive) model. The second neural network and the third neural network can correspond to a NARQ (non-autoregressive) model.

[0160] A method performed by an electronic device as described above may include an operation of acquiring fundamental frequency sequences and energy sequences of an input text according to the prosody of a first voice signal representing a first user's utterance using an AR (autoregressive) model. The method may include an operation of acquiring first information representing the prosody of the input text according to the first voice signal based on the fundamental frequency sequences and the energy sequences. The method may include an operation of acquiring second information representing the pronunciation of the input text by the second user corresponding to a second voice signal representing a second user's utterance based on the energy sequences. The method may include an operation of generating a spectrogram representing the second user's utterance regarding the input text according to the prosody based on the first information and the second information. The method may include an operation of generating a third voice signal based on the spectrogram using an acoustic decoder utilizing adaptive kernel selection.

[0161] For example, the above method may include the operation of displaying parameters for controlling the third voice signal through a display of the electronic device. The parameters may include a parameter associated with the fundamental frequency of a word of the input text according to the prosody of the first voice signal, a parameter associated with the length of the word, a parameter associated with the fundamental frequency of a phoneme of the word, a parameter associated with the length of the phoneme, or a combination thereof.

[0162] For example, the above method may include an operation to change at least one fundamental frequency sequence among the fundamental frequency sequences, at least one energy sequence among the energy sequences, or a combination thereof, based on obtaining user input for at least one parameter among the parameters for controlling the third voice signal displayed through the display.

[0163] For example, the first information may represent the intonation of the first user, the stress pattern of the first voice signal, or a combination thereof. The second information may represent the pronunciation of the second user, the content of the input text, or a combination thereof.

[0164] Methods according to the claims or embodiments described in the specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0165] When implemented in software, a computer-readable storage medium may be provided for storing one or more programs (software modules). One or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. One or more programs include instructions that cause the electronic device to execute methods according to the claims or embodiments described in the specification of this disclosure.

[0166] One or more of these programs (software modules, software) may be stored in random access memory, non-volatile memory including flash memory, ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), magnetic disc storage device, optical storage device (e.g., Compact Disc-ROM (CD-ROM), Digital Versatile Discs (DVDs)), magnetic cassette, or a combination thereof. Or, they may be stored in memory composed of some or all of these. Additionally, the above program may be stored on an attachable storage device accessible via a communication network such as the Internet, Intranet, Local Area Network (LAN), Wide LAN (WLAN), or Storage Area Network (SAN), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure through an external port. Additionally, a separate storage device on a communication network may be connected to a device performing an embodiment of the present disclosure.

[0167] In the specific embodiments of the present disclosure described above, the components of the electronic device included in the present disclosure are expressed in a singular or plural form according to the specific embodiments presented. However, the singular or plural expression of said components is selected to suit the circumstances presented for convenience of explanation, and the present disclosure is not limited to singular or plural components; even if a component is expressed in the plural, it may be composed of a singular form, and even if a component is expressed in the singular form, it may be composed of a plural form.

[0168] Meanwhile, although specific embodiments have been described in the detailed description of the present disclosure, it is understood that various modifications are possible within the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof.

Claims

Claim 1 In an electronic device, a memory that stores instructions and includes one or more storage media; The invention comprises at least one processor including a processing circuit, wherein the instructions, when executed individually or collectively by the at least one processor, receive an input text, a first voice signal representing a first user’s utterance regarding text different from the input text, and a second voice signal representing a second user’s utterance regarding another text different from the input text; using a text encoder, obtain phoneme sequences for the input text; using a first neural network, obtain fundamental frequency sequences and energy sequences of the input text according to the prosody of the first voice signal based on the first voice signal and the phoneme sequences; using a second neural network, obtain first information expressing the prosody of the input text according to the prosody of the first voice signal based on the fundamental frequency sequences and the energy sequences; and using a third neural network, expressing the pronunciation of the input text by the second user corresponding to the second voice signal based on the energy sequences, the second voice signal, and the phoneme sequences. An electronic device that causes the electronic device to acquire second information and, using the first information and the second information, generate a third voice signal representing the second user’s speech in relation to the input text according to the prosody expressed by the first information. Claim 2 The electronic device according to claim 1 further comprises a display, and the instructions cause the electronic device to display parameters associated with the prosody of the input text through the display when executed individually or collectively by the at least one processor, wherein the parameters include a parameter associated with the fundamental frequency of a word of the input text, a parameter associated with the length of the word, a parameter associated with the fundamental frequency of a phoneme of the word, a parameter associated with the length of the phoneme, or a combination thereof. Claim 3 An electronic device according to paragraph 2, wherein the instructions, when executed individually or collectively by the at least one processor, obtain user input for at least one parameter of the parameters associated with the prosody of the input text displayed through the display, and cause the electronic device to change at least one fundamental frequency sequence of the fundamental frequency sequences, at least one energy sequence of the energy sequences, or a combination thereof based on the user input. Claim 4 In claim 1, the electronic device causes the instructions to be executed individually or collectively by the at least one processor to generate the third voice signal, to obtain a spectrogram representing the second user’s speech in the input text according to the prosody expressed by the first information based on a combination of the first information and the second information, and to generate the third voice signal representing the characteristics of the second user based on the spectrogram and the second voice signal using a fourth neural network using sample-adaptive kernel selection. Claim 5 An electronic device according to claim 1, wherein the first information is associated with the intonation of the first user, the stress pattern of the first voice signal, or a combination thereof, and the second information is associated with the pronunciation of the second user, the content of the input text, or a combination thereof. Claim 6 An electronic device according to claim 1, wherein when the first voice synthesis is performed, the first user and the second user are the same, when the second voice synthesis is performed, the first user and the second user are different, and when the third voice synthesis is performed, the language of the input text and the language of the first voice signal and the second voice signal are different. Claim 7 An electronic device according to claim 1, wherein the first neural network corresponds to an AR (autoregressive) model, and the second neural network and the third neural network correspond to a NAR (non-autoregressive) model. Claim 8 A method performed by an electronic device comprises: receiving an input text, a first voice signal representing a first user’s utterance regarding text different from the input text, and a second voice signal representing a second user’s utterance regarding another text different from the input text; using a text encoder to obtain phoneme sequences for the input text; using a first neural network to obtain fundamental frequency sequences and energy sequences of the input text according to the prosodic nature of the first voice signal based on the first voice signal and the phoneme sequences; using a second neural network to obtain first information expressing the prosodic nature of the input text according to the prosodic nature of the first voice signal based on the fundamental frequency sequences and the energy sequences; using a third neural network to obtain second information expressing the pronunciation of the input text by the second user corresponding to the second voice signal based on the energy sequences, the second voice signal, and the phoneme sequences. A method comprising the operation of generating a third voice signal representing the second user’s speech in the input text according to the prosody expressed by the first information, using the first information and the second information. Claim 9 A method according to claim 8, further comprising the operation of displaying parameters associated with the prosodic nature of the input text through a display, wherein the parameters include a parameter associated with the fundamental frequency of a word of the input text, a parameter associated with the length of the word, a parameter associated with the fundamental frequency of a phoneme of the word, a parameter associated with the length of the phoneme, or a combination thereof. Claim 10 A method according to claim 8, further comprising: an operation of obtaining user input for at least one parameter among parameters associated with the prosody of the input text displayed through a display; and an operation of changing at least one fundamental frequency sequence among the fundamental frequency sequences, at least one energy sequence among the energy sequences, or a combination thereof based on the user input. Claim 11 In claim 8, the operation of generating the third voice signal comprises: an operation of acquiring a spectrogram representing the second user’s speech for the input text according to the prosody expressed by the first information, based on a combination of the first information and the second information; and an operation of generating the third voice signal representing the characteristics of the second user based on the spectrogram and the second voice signal, using a fourth neural network that uses sample-adaptive kernel selection. Claim 12 A method according to claim 8, wherein the first information is associated with the intonation of the first user, the stress pattern of the first voice signal, or a combination thereof, and the second information is associated with the pronunciation of the second user, the content of the input text, or a combination thereof. Claim 13 In claim 8, when the first voice synthesis is performed, the first user and the second user are the same; when the second voice synthesis is performed, the first user and the second user are different; and when the third voice synthesis is performed, the language of the input text and the language of the first voice signal and the second voice signal are different. Claim 14 A method according to claim 8, wherein the first neural network corresponds to an AR (autoregressive) model, and the second neural network and the third neural network correspond to a NARQ (non-autoregressive) model. Claim 15 A method performed by an electronic device comprising: acquiring fundamental frequency sequences and energy sequences of an input text according to the prosodic nature of a first voice signal representing a first user’s speech using an AR (autoregressive) model; acquiring first information representing the prosodic nature of the input text according to the first voice signal based on the fundamental frequency sequences and the energy sequences; acquiring second information representing the pronunciation of the input text by the second user corresponding to a second voice signal representing a second user’s speech based on the energy sequences; generating a spectrogram representing the speech of the second user regarding the input text according to the prosodic nature based on the first information and the second information; and generating a third voice signal based on the spectrogram using an acoustic decoder utilizing adaptive kernel selection. Claim 16 A method according to claim 15, further comprising the operation of displaying parameters for controlling the third voice signal through a display of the electronic device, wherein the parameters include a parameter associated with the fundamental frequency of a word of the input text according to the prosody of the first voice signal, a parameter associated with the length of the word, a parameter associated with the fundamental frequency of a phoneme of the word, a parameter associated with the length of the phoneme, or a combination thereof. Claim 17 A method according to claim 16, further comprising an operation to change at least one fundamental frequency sequence among the fundamental frequency sequences, at least one energy sequence among the energy sequences, or a combination thereof, based on obtaining user input for at least one parameter among the parameters for controlling the third voice signal displayed through the display. Claim 18 A method according to paragraph 15, wherein the first information represents the intonation of the first user, the stress pattern of the first voice signal, or a combination thereof, and the second information represents the pronunciation of the second user, the content of the input text, or a combination thereof. Claim 19 In an electronic device, a memory that stores instructions and includes one or more storage media; The apparatus comprises at least one processor including a processing circuit, wherein the instructions, when executed individually or collectively by the at least one processor to generate a third voice signal, receive an input text, a first voice signal representing a first user’s utterance regarding text different from the input text, and a second voice signal representing a second user’s utterance regarding text different from the input text; using a first neural network, obtain fundamental frequency sequences and energy sequences of the input text according to the prosody of the first voice signal; using a second neural network, obtain first information expressing the prosody of the input text according to the prosody of the first voice signal based on the fundamental frequency sequences and the energy sequences; using a third neural network, obtain second information expressing the pronunciation of the input text by the second user corresponding to the second voice signal based on the energy sequences; and based on a combination of the first information and the second information, the input text according to the prosody expressed by the first information An electronic device that causes the electronic device to acquire a spectrogram representing the speech of a second user, and to generate a third speech signal representing the characteristics of the second user based on the spectrogram and the second speech signal using a fourth neural network that uses sample-adaptive kernel selection. Claim 20 A method performed by an electronic device comprises: receiving an input text, a first voice signal representing a first user’s utterance regarding text different from the input text, and a second voice signal representing a second user’s utterance regarding another text different from the input text; using a first neural network to obtain fundamental frequency sequences and energy sequences of the input text according to the prosody of the first voice signal; using a second neural network to obtain first information expressing the prosody of the input text according to the prosody of the first voice signal based on the fundamental frequency sequences and the energy sequences; using a third neural network to obtain second information expressing the pronunciation of the input text by the second user corresponding to the second voice signal based on the energy sequences; and obtaining a spectrogram representing the second user’s utterance regarding the input text according to the prosody expressed by the first information based on a combination of the first information and the second information. A method comprising the operation of generating a third voice signal representing the characteristics of the second user based on the spectrogram and the second voice signal, using a fourth neural network that uses sample-adaptive kernel selection.

Images