System and method for generating speaker and emotion condition information for text-to-speech conversion

Abstract

The present disclosure relates to a method, performed by at least one processor, for generating speaker and emotion condition information for text-to-speech conversion, comprising the steps of: generating a condition generation model on the basis of a machine learning-based learning method; obtaining speaker information corresponding to a speaker; obtaining emotion information corresponding to emotion; and generating speaker and emotion condition information as result data by inputting the obtained speaker information and the obtained emotion information as input data to the generated condition generation model.

Description

System and method for generating speaker and emotional condition information for text-to-speech conversion

[0001] The present disclosure relates to a system and method for generating speaker and emotion condition information for text-to-speech conversion, and more particularly, to a system and method for generating speaker and emotion condition information necessary for generating a voice containing a specific speaker and a specific emotion based on text.

[0002] Today, broadcast programs, including audio content, are actively provided not only through traditional channels like TV and radio, but also through online platforms like YouTube and podcasts. In these contents, naturalness and rich emotional expression, not just the speaker's voice, are crucial. For this purpose, text-to-speech technology is widely utilized.

[0003] Text-to-speech (TTS) technology converts input text into natural, human-like speech. It is utilized in various applications such as announcements, navigation, AI speakers, and virtual assistants. TTS technology involves mapping text information into appropriate acoustic characteristics, considering speech style, intonation, and speed, and synthesizing this into a voice waveform to convert the content of the input text into speech.

[0004] Text-to-speech technology can be broadly categorized into two approaches. The first is a concatenative approach, which combines actual speech data at the syllable or word level to generate sentences. The second is a parametric approach, which synthesizes speech by representing its characteristics as parameters. Recently, deep learning-based neural network models have become the mainstream approach to enhance the naturalness and expressiveness of speech.

[0005] While these technologies can generate human-like voices, there's a growing demand in broadcasting and content production settings to generate voices that reflect specific speakers and emotions. However, existing text-to-speech systems focus on generating voices that are typical of a typical speaker or emotion-neutral, limiting their ability to naturally generate voices that reflect specific speaker styles or emotions.

[0006] Accordingly, there is a growing technological need for designing and utilizing conditional information that can more precisely generate voices that reflect a specific speaker's tone and specific emotions, tailored to the user's intent. In particular, technologies that generate appropriate speaker conditional information and emotional conditional information for text-to-speech conversion and integrate and apply this information are becoming increasingly important.

[0007] The background technology described above is something that the inventor possessed or acquired in the process of deriving the disclosure of the present application, and cannot necessarily be said to be a publicly known technology disclosed to the general public prior to the present application.

[0008] The present disclosure provides a method for generating speaker and emotional condition information for text-to-speech conversion, a computer program stored in a recording medium, and a device (system) to solve the above-described problems.

[0009] The present disclosure can be implemented in various ways, including as a method, a system (device), or a computer program stored on a readable storage medium.

[0010] According to one embodiment of the present disclosure, a method for generating speaker and emotion condition information for text-to-speech conversion performed by at least one processor may include a step of generating a condition generation model based on a machine learning-based learning method, a step of obtaining speaker information corresponding to a speaker, a step of obtaining emotion information corresponding to an emotion, and a step of generating speaker and emotion condition information as result data by inputting the obtained speaker information and the obtained emotion information into a generated condition generation model as input data.

[0011] In addition, the step of obtaining speaker information may include a step of obtaining, as speaker information, at least one of a speaker embedding vector and a speaker voice sample from a voice spoken by a specific speaker, and the step of obtaining emotion information may include a step of obtaining, as emotion information, at least one of an emotion embedding vector and an emotion voice sample from a voice containing a specific emotion.

[0012] In addition, the step of obtaining speaker information may include a step of obtaining information about a speaker who uttered a voice corresponding to each of a plurality of sentences included in a single document, and the step of obtaining emotion information may include a step of generating an emotion prediction model and a step of obtaining emotion information by inputting a plurality of sentences into the emotion prediction model or inputting a plurality of sentences and the obtained information about the speaker into the emotion prediction model.

[0013] In addition, the step of generating an emotion prediction model may include a step of generating input data using a plurality of text-based training sentences, a step of generating correct answer data using information about emotions contained in voices corresponding to each of the plurality of training sentences, and a step of generating an emotion prediction model by training the generated input data based on a regression-based loss function so that the error between the generated correct answer data and the resulting data is minimized.

[0014] In addition, the step of generating input data may include a step of extracting a sentence embedding vector from each of a plurality of training sentences, a step of generating a combined embedding vector as input data by connecting the speaker embedding vector and the extracted sentence embedding vector and the extracted speaker embedding vector from information about a speaker who uttered a voice corresponding to each of the plurality of training sentences, and the step of generating correct answer data may include a step of extracting an emotion embedding vector as correct answer data from information about an emotion contained in a pitch corresponding to each of the plurality of training sentences.

[0015] In addition, the step of generating input data may include a step of tokenizing a plurality of training sentences and a step of inserting an end-of-sentence indicator at the end of each of the tokenized plurality of training sentences and then interconnecting them to generate a single sentence token sequence as input data, and the step of generating correct answer data may include a step of extracting an emotion embedding vector as correct answer data from information about an emotion contained in a voice corresponding to each of the plurality of training sentences.

[0016] In addition, the step of obtaining speaker information may include a step of obtaining information about a speaker who uttered a voice corresponding to each of a plurality of sentences included in a single document, and the step of obtaining emotion information may include a step of generating an emotion search model and a step of obtaining one emotion information from among a plurality of regularly stored emotion information through the generated emotion search model by inputting the plurality of sentences and the obtained information about the speaker into the generated emotion search model.

[0017] Additionally, the generated emotion search model may include a text encoder that receives a specific sentence as input and extracts a text embedding vector corresponding to the specific sentence, and an audio encoder that receives a speech containing a specific emotion as input and extracts an emotion embedding vector corresponding to the speech containing the specific emotion.

[0018] In addition, the step of generating an emotion search model may include a step of extracting a plurality of text embedding vectors corresponding to each of a plurality of training sentences written based on text using a text encoder, a step of extracting a plurality of audio embedding vectors from speech corresponding to each of a plurality of training sentences using an audio encoder, and a step of generating an emotion search model by training based on a contrastive loss function such that the similarity between the text embedding vectors and the audio embedding vectors extracted corresponding to the same training sentence is maximized and the similarity between the text embedding vectors and the audio embedding vectors extracted corresponding to different training sentences is minimized.

[0019] In addition, the step of generating an emotion search model may include a step of extracting a first text embedding vector from a target training sentence among a plurality of training sentences written based on text using a text encoder, a step of extracting a second text embedding vector from training sentences before and after the target training sentence among the plurality of training sentences using the text encoder, a step of generating a target text embedding vector by reflecting the extracted second text embedding vector to the extracted first text embedding vector, a step of extracting a target audio embedding vector from a voice corresponding to the target training sentence using an audio encoder, and a step of generating an emotion search model by training so that the similarity between the generated target text embedding vector and the extracted target audio embedding vector is maximized.

[0020] In addition, the step of generating an emotion search model may include a step of extracting a first text embedding vector from a target training sentence among a plurality of training sentences written based on text using a text encoder, a step of extracting a second text embedding vector from training sentences before and after the target training sentence among the plurality of training sentences using the text encoder, a step of extracting a third text embedding vector from the plurality of training sentences using the text encoder, a step of generating a target text embedding vector by reflecting the extracted second text embedding vector and the extracted third text embedding vector to the extracted first text embedding vector, a step of extracting a target audio embedding vector from a voice corresponding to the target training sentence using an audio encoder, and a step of generating an emotion search model by training so that the similarity between the generated target text embedding vector and the extracted target audio embedding vector is maximized.

[0021] In addition, the step of generating a conditional generation model may include a step of generating first input data using a voice signal corresponding to a voice spoken by a first speaker with a first emotion, a step of generating second input data using a voice signal corresponding to a voice spoken by a second speaker with a second emotion, a step of generating correct answer data using an embedding vector extracted from a voice spoken by the first speaker with a second emotion, and a step of generating a conditional generation model by learning to minimize an error between result data derived by inputting the generated first input data and the generated second input data and the generated correct answer data.

[0022] In addition, the step of generating a conditional generation model may include a step of generating first input data using a first voice embedding vector extracted from a voice spoken by a first speaker with a second emotion, a step of generating second input data using a second voice embedding vector generated by passing an emotion index indicating the first emotion through an embedding layer, a step of generating correct answer data using a correct answer embedding vector extracted from a voice spoken by the first speaker with the first emotion, and a step of generating a conditional generation model by learning to minimize an error between result data derived by inputting the generated first input data and the generated second input data and the generated correct answer data.

[0023] In addition, the step of generating a conditional generation model may include a step of generating a first voice uttered by the first speaker with a second emotion by modulating a voice uttered by the first speaker with a first emotion, and generating first input data using the generated first voice; a step of generating a second voice uttered by the second speaker with a first emotion by modulating a voice uttered by the first speaker with a first emotion, and generating second input data using the generated second voice; a step of generating correct answer data using an embedding vector extracted from the voice uttered by the first speaker with the first emotion; and a step of generating a conditional generation model by learning to minimize an error between result data derived by inputting the generated first input data and the generated second input data and the generated correct answer data.

[0024] In addition, the step of generating a conditional generation model may include a step of generating a first voice spoken by a first speaker with a second emotion by modulating a voice spoken by a first speaker with a first emotion, extracting a first embedding vector from the generated first voice, and generating first input data by removing information about the first emotion from the extracted first embedding vector, a step of generating a second voice spoken by a second speaker with a first emotion by modulating a voice spoken by a first speaker with a first emotion, extracting a second embedding vector from the generated second voice, and generating second input data by removing information about the first emotion from the extracted second embedding vector, a step of generating correct answer data by using the embedding vector extracted from the voice spoken by the first speaker with the first emotion, and a step of generating a conditional generation model by training such that an error between result data derived by inputting the generated first input data and the generated second input data and the generated correct answer data is minimized.

[0025] Additionally, the generated condition generation model may include at least one of a multi-layer perceptron (MLP) model and a transformer-based diffusion model.

[0026] According to one embodiment of the present disclosure, a method for generating emotional information using an emotional prediction model, performed by at least one processor, includes the steps of obtaining one or more target sentences and speaker information corresponding to each of the one or more target sentences, and the step of outputting an emotional prediction result for one or more target sentences obtained from the obtained one or more target sentences and the obtained speaker information using the emotional prediction model, wherein the emotional prediction model may be generated or updated by generating correct answer data using a plurality of training sentences written based on text and information about emotions contained in voices corresponding to each of the plurality of training sentences, and learning to minimize an error between result data derived by inputting the generated plurality of training sentences and the generated correct answer data based on a regression-based loss function.

[0027] A computer program stored in a computer-readable recording medium may be provided to execute the method for generating speaker and emotional condition information for text-to-speech conversion described above on a computer.

[0028] According to one embodiment of the present disclosure, an information processing system includes a memory and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, wherein the at least one program may include instructions for generating a condition generation model based on a machine learning-based learning method, obtaining speaker information corresponding to a speaker, obtaining emotion information corresponding to an emotion, and inputting the acquired speaker information and the acquired emotion information as input data into the generated condition generation model to generate speaker and emotion condition information as result data.

[0029] According to some embodiments of the present disclosure, by generating speaker and emotional condition information that can be utilized in text-to-speech conversion, it is possible to generate a voice that reflects both speaker information and emotional information, thereby enabling voice synthesis that is rich in emotional expression and has personality beyond the level of simple sentence reading.

[0030] According to some embodiments of the present disclosure, since it is possible to express various emotions by the same speaker, or to express the same emotion by multiple speakers, there is an advantage in that the diversity of voices that can be combined is dramatically increased.

[0031] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs (referred to as “one skilled in the art”) from the description of the claims.

[0032] Embodiments of the present disclosure will be described below with reference to the accompanying drawings, wherein like reference numerals represent similar elements, but are not limited thereto.

[0033] FIG. 1 is a diagram illustrating an example of generating speaker and emotion condition information according to one embodiment of the present disclosure.

[0034] FIG. 2 is a schematic diagram showing a configuration in which an information processing system is connected to enable communication with a plurality of user terminals in order to provide a speaker and emotion condition information generation service for text-to-speech conversion according to one embodiment of the present disclosure.

[0035] FIG. 3 is a block diagram showing the internal configuration of a user terminal and an information processing system according to one embodiment of the present disclosure.

[0036] FIG. 4 is a diagram showing an example of obtaining emotional information through an emotional prediction model according to one embodiment of the present disclosure.

[0037] FIG. 5 is a flowchart of a method for generating an emotion prediction model according to one embodiment of the present disclosure.

[0038] FIG. 6 is a diagram showing an example of obtaining emotional information through an emotional search model according to one embodiment of the present disclosure.

[0039] FIG. 7 is a flowchart of a method for generating an emotion search model according to one embodiment of the present disclosure.

[0040] FIG. 8 is a diagram illustrating a structure for performing alignment and contrast learning on an embedding space between a text embedding vector extracted from a text encoder of an emotion retrieval model according to one embodiment of the present disclosure and an emotion embedding vector extracted through an audio encoder.

[0041] FIG. 9 is a flowchart of a method for generating a condition generation model according to one embodiment of the present disclosure.

[0042] FIG. 10 is a flowchart of a method for generating speaker and emotion condition information for text-to-speech conversion according to one embodiment of the present disclosure.

[0043] Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, specific descriptions of widely known functions or configurations will be omitted if they may unnecessarily obscure the gist of the present disclosure.

[0044] In the attached drawings, identical or corresponding components are assigned the same reference numerals. Furthermore, in the description of the embodiments below, duplicate descriptions of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0045] The advantages and features of the disclosed embodiments, and methods for achieving them, will become clearer with reference to the embodiments described below, along with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided solely to ensure the completeness of the disclosure and to fully inform those skilled in the art of the scope of the invention.

[0046] The terms used in this specification will be briefly explained, followed by a detailed description of the disclosed embodiments. The terms used in this specification have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of engineers working in the relevant field, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0047] In this specification, singular expressions include plural expressions unless the context clearly indicates otherwise. Furthermore, plural expressions include singular expressions unless the context clearly indicates otherwise. When a part of the specification is said to include a component, this does not exclude other components, but rather implies that other components may be included, unless otherwise specifically stated.

[0048] Also, the term 'module' or 'part' used in the specification means a software or hardware component, and the 'module' or 'part' performs certain roles. However, the 'module' or 'part' is not limited to software or hardware. The 'module' or 'part' may be configured to reside on an addressable storage medium and may be configured to execute one or more processors. Thus, as an example, the 'module' or 'part' may include at least one of components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, or variables. The functionality provided within the components and 'modules' or 'parts' may be combined into a smaller number of components and 'modules' or 'parts', or further separated into additional components and 'modules' or 'parts'.

[0049] In one embodiment of the present disclosure, a 'module' or 'unit' may be implemented as a processor and a memory. 'Processor' should be broadly construed to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and the like. In some circumstances, a 'processor' may also refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field-programmable gate array (FPGA), and the like. A 'processor' may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or any other such combination of configurations. In addition, 'memory' should be broadly construed to include any electronic component capable of storing electronic information. 'Memory' may refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. Memory is said to be in electronic communication with the processor if the processor can read information from, and / or write information to, the memory. Memory integrated in a processor is in electronic communication with the processor.

[0050] In the present disclosure, the "system" may include, but is not limited to, at least one of a server device and a cloud device. For example, the system may be comprised of one or more server devices. As another example, the system may be comprised of one or more cloud devices. As yet another example, the system may be configured and operated by a combination of a server device and a cloud device.

[0051] FIG. 1 is a diagram illustrating an example of generating speaker and emotion condition information according to one embodiment of the present disclosure. As illustrated in FIG. 1 , the information processing system (100) can receive speaker information (110) and emotion information (120) and generate speaker and emotion condition information (130).

[0052] In one embodiment, speaker information (110) may be information obtained from speech uttered by a specific speaker. For example, speaker information (110) may include at least one of an embedding vector extracted from speech uttered by the speaker, a speaker identification index (label), or a speech signal (waveform) of the speaker. In particular, the speaker embedding vector is extracted through a pretrained model such as Pyannote, ECAPA-TDNN, or TitaNet, and may refer to a high-dimensional vector that compressively expresses the vocal characteristics of a specific speaker.

[0053] Additionally, the speaker information (110) may be a speaker voice sample (reference speech) containing the voice of a specific speaker. That is, the speaker information (110) may be not only a fixed vector but also the raw voice signal itself.

[0054] In one embodiment, the emotion information (120) may be information obtained through a voice or emotion label reflecting a specific emotion. For example, the emotion information (120) may include at least one of an emotion embedding vector, an emotion identification index (label), or an emotion-containing voice signal (waveform). In particular, the emotion embedding vector is a vector representing an emotional state (e.g., happiness, anger, sadness, etc.) and may be extracted by passing an emotion prediction model (emotion retrieval model) or an emotion index through an embedding layer, but is not limited thereto.

[0055] Additionally, the emotional information (120) may be an emotional voice sample (reference speech) expressing a specific emotion. That is, the emotional information (120) may be not only a fixed vector but also the raw voice signal itself.

[0056] In one embodiment, the information processing system (100) can use speaker information (110) and emotion information (120) as conditions to generate speaker and emotion condition information (130) in a combined form.

[0057] Here, the speaker and emotion condition information (130) may be an integrated embedding vector (global condition embedding) that simultaneously reflects information on two conditions, namely, speaker and emotion.

[0058] Speaker and emotion condition information (130) can be generated by a condition generator, and can then be used as an input condition of a text-to-speech (TTS) model, or in some cases, as an input condition of a voice conversion model, thereby enabling voice synthesis that reflects a specific speaker and a specific emotion. Speaker and emotion condition information (130) can be generated based on at least one of a multi-layer perceptron (MLP) model and a transformer-based diffusion model, but is not limited thereto.

[0059] FIG. 2 is a schematic diagram showing a configuration in which an information processing system (230) is connected to a plurality of user terminals (210_1, 210_2, 210_3) so as to be able to communicate with each other in order to provide a speaker and emotion condition information generation service for text-to-speech conversion according to one embodiment of the present disclosure. As illustrated, the plurality of user terminals (210_1, 210_2, 210_3) may be connected to an information processing system (230) capable of generating speaker and emotion condition information for text-to-speech conversion via a network (220). Here, the plurality of user terminals (210_1, 210_2, 210_3) may include terminals of users who will be provided with speaker and emotion condition information generation or related services for text-to-speech conversion. In one embodiment, the information processing system (230) may include one or more server devices and / or databases capable of storing, providing, and executing computer-executable programs and data related to generating speaker and emotional condition information for text-to-speech conversion, or one or more distributed computing devices and / or distributed databases based on cloud computing services.

[0060] The speaker and emotion condition information generation service for text-to-speech conversion provided by the information processing system (230) may be provided to users through applications or web browsers installed on each of a plurality of user terminals (210_1, 210_2, 210_3). For example, the information processing system (230) may provide a service related to speaker and emotion condition information generation for text-to-speech conversion received from user terminals (210_1, 210_2, 210_3) through applications or perform corresponding processing.

[0061] A plurality of user terminals (210_1, 210_2, 210_3) can communicate with an information processing system (230) via a network (220). The network (220) can be configured to enable communication between the plurality of user terminals (210_1, 210_2, 210_3) and the information processing system (230). Depending on the installation environment, the network (220) can be configured as, for example, a TCP / IP-based network, Ethernet, a wired home network (Power Line Communication), a telephone line communication device, and a wired network such as RS-serial communication, a mobile communication network, a wireless network such as WLAN (Wireless LAN), Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and may include not only a communication method utilizing a communication network (e.g., a mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.) that the network (220) may include, but also short-range wireless communication between user terminals (210_1, 210_2, 210_3).

[0062] In FIG. 2, a mobile phone terminal (210_1), a tablet terminal (210_2), and a PC terminal (210_3) are illustrated as examples of user terminals, but are not limited thereto, and the user terminals (210_1, 210_2, 210_3) may be any computing device capable of wired and / or wireless communication and capable of installing and executing applications or web browsers. For example, the user terminals may include AI speakers, smartphones, mobile phones, navigation systems, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, game consoles, wearable devices, IoT (Internet of Things) devices, VR (virtual reality) devices, AR (augmented reality) devices, MR (Mixed reality) devices, set-top boxes, etc. In addition, although FIG. 2 illustrates three user terminals (210_1, 210_2, 210_3) communicating with the information processing system (230) via the network (220), this is not limited thereto, and a different number of user terminals may be configured to communicate with the information processing system (230) via the network (220).

[0063] FIG. 3 is a block diagram showing the internal configuration of a user terminal (210) and an information processing system (230) according to one embodiment of the present disclosure. The user terminal (210) may refer to any computing device capable of executing an application or a web browser and capable of wired / wireless communication, and may include, for example, a mobile phone terminal (210_1), a tablet terminal (210_2), a PC terminal (210_3) of FIG. 2. As illustrated, the user terminal (210) may include a memory (312), a processor (314), a communication module (316), and an input / output interface (318). Similarly, the information processing system (230) may include a memory (332), a processor (334), a communication module (336), and an input / output interface (338). As illustrated in FIG. 3, the user terminal (210) and the information processing system (230) may be configured to communicate information and / or data via a network (220) using respective communication modules (316, 336). In addition, the input / output device (320) may be configured to input information and / or data to the user terminal (210) or output information and / or data generated from the user terminal (210) via the input / output interface (318).

[0064] The memory (312, 332) may include any non-transitory computer-readable recording medium. According to one embodiment, the memory (312, 332) may include a permanent mass storage device such as a read-only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. As another example, a permanent mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be included in the user terminal (210) or the information processing system (230) as a separate permanent storage device distinct from the memory. In addition, the memory (312, 332) may store an operating system and at least one program code (e.g., code for an application installed in the user terminal (210).

[0065] These software components may be loaded from a computer-readable recording medium separate from the memory (312, 332). This separate computer-readable recording medium may include a recording medium directly connectable to the user terminal (210) and the information processing system (230), and may include, for example, a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. As another example, the software components may be loaded into the memory (312, 332) through a communication module other than a computer-readable recording medium. For example, at least one program may be loaded into the memory (312, 332) based on a computer program that is installed by files provided by developers or a file distribution system that distributes installation files of applications through a network (220).

[0066] The processor (314, 334) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (314, 334) by a memory (312, 332) or a communication module (316, 336). For example, the processor (314, 334) may be configured to execute instructions received according to program code stored in a storage device such as the memory (312, 332).

[0067] The communication module (316, 336) may provide a configuration or function for the user terminal (210) and the information processing system (230) to communicate with each other via the network (220), and may provide a configuration or function for the user terminal (210) and / or the information processing system (230) to communicate with another user terminal or another system (e.g., a separate cloud system, etc.). For example, a request or data (e.g., a request for generating speaker and emotional condition information for text-to-speech conversion, etc.) generated by the processor (314) of the user terminal (210) according to a program code stored in a recording device such as a memory (312) may be transmitted to the information processing system (230) via the network (220) under the control of the communication module (316). Conversely, a control signal or command provided under the control of the processor (334) of the information processing system (230) may be received by the user terminal (210) through the communication module (316) of the user terminal (210) via the communication module (336) and the network (220). For example, the user terminal (210) may receive the results of generating speaker and emotion condition information for text-to-speech conversion from the information processing system (230) through the communication module (316).

[0068] The input / output interface (318) may be a means for interfacing with an input / output device (320). As an example, the input device may include a device such as a camera, keyboard, microphone, mouse, etc., including an audio sensor and / or an image sensor, and the output device may include a device such as a display, a speaker, a haptic feedback device, etc. As another example, the input / output interface (318) may be a means for interfacing with a device that has a configuration or function integrated into one for performing input and output, such as a touch screen. For example, when the processor (314) of the user terminal (210) processes a command of a computer program loaded into the memory (312), a service screen configured using information and / or data provided by the information processing system (230) or another user terminal may be displayed on the display through the input / output interface (318). In FIG. 3, the input / output device (320) is illustrated as not being included in the user terminal (210), but is not limited thereto, and may be configured as a single device with the user terminal (210). In addition, the input / output interface (338) of the information processing system (230) may be a means for interfacing with a device (not shown) for input or output that is connected to the information processing system (230) or that the information processing system (230) may include. In FIG. 3, the input / output interfaces (318, 338) are illustrated as elements configured separately from the processors (314, 334), but are not limited thereto, and the input / output interfaces (318, 338) may be configured to be included in the processors (314, 334).

[0069] The user terminal (210) and the information processing system (230) may include more components than those shown in FIG. 3. However, it is not necessary to explicitly illustrate most of the conventional components. According to one embodiment, the user terminal (210) may be implemented to include at least some of the input / output devices (320) described above. In addition, the user terminal (210) may further include other components, such as a transceiver, a Global Positioning System (GPS) module, a camera, various sensors, a database, etc. For example, if the user terminal (210) is a smartphone, it may include components that a smartphone generally includes, and various components, such as an acceleration sensor, a gyro sensor, a camera module, various physical buttons, buttons using a touch panel, input / output ports, and a vibrator for vibration, may be implemented to be further included in the user terminal (210). According to one embodiment, the processor (314) of the user terminal (210) may be configured to operate an application, etc. At this time, code associated with the application and / or program may be loaded into the memory (312) of the user terminal (210).

[0070] While a program for an application, etc. is running, the processor (314) may receive text, images, videos, voices and / or actions, etc. input or selected through input devices such as a camera, microphone, including a touch screen, keyboard, audio sensor and / or image sensor connected to an input / output interface (318), and may store the received text, images, videos, voices and / or actions, etc. in the memory (312) or provide them to the information processing system (230) through the communication module (316) and the network (220). For example, the processor (314) may receive data related to the generation of speaker and emotional condition information for text-to-speech conversion by a user and provide the data to the information processing system (230) through the communication module (316) and the network (220).

[0071] The processor (314) of the user terminal (210) may be configured to manage, process, and / or store information and / or data received from an input device (320), another user terminal, an information processing system (230), and / or multiple external systems. The information and / or data processed by the processor (314) may be provided to the information processing system (230) via a communication module (316) and a network (220). The processor (314) of the user terminal (210) may transmit information and / or data to an input / output device (320) via an input / output interface (318) and output the information and / or data.

[0072] The processor (334) of the information processing system (230) may be configured to manage, process, and / or store information and / or data received from multiple user terminals (210) and / or multiple external systems. Information and / or data processed by the processor (334) may be provided to the user terminal (210) via a communication module (336) and a network (220).

[0073] The processor (334) of the information processing system (230) may be configured to output processed information and / or data through an output device (320) such as a display output capable device (e.g., a touch screen, a display, etc.) or a voice output capable device (e.g., a speaker) of the user terminal (210).

[0074] FIG. 4 is a diagram illustrating an example of how emotional information is obtained through an emotional prediction model (420) according to one embodiment of the present disclosure. As illustrated in FIG. 4, the emotional prediction module (420) can obtain the emotional information necessary for generating speaker and emotional condition information.

[0075] Here, a single document (410) input to the emotion prediction module (420) may include multiple sentences (411_1, 411_2, ..., 411_N) and speaker information (412_1, 412_2, ..., 412_N) corresponding to each sentence. Here, the document (410) may be a single voice-based content or script, and each sentence within the document (410) may be uttered by the same speaker or different speakers.

[0076] Each of the plurality of sentences (411_1, 411_2, ..., 411_N) is text information, and each of the plurality of speaker information (412_1, 412_2, ..., 412_N) corresponding to each of the plurality of sentences (411_1, 411_2, ..., 411_N) may be information including phonetic characteristics of the speaker who uttered the sentence. For example, the plurality of speaker information (412_1, 412_2, ..., 412_N) may be speaker embedding vectors, but is not limited thereto. Here, the plurality of speaker information (412_1, 412_2, ..., 412_N) may be selectively configured, and in some cases, only the plurality of sentences (411_1, 411_2, ..., 411_N) may be included in one document (410) input to the emotion prediction model (420).

[0077] The emotion prediction module (420) uses a plurality of sentences (411_1, 411_2, ..., 411_N) and a plurality of speaker information (412_1, 412_2, ..., 412_N)) as input data to predict the emotions inherent in each of the plurality of sentences (411_1, 411_2, ..., 411_N), and accordingly derives an emotion prediction result (430).

[0078] The emotion prediction result (430) may be composed of multiple emotion information (431_1, 431_2, ..., 431_N) corresponding to each of multiple sentences (411_1, 411_2, ..., 411_N). Here, the multiple emotion information (431_1, 431_2, ..., 431_N) may be emotion labels or emotion embedding vectors, but are not limited thereto. When one document (410) is a script, the multiple emotion information (431_1, 431_2, ..., 431_N) may include fingerprint text related to emotions included in the script.

[0079] Figure 5 is a flowchart of a method for generating an emotion prediction model according to one embodiment of the present disclosure. Method (500) may be performed by at least one processor of a user terminal or an information processing system. Alternatively, the steps of method (500) may be performed separately by at least one processor of the user terminal and at least one processor of the information processing system.

[0080] In method (500), first, the processor can generate input data using multiple learning sentences written in text-based form and included in one document (S510).

[0081] For example, the processor can extract a sentence embedding vector from each of a plurality of training sentences, extract a speaker embedding vector from information about the speaker who uttered the speech corresponding to each of the plurality of training sentences, and generate a combined embedding vector as input data by concatenating the sentence embedding vector and the speaker embedding vector. This combined embedding vector can be utilized as input data when attempting to generate a transformer-based emotion prediction model.

[0082] As another example, the processor can tokenize multiple training sentences, insert a sentence-end indicator (e.g., a sentence end token) at the end of each tokenized training sentence, and then concatenate them to generate a single-sentence token sequence as input data. This single-sentence token sequence can be used as input data when creating a sentiment prediction model using the GPT2 architecture.

[0083] Thereafter, the processor can generate correct answer data using information about the emotions contained in the pitches corresponding to each of the multiple training sentences (S520). For example, the processor can extract an emotion embedding vector as correct answer data from information about the emotions contained in the pitches corresponding to each of the multiple training sentences.

[0084] Thereafter, the processor can create a hypothesis prediction model by learning using the above input data and the above correct answer data (S530).

[0085] In one embodiment, the processor can create an emotion prediction model by learning to minimize the error between the output data and the correct data derived by inputting input data based on a regression-based loss function.

[0086] For example, the processor can set the emotion embedding vector generated as the correct answer data as the target embedding vector, and train based on the MSE loss function so that the error between the embedding vector derived by inputting input data and the target embedding vector is minimized.

[0087] Here, the processor can be set to not inflict loss on tokens (time-steps) without emotion when the emotion prediction model has the GPT2 structure, thereby focusing on learning only in locations where emotion expression is required.

[0088] FIG. 6 is a diagram illustrating an example of how emotional information is acquired through an emotional search model (620) according to one embodiment of the present disclosure. As illustrated in FIG. 6, the emotional search module (320) can acquire the emotional information necessary for generating speaker and emotional condition information.

[0089] Here, a single document (610) input to the emotion search module (620) may include multiple sentences (611_1, 611_2, ..., 611_N) and speaker information (612_1, 612_2, ..., 612_N) corresponding to each sentence. Here, the document (610) may be a single voice-based content or script, and each sentence in the document (610) may be uttered by the same speaker or different speakers.

[0090] Each of the plurality of sentences (611_1, 611_2, ..., 611_N) is text information, and each of the plurality of speaker information (612_1, 612_2, ..., 612_N) corresponding to each of the plurality of sentences (611_1, 611_2, ..., 611_N) may be information including phonetic characteristics of the speaker who uttered the sentence. For example, the plurality of speaker information (612_1, 612_2, ..., 612_N) may be speaker embedding vectors, but is not limited thereto. Here, the plurality of speaker information (612_1, 612_2, ..., 612_N) may be selectively configured, and in some cases, only the plurality of sentences (611_1, 611_2, ..., 611_N) may be included in one document (610) input to the emotion prediction model (620).

[0091] The emotion search model (620) can input a plurality of sentences (611_1, 611_2, ..., 611_N) and a plurality of speaker information (612_1, 612_2, ..., 612_N) corresponding to each of the plurality of sentences (611_1, 611_2, ..., 611_N), and search for emotion information most suitable for the input among a plurality of emotion expressions stored in an emotion database. At this time, the emotion search model (620) can include a text encoder that extracts a text embedding vector corresponding to the sentence (611), and an audio encoder that extracts an emotion embedding vector from a voice reflecting a specific emotion, and can operate by searching for appropriate emotion information through a similarity comparison between the two embeddings.

[0092] The emotion search result (630) may include multiple emotion information (631_1, ..., 631_N) indicating emotions appropriate for each of multiple sentences (611_1, 611_2, ..., 611_N). The multiple emotion information (631_1, ..., 631_N) may be composed of emotion labels, emotion embedding vectors, or valence-arousal values, and when expressed in the form of speech embedding, may be extracted through pre-learning models such as emotion2vec, ECAPA-TDNN, and Pyannote.

[0093] FIG. 7 is a flowchart of a method for generating an emotion search model according to an embodiment of the present disclosure, and FIG. 8 is a diagram illustrating a structure for performing alignment and contrast learning on an embedding space for a text embedding vector extracted from a text encoder (810) of an emotion search model according to an embodiment of the present disclosure and an emotion embedding vector extracted through an audio encoder (820). The method (700) may be performed by at least one processor of a user terminal or an information processing system. Alternatively, the steps of the method (700) may be performed separately by at least one processor of the user terminal and at least one processor of the information processing system.

[0094] In method (700), first, the processor may generate a text embedding vector by analyzing a sentence (830) using a text encoder (810) (S710). For example, the processor may extract a plurality of text embedding vectors corresponding to each of a plurality of text-based learning sentences using the text encoder (710).

[0095] In one embodiment, the processor may extract a text embedding vector for each of a plurality of training sentences by considering adjacent training sentences.

[0096] For example, the processor may use a text encoder (820) to extract a first text embedding vector from a target learning sentence among a plurality of learning sentences written based on text, extract a second text embedding vector from learning sentences before and after the target learning sentence among the plurality of learning sentences, and generate a target text embedding vector by reflecting the second text embedding vector on the first text embedding vector. For example, the processor may extract a text embedding vector in which information about surrounding sentences is reflected by reflecting the second text embedding vector on the first text embedding vector through cross attention.

[0097] As another example, the processor may use a text encoder (810) to extract a first text embedding vector from a target learning sentence among a plurality of learning sentences written based on text, extract a second text embedding vector from learning sentences before and after the target learning sentence among the plurality of learning sentences, extract a third text embedding vector from the plurality of learning sentences, and generate a target text embedding vector by reflecting the second text embedding vector and the third text embedding vector on the first text embedding vector. For example, the processor may extract a text embedding vector in which information about surrounding sentences and information about the entire document are appropriately reflected by reflecting the second text embedding vector and the third text embedding vector on the first text embedding vector through cross attention.

[0098] Thereafter, the processor may extract an audio embedding vector by analyzing the speech (840) using the audio encoder (820) (S720). For example, the processor may extract a target audio embedding vector from the speech corresponding to the target learning sentence using the audio encoder (820).

[0099] Thereafter, the processor can create an emotion search model by training using the target text embedding vector extracted through the text encoder (810) and the target audio embedding vector extracted through the audio encoder (820) as training data (S730). For example, the processor can create an emotion search model by training so that the similarity between the target text embedding vector and the target audio embedding vector is maximized.

[0100] Figure 9 is a flowchart of a method for generating a condition generation model according to one embodiment of the present disclosure. Method (900) may be performed by at least one processor of a user terminal or an information processing system. Alternatively, the steps of method (900) may be performed separately by at least one processor of the user terminal and at least one processor of the information processing system.

[0101] In the method (900), first, the processor can generate first input data and second input data using information about the speaker and emotion (S910).

[0102] Here, the information about the speaker used to generate the first input data and the second input data may include, but is not limited to, a voice signal corresponding to a voice spoken by the speaker, a voice embedding vector extracted from the voice spoken by the speaker, and a speaker index (or speaker label) indicating the speaker.

[0103] Additionally, the information about emotions used to generate the first input data and the second input data herein may include, but is not limited to, a voice signal corresponding to an emotional voice, a voice embedding vector extracted from the emotional voice, and an emotion index (or emotion label) indicating an emotion.

[0104] That is, the processor can generate first input data and second input data by combining information about the speaker and information about emotions as described above.

[0105] For example, the processor may generate first input data using a voice signal corresponding to a voice spoken by a first speaker with a first emotion, and may generate second input data using a voice signal corresponding to a voice spoken by a second speaker with a second emotion.

[0106] As another example, the processor may generate first input data using a first speech embedding vector extracted from speech uttered by a first speaker with a second emotion, and may generate second input data using a second speech embedding vector generated by passing an emotion index indicating the first emotion through an embedding layer.

[0107] As another example, the processor may generate a first voice uttered by a first speaker with a second emotion by modulating a voice uttered by a first speaker with a first emotion, and generate first input data using the first voice, and may generate a second voice uttered by a second speaker with a first emotion by modulating a voice uttered by the first speaker with the first emotion, and generate second input data using the second voice.

[0108] As another example, the processor may generate first input data by removing information about a first emotion from a first embedding vector extracted from a first voice, and may generate second input data by removing information about the first emotion from a second embedding vector extracted from a second voice.

[0109] Thereafter, the processor can generate correct answer data based on speaker and emotional condition information (S920). For example, the processor can generate an embedding vector extracted from a speech uttered by a specific speaker with a specific emotion as correct answer data.

[0110] Thereafter, the processor can generate a condition generation model using the first input data, the second input data, and the correct answer data (S930). For example, the processor can train the model to minimize the error between the result data and the correct answer data derived by inputting the first and second input data.

[0111] For example, if the condition generation model is a multi-layer perceptron (MLP) model, the processor can be trained to minimize the error between the result data and the correct answer data derived by inputting the first input data and the second input data based on a regression-based loss function (e.g., MSE loss, cosine-distance loss).

[0112] As another example, if the condition generation model is a transformer-based diffusion model, the processor can be trained to minimize the error between the result data and the correct data derived by inputting the first input data and the second input data based on a general loss function (e.g., MSE loss, etc.) used in the diffusion model.

[0113] Figure 10 is a flowchart of a method for generating speaker and emotional condition information for text-to-speech conversion according to one embodiment of the present disclosure. Method (1000) may be performed by at least one processor of a user terminal or an information processing system. Alternatively, the steps of method (1000) may be performed separately by at least one processor of the user terminal and at least one processor of the information processing system.

[0114] In the method (1000), first, the processor can generate a condition generation model based on a machine learning-based learning method (S1010).

[0115] In one embodiment, the processor can generate a condition generation model that receives speaker and emotion information as input data and derives speaker and emotion condition information by learning using learning data that inputs information related to the speaker and information about the emotion and the corresponding speaker and emotion condition information as correct answer data.

[0116] Thereafter, the processor can obtain speaker information (S1020). For example, the processor can obtain at least one of a speaker embedding vector and a speaker reference speech sample from a speech uttered by a specific speaker as speaker information.

[0117] Thereafter, the processor can obtain emotional information corresponding to the emotional speech (S1030). For example, the processor can obtain at least one of an emotional embedding vector and an emotional speech sample (Emotion Reference Speech) from a speech containing a specific emotion as emotional information.

[0118] Thereafter, the processor can generate speaker and emotion condition information (global condition embedding) as result data by inputting speaker information and emotion information as input data into a condition generation model (S1040).

[0119] The above-described method may be provided as a computer program stored on a computer-readable recording medium for execution on a computer. The medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program instructions, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0120] The methods, operations, or techniques of the present disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will appreciate that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software will depend on the particular application and the design requirements imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementations should not be construed as departing from the scope of the present disclosure.

[0121] In a hardware implementation, the processing units used to perform the techniques may be implemented within one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, a computer, or a combination thereof.

[0122] Accordingly, the various exemplary logical blocks, modules, and circuits described in connection with the present disclosure may be implemented or performed by any combination of a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or those designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0123] In a firmware and / or software implementation, the techniques may be implemented as instructions stored on a computer-readable medium, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, a compact disc (CD), a magnetic or optical data storage device, etc. The instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described herein.

[0124] When implemented in software, the techniques may be stored on or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium.

[0125] For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of media. Disk and disc, as used herein, includes compact discs, laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks usually reproduce data magnetically, whereas discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0126] A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. Alternatively, the processor and the storage medium may reside as discrete components in the user terminal.

[0127] While the embodiments described above have been described as utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the present disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or distributed computing environment. Furthermore, aspects of the present disclosure may be implemented in multiple processing chips or devices, and storage may be similarly affected across multiple devices. Such devices may include personal computers, network servers, and portable devices.

[0128] While the present disclosure has been described in connection with certain embodiments herein, various modifications and variations may be made without departing from the scope of the present disclosure, which would be apparent to those skilled in the art. Furthermore, such modifications and variations are intended to fall within the scope of the claims appended to this specification.

Claims

1. A method for generating speaker and emotion condition information for text-to-speech conversion performed by at least one processor, A step of creating a condition generation model based on a machine learning-based learning method; A step of obtaining speaker information corresponding to the speaker; A step of acquiring emotional information corresponding to an emotion; and A step of generating speaker and emotion condition information as result data by inputting the acquired speaker information and the acquired emotion information as input data into the generated condition generation model. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

2. In paragraph 1, The step of acquiring the above speaker information is, As speaker information, a step of obtaining at least one of a speaker embedding vector and a speaker voice sample from a voice spoken by a specific speaker. Includes, The step of acquiring the above-mentioned emotional information is, As emotional information, a step of obtaining at least one of an emotional embedding vector and an emotional voice sample from a voice containing a specific emotion. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

3. In paragraph 1, The step of acquiring the above speaker information is, A step of obtaining information about the speaker who uttered the voice corresponding to each of multiple sentences included in a single document. Includes, The step of acquiring the above-mentioned emotional information is, Steps for generating an emotion prediction model; and A step of acquiring emotional information by inputting the plurality of sentences into the emotion prediction model or by inputting the plurality of sentences and the acquired information regarding the speaker into the emotion prediction model. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

4. In paragraph 3, The steps for generating the above emotion prediction model are: A step of generating input data using multiple training sentences written in text-based format; A step of generating correct answer data using information about emotions contained in the voice corresponding to each of the plurality of learning sentences; and A step of generating an emotion prediction model by training it based on a regression-based loss function so that the error between the result data derived by inputting the generated input data and the generated correct answer data is minimized. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

5. In paragraph 4, The step of generating the above input data is, A step of extracting sentence embedding vectors from each of the plurality of training sentences above; Speaker embedding vectors derived from information regarding the speaker who uttered the speech corresponding to each of the plurality of learning sentences above; and A step of generating a combined embedding vector as input data by concatenating the extracted sentence embedding vector and the extracted speaker embedding vector. Includes, The step of generating the above correct answer data is, A step of extracting an emotion embedding vector as correct answer data from information regarding the emotion contained in the pitch corresponding to each of the plurality of learning sentences above. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

6. In Paragraph 4, The step of generating the above input data is, The step of tokenizing the above plurality of learning sentences; and A step of generating a single sentence token sequence as input data by inserting an indicator pointing to the end of the sentence at the end of each of the aforementioned multiple tokenized training sentences and then interconnecting them. Includes, The step of generating the above correct answer data is, A step of extracting an emotion embedding vector as correct answer data from information regarding the emotion contained in the voice corresponding to each of the plurality of training sentences above. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

7. In paragraph 1, The step of acquiring the above speaker information is, A step of obtaining information about the speaker who uttered the voice corresponding to each of multiple sentences included in a single document. Includes, The step of acquiring the above-mentioned emotional information is, Steps for generating an emotion search model; and A step of obtaining one of the plurality of emotional information items stored regularly through the generated emotional search model by inputting the plurality of sentences and the information about the acquired speaker into the generated emotional search model. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

8. In paragraph 7, The above generated emotion retrieval model is, A text encoder that receives a specific sentence as input and extracts a text embedding vector corresponding to the specific sentence; and An audio encoder that receives a voice containing a specific emotion as input and extracts an emotion embedding vector corresponding to the voice containing the specific emotion. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

9. In paragraph 8, The step of generating the above emotion search model is, A step of extracting a plurality of text embedding vectors corresponding to each of a plurality of training sentences written based on text using the above text encoder; and A step of extracting a plurality of audio embedding vectors from speech corresponding to each of the plurality of training sentences using the above audio encoder; and A step of generating an sentiment search model by training it based on a contrast loss function such that the similarity between text embedding vectors and audio embedding vectors extracted corresponding to the same training sentence is maximized, and the similarity between text embedding vectors and audio embedding vectors extracted corresponding to different training sentences is minimized. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

10. In paragraph 8, The step of generating the above emotion search model is, A step of extracting a first text embedding vector from a target learning sentence among a plurality of learning sentences written based on text using the above text encoder; A step of extracting a second text embedding vector from the training sentences before and after the target training sentence among the plurality of training sentences using the text encoder; A step of generating a target text embedding vector by reflecting the extracted second text embedding vector into the extracted first text embedding vector; A step of extracting a target audio embedding vector from a voice corresponding to the target learning sentence using the audio encoder; and A step of creating an emotion search model by learning to maximize the similarity between the generated target text embedding vector and the extracted target audio embedding vector. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

11. In Paragraph 8, The step of generating the above emotion search model is, A step of extracting a first text embedding vector from a target learning sentence among a plurality of learning sentences written based on text using the above text encoder; A step of extracting a second text embedding vector from the training sentences before and after the target training sentence among the plurality of training sentences using the text encoder; A step of extracting a third text embedding vector from the plurality of training sentences using the above text encoder; A step of generating a target text embedding vector by reflecting the extracted second text embedding vector and the extracted third text embedding vector on the extracted first text embedding vector; A step of extracting a target audio embedding vector from a voice corresponding to the target learning sentence using the audio encoder; and A step of creating an emotion search model by learning to maximize the similarity between the generated target text embedding vector and the extracted target audio embedding vector. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

12. In paragraph 1, The steps for creating the above condition generation model are: A step of generating first input data using a voice signal corresponding to a voice uttered by a first speaker with a first emotion; A step of generating second input data using a voice signal corresponding to a voice uttered by a second speaker with a second emotion; A step of generating correct answer data using an embedding vector extracted from the voice uttered by the first speaker with the second emotion; and A step of creating a condition generation model by learning to minimize the error between the result data derived by inputting the first input data and the second input data and the generated correct answer data. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

13. In paragraph 1, The steps for creating the above condition generation model are: A step of generating first input data using a first voice embedding vector extracted from a voice uttered by a first speaker with a second emotion; A step of generating second input data using a second speech embedding vector generated by passing an emotion index indicating a first emotion through an embedding layer; A step of generating correct answer data using a correct answer embedding vector extracted from the voice uttered by the first speaker with the first emotion; and A step of creating a condition generation model by learning to minimize the error between the result data derived by inputting the first input data and the second input data and the generated correct answer data. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

14. In paragraph 1, The steps for creating the above condition generation model are: A step of generating a first voice uttered by a first speaker with a second emotion by modulating a voice uttered by a first speaker with a first emotion, and generating first input data using the generated first voice; A step of generating a second voice uttered by a second speaker with the first emotion by modulating the voice uttered by the first speaker with the first emotion, and generating second input data using the generated second voice; A step of generating correct answer data using an embedding vector extracted from the voice uttered by the first speaker with the first emotion; and A step of creating a condition generation model by learning to minimize the error between the result data derived by inputting the first input data and the second input data and the generated correct answer data. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

15. In paragraph 1, The steps for creating the above condition generation model are: A step of generating a first voice uttered by a first speaker with a second emotion by modulating a voice uttered by a first speaker with a first emotion, extracting a first embedding vector from the generated first voice, and generating first input data by removing information about the first emotion from the extracted first embedding vector; A step of generating a second voice uttered by a second speaker with the first emotion by modulating the voice uttered by the first speaker with the first emotion, extracting a second embedding vector from the generated second voice, and generating second input data by removing information regarding the first emotion from the extracted second embedding vector; A step of generating correct answer data using an embedding vector extracted from the voice uttered by the first speaker with the first emotion; and A step of creating a condition generation model by learning to minimize the error between the result data derived by inputting the first input data and the second input data and the generated correct answer data. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

16. In paragraph 1, The above-mentioned condition generation model is, At least one of a multi-layer perceptron (MLP) model and a transformer-based diffusion model. A method for generating speaker and emotion condition information for text-to-speech conversion, including:

17. A method for generating emotional information using an emotional prediction model, performed by at least one processor, A step of obtaining one or more target sentences and speaker information corresponding to each of the one or more target sentences; and A step of outputting an emotion prediction result for one or more target sentences obtained from one or more target sentences obtained and the speaker information obtained using the emotion prediction model, The above emotion prediction model generates correct answer data using information regarding emotions contained in speech corresponding to each of the plurality of text-based training sentences and, based on a regression-based loss function, is generated or updated by training such that the error between the result data derived by inputting the generated plurality of training sentences and the generated correct answer data is minimized. Method for generating emotion information using an emotion prediction model.

18. A computer program stored on a computer-readable recording medium for executing a method according to any one of paragraphs 1 through 17 on a computer.

19. As an information processing system, memory; and At least one processor connected to the memory and configured to execute at least one computer-readable program contained in the memory. Includes, At least one program above, Create a condition generation model based on a machine learning-based learning method, and Acquire speaker information corresponding to the speaker, and Acquire emotional information corresponding to the emotion, and An information processing system including commands for generating speaker and emotion condition information as result data by inputting the acquired speaker information and the acquired emotion information into the generated condition generation model as input data.

Images