Method for providing character video call by using natural language processing model, computing system for performing same, and computer-readable recording medium for implementing same

Abstract

The present invention relates to a method for providing a character video call by using a natural language processing model, a computing system for performing same, and a computer-readable recording medium for implementing same, and, more specifically, to a method for providing a character video call by using a natural language processing model, a computing system for performing same, and a computer-readable recording medium for implementing same, the method comprising inputting, into a deep learning-type natural language processing model, a prompt that includes information related to the video, voice, way of speaking, personality and background of a character who appears in a work such as an actual animation, thereby enabling a user to have an experience as if making a video call with a corresponding character in real time.

Description

A method for providing character video calls using a natural language processing model, a computing system for performing the same, and a computer-readable recording medium for implementing the same

[0001] The present invention relates to a method for providing a character video call using a natural language processing model, a computing system for performing the same, and a computer-readable recording medium for implementing the same, and more particularly, to a method for providing a character video call using a natural language processing model, a computing system for performing the same, and a computer-readable recording medium for implementing the same, which can provide a user with an experience as if making a real-time video call with the character by inputting a prompt including information related to the image, voice, speech, personality, background, etc. of a character appearing in an actual animation-like work into a deep learning-type natural language processing model, and to a computing system for performing the same, and to a computer-readable recording medium for implementing the same.

[0002] AI language models, which learn language comprehension and knowledge from large amounts of text, have garnered explosive interest since the announcement of GPT-3 (Generative Pre-trained Transformers) by OpenAI in July 2020. Leveraging the massive computing and data power of the GPT-3 learning model, it has become possible to apply it to applied tasks with small amounts of training data. Its potential applications in various fields has led to active research in this area.

[0003] The Generative AI Language Model, one of the many types of artificial intelligence language models, is a model that learns a large amount of data in advance and predicts the next word that is most suitable for a given string of words. It learns with a neural network to predict which word is the best by using the previous words that are already known, and it enables the appropriate selection of words or sentences or the selection of better candidates among several candidates that must be generated.

[0004] Generative AI language models perform well in language generation tasks such as automatic translation, sentence summarization, dialogue or chatbot, and story generation, which require understanding an input language and then generating a new language. Examples of such models include OpenAI's GPT series of models, XLNet jointly developed by Carnegie Mellon University (CMU) and Google Brain, and Facebook's BART (Bidirectional and Auto-Regressive Transformer).

[0005] In particular, among artificial intelligence language models, a language model trained with a massive amount of text data is called a Large Language Model (LLM). While conventional natural language processing technology analyzes and processes human natural language by considering some words and sentences, LLM processes by considering the entire context, so it can output more natural and context-appropriate answers, so it can be said to be an advanced model. Recently, as the performance of LLM has significantly improved, it is attracting attention because it can be applied in various fields.

[0006] Meanwhile, although many existing technologies, such as chatbot services using artificial intelligence language models, have been invented, there is no existing technology that provides users with an experience similar to video calling with a familiar person or directly video calling with a character of their choice.

[0007] The present invention provides a method for providing a character video call using a natural language processing model, a computing system for performing the same, and a computer-readable recording medium for implementing the same. More specifically, the present invention provides a method for providing a character video call using a natural language processing model, a computing system for performing the same, and a computer-readable recording medium for implementing the same, which can provide a user with an experience as if making a real-time video call with the character by inputting a prompt including information related to the image, voice, speech, personality, background, etc. of a character appearing in an actual animation-like work into a deep learning-type natural language processing model, and a computing system for performing the same, and a computer-readable recording medium for implementing the same.

[0008] In order to solve the above problem, in one embodiment of the present invention, a method for providing a character video call using a deep learning-type natural language processing model executed in a computing system including one or more memories and one or more processes, the method comprising: a dialogue input step of receiving text input information converted into a text format by an STT model from audio input information inputted through a user terminal; a first prompt generation step of generating a first prompt to be inputted into the deep learning-type natural language processing model, wherein the first prompt includes text input information, a dialogue history, and preprocessing information; and a text output information reception step of receiving text output information outputted by the first prompt being inputted into the natural language processing model; wherein the text output information is converted into audio output information in an audio format by a TTS model and played back on the user terminal, thereby delivering the voice of the corresponding character to the user, and the first prompt generation step includes: a feature information extraction step of inputting information including the text input information into an embedding model to extract feature information in a vector format; And, for a database including work information related to a work in which the character appears, a similarity information derivation step for deriving similar information similar to the feature information using a similarity-based search algorithm is provided; and the preprocessing information provides a method for providing a character video call, wherein the preprocessing information includes the similar information.

[0009] In one embodiment of the present invention, the work information may include: episode-by-episode information of the work; world view information of the work; character information in the work; user memory information derived from past conversations between the character and the user; and sub-character information about characters surrounding the character in the work.

[0010] In one embodiment of the present invention, the similar information derivation step derives a matching rate for each item of the work information, and the first prompt generation step can determine the preprocessing information based on the derived matching rate for each item and a preset rule.

[0011] The above first prompt generation step further includes a user information input step for receiving pre-stored user information; and a lesson information input step for receiving pre-stored lesson information; and the first prompt may further include the user information and the lesson information.

[0012] In one embodiment of the present invention, the character video call providing method further includes a second prompt generation step of converting the text output information into text emotion information in a text form that can be derived from the text output information, and inputting a second prompt including the text emotion information into the natural language processing model, wherein the natural language processing model receives the second prompt and outputs emotion output information, the TTS model receives the emotion output information and outputs character emotion information, and the character emotion information can be combined with the audio output information and transmitted to the user terminal.

[0013] In one embodiment of the present invention, the audio output information is transmitted to the user terminal together with the video output information, and the video output information includes emotional expression information determined based on the emotional output information among a plurality of pre-stored emotional expressions, and the emotional expression information is information about an emotional expression visually displayed through the corresponding character, and the corresponding character can display the emotional expression before the audio output information is output from the user terminal.

[0014] In one embodiment of the present invention, the character video call providing method further includes a third prompt generation step of generating a third prompt including M (M is a natural number greater than or equal to 2) conversation histories included in the conversation history; and a user memory generation step of generating user memory information based on N (N is a natural number less than M) pieces of summary conversation information obtained by inputting the third prompt into the natural language processing model; wherein the user memory information may include the summary conversation information and summary conversation feature information in a vector form by inputting the summary conversation information into an embedding model.

[0015] In one embodiment of the present invention, the character video call providing method may further include an instant response generation step of generating text output information based on the conversation content by applying a similarity-based search algorithm to the vector-type feature information derived in the feature information extraction step; and a preset DB set storing conversation content exchanged between the character and the user in the past; and deriving conversation content satisfying preset criteria.

[0016] In order to solve the above problem, in one embodiment of the present invention, a computing system including one or more memories and one or more processes and performing a method for providing a character video call using a deep learning-type natural language processing model, the computing system includes: a dialogue input unit that receives text input information converted into a text format by an STT model from audio input information inputted through a user terminal; a first prompt generation unit that generates a first prompt to be inputted into the deep learning-type natural language processing model, wherein the first prompt includes text input information, a dialogue history, and preprocessing information; and a text output information reception unit that receives text output information outputted by the first prompt being inputted into the natural language processing model, wherein the text output information is converted into audio output information in an audio format by a TTS model and played back on the user terminal, thereby delivering the voice of the corresponding character to the user, and the first prompt generation unit includes: a feature information extraction unit that inputs information including the text input information into an embedding model to extract feature information in a vector format; And, a computing system is provided, including a similarity information extraction unit that extracts similar information similar to the feature information using a similarity-based search algorithm for a database including work information related to a work in which the character appears, wherein the preprocessing information includes the similar information.

[0017] In order to solve the above problem, in one embodiment of the present invention, a computer-readable recording medium for implementing a method for providing a character video call using a deep learning type natural language processing model, which is performed in a computing system including one or more memories and one or more processors, wherein the computer-readable recording medium stores commands that cause the computing system to perform the following steps, wherein the following steps include: a dialogue input step of receiving text input information converted into a text type by an STT model from audio input information in the form of audio input input through a user terminal; a first prompt generation step of generating a first prompt to be input to a deep learning type natural language processing model, wherein the first prompt includes text input information, a dialogue history, and preprocessing information; And a text output information receiving step for receiving text output information that is input to the natural language processing model and output by the first prompt; wherein the text output information is converted into audio output information in an audio format by the TTS model and played back on a user terminal to deliver the voice of the corresponding character to the user; and the first prompt generation step includes a feature information extraction step for inputting information including the text input information into an embedding model to extract feature information in a vector format; and a similarity information derivation step for deriving similar information similar to the feature information using a similarity-based search algorithm for a database including work information related to works in which the corresponding character appears; wherein the preprocessing information includes the similar information.

[0018] According to one embodiment of the present invention, by providing a UI similar to an actual video call, it is possible to provide a user with an experience of actually making a video call with their favorite character or character.

[0019] According to one embodiment of the present invention, by including information about the work in which the character appears, as well as information about the relationship between the character and surrounding characters, in the prompt, an experience similar to talking to a character in an actual work can be provided.

[0020] According to one embodiment of the present invention, by inputting separate lesson information, the effect of the user naturally accepting the content included in the lesson information can be achieved.

[0021] According to one embodiment of the present invention, the user's guardian can view the conversation between the user and the character, thereby having the effect of understanding the user's true feelings.

[0022] According to one embodiment of the present invention, by using a generative AI language model, it is possible to provide a natural answer to a user's question or utterance.

[0023] FIG. 1 schematically illustrates the configuration of a computing system that performs a method for providing a character video call using a natural language processing model according to one embodiment of the present invention.

[0024] Figure 2 schematically illustrates the execution process of a dialogue input step according to one embodiment of the present invention.

[0025] FIG. 3 schematically illustrates information included in a first prompt according to one embodiment of the present invention.

[0026] Figure 4 schematically illustrates an interface for inputting user information and lesson information according to one embodiment of the present invention.

[0027] Figure 5 schematically illustrates the execution process of a text output information reception step according to one embodiment of the present invention.

[0028] Figure 6 schematically illustrates the execution process of the first prompt generation step according to one embodiment of the present invention.

[0029] Figure 7 schematically illustrates information for each item included in work information according to one embodiment of the present invention.

[0030] Figure 8 schematically illustrates a process for determining similar information according to one embodiment of the present invention.

[0031] Figure 9 schematically illustrates the execution process of a similar information derivation step according to one embodiment of the present invention.

[0032] Figure 10 schematically illustrates a process in which a second prompt is generated and emotional output information is derived according to one embodiment of the present invention.

[0033] FIG. 11 schematically illustrates a process in which a user terminal receives audio output information and video output information according to one embodiment of the present invention.

[0034] Figure 12 schematically illustrates the execution process of the third prompt generation step and the user memory generation step according to one embodiment of the present invention.

[0035] Figure 13 schematically illustrates the execution process of an instant answer generation step according to one embodiment of the present invention.

[0036] Figure 14 schematically illustrates the internal configuration of a computing device according to one embodiment of the present invention.

[0037] Hereinafter, various embodiments and / or aspects are now disclosed with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of one or more aspects. However, it will be apparent to one skilled in the art that such aspects may be practiced without these specific details. The following description and the attached drawings detail specific exemplary aspects of one or more aspects. However, these aspects are exemplary, and it is to be understood that any of the various methods within the principles of the various aspects may be utilized, and the description is intended to encompass all such aspects and their equivalents.

[0038]

[0039] Additionally, various aspects and features will be presented by systems that may include a number of devices, components, and / or modules. It is also to be understood and appreciated that various systems may include additional devices, components, and / or modules, and / or may not include all of the devices, components, and modules discussed in connection with the drawings.

[0040] The terms "embodiment," "example," "aspect," and "example" used herein may not be construed as implying that any aspect or design described is better or advantageous than other aspects or designs. The terms "part," "component," "module," "system," and "interface" used below generally refer to computer-related entities, and may refer to, for example, hardware, a combination of hardware and software, or software.

[0041] Additionally, it should be understood that the terms "comprises" and / or "comprising" imply the presence of the features and / or components, but do not preclude the presence or addition of one or more other features, components and / or groups thereof.

[0042] Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. The term and / or includes a combination of a plurality of related described items or any of a plurality of related described items.

[0043] Additionally, in the embodiments of the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in the embodiments of the present invention.

[0044] Additionally, unless otherwise stated, the “user terminal” referred to below may be implemented as a computer, portable terminal, or tablet PC that can connect to a server or other terminal via a network. Here, the computer may include, for example, a notebook, desktop, laptop, etc. equipped with a web browser, and the portable terminal may include, for example, a wireless communication device that ensures portability and mobility, such as a smart phone, a PCS (Personal Communication System), a GSM (Global System for Mobile communications), a PDC (Personal Digital Cellular), a PHS (Personal Handyphone System), a PDA (Personal Digital Assistant), an IMT (International Mobile Telecommunication)-2000, a CDMA (Code Division Multiple Access)-2000, a W-CDMA (W-Code Division Multiple Access), a Wibro (Wireless Broadband Internet) terminal, and all kinds of handheld-based wireless communication devices capable of collecting biometric information of a client, etc.

[0045]

[0046] FIG. 1 schematically illustrates the configuration of a computing system (2000) that performs a method for providing a character video call using a natural language processing model according to one embodiment of the present invention.

[0047]

[0048] As illustrated in FIG. 1, a computing system (2000) comprising one or more memories and one or more processes and performing a method for providing a character video call using a deep learning-type natural language processing model, the computing system comprises: a dialogue input unit (2100) for receiving text input information converted into text by an STT model from audio input information in the form of audio input input through a user terminal (1000); a first prompt generation unit (2200) for generating a first prompt to be input to the deep learning-type natural language processing model, wherein the first prompt includes text input information, a dialogue history, and preprocessing information; And the first prompt includes a text output information receiving unit (2300) that receives text output information inputted and outputted by the natural language processing model, and the text output information is converted into audio output information in audio form by the TTS model and played back on the user terminal (1000) to deliver the voice of the corresponding character to the user, and the first prompt generating unit (2200) includes a feature information extracting unit (2210) that inputs information including the text input information into an embedding model to extract feature information in vector form; and a similarity information extracting unit (2220) that uses a similarity-based search algorithm to derive similar information similar to the feature information from a database (2400) that includes work information related to works in which the corresponding character appears, and the preprocessing information includes the similar information.

[0049]

[0050] Specifically, the computing system (2000) of the present invention, as illustrated in FIG. 1, communicates with a user terminal (1000) to perform a character video call provision method, and the user can be provided with an experience similar to making a real-time video call with a character appearing in a work such as an actual animation through a UI, animation, or video displayed on the user terminal (1000). As an embodiment of the present invention, the following description assumes that the work is an animation, but is not limited thereto, and the present invention can also be applied to works of various genres such as movies, dramas, comics, games, and novels. In addition, the character described below corresponds to a concept including a character appearing in the work, or a character appearing.

[0051] More specifically, when a user executes an application stored in a user terminal (1000), a list of multiple friends with whom the user can make a video call, i.e., a list of multiple characters, is displayed on the user terminal (1000), and the user can select any character from the list and make a video call with the character. In other words, when an icon for making a video call with a specific character is selected, an interface that appears to make a video call with the character is displayed on the user terminal (1000). The interface includes a layer that displays an image captured by a camera provided in the user terminal (1000) and an image of the character, and the user's speech can be input by manipulating the interface. At this time, the user's speech is input through a microphone provided in the user terminal (1000).

[0052]

[0053] Audio input information in the form of audio input through the user terminal (1000) is converted into text input information in the form of text by the STT (Speech To Text) model, and a dialogue input step of receiving the text input information is performed by the dialogue input unit (2100). The STT model may be included in the user terminal (1000) as one embodiment of the present invention, may be included in the computing system (2000) of the present invention as another embodiment of the present invention, and may be included in a server provided externally other than the user terminal (1000) or the computing system (2000) as yet another embodiment of the present invention.

[0054]

[0055] When the text input information is received by the dialogue input unit (2100), a first prompt generation step of generating a first prompt is performed by the first prompt generation unit (2200). The first prompt is a prompt input to the deep learning-type natural language processing model used in the present invention, and includes the text input information. In a preferred embodiment, the natural language processing model may correspond to a large language model (LLM), and the following description will assume that the natural language processing model of the present invention is a large language model. However, this is only one example, and various types of natural language processing models may be used.

[0056] More specifically, the first prompt includes the text input information, conversation history, and preprocessing information. The conversation history includes a conversation history between the character with whom the user is making a video call and the user during the video call. That is, when the user and the character start a video call through the application, there is no conversation history, but as the video call progresses, the conversation contents exchanged between the user and the character are stored as the conversation history. The preprocessing information includes information related to the character and information related to the work in which the character appears, and a specific description of the preprocessing information will be described later in the description of FIG. 3.

[0057] The above first prompt generation step includes a feature information extraction step for extracting feature information of text input information; and a similar information extraction step for deriving similar information similar to the feature information from a database (2400). In one embodiment of the present invention, the database (2400) may be provided within the computing system (2000), as illustrated in FIG. 1, but in another embodiment of the present invention, the database (2400) may be provided outside the computing system (2000) and may communicate directly or indirectly with the computing system (2000). A detailed description of the feature information extraction step and the similar information extraction step will be provided below.

[0058]

[0059] When the first prompt is generated by the first prompt generation unit (2200), the computing system (2000) inputs the first prompt into the macro language model, and the text output information receiving unit (2300) performs a text output information receiving step of receiving text output information output from the macro language model. At this time, the text output information is data in the form of text. The text output information is converted into audio output information in the form of audio by the TTS (Text To Speech) model, and the audio output information is output through a speaker or the like provided in the user terminal (1000). The audio output information is output by reflecting the voice, tone, intonation, etc. of the corresponding character, and thereby can provide the user with the experience of conversing with an actual character.

[0060] The above TTS model may be included in the user terminal (1000) as one embodiment of the present invention, may be included in the computing system (2000) as another embodiment of the present invention, and may be included in a server provided externally other than the user terminal (1000) or the computing system (2000) as yet another embodiment of the present invention. That is, when the text output information is input to the TTS model, the TTS model outputs audio output information, and the output audio output information is transmitted to the user terminal (1000) to transmit the character's voice to the user.

[0061] In addition, the above-mentioned large language model may be included in the computing system (2000) of the present invention as one embodiment of the present invention, and may be included in a server or the like provided externally other than the computing system (2000) as another embodiment of the present invention.

[0062]

[0063] Figure 2 schematically illustrates the execution process of a dialogue input step according to one embodiment of the present invention.

[0064]

[0065] Schematically, Fig. 2 illustrates an interface displayed on a user terminal (1000) and a process performed in an STT model when a dialogue input step is performed.

[0066]

[0067] Specifically, as illustrated in FIG. 2, the interface for a video call between a user and a character includes a layer for displaying a character's image (displayed in the center of the user terminal (1000)) and a layer for displaying an image captured through the user terminal (1000) (displayed in the lower left of the user terminal (1000). When the user selects and inputs a microphone-shaped icon and speaks, audio input information corresponding to the spoken content is input through the microphone of the user terminal (1000), and the input audio input information is transmitted to the STT model.

[0068] According to one embodiment of the present invention, when the STT model converts the audio input information into text input information and transmits it to the user terminal (1000), the text input information can be displayed on the interface, and preferably, when the user speaks, an animation effect indicating that recording is being performed can be displayed on the interface. When the user's speech is finished, the user can select and input an icon capable of performing a function of sending a message, and when the user terminal (1000) receives a selection input for the icon, the text input information is transmitted to the computing system (2000) through the dialogue input unit (2100). At this time, it is preferable that the user terminal (1000) display a graphic that can inform the user that a message is being transmitted.

[0069]

[0070] FIG. 3 schematically illustrates information included in a first prompt according to one embodiment of the present invention, and FIG. 4 schematically illustrates an interface into which user information and lesson information are input according to one embodiment of the present invention.

[0071]

[0072] As illustrated in Figure 3, the preprocessing information includes work information, and the work information includes: episode information of the work; world view information of the work; character information in the work; user memory information derived from past conversations between the character and the user; and sub-character information about characters surrounding the character in the work.

[0073] In addition, as illustrated in FIG. 4, the first prompt generation step further includes a pre-stored user information input step; and a lesson information input step for receiving pre-stored lesson information; and the first prompt further includes the user information and the lesson information.

[0074]

[0075] Specifically, the first prompt as described above includes text output information, conversation history, and preprocessing information, and may further include user information and lesson information received through the user information input step and the lesson information input step.

[0076] More specifically, as illustrated in FIG. 4, a user can input user information and lesson information through a user terminal (1000). FIG. 4 schematically illustrates an interface for inputting the user information and the lesson information. In the interface for inputting user information (left side of FIG. 4), as an embodiment of the present invention, the user's basic information and the user's detailed information can be input. The basic information includes information such as the user's name and date of birth, and the detailed information includes information about the user's characteristics, for example, what the user likes or dislikes. The detailed information can be directly entered by the user in a descriptive form, as illustrated in FIG. 4, but in another embodiment of the present invention, it can be set in a manner in which the user selects from among a plurality of pre-classified options.

[0077] In the interface for inputting lesson information (right side of Fig. 4), as an embodiment of the present invention, lesson information including what the character would like the child to do can be input. For example, if the lesson information is set to "Brush your teeth well," the character can naturally teach the user a lesson or life guideline by saying something like, "I didn't brush my teeth properly, so my teeth rotted, and I went to the dentist, but the treatment was so painful and scary. Gil-dong should brush his teeth well so he doesn't have to go to the dentist." In particular, the character can combine the user information and lesson information to say something like, "I went to the dentist and saw many 5-year-old friends like Gil-dong. I think Gil-dong would go to the dentist if he didn't brush his teeth properly." In addition, the large language model of the present invention can provide the user with a realistic answer by outputting text output information that takes into account the preprocessing information described below, the user information, and the lesson information.

[0078]

[0079] As described above, the first prompt includes information that can serve as the basis for speech output by the character. The preprocessing information included in the first prompt includes: episode-specific information of the work; worldview information of the work; character information in the work; user memory information derived from past conversations between the character and the user; and sub-character information regarding the character's surrounding characters in the work.

[0080] More specifically, the preprocessing information derived after the preprocessing process is performed on the work information previously stored in the database (2400) is included in the first prompt. That is, some of the information from among the episode information, world view information, character information, sub-character information, user information, and user memory information stored in the database (2400) is extracted and included in the first prompt as the preprocessing information.

[0081] The above episode-specific information is episode-specific information of the work, and as shown in Figure 7A, it includes information on the content of each episode, such as Episode 1, Episode 2, Episode 3, etc. of the work. For example, if Episode 2 contains content related to a sports day, and the first prompt including the work information is input into the macro language model, the macro language model can output text information such as, "At the last sports day, Krong got hurt..."

[0082] The above world view information includes information about the world view of the work, or in other words, background information about the work. For example, information such as "Pororo's village is covered in snow" may be included. The above character information includes information about the character. For example, information about Pororo's gender, birthday, personality, experience, etc. may be included as the character information. The above sub-character information includes information about the gender, birthday, personality, experience, etc. of characters other than the character selected by the user, i.e., surrounding characters, that appear in the work. The above user information corresponds to information entered through the user information input step described above.

[0083] The above user memory information includes information about the conversation history between the user and the character. More specifically, the aforementioned conversation history is information corresponding to all conversations exchanged between the user and the character from the start of the current video call to the end of the call, and the user memory information includes summarized information about the conversation history exchanged between the user and the character in past video calls.

[0084] In one embodiment of the present invention, the user's guardian can access the conversation history or user memory information, thereby gaining insight into the user's inner thoughts. For example, if the user discusses with a character the fact that they were scolded by their mother and their feelings at that time, the user's guardian can access the conversation history and gain insight into the user's inner thoughts.

[0085]

[0086] In summary, all information related to the character making a video call with the user is stored in a database (2400), a preprocessing step is performed on the database (2400) to derive preprocessing information, and the first prompt generation unit (2200) includes the derived preprocessing information in the first prompt to generate the first prompt. The preprocessing step includes a process of extracting information similar to text input information entered by the user from among a plurality of pieces of information included in the database (2400).

[0087]

[0088] Figure 5 schematically illustrates the execution process of a text output information reception step according to one embodiment of the present invention.

[0089]

[0090] As illustrated in Fig. 5, when the first prompt is input into the macro language model, the macro language model outputs text output information. The text output information receiving unit (2300) performs a text output information receiving step of receiving the output text output information, and then transmits the output text output information to a TTS model, and the TTS model converts the text output information into audio output information in audio format. As described above, the macro language model corresponds to a preferred embodiment of a deep learning-type natural language processing model.

[0091]

[0092] Figure 6 schematically illustrates the execution process of the first prompt generation step according to one embodiment of the present invention.

[0093]

[0094] As illustrated in FIG. 6, the first prompt generation step includes a feature information extraction step of inputting information including the text input information into an embedding model to extract feature information in a vector form; and a similar information extraction step of using a similarity-based search algorithm to derive similar information similar to the feature information from a database (2400) containing work information related to a work in which the character appears.

[0095]

[0096] In outline, the above feature information extraction step and the above similar information extraction step are included in the above preprocessing step.

[0097]

[0098] Specifically, when text input information is received through the dialogue input step, a feature information extraction step is performed by the feature information extraction unit (2210). More specifically, the feature information extraction step is a step of extracting feature information from information including the text input information, wherein the feature information is in vector form and is performed by inputting information including the text input information into an embedding model. The embedding model is preferably included in the computing system (2000), and can be designed and implemented using known techniques.

[0099] Based on the output feature information, the similar information extraction unit (2220) extracts similar information from among the plurality of pieces of information contained in the database (2400). At this time, the similar information can be extracted using a similarity-based search algorithm (Similar Search Algorithm, SAA). The similar information includes, with reference to the description of FIG. 3, episode-specific information, world view information, character information, sub-character information, and user memory information.

[0100] For example, when the text input information 'What do you like?' is entered, the search algorithm (SAA) based on the similarity is used to search the database (2400) for information related to 'What do you like?' and derive similar information related thereto. Examples of the similar information include 'Episode 6: The best gift Pororo received' as episode information, 'Pororo likes strawberry ice cream' as character information, and 'Pororo used to say he likes baseball' as user memory information.

[0101] That is, through the aforementioned process, similar information similar to the text input information can be derived in the first prompt generation step, and the preprocessing information including the similar information can be included in the first prompt to generate the first prompt. Additional descriptions of the aforementioned feature information extraction step and similar information extraction step will be provided later in the description of FIGS. 7 to 9.

[0102]

[0103] FIG. 7 schematically illustrates information for each item included in work information according to one embodiment of the present invention, FIG. 8 schematically illustrates a process for determining similar information according to one embodiment of the present invention, and FIG. 9 schematically illustrates a process for performing a similar information derivation step according to one embodiment of the present invention.

[0104]

[0105] Specifically, FIGS. 7A and 7B schematically illustrate one embodiment of work information stored in a database (2400). As illustrated in FIGS. 7A and 7B, each piece of work information is stored in a vector format, and according to one embodiment of the present invention, it is preferable to store it by class. Here, the class refers to an item that distinguishes what type of work information it is, such as whether the work information is user memory information or episode-specific information. It is preferable that such work information be pre-stored in the database (2400) before the user starts a video call. Meanwhile, FIGS. 7A and 7B schematically illustrate the contents of the work information for convenience of explanation, and in the actual invention, the work information is not limited to the configuration illustrated in FIGS. 7A and 7B.

[0106] Thereafter, as illustrated in Figure 8, when the text input information "How is baseball?" is received, feature information of the text input information is derived through an embedding model. Thereafter, referring to the description of Figure 6, the similarity-based search algorithm (SAA) is used to derive work information similar to the derived feature information, i.e., similar information. Although the similar information illustrated in Figure 8 is depicted without class distinction, it is preferable to derive similar information by class.

[0107] In other words, it is desirable to derive certain similar information from all classes, such as deriving certain similar information from the similar memory information class and deriving certain similar information from the episode information. For example, in Fig. 8, "Pororo likes baseball and basketball" and "Pororo has participated in a drawing contest" may be similar information included in the user memory information class, and "Luffy does not like baseball" may be similar information included in the sub-character information class.

[0108] Additionally, as an embodiment of the present invention, if there is no information satisfying the criteria for deriving similar information from a certain class, similar information may not be derived from that class. The description of Figure 9 will describe the similar information derivation step in more detail.

[0109]

[0110] As illustrated in Figure 9, the similar information derivation step derives a matching rate for each item of the work information, and the first prompt generation step determines preprocessing information based on the derived matching rate and preset rules.

[0111]

[0112] Specifically, as described above, it is preferable that the similarity information derivation step derives a predetermined amount of similar information for each class. In one embodiment of the present invention, the matching rate can be derived by comparing the vector of information stored in the database (2400) with the feature vector of the text input information, and can be derived using a known similarity derivation technique, such as a method of deriving cosine similarity. That is, the matching rate between each of a plurality of pieces of information stored in the user memory information class of the database (2400) and the feature information can be derived, and among them, information satisfying a preset rule can be determined as similar information. An example of the preset rule may include the top three pieces of information from which a matching rate of 65% or more is derived, and this corresponds to a configuration that can be changed at any time according to the inventor's intention.

[0113] Additionally, if there is no information satisfying the above-described rules in a specific class, similar information may not be derived from that class. By generating a first prompt based on preprocessing information containing similar information derived through this process, an actual character can engage in a conversation with the user based on its own characteristics.

[0114] In other words, unlike conventional chatbots that generate answers based solely on simple conversation history, the technical feature of this system is that it generates answers based on the character's experiences within the work. While these experiences are not included in the conversation history, they are experiences known to users who have watched the work. Therefore, it can provide users with a greater sense of intimacy and realism than conventional chatbot technology. Furthermore, this configuration can provide users with the aforementioned lesson information more effectively.

[0115]

[0116] Figure 10 schematically illustrates a process in which a second prompt is generated and emotional output information is derived according to one embodiment of the present invention.

[0117]

[0118] As illustrated in FIG. 10, the character video call providing method further includes a second prompt generation step of converting the text output information into text-type emotional information that can be derived from the text output information, and inputting a second prompt including the text emotional information into the macro language model, wherein the macro language model receives the second prompt and outputs emotional output information, the TTS model receives the emotional output information and outputs character emotional information, and the character emotional information is combined with the audio output information and transmitted to the user terminal (1000).

[0119]

[0120] Specifically, when text output information is output from the large language model, the computing system (2000) converts the text output information into text emotion information. The text emotion information is in text form and refers to information expressing emotion. For example, the text output information 'Like!' can be converted into emotion information 'Happy'. In one embodiment of the present invention, the conversion process can be performed through a Text To Emotion model, etc., and in another embodiment of the present invention, it can be performed by extracting emotion information included in the text output information generated from the large language model.

[0121] A second prompt generation step is performed in which a second prompt including such textual sentiment information is generated by a second prompt generation unit (not shown), and the second prompt generation unit inputs the second prompt into the macro language model. The macro language model into which the second prompt is input outputs sentiment output information, and the sentiment output information corresponds to information in the form of text.

[0122] Meanwhile, as another embodiment of the present invention, the text sentiment information can be derived without the macro-language model. In other words, text sentiment information can be derived or converted from text input information without text output information.

[0123]

[0124] FIG. 11 schematically illustrates a process in which a user terminal (1000) receives audio output information and video output information according to one embodiment of the present invention.

[0125]

[0126] As illustrated in FIG. 11, the audio output information is transmitted to the user terminal (1000) together with the video output information, and the video output information includes emotional expression information determined based on the emotional output information among a plurality of pre-stored emotional expressions, and the emotional expression information is information about an emotional expression visually displayed through a corresponding character, and the corresponding character displays the emotional expression before the audio output information is output from the user terminal (1000).

[0127]

[0128] Specifically, the user terminal (1000) displays a character and outputs a response to the user's speech, and the character displays actions and emotional expressions that match the audio output information being output. In addition, the audio output information may be output with a tone, intonation, or interjection that matches the audio output information.

[0129] More specifically, the character of the present invention may not say the audio output information such as 'Good!' in a dry manner, but may say the audio output information in an excited tone, and as one embodiment of the present invention, may say the audio output information with an exclamation such as 'Yahoo!'. Information that can indirectly express emotions, such as such tone, intonation, or exclamation, is included in the character emotion information, and the character emotion information is generated based on the emotion output information and output from the TTS model.

[0130] That is, the TTS model receives text output information and emotion output information and outputs audio output information and character emotion information, but outputs the audio output information and the character emotion information in a combined form. The combined audio output information and character emotion information are transmitted to the user terminal (1000) and provided to the user.

[0131] Meanwhile, the character displayed on the user terminal (1000) may perform a certain action or make a certain facial expression when the audio output information is output. In this way, information related to the movement of the character, such as an action or facial expression, is included in the video output information. The video output information is generated in the computing system (2000) and can be transmitted to the user terminal (1000) together with the audio output information. Preferably, before the character speaks a word corresponding to the audio output information, an emotional expression corresponding to the emotional output information included in the video output information can be displayed. By displaying the emotional expression faster than the voice, the awkwardness felt by the user can be minimized, and the effect of providing the user with a more realistic conversation-like experience can be achieved.

[0132] The above emotional output information may be determined based on the emotional output information among a plurality of previously stored emotional expressions. The plurality of previously stored emotional expressions include joy, sadness, anger, etc. In addition, according to one embodiment of the present invention, there may be a plurality of different actions, movements, or facial expressions for a single emotional expression of joy, and the image output information may also be transmitted to the user terminal (1000) by randomly selecting one of the plurality of different actions, movements, or facial expressions.

[0133]

[0134] Figure 12 schematically illustrates the execution process of the third prompt generation step and the user memory generation step according to one embodiment of the present invention.

[0135]

[0136] As illustrated in FIG. 12, the character video call providing method further includes a third prompt generation step of generating a third prompt including M (M is a natural number greater than or equal to 2) conversation histories included in the conversation history; and a user memory generation step of generating user memory information based on N (N is a natural number less than M) pieces of summary conversation information obtained by inputting the third prompt into the large language model; wherein the user memory information includes the summary conversation information and summary conversation feature information in a vector form by inputting the summary conversation information into an embedding model.

[0137]

[0138] Specifically, Chat Pieces illustrated in FIG. 12 represent conversation history. A third prompt generation unit (not shown) performs a third prompt generation step of generating a third prompt based on M conversation histories included in the conversation history. When the third prompt including the M conversation histories is input into the macro language model, the macro language model outputs summarized conversation information. The summarized conversation information includes a conversation history in which the M conversation histories are summarized into N pieces. In other words, it is reasonable that the summarized conversation information has a smaller size than the conversation history.

[0139] When the above-mentioned summary dialogue information is output, a user memory generation unit (not shown) performs a user memory generation step of generating a user memory based on the above-mentioned summary dialogue information. At this time, the user memory information includes the above-mentioned summary dialogue information and summary dialogue feature information corresponding to the above-mentioned summary dialogue information, referring to FIG. 7A. The above-mentioned summary dialogue feature information can be derived by the above-mentioned embedding model. Through such a configuration, the effect of efficiently managing user memory information can be achieved.

[0140]

[0141] Figure 13 schematically illustrates the execution process of an instant answer generation step according to one embodiment of the present invention.

[0142]

[0143] As illustrated in FIG. 13, the character video call providing method further includes an instant response generation step of generating text output information based on the conversation content by applying a similarity-based search algorithm to the vector-type feature information derived in the feature information extraction step; and a preset DB set storing conversation content exchanged between the character and the user in the past; and deriving conversation content satisfying preset criteria.

[0144]

[0145] Specifically, if the large-scale language model used in the present invention is located outside the computing system (2000), costs may be incurred in using the large-scale language model. In other words, the configuration of the present invention described in FIG. 13 is intended to reduce such costs.

[0146] More specifically, when text input information is input and feature information of the text input information is extracted through the feature information extraction step, a conversation content that satisfies a preset criterion is searched for in a preset DB set through a search algorithm based on the similarity. The preset DB set refers to a set in which conversation content exchanged between the character and the user in the past is stored. In one embodiment of the present invention, the aforementioned conversation history, user memory information, or summary conversation information may correspond to the DB set, but in another embodiment of the present invention, it may correspond to a separate DB set for the corresponding configuration. The DB set is included in the database (2400).

[0147] The search algorithm based on the above similarity can retrieve conversation content similar to the above characteristic information from the above database set. For example, if the question "When do you like Pororo the best?" is received as text input, the previously entered question "When are you the happiest?" can be searched for and found, and the past answer to the question, "I like playing with my friends the best," can be searched from the above database set to derive the text output information "I like playing with my friends the best" without using a large language model. Through such a configuration, the effect of quickly answering the user's question can be achieved while saving costs.

[0148]

[0149] Figure 14 schematically illustrates the internal configuration of a computing device (11000) according to one embodiment of the present invention.

[0150]

[0151] The computing system (2000) illustrated in FIG. 1 described above may include components of the computing device (11000) illustrated in FIG. 14.

[0152] As illustrated in FIG. 14, the computing device (11000) may include at least one processor (11100), memory (11200), peripheral interface (11300), input / output subsystem (I / O subsystem) (11400), power circuit (11500), and communication circuit (11600).

[0153] The memory (11200) may include, for example, high-speed random access memory, a magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory (11200) may include software modules, instruction sets, or other various data required for the operation of the computing device (11000).

[0154] At this time, access to the memory (11200) from other components such as the processor (11100) or peripheral interface (11300) may be controlled by the processor (11100).

[0155] The peripheral interface (11300) may couple input and / or output peripherals of the computing device (11000) to the processor (11100) and memory (11200). The processor (11100) may execute software modules or instruction sets stored in the memory (11200) to perform various functions for the computing device (11000) and process data.

[0156] The input / output subsystem can couple various input / output peripherals to the peripheral interface (11300). For example, the input / output subsystem can include a controller for coupling peripherals such as a monitor, a keyboard, a mouse, a printer, or, as needed, a touchscreen or sensor to the peripheral interface (11300). In another aspect, the input / output peripherals can be coupled to the peripheral interface (11300) without going through the input / output subsystem.

[0157] The power circuit (11500) may supply power to all or part of the components of the terminal. For example, the power circuit (11500) may include a power management system, one or more power sources such as a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, and distribution.

[0158] The communication circuit (11600) may enable communication with another computing device using at least one external port.

[0159] Alternatively, as described above, the communication circuit (11600) may enable communication with other computing devices by transmitting and receiving RF signals, also known as electromagnetic signals, including RF circuits, as needed.

[0160] The embodiment of FIG. 14 is only an example of a computing device (11000), and the computing device (11000) may have some components illustrated in FIG. 14 omitted, may further include additional components not illustrated in FIG. 14, or may have a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor, in addition to the components illustrated in FIG. 14, and may include a circuit for RF communication of various communication methods (WiFi, 3G, LTE, Bluetooth, NFC, Zigbee, etc.) in the communication circuit (11600). Components that can be included in the computing device (11000) may be implemented as hardware including one or more signal processing or application-specific integrated circuits, software, or a combination of both hardware and software.

[0161] Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed by various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application exclusively for mobile terminals. An application to which the present invention is applied may be installed on a computing device (11000) through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file at the request of the computing device (11000).

[0162]

[0163] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0164] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed across network-connected computing devices and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0165] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0166]

[0167] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0168] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

A method for providing a character video call using a deep learning-type natural language processing model performed on a computing system including 1.1 or more memory and 1 or more processes, A dialogue input stage in which audio input information in the form of audio input through a user terminal is received as text input information converted into text by an STT model; A first prompt generation step for generating a first prompt to be input into a deep learning-type natural language processing model, wherein the first prompt includes text input information, conversation history, and preprocessing information; and The above first prompt includes a text output information receiving step for receiving text output information input to the natural language processing model; The above text output information is converted into audio output information in audio format by the TTS model and played on the user terminal, thereby delivering the character's voice to the user. The above first prompt generation step is: A feature information extraction step for extracting feature information in vector form by inputting information including the above text input information into an embedding model; and A similar information derivation step for deriving similar information similar to the above characteristic information using a similarity-based search algorithm for a database containing work information related to a work in which the character appears; A method for providing a character video call, wherein the above preprocessing information includes the above similar information.

2. In claim 1, The above work information is, A method for providing a character video call, comprising: episode-by-episode information of the above work; world view information of the above work; character information in the above work; user memory information derived from past conversations between the character and the user; and sub-character information about characters surrounding the character in the above work.

3. In claim 2, The above similar information derivation step derives the matching rate for each item of the above work information, A method for providing a character video call, wherein the first prompt generation step determines the preprocessing information based on the derived item-by-item matching rate and preset rules.

4. In claim 1, The above first prompt generation step is: A user information input step for receiving previously stored user information; and Further comprising a lesson information input step for inputting pre-saved lesson information; A method for providing a character video call, wherein the first prompt further includes the user information and the lesson information.

5. In claim 1, The above character video call provision method is: It further includes a second prompt generation step of converting the text output information into text emotion information in a text form that can be derived from the text output information, and inputting a second prompt including the text emotion information into the natural language processing model. The above natural language processing model receives the second prompt as input and outputs emotional output information. The above TTS model receives the above emotional output information and outputs character emotional information. A method for providing a character video call, wherein the above character emotion information is combined with the above audio output information and transmitted to the user terminal.

6. In claim 5, The above audio output information is transmitted to the user terminal together with the video output information. The above video output information includes emotional expression information determined based on the emotional output information among multiple pre-stored emotional expressions, The above emotional expression information is information about an emotional expression visually displayed through the character, and the character displays the emotional expression before the audio output information is output from the user terminal. A method for providing a character video call.

7. In claim 1, The above character video call provision method is: A third prompt generation step for generating a third prompt including M (M is a natural number greater than or equal to 2) conversation histories included in the above conversation history; and It further includes a user memory generation step of generating user memory information based on N (N is a natural number less than M) summary conversation information obtained by inputting the third prompt to the natural language processing model; A method for providing a character video call, wherein the above user memory information includes the above summary conversation information and the above summary conversation information entered into an embedding model to include the vector-type summary conversation feature information.

8. In claim 1 The above character video call provision method is: A method for providing a character video call, further comprising: applying a similarity-based search algorithm to feature information in vector form derived from the feature information extraction step; and a preset DB set storing conversations exchanged between the character and the user in the past; and generating text output information based on the conversation content by extracting conversation content that satisfies preset criteria; and generating an instant response based on the conversation content. A computing system including memory of 9.1 or more and one or more processes, and performing a method for providing a character video call using a natural language processing model in the form of deep learning, A dialogue input unit that receives text input information converted into text format by an STT model from audio input information in the form of audio input entered through a user terminal; A first prompt generation unit that generates a first prompt to be input into a deep learning-type natural language processing model, wherein the first prompt includes text input information, conversation history, and preprocessing information; and The above first prompt includes a text output information receiving unit that receives text output information input to the natural language processing model and outputs it; The above text output information is converted into audio output information in audio format by the TTS model and played on the user terminal, thereby delivering the character's voice to the user. The above first prompt generation unit is, A feature information extraction unit that inputs information including the above text input information into an embedding model to extract feature information in vector form; and A similar information extraction unit that extracts similar information similar to the above characteristic information using a similarity-based search algorithm for a database containing information on works related to works in which the character appears; A computing system wherein the above preprocessing information includes the above similar information. A computer-readable recording medium for implementing a method for providing a character video call using a deep learning type natural language processing model, which is performed on a computing system including 10.1 or more memory and 1 or more processors, The above computer-readable recording medium stores instructions that cause the computing system to perform the following steps: The steps below are: A dialogue input stage in which audio input information in the form of audio input through a user terminal is received as text input information converted into text by an STT model; A first prompt generation step for generating a first prompt to be input into a deep learning-type natural language processing model, wherein the first prompt includes text input information, conversation history, and preprocessing information; and The above first prompt includes a text output information receiving step for receiving text output information input to the natural language processing model; The above text output information is converted into audio output information in audio format by the TTS model and played on the user terminal, thereby delivering the character's voice to the user. The above first prompt generation step is: A feature information extraction step for extracting feature information in vector form by inputting information including the above text input information into an embedding model; and A similar information derivation step for deriving similar information similar to the above characteristic information using a similarity-based search algorithm for a database containing information on works related to works in which the character appears; The above preprocessing information is a computer-readable recording medium including the above similar information.

Images