Method and system for generating avatar based on text

Abstract

The present disclosure relates to a text-based avatar generation method performed by at least one processor of an information processing system. The avatar generation method includes the steps of synthesizing the voice of an avatar based on input text, generating a lip-sync animation based on meta-information of the synthesized voice, estimating the motion of an avatar based on input text, and synthesizing the lip-sync animation and the estimated motion to generate a full-body animation of an avatar.

Description

Technology Field

[0001] The present disclosure relates to a text-based avatar generation method and system, and specifically, to a method and system for generating a full-body animation of an avatar by estimating and synthesizing lip-sync animations and movements of an avatar based on text. Background Technology

[0003] As interest in virtual space increases, so does interest in avatars, which are characters that take the place of users in virtual space. Many avatar service providers are making great efforts to create avatars in a virtual 3D space that behave similarly to people in the real world, but the movements of avatars are often unnatural.

[0004] For example, in the case of text-based avatars or avatar animations, there are issues where the avatar's movements themselves are unnatural or the transitions between movements are unnatural. Additionally, when playing full-body avatar animations, there is a problem with a lack of realism in the movements, such as giving the impression that the face and body parts are moving independently.

[0005] Furthermore, in the process of generating avatars or avatar animations based on text, there is a problem in that it is inefficient because extracting information or parameters necessary for implementing avatar animation from the text often involves numerous complex computational processes, and significant errors occur in the results of full-body avatar animation. The problem to be solved

[0007] The present disclosure provides a text-based avatar generation method, a computer program stored on a recording medium, and a device (system) for solving the above-mentioned problems. means of solving the problem

[0009] The present disclosure may be implemented in various ways, including a method, an apparatus (system), or a computer program stored on a readable storage medium.

[0010] According to one embodiment of the present disclosure, a text-based avatar generation method performed by at least one processor of an information processing system comprises the steps of synthesizing the voice of an avatar based on input text, generating a lip-sync animation based on meta-information of the synthesized voice, estimating the motion of an avatar based on input text, and synthesizing the lip-sync animation and the estimated motion to generate a full-body animation of an avatar.

[0011] A computer program stored on a computer-readable recording medium is provided for executing a text-based avatar generation method according to one embodiment of the present disclosure on a computer.

[0012] An information processing system according to one embodiment of the present disclosure comprises a memory and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, and the at least one program includes instructions for synthesizing the voice of an avatar based on input text, generating a lip-sync animation based on meta-information of the synthesized voice, estimating the motion of an avatar based on input text, and synthesizing the lip-sync animation and the estimated motion to generate a full-body animation of an avatar. Effects of the invention

[0014] According to some embodiments of the present disclosure, by including both facial expressions and body gestures and movements in a motion or motion unit representing the movement of an avatar, natural motion in which the face and body are not separated can be realized.

[0015] According to some embodiments of the present disclosure, in generating a full-body animation of an avatar, the movement of not only large muscles but also small muscles can be reflected to generate an animation that moves in detail.

[0016] According to some embodiments of the present disclosure, by estimating the motion unit ID assigned to each motion unit rather than the motion of the avatar or character itself, motions for various avatars or characters can be estimated or synthesized by changing only the resources without retraining the model.

[0017] According to some embodiments of the present disclosure, in estimating motion, by using separate machine learning models trained for each emotion, a more suitable motion can be estimated for each emotion.

[0018] According to some embodiments of the present disclosure, by estimating not only the motion (motion unit ID) but also the playback time information (and connection relationship) of each motion, it is possible to generate an avatar animation that operates naturally without interruption.

[0019] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art to which the present disclosure pertains from the description in the claims (“person skilled in the art”). Brief explanation of the drawing

[0021] Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, wherein similar reference numerals indicate similar elements, but are not limited thereto. FIG. 1 shows the configuration of a text-based avatar generation system according to one embodiment of the present disclosure. FIG. 2 is a block diagram showing the internal configuration of an information processing system according to one embodiment of the present disclosure. FIG. 3 is a diagram showing the internal configuration of a processor of an information processing system according to one embodiment of the present disclosure. FIG. 4 shows the configuration of a text-based avatar generation system according to another embodiment of the present disclosure. FIG. 5 is a drawing showing the internal configuration of a motion estimation unit according to one embodiment of the present disclosure. FIG. 6 is a drawing showing the internal configuration of a motion estimation unit according to another embodiment of the present disclosure. FIG. 7 is a diagram showing a data format that a motion estimation unit learns to output a motion playback time together with a motion unit ID according to one embodiment of the present disclosure. FIG. 8 is a drawing showing blend shape information used by a lip generation unit when generating a lip sync animation according to one embodiment of the present disclosure. FIG. 9 is a flowchart illustrating an example of a text-based avatar generation method according to one embodiment of the present disclosure. Specific details for implementing the invention

[0022] Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, specific descriptions regarding well-known functions or configurations will be omitted if there is a risk that the gist of the present disclosure may be unnecessarily obscured.

[0023] In the attached drawings, identical or corresponding components are assigned the same reference numerals. Additionally, in the description of the following embodiments, the description of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0024] The advantages and features of the disclosed embodiments and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various different forms, and the embodiments provided are merely to make the present disclosure complete and to fully inform those skilled in the art of the scope of the invention.

[0025] The terms used in this specification will be briefly explained, and the disclosed embodiments will be described in detail. The terms used in this specification have been selected to be as generally used as possible, taking into account their functions in this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0026] In this specification, singular expressions include plural expressions unless the context clearly specifies them as singular. Additionally, plural expressions include singular expressions unless the context clearly specifies them as plural. Throughout the specification, when a part is described as including a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0027] Additionally, the terms 'module' or 'part' as used in the specification refer to software or hardware components, and the 'module' or 'part' performs certain roles. However, the meaning of 'module' or 'part' is not limited to software or hardware. The 'module' or 'part' may be configured to reside in an addressable storage medium or configured to run on one or more processors. Thus, as an example, the 'module' or 'part' may include components such as software components, object-oriented software components, class components, and task components, and at least one of processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. The components and the functions provided within the 'module' or 'part' may be combined into a smaller number of components and 'modules' or 'parts', or further separated into additional components and 'modules' or 'parts'.

[0028] According to one embodiment of the present disclosure, a ‘module’ or ‘part’ may be implemented as a processor and memory. The term ‘processor’ should be broadly interpreted to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the term ‘processor’ may refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. The term ‘processor’ may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a DSP core, or any other combination of such configurations. Additionally, the term ‘memory’ should be broadly interpreted to include any electronic component capable of storing electronic information. 'Memory' may refer to various types of processor-readable media, such as Random Access Memory (RAM), Read-Only Memory (ROM), Non-Volatile Random Access Memory (NVRAM), Programmable Read-Only Memory (PROM), Erasable-Programmable Read-Only Memory (EPROM), Electrically Erasable PROM (EEPROM), Flash Memory, Magnetic or Optical Data Storage Devices, Registers, etc. If a processor can read information from memory and / or write information to memory, the memory is said to be in an electronic communication state with the processor. Memory integrated into a processor is in an electronic communication state with the processor.

[0029] In the present disclosure, the 'system' may include at least one of a server device and a cloud device, but is not limited thereto. For example, the system may be composed of one or more server devices. As another example, the system may be composed of one or more cloud devices. As yet another example, the system may be configured and operated with both a server device and a cloud device.

[0030] In the present disclosure, "avatar" may refer to a virtual two-dimensional or three-dimensional object to which a user of any service or application has given themselves an appearance. For example, an avatar may refer to a two-dimensional or three-dimensional object or character that mimics the user's face or full body appearance, or embodies the face or full body appearance set by the user. In the present disclosure, an avatar may be expressed through lip-sync animations, facial and full-body motion animations, etc., generated based on synthesized speech extracted from text input by the user through an application or automatically generated by a service platform (e.g., conversational text input by the user or automatically generated), meta-information associated with the synthesized speech, sentiment information, etc.

[0031] In the present disclosure, a ‘machine learning model’ may include any model used to infer an answer to a given input. According to one embodiment, the machine learning model may include an artificial neural network model comprising an input layer, a plurality of hidden layers, and an output layer. Here, each layer may include a plurality of nodes. In the present disclosure, some or all of the plurality of machine learning models described as separate models may be implemented as a single model, and the machine learning model described as a single model in the present disclosure may be implemented as a plurality of machine learning models.

[0032] In the present disclosure, 'motion' may include animations representing facial expressions and body gestures or movements of an avatar, and in one embodiment, motion may be an animation representing an expression, gesture, and / or movement suitable for the avatar while speaking specific text. According to one embodiment, motion may be composed of a combination or connection of one or more motion units. In the present disclosure, 'motion' may refer to each motion unit or a motion unit ID assigned to each motion unit. In one embodiment, motion may include the avatar taking a basic expression, movement, or gesture without taking any expression, movement, or gesture (no motion).

[0033] In the present disclosure, 'each of a plurality of A' or 'each of a plurality of A' may refer to each of all components included in a plurality of A, or each of some components included in a plurality of A.

[0034] FIG. 1 shows the configuration of a text-based avatar generation system (100) according to one embodiment of the present disclosure. The text-based avatar generation system (100) can generate a full-body animation (160) of an avatar by performing the processes of speech synthesis (120), lip-sync animation generation (130), motion estimation (140), and video synthesis (150) based on input text (110).

[0035] A text-based avatar generation system (100) receives input text (110) (e.g., "Hello. Nice to meet you") and performs speech synthesis (120) (TTS: Text-to-Speech) based on the input text (110) to generate a synthesized voice. Here, the synthesized voice may be a voice generated as a result of the avatar speaking the input text (110) (e.g., the voice of the avatar speaking "Hello. Nice to meet you") and may be a natural conversational voice, but is not limited thereto. In the speech synthesis (120) process, one of various speech synthesis models including a statistical-based model (e.g., HMM model) and a deep learning-based model (e.g., End-to-End model) may be used.

[0036] According to one embodiment, a text-based avatar generation system (100) may receive voice characteristic information of a user (e.g., a voice sample of the user, voice information of the user, etc.) to generate a synthesized sound that imitates the voice of the user, or may use voice characteristic information of an avatar (e.g., a voice sample of the avatar or voice information of the avatar, etc.) to generate a synthesized sound that reflects the voice characteristics of the avatar, but is not limited thereto. For example, the text-based avatar generation system (100) may generate a synthesized sound that is not dependent on the user or the speaker.

[0037] According to one embodiment, a text-based avatar generation system (100) may use meta-information for synthesized speech. Here, the meta-information may include, but is not limited to, phoneme information, playback time information, speech intensity information, etc. of the synthesized speech, and the meta-information may further include any information that can be extracted from the synthesized speech, such as emotion information. Examples of meta-information may include information on a plurality of phonemes included in the synthesized speech, information on the playback start time and playback length of each phoneme, information on the total time the synthesized speech is played, information on intonation such as intonation and stress, and information on speech intensity.

[0038] A text-based avatar generation system (100) can generate a lip-sync animation (130) based on meta-information of synthesized voice. A lip-sync animation may refer to an animation that shows a change in the mouth shape of an avatar according to the utterance of synthesized voice. According to one embodiment, a blendshape-based animation technique may be used to generate the lip-sync animation (130), but is not limited thereto. The generation of the lip-sync animation (130) using a blendshape-based animation technique will be described in detail later with reference to FIG. 8.

[0039] A text-based avatar generation system (100) can estimate the motion of an avatar (140) based on input text (110). Here, the motion may be an animation representing an expression, gesture, and / or movement suitable for the avatar while speaking the input text (110). According to one embodiment, the motion may be composed of a combination or connection of one or more motion units, and each motion or each motion unit may include an animation representing the avatar's facial expression and body gesture / movement. Rather than implementing the movements of the face and body separately, a more natural motion can be achieved by including both the facial expression and the body gesture / movement in the motion or motion unit. According to one embodiment, each motion unit may be assigned a motion unit ID, and the text-based avatar generation system (100) can estimate one or more motion unit IDs based on the input text (110). If the motion unit ID assigned to each motion unit is estimated rather than the motion of the avatar or character itself, it may be possible to generate motions for various avatars or characters by changing only the resources without retraining the motion estimation model used for motion estimation (140).

[0040] According to one embodiment, the face portion of the motion or motion unit may be generated using a blend shape-based animation technique. For example, motion of the face portion may be generated by a method of combining a plurality of blend shapes (or morphing targets) in a morphing manner. As a specific example, a plurality of blend shapes may be defined, including a first blend shape with the left eye closed, a second blend shape with the right eye closed, a third blend shape with the left lip raised in a smile, a fourth blend shape with the right lip raised in a smile, and the like; and a facial expression may be generated by blending each blend shape with a weight value (e.g., 0 or more, 1 or less). A face portion of the motion may be generated by combining a plurality of such facial expressions in a morphing manner.

[0041] According to some embodiments, a text-based avatar generation system (100) may perform motion estimation (140) by considering additional information in addition to the input text (110), and a machine learning model may be used for motion estimation (140). This will be described in detail later with reference to FIGS. 3 to 7.

[0042] A text-based avatar generation system (100) can generate a full-body animation of an avatar by performing video synthesis (150) on lip-sync animation and estimated motion. For example, a full-body animation of an avatar can be generated through a process such as adding a blend shape of a lip-sync animation to a blend shape format for expressing the face part of the avatar's motion. In one embodiment, the generated full-body animation may be an animation in which the avatar moves in detail by reflecting not only large muscles but also small muscles.

[0043] According to one embodiment, the full-body animation can be generated in a data form applicable to various avatars, rather than in the form of an animation generated for a specific avatar. Accordingly, the generated full-body animation data can be applied to various avatars, characters (people, animals, etc.).

[0044] According to one embodiment, the full-body animation may include synthesized sound. According to another embodiment, the full-body animation may be output together with synthesized sound when it is output on a display (e.g., a display of a user terminal). In one embodiment, the playback length of the full-body animation may be related to the playback length of the synthesized sound. For example, the playback length of the full-body animation may be equal to the playback length of the synthesized sound, and for a more natural effect, the playback length of the full-body animation may be longer than the playback length of the synthesized sound. As a specific example, after the avatar finishes speaking, a knot-tying motion may be performed for a predetermined time (e.g., 1 to 2 seconds).

[0045] As described above, by generating a lip-sync animation (130) that is like speaking the input text (110), estimating a motion based on the input text (110) (140), and synthesizing the lip-sync animation and the motion to generate a full-body animation (160) of the avatar, a natural animation can be generated that appears as if the avatar is actually speaking the input text (110) and performing actions.

[0046] FIG. 2 is a block diagram showing the internal configuration of an information processing system (200) according to one embodiment of the present disclosure. The information processing system (200) may include a memory (210), a processor (220), a communication module (230), and an input / output interface (240). The information processing system (200) may be configured to communicate information and / or data through a network using the communication module (230).

[0047] The memory (210) may include any non-transient computer-readable recording medium. According to one embodiment, the memory (210) may include a non-perishable permanent mass storage device such as RAM (random access memory), ROM (read only memory), disk drive, SSD (solid state drive), flash memory, etc. As another example, a non-perishable permanent mass storage device such as ROM, SSD, flash memory, disk drive, etc. may be included in the information processing system (200) as a separate permanent storage device distinct from the memory. Additionally, the memory (210) may store an operating system and at least one program code (e.g., code for voice synthesis and full-body animation of an avatar installed and run on the information processing system (200), etc.).

[0048] These software components may be loaded from a computer-readable recording medium separate from the memory (210). This separate computer-readable recording medium may include a recording medium that can be directly connected to the information processing system (200), for example, a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. As another example, the software components may be loaded into the memory (210) via a communication module (230) rather than a computer-readable recording medium. For example, at least one program may be loaded into the memory (210) based on a computer program (e.g., a program for voice synthesis and full-body animation of an avatar, etc.) that is installed by files provided through the communication module (230) by developers or a file distribution system that distributes installation files for applications.

[0049] The processor (220) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to a user terminal (not shown) or another external system by memory (210) or a communication module (230). For example, the processor (220) may synthesize the voice of an avatar based on input text, generate a lip-sync animation based on meta-information of the synthesized voice, estimate the motion of an avatar based on input text, and synthesize the lip-sync animation and the estimated motion to generate a full-body animation of an avatar.

[0050] The communication module (230) may provide a configuration or function for a user terminal (not shown) and an information processing system (200) to communicate with each other via a network, and may provide a configuration or function for the information processing system (200) to communicate with an external system (e.g., a separate cloud system). For example, control signals, commands, data, etc. provided under the control of the processor (220) of the information processing system (200) may be transmitted to the user terminal and / or the external system through the communication module (230) and the network, and through the communication module of the user terminal and / or the external system. For example, the user terminal may receive a full-body animation (and synthesized voice) of a generated avatar.

[0051] Additionally, the input / output interface (240) of the information processing system (200) may be a means for interfacing with a device (not shown) for input or output that is connected to the information processing system (200) or that the information processing system (200) may include. In FIG. 2, the input / output interface (240) is shown as an element configured separately from the processor (220), but is not limited thereto, and the input / output interface (240) may be configured to be included in the processor (220). The information processing system (200) may include more components than those shown in FIG. 2. However, there is no need to clearly illustrate most of the prior art components.

[0052] The processor (220) of the information processing system (200) may be configured to manage, process, and / or store information and / or data received from a plurality of user terminals and / or a plurality of external systems. According to one embodiment, the processor (220) may receive input text from a user terminal. Then, based on the input text, the voice of the avatar may be synthesized, a lip-sync animation may be generated based on the meta-information of the synthesized voice, the motion of the avatar may be estimated based on the input text, and the lip-sync animation and the estimated motion may be synthesized to generate a full-body animation of the avatar.

[0053] FIG. 3 is a diagram showing the internal configuration of a processor (220) of an information processing system according to one embodiment of the present disclosure. According to one embodiment, the processor (220) may include a speech synthesis unit (310), an emotion analysis unit (320), a lip generation unit (330), a motion estimation unit (340), and an image synthesis unit (350). The internal configuration of the processor (220) of the information processing system illustrated in FIG. 3 is merely an example and may be implemented differently. For example, at least some of the configurations of the processor (220) may be omitted or other configurations may be added, and at least some of the processes performed by the processor (220) may be performed by the processor of the user terminal.

[0054] According to one embodiment, the processor (220) may receive input text, and the input text may be provided to at least one of a speech synthesis unit (310), an emotion analysis unit (320), and a motion estimation unit (340). According to one embodiment, the input text may be received from a user terminal, an external system, or another application (e.g., a conversation generation application), or may be generated by a conversation generation unit (not shown) of the processor (220).

[0055] The speech synthesis unit (310) can generate a synthesized voice by synthesizing the voice of the avatar based on the input text. Here, the synthesized voice may refer to the voice of the avatar speaking the input text. According to one embodiment, the synthesized voice may be a natural conversational voice, but is not limited thereto. For example, in another embodiment, the synthesized voice may be a guidance voice for guidance. The speech synthesis unit (310) can generate a synthesized voice using any speech synthesis model, such as a statistical-based model (e.g., HMM model) or a deep learning-based model (e.g., End-to-End model).

[0056] According to one embodiment, the voice synthesis unit (310) may receive voice characteristic information of a user and generate a synthesized sound that mimics the user's voice, or receive voice characteristic information of an avatar and generate a synthesized sound that reflects the voice characteristics of the avatar, but is not limited thereto. For example, the voice synthesis unit (310) may generate a synthesized sound that is not dependent on the user or the speaker.

[0057] Additionally, the voice synthesis unit (310) can synthesize the voice of the avatar by further utilizing the emotion information extracted by the emotion analysis unit (320). For example, if the emotion of 'joy' is extracted from the input text by the emotion analysis unit (320), a synthesized voice reflecting the emotion information of 'joy' can be generated.

[0058] According to one embodiment, the speech synthesis unit (310) may be provided with meta-information for the synthesized speech to at least one of the lip generation unit (330) and the motion estimation unit (340). Here, the meta-information may include, but is not limited to, phoneme information, playback time information, and speech intensity information of the synthesized speech, and the meta-information may further include any information that can be extracted from the synthesized speech, such as emotion information. Examples of meta-information may include information on a plurality of phonemes included in the synthesized speech, information on the playback start time and playback length of each phoneme, information on the total time the synthesized speech is played, information on intonation such as intonation and stress, and information on speech intensity.

[0059] The emotion analysis unit (320) can extract emotion information from the input text. For example, the emotion analysis unit (320) can infer emotion information (e.g., joy, neutrality, sadness, anger, etc.) that is presumed to be felt by the person speaking the input text. According to one embodiment, the emotion analysis unit (320) can extract emotion information using a machine learning model trained to infer emotion information from the input text. In addition, in one embodiment, the emotion analysis unit (320) can receive not only the input text but also additional conversation content before and after the input text, and extract emotion information for the input text by considering the overall conversation content or context containing the input text. The emotion information extracted by the emotion analysis unit (320) can be provided to at least one of the speech synthesis unit (310) and the motion estimation unit (340).

[0060] The lip generation unit (330) can generate a lip sync animation based on meta-information of the synthesized sound extracted by the speech synthesis unit (310). The lip sync animation may refer to an animation showing a change in the mouth shape of an avatar according to the utterance of the synthesized sound. According to one embodiment, a blendshape-based animation technique may be used to generate the lip sync animation, but is not limited thereto. The generation of the lip sync animation using a blendshape-based animation technique will be described in detail later with reference to FIG. 8.

[0061] The motion estimation unit (340) can estimate the motion of the avatar based on the input text. The motion of the avatar may be an animation representing an expression, gesture, and / or movement suitable for the avatar while speaking the input text. According to one embodiment, the motion may be composed of a combination or connection of one or more motion units, and each motion or each motion unit may include an animation representing the avatar's facial expression and body gesture / movement. According to one embodiment, each motion unit may be assigned a motion unit ID, and the motion estimation unit (340) can estimate one or more motion unit IDs based on the input text.

[0062] According to one embodiment, the motion estimation unit (340) may receive not only input text but also additional meta-information of emotion information and / or synthesized sound, and may estimate the motion of the avatar based on the received information. Here, the emotion information may be emotion information extracted from the input text by the emotion analysis unit (320) or emotion information included in the meta-information of the synthesized sound. Additionally, according to one embodiment, the motion estimation unit (340) may estimate not only the motion (motion unit ID) but also playback time information of each motion.

[0063] In one embodiment, the motion estimation unit (340) may use a machine learning model to estimate motion (and motion playback time information) based on input text (and emotion information). The estimation of motion using a machine learning model will be described in detail later with reference to FIGS. 5 to 7.

[0064] The video synthesis unit (350) can complete the motion by extracting motion unit files corresponding to each motion unit ID estimated by the motion estimation unit (340) from the motion unit DB (360) and appropriately combining, connecting, or synthesizing the extracted motion unit files.

[0065] Additionally, the video synthesis unit (350) can generate a full-body animation of the avatar by synthesizing the generated lip sync animation and the motion of the avatar. For example, the video synthesis unit (350) can generate a full-body animation of the avatar by appropriately mixing the lip sync animation and the motion of the avatar through methods such as adding the blend shape of the lip sync animation to the blend shape format for expressing the face part of the motion of the avatar, or synthesizing the lip sync animation and the motion. Any 3D graphic technique such as blendshape, morphing, blending, or interpolation may be used when the video synthesis unit (350) connects or combines each motion unit file or synthesizes the lip sync animation and the motion.

[0066] FIG. 4 illustrates the configuration of a text-based avatar generation system (400) according to another embodiment of the present disclosure. According to one embodiment, the text-based avatar generation system (400) may include a conversation generation unit (410), a speech synthesis unit (310), an emotion analysis unit (320), a lip generation unit (330), a motion estimation unit (340), and an image synthesis unit (350). In FIG. 4, configurations that overlap with FIG. 3 are briefly described based on the embodiment shown in FIG. 4.

[0067] According to one embodiment, the dialogue generation unit (410) can generate input text using any language generation model. Here, the input text generated by the dialogue generation unit (410) can be input text for generating a full-body animation. In one embodiment, the dialogue generation unit (410) can generate a second text based on a first text using any language generation model. For example, it can receive a first text input by a user from a user terminal and generate a second text (e.g., a response to the first text) based thereon. Here, the second text generated by the dialogue generation unit (410) can be input text for generating a full-body animation.

[0068] Additionally, the conversation generation unit (410) can generate additional information associated with the input text in addition to the input text. For example, the conversation generation unit (410) can generate a command phrase “(Anna raises her arms in joy)” along with the input text “Today I feel so good”, such as “Today I feel so good (Anna raises her arms in joy).” According to one embodiment, the generated additional information is provided to the speech synthesis unit (310), the emotion analysis unit (320), and / or the motion estimation unit (340) and can be used to generate synthesized speech reflecting the additional information, extract emotion information, or estimate motion.

[0069] According to one embodiment, the dialogue generation unit (410) may be omitted. For example, text generated by another application (e.g., a language generation application) may be received as input text, or text directly entered by a user may be received as input text.

[0070] The emotion analysis unit (320) can extract emotion information from the input text. For example, the emotion analysis unit (320) can infer emotion information (e.g., joy, neutrality, sadness, anger, etc.) that is presumed to be felt by the person speaking the input text. The emotion information extracted by the emotion analysis unit (320) can be provided to the speech synthesis unit (310) and the motion estimation unit (340).

[0071] The voice synthesis unit (310) can synthesize the voice of the avatar based on input text and emotion information. For example, the voice synthesis unit (310) can synthesize a voice such as the avatar speaking the input text with the extracted emotion. Text-to-speech (TTS) meta-information can be extracted from the voice synthesized by the voice synthesis unit (310), and the extracted TTS meta-information can be provided to the lip generation unit (330) and the motion estimation unit (340).

[0072] The lip generation unit (330) can generate a lip sync animation based on TTS meta-information. For example, the lip generation unit (330) can generate a lip sync animation that shows a change in mouth shape (or a change in lip movement, etc.) as the avatar speaks a synthesized sound based on TTS meta-information. In one embodiment, the lip sync animation can be generated using a blend shape-based animation technique.

[0073] The motion estimation unit (340) can estimate the motion of an avatar based on input text, TTS metadata, and emotion information. For example, the motion estimation unit (340) can estimate the motion unit ID and motion playback time information constituting the motion based on input text, emotion information, and TTS metadata using a machine learning model. An embodiment in which the motion estimation unit (340) estimates the motion unit ID (and motion playback time information) constituting the motion based on input text, emotion information, and / or TTS metadata using a machine learning model will be described in detail later with reference to FIGS. 5 to 7.

[0074] The video synthesis unit (350) can complete the motion by extracting motion unit files corresponding to each motion unit ID estimated by the motion estimation unit (340) from the motion unit DB (360) and appropriately combining, connecting, or synthesizing the extracted motion unit files. Additionally, the video synthesis unit (350) can generate a full-body animation of the avatar by synthesizing the generated lip sync animation and the motion of the avatar.

[0075] Synthetic voice and full-body animation generated by the text-based avatar generation system (400) can be transmitted to a user terminal, and the user terminal can output the received synthetic voice and full-body animation through an output device. For example, the user terminal can output the synthetic voice through a speaker while simultaneously displaying the full-body animation on a display. Through this, the user can perceive that the avatar is making facial expressions, mouth shapes, and movements included in the full-body animation while speaking the synthetic voice.

[0076] FIG. 5 is a diagram showing the internal configuration of a motion estimation unit (340) according to one embodiment of the present disclosure. According to one embodiment, the motion estimation unit (340) can estimate the motion unit ID of an avatar based on input text and emotion information, and the motion estimation unit (340) may include an emotion branching unit (510) and an emotion-specific machine learning model (520, 530, 540, 550). Here, the emotion information received by the motion estimation unit (340) may be emotion information extracted by the emotion analysis unit (320) based on the input text, emotion information included in the meta-information of the synthesized sound, or emotion information entered by a user.

[0077] The emotion-specific machine learning models (520, 530, 540, 550) may include one or more machine learning models trained for each emotion. For example, the machine learning model (joy) (520) may be a model trained to receive input text associated with the emotion of 'joy' and infer a motion unit ID associated with the emotion of 'joy', and the machine learning model (sadness) (540) may be a model trained to receive input text associated with the emotion of 'sadness' and infer a motion unit ID associated with the emotion of 'sadness'.

[0078] According to one embodiment, the emotion branching unit (510) can determine the machine learning model (520, 530, 540, 550) to use based on the received emotion information. For example, if the received emotion information is 'joy', it can be determined to use the machine learning model (joy) (520). Then, a motion unit ID reflecting the emotion information can be estimated based on the input text by the machine learning model (520, 530, 540, 550) determined to be used by the emotion branching unit (510).

[0079] The motion unit ID inferred by each emotion-specific machine learning model (520, 530, 540, 550) may be a motion unit ID corresponding to a motion representing each emotion. According to one embodiment, even if the motions perform the same or similar actions, various motions may be implemented for each emotion. For example, as shown in FIG. 5, for the same or similar actions of raising one hand to the chest, various motions may be implemented for each emotion.

[0080] In FIG. 5, the motion estimation unit (340) is illustrated as receiving input text and emotion information to estimate a motion unit ID, but is not limited thereto. For example, the motion estimation unit (340) may receive only input text without emotion information. In this case, the emotion branching unit (510) may extract emotion information from the input text in a manner similar to the emotion analysis unit (320). For example, the emotion branching unit (510) may extract emotion information using a machine learning model trained to infer emotion information from the input text. Subsequently, the emotion branching unit (510) may determine the machine learning model (520, 530, 540, 550) to use based on the extracted emotion information, and the subsequent process may be performed as described above.

[0081] According to one embodiment, the motion estimation unit (340) may receive input text (and emotion information) as well as previous conversation or conversation context information as input. In this case, the emotion branching unit (510) may extract emotion information by considering additional information or determine a machine learning model (520, 530, 540, 550) to use, and the received additional information may be input into the machine learning model (520, 530, 540, 550) together with the input text (and emotion information) and used to estimate the motion unit ID.

[0082] FIG. 6 is a diagram showing the internal configuration of a motion estimation unit (340) according to another embodiment of the present disclosure. According to one embodiment, the motion estimation unit (340) can estimate the motion unit ID of an avatar based on input text and emotion information using a single machine learning model (610). Here, the emotion information received by the motion estimation unit (340) may be emotion information extracted by the emotion analysis unit (320) based on the input text, emotion information included in the meta-information of the synthesized sound, or emotion information entered by a user.

[0083] A single machine learning model (610) may be a model trained to estimate a motion unit ID by receiving input text and emotion information as input. That is, the single machine learning model (610) can estimate a motion that is suitable for the input text and reflects emotion information. Therefore, according to the single machine learning model (610) that estimates a motion unit ID based on input text and emotion information, even if the input text is the same, a different motion unit ID (or motion) can be estimated depending on the received emotion information.

[0084] In FIG. 6, the motion estimation unit (340) is illustrated as receiving input text and emotion information to estimate a motion unit ID, but is not limited thereto. For example, the motion estimation unit (340) may receive only input text without emotion information. In this case, the single machine learning model (610) may estimate the motion unit ID based only on the input text. Even in this case, the single machine learning model (610) may estimate the motion unit ID by reflecting the emotion information identified by the input text. According to the single machine learning model (610) that estimates the motion unit ID by receiving only the input text as input, if the input text is the same, the same motion unit ID (or motion) may be estimated.

[0085] According to one embodiment, the motion estimation unit (340) may receive input text (and emotion information) as well as previous conversation or conversation context information as input, and the received additional information may be input into a single machine learning model (610) together with the input text (and emotion information) and used to estimate the motion unit ID.

[0086] As shown in FIGS. 5 and 6, when the motion estimation unit (340) estimates the motion unit ID assigned to each motion unit rather than the motion of the avatar or character itself, it is possible to estimate motion for various avatars or characters by changing only the resources without retraining the machine learning model.

[0087] FIG. 7 is a diagram illustrating a data format in which a motion estimation unit is trained to output a motion playback time along with a motion unit ID according to one embodiment of the present disclosure. According to one embodiment, the motion estimation unit can output the motion playback time along with the motion unit ID using a machine learning model. To train the machine learning model to output the motion playback time along with the motion unit ID, text and TTS metadata (710, 730), playback time information, and motion unit ID (720, 730) can all be included in the training data. That is, the machine learning model can be trained to output playback time information and motion unit ID (720, 730) when text and TTS metadata (710, 730) are input. Here, the text may include not only a single sentence but also multiple sentences.

[0088] For example, an example of a training data format is illustrated in FIG. 7. Training data 1 (710) includes text 1 "Hello. Nice to meet you. It's Tta-bong." and meta-information of the synthesized sound for text 1 (e.g., syllable or phoneme information, playback time information per syllable or phoneme). The emotion information included in training data 1 (710) may be emotion information included in the meta-information of the synthesized sound, or emotion information extracted from the emotion analysis unit based on the text. Training data 2 (720) includes motion unit ID 1 "Unit 1 (Greeting motion), Unit 45 (Neutral motion), Unit 78 (Tta-bong motion), Unit 90 (Idle motion)" and playback time information 1 of the motion corresponding to the motion unit ID. The playback time information of the motion may include time information when playback of each motion begins and playback length information. Between each motion, a connection operation may be performed to ensure a natural transition between motions, and an idle motion may be added at the point where all motions for the text end to wrap up the motion. The motion playback time information estimated by the motion estimator may be estimated by considering the connection relationships of motion actions so that the connection to the next motion can be naturally maintained. To perform this estimation considering connection relationships, the training data may include data containing the connection relationships of motions. For example, data in the form of "1-connect-45-connect-78-connect-90" containing the connection relationships of each motion may be included in the training data.

[0089] Another training data, training data 3 (730), similar to training data 1 (710), includes meta-information of text 2 and synthesized sound for text 2 (e.g., phoneme or syllable information, playback time information per phoneme or syllable), and training data 4 (740), similar to training data 2 (720), includes motion unit ID 2 and playback time information 2 of the motion.

[0090] The machine learning model can be trained using training data in the form of pairs of training data 1 (710) and training data 2 (720), and pairs of training data 3 (730) and training data 4 (740), and the motion estimation unit can use the machine learning model trained in this way to estimate the motion unit ID and playback time information (and connection relationship) of each motion based on the metadata of the input text and synthesized sound.

[0091] As described above, the motion estimation unit estimates the motion unit ID and playback time information (and connection relationships) of each motion, and synthesizes the motion and generates the animation based on this, thereby enabling the generation of a seamless and natural avatar animation.

[0092] FIG. 8 is a diagram showing blend shape information used by a lip generation unit when generating a lip sync animation according to one embodiment of the present disclosure. According to one embodiment, the animation of a face portion of a motion or motion unit and / or a lip sync animation may be generated using a blend shape-based animation technique. For example, the motion of a face portion and / or a lip sync animation may be generated by a method of combining a plurality of blend shapes (or morphing targets) in a morphing manner.

[0093] As a specific example of generating motion of a face part using a blend shape-based animation technique, a plurality of blend shapes may be specified, including a first blend shape with the left eye closed, a second blend shape with the right eye closed, a third blend shape with the left lip raised and smiling, and a fourth blend shape with the right lip raised and smiling. A facial expression may be generated by blending each blend shape with a weight value (e.g., 0 or greater, 1 or less). A face part of motion may be generated by combining multiple such facial expressions in a morphing manner.

[0094] According to one embodiment, the lip sync animation may be generated in a manner somewhat different from the motion of the face part. For example, for a plurality of phonemes (810, 840), a plurality of mouth shape images (viseme) (820) representing the mouth shape (830) when each phoneme (810, 840) is pronounced may be specified. The lip generation unit may extract mouth shape images (820) representing mouth shapes corresponding to phoneme information included in meta-information for the synthesized sound, and may generate an animation by combining the plurality of mouth shape images (820) so that a mouth shape image (820) representing the mouth shape (830) corresponding to the phoneme (810, 840) is played at the time when a specific phoneme (810, 840) is pronounced. According to one embodiment, in generating the lip sync animation, the lip generation unit may generate a natural lip sync animation by using any 3D graphic technique, such as interpolation or morphing techniques.

[0095] According to one embodiment, to generate general-purpose motion or animation, a predefined blend shape format or a blend shape API (application programming interface), etc., may be used.

[0096] FIG. 9 is a flowchart illustrating an example of a text-based avatar generation method (900) according to one embodiment of the present disclosure. According to one embodiment, the method (900) may be initiated by a processor of an information processing system synthesizing the voice of an avatar based on input text (S910). In one embodiment, meta-information may be extracted from the synthesized voice, and the meta-information may include at least one of playback time information, phoneme information, and speech intensity information of the synthesized voice.

[0097] Then, the processor can generate a lip sync animation based on meta-information of the synthesized voice (S920). For example, the processor can generate a lip sync animation by extracting one or more mouth shape images for the lip sync animation corresponding to phoneme information included in the meta-information and synthesizing the extracted one or more mouth shape images. According to one embodiment, the lip sync animation may be an animation that includes the mouth shape of an avatar that changes based on meta-information of the synthesized voice.

[0098] The processor can estimate the motion of the avatar based on the input text (S930). For example, the processor can extract one or more motion unit IDs corresponding to the input text and extract one or more motion units corresponding to each motion unit ID. Additionally, the processor can estimate the motion of the avatar based on the input text and meta information by additionally considering meta information in addition to the input text. Furthermore, the processor can estimate not only the motion unit ID corresponding to the input text but also the playback time of the motion unit corresponding to the motion unit ID. In one embodiment, the playback time of the motion unit may be the playback time up to a point where the connection with the next motion unit is natural, and when generating a motion animation, a more natural animation can be created by playing only the corresponding time of the motion unit and connecting it to the next unit.

[0099] According to one embodiment, a processor can extract emotion information from input text and estimate motion of an avatar based on at least one of the input text and emotion information. In one embodiment, to estimate motion of an avatar based on input text (and / or emotion information), the processor may use a machine learning model trained to estimate motion associated with input text using input text (and / or emotion information) as input.

[0100] According to one embodiment, the input text used in steps S910 to S930 may be generated by a language generation model. For example, text generated by a language generation model as an answer to a specific text may be the input text.

[0101] After that, the processor can synthesize the lip-sync animation and the estimated motion to generate a full-body animation of the avatar (S940). The process of synthesizing the lip-sync animation and the estimated motion to generate a full-body animation of the avatar may include a process of synchronizing the playback time of the lip-sync animation with the playback time of the estimated motion.

[0102] The flowchart illustrated in FIG. 9 and the description above are merely examples and may be implemented differently in some embodiments. For example, in some embodiments, the order of each step may be changed, some steps may be repeated, some steps may be omitted, or some steps may be added.

[0103] The method described above may be provided as a computer program stored on a computer-readable recording medium for execution on a computer. The medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or multiple hardware components combined, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Furthermore, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0104] The methods, operations, or techniques of the present disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various exemplary logical blocks, modules, circuits, and algorithmic steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate such interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functional aspects. Whether such functions are implemented in hardware or in software depends on the design requirements imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementations should not be construed as departing from the scope of the present disclosure.

[0105] In a hardware implementation, the processing units used to perform the techniques may be implemented in one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described in this disclosure, computers, or a combination thereof.

[0106] Accordingly, the various exemplary logic blocks, modules, and circuits described in connection with the present disclosure may be implemented or performed by any combination of general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or those designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors coupled with a DSP core, or any other combination of configurations.

[0107] In firmware and / or software implementations, techniques may be implemented as instructions stored on a computer-readable medium such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. The instructions may be executable by one or more processors, and may cause the processor(s) to perform specific aspects of the functions described in this disclosure.

[0108] Where implemented in software, the techniques may be stored on a computer-readable medium as one or more instructions or code, or transmitted through a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transmission of a computer program from one place to another. Storage media may be any available medium accessible by a computer. As a non-limiting example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium accessible by a computer that can be used to transfer or store desired program code in the form of instructions or data structures. Additionally, any connection is appropriately referred to as a computer-readable medium.

[0109] For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of a medium. As used herein, disk and disc include CD, laser disc, optical disc, DVD (digital versatile disc), floppy disk, and Blu-ray disc, wherein disks usually play data magnetically, whereas discs play data optically using a laser. The above combinations should also be included within the scope of computer-readable media.

[0110] The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other known form of storage medium. An exemplary storage medium may be connected to a processor so that the processor can read information from the storage medium or write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and the storage medium may exist within an ASIC. The ASIC may exist within a user terminal. Alternatively, the processor and the storage medium may exist as separate components within the user terminal.

[0111] Although the embodiments described above have been described as utilizing aspects of the subject matter disclosed herein in one or more standalone computer systems, the present disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or a distributed computing environment. Furthermore, aspects of the subject matter in the present disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across a plurality of devices. Such devices may include PCs, network servers, and portable devices.

[0112] Although the present disclosure has been described in relation to some embodiments, various modifications and changes may be made without departing from the scope of the present disclosure as understood by a person skilled in the art to which the invention of the present disclosure pertains. Furthermore, such modifications and changes should be considered to fall within the scope of the claims appended to this specification. Explanation of the symbols

[0114] 100: Text-based Avatar Creation System 110: Input text 120: Speech synthesis 130: Create Lip Sync Animation 140: Motion Estimation 150: Video Synthesis 160: Full body animation

Claims

Claim 1 A text-based avatar generation method performed by at least one processor of an information processing system, comprising: a step of synthesizing the voice of an avatar based on input text using a Text-to-Speech (TTS) model and extracting TTS meta-information of the synthesized voice, wherein the TTS meta-information includes playback time information of the synthesized voice; a step of generating a lip-sync animation synchronized with the synthesized voice based on the TTS meta-information; and a step of estimating the motion of the avatar synchronized with the synthesized voice based on the TTS meta-information and the input text. A method for generating an avatar, comprising the step of synthesizing the lip sync animation and the motion of the avatar by synchronizing the playback time of the lip sync animation and the playback time of the motion of the avatar, and the step of estimating the motion of the avatar comprises the step of outputting the motion of the avatar corresponding to the input text by a machine learning model that has been pre-trained to estimate the motion associated with the input text based on the input text, wherein the machine learning model is pre-trained using training data that includes connection relationship data including a training text and connection relationships between motions including each motion corresponding to the training text, and the connection relationship data includes a sequence composed of motion unit IDs arranged in the order in which the motions are performed. Claim 2 A method for generating an avatar according to claim 1, wherein the step of estimating the motion of the avatar comprises: a step of extracting emotion information from the input text; and a step of estimating the motion of the avatar synchronized with the synthesized voice based on the TTS meta information, the input text, and the emotion information. Claim 3 A method for generating an avatar according to claim 1, wherein the step of estimating the motion of the avatar comprises: a step of extracting a motion unit ID based on the TTS meta information and the input text; and a step of extracting a motion unit corresponding to the motion unit ID. Claim 4 A method for generating an avatar according to claim 1, wherein the step of estimating the motion of the avatar comprises: a step of extracting a motion unit ID corresponding to the input text; and a step of estimating the playback time of a motion unit corresponding to the motion unit ID based on the TTS meta information. Claim 5 A method for generating an avatar according to claim 1, wherein the step of generating a full-body animation of the avatar comprises generating a full-body animation of the avatar that includes, at the end, an idle motion performed for a predetermined time as a motion that concludes the motion of the avatar, and the motions including each motion corresponding to the learning text include the idle motion. Claim 6 A method for generating an avatar according to claim 1, wherein the step of estimating the motion of the avatar comprises: a step of extracting emotional information based on the input text; and a step of outputting the motion of the avatar corresponding to the emotional information by one or more machine learning models trained to estimate the motion associated with the emotional information based on the emotional information. Claim 7 A method for generating an avatar according to claim 1, wherein the step of generating the lip sync animation comprises: a step of extracting one or more mouth shape images for a lip sync animation corresponding to phoneme information included in the TTS meta information; and a step of synthesizing the one or more extracted mouth shape images to generate the lip sync animation. Claim 8 A method for generating an avatar according to claim 1, wherein the TTS meta information further includes at least one of phoneme information and speech intensity information of the synthesized voice, and the step of generating a lip sync animation based on the TTS meta information includes the step of generating a lip sync animation including a changing mouth shape based on the TTS meta information. Claim 9 A method for generating an avatar according to claim 1, further comprising the step of generating the input text by a language generation model. Claim 10 A computer program stored on a computer-readable recording medium for executing a method according to any one of paragraphs 1 through 9 on a computer. Claim 11 As an information processing system, memory; and includes at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, wherein the at least one program includes instructions for synthesizing the voice of an avatar based on input text using a Text-to-Speech (TTS) model, extracting TTS metadata of the synthesized voice - wherein the TTS metadata includes playback time information of the synthesized voice -, generating a lip-sync animation synchronized with the synthesized voice based on the TTS metadata, estimating the motion of the avatar synchronized with the synthesized voice based on the TTS metadata and the input text, and synthesizing the lip-sync animation and the motion of the avatar to generate a full-body animation of the avatar by synthesizing the playback time of the lip-sync animation and the playback time of the motion of the avatar, and wherein the at least one program includes instructions for outputting the motion of the avatar corresponding to the input text by a machine learning model pre-trained to estimate the motion associated with the input text based on the input text, and wherein the machine learning model includes motions including a training text and each motion corresponding to the training text. An information processing system that is pre-learned using learning data including connection relationship data including connection relationship between, wherein the connection relationship data includes a sequence composed of motion unit IDs arranged in the order in which the motions are performed.

Images