Systems and methods for generating context-driven animated emoji

Abstract

The present disclosure relates to a method and system for providing contextually animated emojis. The method comprises obtaining an input text message by a user. The method comprises obtaining at least one message from a message thread among users. The method comprises obtaining, audio vector embeddings by encoding the obtained input text message and each of the obtained at least one message. The method comprises obtaining lip vertices, and generating a series of face expression codes based on the audio vector embeddings and the lip vertices. The method comprises obtaining guide pose vectors, and generating a series of joint angle vectors based on the audio vector embeddings and the guide pose vectors. The method comprises animating an emoji based on the series of face expression codes and the series of joint angle vectors.

Description

SYSTEMS AND METHODS FOR GENERATING CONTEXT-DRIVEN ANIMATED EMOJI

[0001] The present disclosure generally relates to computer-implemented methods and systems for providing personalized and context-sensitive emoji animations.

[0002] Nowadays, users can send and receive different types of media in the form of voice messages, images, and videos in various chat applications. These, in addition to plain text, provide a rich source of multi-modal context. Emojis also help convey additional emotion and meaning to these types of messages. Typically, users manually select one or more emojis from a predefined set based on their preferences to complement the context of their conversations. The related art has automated this manual selection of emojis for a more personalized and meaningful approach towards animation and sharing of emojis.

[0003] In accordance with a related art, a method for generating full-body gestures for virtual agents in social virtual reality (VR) applications is provided. It presents a transformer-based network that takes in text one sentence at a time and generates 3D pose sequences for virtual agents corresponding to gestures and emotions aligned with that text. On the other hand, in accordance with a related art, a method for generating highly realistic and expressive 3D facial animations for avatars / emojis based on diverse multi-media inputs, such as audio, text, image, and video is provided. It presents a latent diffusion model for generating facial animations with lip-sync speech and nuanced human emotions contained in text, images, and music. The model takes audio feature, text, and image encodings to generate sequential facial expressions and head poses. Also, in accordance with a related art, a multimodal-based reaction response generation method is provided. It can be implemented in an intelligent conversational agent capable of generating multimodal output data based on multimodal input data, such as text, speech, images, visual or auditory inputs. Example of output data is an animation sequence of the avatar of the intelligent conversational agent, wherein the animation sequence is built to an animation template marked with corresponding facial expression type, body movement type, emotion type, emotion level, among others, based on extracted features from the input data.

[0004] According to an embodiment of the disclosure, a method for context-driven emoji animation may be disclosed. The method may comprise obtaining, via a user-input interface, an input text message by a user. The method may comprise obtaining at least one message from a message thread among users, wherein each of the obtained at least one message is in a format comprising at least one of text, audio, and image. The method may comprise obtaining, audio vector embeddings by encoding the obtained input text message and each of the obtained at least one message. The method may comprise obtaining lip vertices from the audio vector embeddings, and generating a series of face expression codes based on the audio vector embeddings and the obtained lip vertices. The method may comprise obtaining guide pose vectors from the audio vector embeddings, and generating a series of joint angle vectors based on the audio vector embeddings and the obtained guide pose vectors. The method may comprise animating an emoji based on the generated series of face expression codes and the generated series of joint angle vectors.

[0005] According to an embodiment of the disclosure, a system for providing context-driven emoji animation is provided. The system may comprise a user-input interface for inputting a text message by a user. The system may comprise at least one processor comprising processing circuitry. The system may comprise a memory in which one or more instructions are stored. In an embodiment, the one or more instructions, when executed by the at least one processor individually or collectively, cause the system to obtain, via the user-input interface, an input text message. In an embodiment, the one or more instructions, when executed by the at least one processor individually or collectively, cause the system to obtain at least one message from a message thread among users, wherein each of the obtained at least one message is in a format comprising at least one of text, audio, and image. In an embodiment, the one or more instructions, when executed by the at least one processor individually or collectively, cause the system to obtain audio vector embeddings by encoding the obtained input text message and each of the obtained at least one message. In an embodiment, the one or more instructions, when executed by the at least one processor individually or collectively, cause the system to obtain lip vertices from the audio vector embeddings, and generat a series of face expression codes based on the audio vector embeddings and the obtained lip vertices. In an embodiment, the one or more instructions, when executed by the at least one processor individually or collectively, cause the system to obtain guide pose vectors from the audio vector embeddings, and generat a series of joint angle vectors based on the audio vector embeddings and the obtained guide pose vectors. In an embodiment, the one or more instructions, when executed by the at least one processor individually or collectively, cause the system to animate an emoji based on the generated series of face expression codes and the generated series of joint angle vectors.

[0006] The accompanying drawings, which are included to provide a further understanding of the disclosure, are incorporated herein to illustrate embodiments of the disclosure. Along with the description, they also serve to explain the principle of the disclosure. In the drawings:

[0007] FIG. 1 is a flowchart illustrating a method for context-driven emoji animation based on multimodal conversation context and user input text message according to an embodiment of the present disclosure.

[0008] FIG. 2 is a diagram illustrating a system for context-driven emoji animation based on multimodal conversation context and user input text message according to an embodiment of the present disclosure.

[0009] FIG. 3 is a diagram illustrating the text encoder, image encoder, and phoneme encoder of the multimodal embedding module according to one aspect of the present disclosure.

[0010] FIG. 4 is a diagram illustrating the audio encoder of the multimodal embedding module according to one aspect of the present disclosure.

[0011] FIG. 5 is a diagram illustrating the face motion module comprising the face motion model and lip regressor model.

[0012] FIG. 6 is a diagram illustrating the body motion module comprising the body motion model and guide pose generator model.

[0013] FIG. 7 is a diagram illustrating an exemplary embodiment of the operation of the system.

[0014] FIG. 8 is a diagram illustrating an exemplary embodiment of the operation of the system.

[0015] FIG. 9 is a diagram illustrating an exemplary embodiment of the operation of the system.

[0016] FIG. 10 is a diagram illustrating an exemplary embodiment of the operation of the system.

[0017] FIG. 11 is a diagram illustrating an exemplary embodiment of the operation of the system.

[0018] FIG. 12 is a flowchart illustrating a method for context-driven emoji animation according to an embodiment of the present disclosure.

[0019] It should be appreciated that the blocks in each flowchart and combinations of the flowcharts may be performed by one or more computer programs which include computer-executable instructions. The entirety of the one or more computer programs may be stored in a single memory or the one or more computer programs may be divided with different portions stored in different multiple memories.

[0020] Any of the functions or operations described herein can be processed by one processor or a combination of processors. The one processor or the combination of processors is circuitry performing processing and includes circuitry like an application processor (AP), a communication processor (CP), a graphical processing unit (GPU), a neural processing unit (NPU), a microprocessor unit (MPU), a system on chip (SoC), an IC, or the like.

[0021] The processor may include various processing circuitry and / or multiple processors. For example, as used herein, including the claims, the term “processor” may include various processing circuitry, including at least one processor, wherein one or more of at least one processor, individually and / or collectively in a distributed manner, may be configured to perform various functions described herein. As used herein, when “a processor”, “at least one processor”, and “one or more processors” are described as being configured to perform numerous functions, these terms cover situations, for example and without limitation, in which one processor performs some of recited functions and another processor(s) performs other of recited functions, and also situations in which a single processor may perform all recited functions. Additionally, the at least one processor may include a combination of processors performing various of the recited / disclosed functions, e.g., in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0022] FIG. 1 illustrates a method for context-driven emoji animation based on context from multimodal conversation and user input text message according to the exemplary embodiment of the present disclosure. Replying to the current message thread, a user inputs a text message by, for example, typing on a keyboard or touch-screen interface or voice command queries via a microphone (operation S100). At least one message of various modalities (text, image, audio) from a message thread among users in, for example, a chat application, is obtained (e.g. captured) and read by a computing unit, for example, a smartphone (operation S102). Each of the captured at least one message is encoded by a multimodal embedding module into audio vector embeddings. In an embodiment, vector embeddings refer to lower-dimensional vector representations of input data. Audio vector embeddings are a specific type of vector embedding that encode semantic, prosodic, emotional, and timbral features of the input. In an embodiment, audio vector embeddings may be generated by combining content vector embeddings derived from a user-inputted text message with context vector embeddings derived from prior multimodal messages. Depending on the modality, a specific encoder within the multimodal embedding module is used to encode each of the captured at least one message. Messages in text, audio, and image modality are encoded by a text encoder, audio encoder, and image encoder, respectively. Image messages may be in at least one of the forms: static images, animated images, for example, a GIFs, and videos. Additionally, the user-inputted text message is encoded by a phoneme encoder within the multimodal embedding module (operation S104). The goal of the various encoders of the multimodal embedding module is to understand the emotion, mood, and tone of the user and among users as expressed from the conversation and input text and translate the context and content in the form of the audio vector embeddings.

[0023] Next, lip vertices are obtained by a face motion model of a face motion module. The obtained lip vertices are used in combination with the audio vector embeddings as input by the face motion model to generate a series of face expression codes (operation S106). Further, guide pose vectors are obtained by a body motion model of a body motion module. The obtained guide pose vectors are used in combination with the audio vector embeddings as input by the body motion model to generate a series of joint angle vectors, or in layman’s terms, body motion sequences (operation S108). Both the generated series of face expression codes and series of joint angle vectors are processed by an emoji renderer to animate, for example, a full-body emoji or avatar, which can be displayed on, for example, a smartphone’s screen, computer monitor, virtual reality headset, AR glasses, television displays, chatbot applications, among others (operation S110).

[0024] An overview of the system 100 for generating context-driven animated emojis is demonstrated in FIG. 2. The conversation thread and user-inputted text message from a chat application 200 are taken as input to the multimodal embedding module 202. The multimodal embedding module comprises a text encoder 204, an image encoder 206, and an audio encoder 208 to encode, depending on their modalities, each message from the conversation thread into audio vector embeddings 210. A text encoder 204 is used for messages in text modality. An audio encoder 208 is used for messages in audio modality. And an image encoder 206 is used is used for messages in image modality, which includes at least one of: static images, animated images, for example, a GIFs, and videos. Additionally, the multimodal embedding module 202 comprises a phoneme encoder 212 to encode the user-inputted text message into audio vector embeddings 210. The system comprises a face motion module 214 for obtaining lip vertices and generating face expression codes 216 based on the obtained lip vertices and audio vector embeddings 210. The system also comprises a body motion module 218 for obtaining guide pose vectors and generating joint angle vectors 220, which represents body motion sequences, based on the obtained guide pose vectors and audio vector embeddings 210. Lastly, the system comprises an emoji renderer 222 for processing the face expression codes and joint angle vectors to fully animate an emoji or an avatar 224.

[0025] FIG. 3 shows an embodiment of the present disclosure of how the messages with different modalities are transformed into audio vector embeddings. The messages identified as being in text modality 321, 322, 323, 324, 325 are inputted to the text encoder 302 and encoded into context vector embeddings 304. The messages identified as being in image modality 306 are inputted to the image encoder 308 and encoded into context vector embeddings 304 as well. An example of a text encoder 302 and image encoder 308 is CLIP (Contrastive Language-Image Pre-Training) model. The user-inputted text message 310 is forwarded to the phoneme encoder 312, which converts the input text 310 into phoneme tokens and then embeds each token into vectors called content vector embeddings 314. The context vector embeddings 304 from the text encoder 302 and image encoder 308 are concatenated together and combined with the content vector embeddings 314, after which the combined vector embeddings are inputted to the contextual text-to-speech encoder 316 to output audio vector embeddings 318. In this way, the transformed output would have the same structure regardless of the various modalities presented in the input data.

[0026] FIG. 4 presents a simple illustration of how audio signals are transformed into audio vector embeddings. Given that speech data can already be extracted from the audio signals 400, the audio encoder 402 takes directly the audio signals 400 as input and outputs the audio vector embeddings 404. Specifically, the audio encoder 402, for example, wav2vec model, encodes the audio data into semantically meaningful representation vectors. Audio data may, for example, come from speech or the audio stream of a video.

[0027] In accordance with one aspect of the present disclosure, the operations of obtaining lip vertices and generating face expression codes are further illustrated in FIG. 5. The face motion module 500 comprises mainly of two components: lip regressor 502 and face motion model 504. The lip regressor 502 takes the audio vector embeddings 506 as input and outputs a series of lip vertices 508. The lip regressor 502 can be any model or method, which can generate lip geometry audio-synchronized with the speech from the audio data. The series of lip vertices 508 and the audio vector embeddings 506 are then used by the face motion model 504 to generate the series of face expression codes 510.

[0028] In an aspect of the present disclosure, the operations of obtaining guide pose vectors and generating joint angle vectors are further illustrated in FIG. 6. The body motion module 600 comprises mainly of two components: a guide pose generator 602 and a body motion model 604. An example of a body motion model 604 is a denoising diffusion probabilistic model. The guide pose generator 602 retrieves a collection of reference guide pose data 606 from the memory of the computing unit. This reference guide pose data 606 are used in conjunction with the audio vector embeddings 608 by the guide pose generator 602 to generate the guide pose vectors 610. The guide pose generator 602 can be, for example, any model or method conditioned to autoregressively generate guide poses. The audio vector embeddings 608 are then used as input to the body motion model 604 along with the generated guide pose vectors 610 to produce a series of joint angle vectors 612. In an aspect, at inference time, noise data 614 can be used as additional input to the body motion model 604 to vary the outputs of the body motion sequence given the same set of inputs. Here, users can always expect diverse body movements of their emoji response.

[0029] FIG. 7 is a diagram illustrating an exemplary embodiment of the operation of the system.

[0030] In an embodiment of the present disclosure, the system comprises a computing device 700. The computing device 700 may include a display, memory storage, and a user-input interface. For example, the user-input interface may comprise one or more input devices configured to receive instructions from the user.

[0031] In an embodiment, the computing device 700 includes a processing unit (e.g., a processor) and a memory (e.g., memory storage) configured to store and execute an inference method that renders a context-driven animated emoji based on messages from the current conversation.

[0032] In an embodiment, the computing device 700 obtains (e.g., receives) a user input indicating how many prior messages to consider for context (e.g., context window size 702). The computing device 700 may also receive a user instruction 704 to initiate the rendering process via one or more input devices, such as a keyboard, mouse, touch screen, or microphone for voice command queries.

[0033] In an embodiment, the computing device 700 outputs the rendered animated emoji 706 through one or more visual output devices. For example, such visual output devices may include a mobile device screen, computer monitor, virtual reality headset, or television display.

[0034] FIG. 8 is a diagram illustrating an exemplary embodiment of the operation of the system.

[0035] In an embodiment of the present disclosure, the system 100 (e.g., system for inference 806) comprises a computing device (e.g., smart phone) that includes a chat application and an augmented reality (AR) application.

[0036] Conversation context 802, comprising one or more messages in various modalities (e.g., text, images, audio, or video), is read by the system 806 from the chat application. The context window size may be specified by the user via input device such as a keyboard or a voice typing interface. If the user does not specify the context window size, a defined (e.g., predetermined or preset) default value is used.

[0037] In an embodiment, the input text message 804 is received from the user via a keyboard or a voice typing interface.

[0038] Using the conversation context 802 obtained from the chat application, the system 806 generates an animation based on the input text message 804 and renders an animated full-body emoji 808 using an emoji rendering application such as an AR Emoji engine.

[0039] This embodiment may also be applied to other similar computing devices, including but not limited to virtual reality (VR) headsets and AR glasses.

[0040] FIG. 9 is a diagram illustrating an exemplary embodiment of the operation of the system. Using an input speech message 902 obtained from a voice message, the system 904 generates an animation based on the input speech message 902 and renders an animated full-body emoji 906.

[0041] In an embodiment of the present disclosure, the system may use speech data as the only input. For example, in scenarios where the user does not choose to type and include prior messages from a message thread, the user may choose to trigger the microphone of the user device to record a voice message. The recorded voice message is then used by the system to animate the emoji, such that the overall emotion or mood of the user (or speaker in the audio) is captured by the emoji animation. The facial and body movements of the emoji may match the expressions in the speech data. The facial movement of the emoji, for example, the specific movements of the mouth and eye may, as well, match each phoneme or word identified in the speech data. After which, the recorded voice message complemented by the animated emoji are sent to the user’s current conversation.

[0042] FIG. 10 is a diagram illustrating an exemplary embodiment of the operation of the system. Using conversation context 1002 obtained from messages in various modalities (e.g., text, images, audio and video), the system 1004 generates an animation based on the conversation context 1002 and renders an animated full-body emoji 1006.

[0043] In an embodiment, the system uses only the conversation context as input. This can be useful in scenarios where the user wants to send a personalized reaction without having to type a text or send a voice message.

[0044] FIG. 11 is a diagram illustrating an exemplary embodiment of the operation of the system. Using inputs 1102 (e.g., conversation context, speech, text), the system 1104 generates an animation based on the inputs 1102 and renders either an animated face-only emoji 1106 or full-body still emoji 1108.

[0045] In an embodiment, the output of the system may be an animated full-body emoji. In an alternative embodiment, the output of the system may be, for example, a face-only rendered emoji, whether animated or still, or a rendered still image full-body of the emoji. However, the output is not limited to these examples.

[0046] In an embodiment of the present disclosure, the emoji used for animation can be manually selected by the user, automatically selected based on context, or already pre-selected based on user default settings. The range of prior messages from the message conversation to be included for understanding the context, hereby termed as context window, can be specified by the user, for example, by the number of messages, by manual selection of each message, or by inputting a range of date and time. If not specified, a preset default number of messages will be used.

[0047] FIG. 12 is a flowchart illustrating a method for context-driven emoji animation according to an embodiment of the present disclosure.

[0048] In an embodiment, at operation S1201, the method includes obtaining, via a user-input interface, an input text message by a user. At operation S1202, the method includes obtaining at least one message from a message thread among users, wherein each of the obtained at least one message is in a format comprising at least one of text, audio, and image. At operation S1203, the method includes obtaining, audio vector embeddings by encoding the obtained input text message and each of the obtained at least one message. At operation S1204, the method includes obtaining lip vertices from the audio vector embeddings, and generating a series of face expression codes based on the audio vector embeddings and the obtained lip vertices. At operation S1205, the method includes obtaining guide pose vectors from the audio vector embeddings, and generating a series of joint angle vectors based on the audio vector embeddings and the obtained guide pose vectors. At operation S1206, the method includes animating an emoji based on the generated series of face expression codes and the generated series of joint angle vectors.The present disclosure relates to a computer-implemented method and system for providing contextually animated emojis based on the user’s conversation and user-inputted text message. Given a user conversation comprising one or messages in various formats, such as text, audio, image, animation, or video, a multimodal embedding module uses modality-specific models to encode each message with different formats and output audio vector embeddings. The multimodal embedding module comprises a text encoder, a phoneme encoder, an image encoder, and an audio encoder, among others. A face motion model processes the audio vector embeddings, in combination with lip vertices, to generate a series of face expression codes. A body motion model also processes the audio vector embeddings, in combination with guide pose vectors, to generate a series of high-frequency motion sequence, i.e., joint angle vectors. In this way, the context from the conversation and user-inputted text message is embedded in these face expression codes and high-frequency motion sequence, which are then processed by an emoji renderer to contextually animate the emoji. The present disclosure specifically discloses the method of incorporating user-inputted text message and encoding this into audio vector embeddings to translate the emotions of the user and content of the message into the animated emojis or avatars. The present disclosure also discloses the specific method of transforming the multimodal conversation messages and user-inputted text message to output data having a unified single form, which is the audio vector embeddings. These audio vector embeddings are used, in general, to contextualize and animate the user’s emoji message.

[0049] The present disclosure relates to a computer-implemented method and system for generating animated emojis based on conversation context and user input text message. Given a conversation consisting of text, speech, image, animation (GIF), and / or video, the computer-implemented method and system outputs an animation of an emoji that conveys the context?emotion, mood, tone?of the conversation and the expression in the user’s input text response. A multimodal embedding module uses modality-specific models to encode into audio vector embeddings each message in different formats in the conversation. The multimodal embedding module comprises a text encoder, a phoneme encoder, an image encoder, and an audio encoder, among others. After obtaining the audio vector embeddings, a face motion model within the face motion module processes the audio vector embeddings, in combination with lip vertices, to generate a series of face expression codes. On the other hand, a body motion model within the body motion module processes the same audio vector embeddings, in combination with guide pose vectors, to generate a series of high-frequency motion sequence, i.e., joint angle vectors. Therefore, the context and content from the conversation and user’s intended response is embedded in these face expression codes and high-frequency motion sequence, which are then processed by an emoji renderer to contextually animate the emoji.

[0050] In one aspect of the present disclosure, the text encoder and the image encoder encode the conversation message in text and image format, respectively, into context vector embeddings. The phoneme encoder encodes the user’s input text response into content vector embeddings. A text-to-speech encoder of the multimodal embedding module takes the context vector embeddings and content vector embeddings as input to produce the audio vector embeddings.

[0051] In an aspect of the present disclosure, the face motion module further comprises a lip regressor model for generating the lip vertices based on the audio vector embeddings. In a further aspect of the present disclosure, the body motion module further comprises a guide pose generator model for generating the guide pose vectors using the audio vector embeddings and a collection of reference guide pose data as input. Noise vectors can also be used as additional input to the body motion model to generate more variations of the body motions of the animated emoji.

[0052] According to an embodiment of the disclosure, a computer-implemented method for context-driven emoji animation based on multimodal conversation context and user-inputted text message is provided. The method may comprise inputting, via a user-input interface, a text message by a user. The method may comprise capturing, by a computing unit, at least one message from a message thread among users. In an embodiment, each of the captured at least one message is in the format selected from, but not limited to, text, audio, and image. The method may comprise encoding, by a multimodal embedding module, the inputted text message and each of the captured at least one message into audio vector embeddings. The method may comprise encoding, by a text encoder of the multimodal embedding module, the captured at least one message in text format into audio vector embeddings. The method may comprise encoding, by a phoneme encoder of the multimodal embedding module, the inputted text message into audio vector embeddings. The method may comprise encoding, by an image encoder of the multimodal embedding module, the captured at least one message in image format into audio vector embeddings. The method may comprise encoding, by an audio encoder of the multimodal embedding module, the captured at least one message in audio format into audio vector embeddings. The method may comprise obtaining lip vertices and generating, by a face motion model of a face motion module, a series of face expression codes based on the audio vector embeddings and the obtained lip vertices. The method may comprise obtaining guide pose vectors and generating, by a body motion model of a body motion module, a series of joint angle vectors based on audio vector embeddings and the obtained guide pose vectors. The method may comprise animating, by an emoji renderer, an emoji based on the generated series of face expression codes and series of joint angle vectors.

[0053] In an embodiment, the captured at least one message in the image format includes at least one of: a static image, animated images, or video. In an embodiment, the step of encoding the captured at least one message in text and image format and the step of encoding the inputted text message further comprise encoding, by the text encoder, the captured at least one message in text format into context vector embeddings. The step of encoding the captured at least one message in text and image format and the step of encoding the inputted text message further comprise encoding, by the image encoder, the captured at least one message in image format into context vector embeddings. The step of encoding the captured at least one message in text and image format and the step of encoding the inputted text message further comprise encoding, by the phoneme encoder, the inputted text message into content vector embeddings. The step of encoding the captured at least one message in text and image format and the step of encoding the inputted text message further comprise combining and encoding, by a text-to-speech encoder of the multimodal embedding module, the context vector embeddings and content vector embeddings into the audio vector embeddings.

[0054] In an embodiment, the step of obtaining lip vertices further comprises generating, by a lip regressor model of the face motion module, lip vertices based on the audio vector embeddings.

[0055] In an embodiment, the step of obtaining guide pose vectors further comprises generating, by a guide pose generator model of the body motion module, guide pose vectors based on the audio vector embeddings and a collection of reference guide pose data.

[0056] In an embodiment, the method may comprise adding noise vectors as additional input to the body motion module and generating the series of joint angle vectors in further variations.

[0057] According to an embodiment of the disclosure, a context-driven emoji animation system using multimodal conversation context and user input text message is provided. The system may comprise a user-input interface for inputting a text message by a user. The system may comprise a computing unit for capturing at least one message from a message thread among users, wherein each of the captured at least one message is in the format selected from, but not limited to, text, audio, and image. The system may comprise a multimodal embedding module for encoding the inputted text message and each of the captured at least one message into audio vector embeddings. The system may comprise a text encoder for encoding the captured at least one message in text format into audio vector embeddings. The system may comprise a phoneme encoder for encoding the inputted text message into audio vector embeddings. The system may comprise an image encoder for encoding the captured at least one message in image format into audio vector embeddings. The system may comprise an audio encoder for encoding the captured at least one message in audio format into audio vector embeddings. The system may comprise a face motion module comprising a face motion model for obtaining lip vertices and generating a series of face expression codes based on the audio vector embeddings and the obtained lip vertices. The system may comprise a body motion module comprising a body motion model for obtaining guide pose vectors and generating a series of joint angle vectors based on audio vector embeddings and the obtained guide pose vectors. The system may comprise an emoji renderer for animating an emoji based on the generated series of face expression codes and series of joint angle vectors.

[0058] In an embodiment, the captured at least one message in the image format includes at least one of: a static image, animated images, or video.

[0059] In an embodiment, the system may comprise the text encoder encoding the captured at least one message in text format into context vector embeddings. In an embodiment, the system may comprise the image encoder encoding the captured at least one message in image format into context vector embeddings. In an embodiment, the system may comprise the phoneme encoder encoding the inputted text message into content vector embeddings.

[0060] In an embodiment, the multimodal embedding module further comprises a text-to-speech encoder for combining and encoding the context vector embeddings and content vector embeddings into the audio vector embeddings.

[0061] In an embodiment, the face motion module further comprises a lip regressor model for generating lip vertices based on the audio vector embeddings.

[0062] In an embodiment, the body motion module further comprises a guide pose generator model for generating guide pose vectors based on the audio vector embeddings and a collection of reference guide pose data.

[0063] In an embodiment, noise vectors is provided as additional input to the body motion module for generating the series of joint angle vectors in further variations.

[0064] According to an embodiment of the disclosure, a method for context-driven emoji animation may be disclosed. The method may comprise obtaining, via a user-input interface, an input text message by a user. The method may comprise obtaining at least one message from a message thread among users, wherein each of the obtained at least one message is in a format comprising at least one of text, audio, and image. The method may comprise obtaining, audio vector embeddings by encoding the obtained input text message and each of the obtained at least one message. The method may comprise obtaining lip vertices from the audio vector embeddings, and generating a series of face expression codes based on the audio vector embeddings and the obtained lip vertices. The method may comprise obtaining guide pose vectors from the audio vector embeddings, and generating a series of joint angle vectors based on the audio vector embeddings and the obtained guide pose vectors. The method may comprise animating an emoji based on the generated series of face expression codes and the generated series of joint angle vectors.

[0065] In an embodiment, the obtaining the audio vector embeddings may comprise encoding, by a phoneme encoder of a multimodal embedding module, the obtained input text message into the audio vector embeddings. In an embodiment, the obtaining the audio vector embeddings may comprise encoding, by a text encoder of the multimodal embedding module, the obtained at least one message in text format into the audio vector embeddings. In an embodiment, the obtaining the audio vector embeddings may comprise encoding, by an image encoder of the multimodal embedding module, the obtained at least one message in image format into the audio vector embeddings. In an embodiment, the obtaining the audio vector embeddings may comprise encoding, by an audio encoder of the multimodal embedding module, the obtained at least one message in audio format into the audio vector embeddings.

[0066] In an embodiment, the obtained at least one message in the image format includes at least one of: a static image, animated images, or video.

[0067] In an embodiment, the encoding the obtained at least one message in text and image format and the encoding the input text message may comprise encoding, by the text encoder, the obtained at least one message in text format into context vector embeddings. In an embodiment, the encoding the obtained at least one message in text and image format and the encoding the input text message may comprise encoding, by the image encoder, the obtained at least one message in image format into the context vector embeddings. In an embodiment, the encoding the obtained at least one message in text and image format and the encoding the input text message may comprise encoding, by the phoneme encoder, the input text message into content vector embeddings. In an embodiment, the encoding the obtained at least one message in text and image format and the encoding the input text message may comprise combining and encoding, by a text-to-speech encoder of the multimodal embedding module, the context vector embeddings and the content vector embeddings into the audio vector embeddings.

[0068] In an embodiment, the obtaining lip vertices may comprise generating, by a lip regressor model of a face motion module, the lip vertices based on the audio vector embeddings.

[0069] In an embodiment, the obtaining guide pose vectors may comprise generating, by a guide pose generator model of a body motion module, the guide pose vectors based on the audio vector embeddings and a collection of reference guide pose data.

[0070] In an embodiment, the method may comprise adding noise vectors as additional input to the body motion module and generating the series of joint angle vectors in further variations.

[0071] According to an embodiment of the disclosure, a system for providing context-driven emoji animation is provided. The system may comprise a user-input interface for inputting a text message by a user. The system may comprise at least one processor comprising processing circuitry. The system may comprise a memory in which one or more instructions are stored. In an embodiment, the one or more instructions, when executed by the at least one processor individually or collectively, cause the system to obtain, via the user-input interface, an input text message. In an embodiment, the one or more instructions, when executed by the at least one processor individually or collectively, cause the system to obtain at least one message from a message thread among users, wherein each of the obtained at least one message is in a format comprising at least one of text, audio, and image. In an embodiment, the one or more instructions, when executed by the at least one processor individually or collectively, cause the system to obtain audio vector embeddings by encoding the obtained input text message and each of the obtained at least one message. In an embodiment, the one or more instructions, when executed by the at least one processor individually or collectively, cause the system to obtain lip vertices from the audio vector embeddings, and generate a series of face expression codes based on the audio vector embeddings and the obtained lip vertices. In an embodiment, the one or more instructions, when executed by the at least one processor individually or collectively, cause the system to obtain guide pose vectors from the audio vector embeddings, and generate a series of joint angle vectors based on the audio vector embeddings and the obtained guide pose vectors. In an embodiment, the one or more instructions, when executed by the at least one processor individually or collectively, cause the system to animate an emoji based on the generated series of face expression codes and the generated series of joint angle vectors.

[0072] In an embodiment, the system may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system to encode, by a phoneme encoder of a multimodal embedding module, the obtained input text message into the audio vector embeddings. In an embodiment, the system may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system to encode, by a text encoder of the multimodal embedding module, the obtained at least one message in text format into the audio vector embeddings. In an embodiment, the system may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system to encode, by an image encoder of the multimodal embedding module, the obtained at least one message in image format into the audio vector embeddings. In an embodiment, the system may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system to encode, by an audio encoder of the multimodal embedding module, the obtained at least one message in audio format into the audio vector embeddings.

[0073] In an embodiment, the obtained at least one message in the image format includes at least one of: a static image, animated images, or video.

[0074] In an embodiment, the system may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system to encode, by the text encoder, the obtained at least one message in text format into context vector embeddings. In an embodiment, the system may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system to encode, by the image encoder, the obtained at least one message in image format into the context vector embeddings. In an embodiment, the system may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system to encode, by the phoneme encoder, the input text message into content vector embeddings.

[0075] In an embodiment, the system may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system to combine and encode, by a text-to-speech encoder of the multimodal embedding module, the context vector embeddings and the content vector embeddings into the audio vector embeddings.

[0076] In an embodiment, the system may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system to generate, by a lip regressor model of a face motion module, the lip vertices based on the audio vector embeddings.

[0077] In an embodiment, the system may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system to generate, by a guide pose generator model of a body motion module, the guide pose vectors based on the audio vector embeddings and a collection of reference guide pose data.

[0078] In an embodiment, the system may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system to add noise vectors as additional input to the body motion module for generating the series of joint angle vectors in further variations.

[0079] The system for generating context-driven animated emojis can be implemented in various configurations. For example, the multimodal embedding module, face motion module, and body motion module may reside locally within a user device, remotely in cloud computing units, or a combination thereof. The system may employ a single model architecture or a combination of machine learning models or neural networks including, but not limited to, recurrent, convolutional, recursive, long / short term memory (LSTM), denoising autoencoder, variational autoencoder, generative adversarial networks (GAN), fully connected, and feed-forward neural networks.

[0080] Computer software programs and algorithms?those including machine learning and neural networks?may be written in any of various suitable programming languages, such as C, C++, C#, Pascal, Fortran, Perl, MATLAB (from MathWorks, www.mathworks.com), SAS, SPSS, JavaScript, CoffeeScript, Objective-C, Objective-J, Ruby, Python, Erlang, Lisp, Scala, Clojure, and Java. The computer software programs may be an independent application with data input and data display modules. Alternatively, the computer software programs may be classes that may be instantiated as distributed objects. The computer software programs may also be component software such as Java Beans (from Oracle) or Enterprise Java Beans (EJB from Oracle).

[0081] It is contemplated for embodiments described herein to extend to individual elements and concepts described herein, independently of other concepts, ideas or system, as well as for embodiments to include combinations of elements recited anywhere in this application. It is to be understood that the disclosure is not limited to the embodiments described in detail herein with reference to the accompanying drawings. As such, many variations and modifications will be apparent to practitioners skilled in this art. Accordingly, it is intended that the scope of the disclosure be defined by the following claims and their equivalents. Moreover, it is contemplated that a feature described either individually or as part of an embodiment may be combined with other individually described features, or parts of other embodiments, even if the other features and embodiments make no mention of the said feature. Hence, the absence of describing combinations should not preclude the inventor from claiming rights to such combinations.

Claims

1.A method for context-driven emoji animation, the method comprising:obtaining, via a user-input interface, an input text message by a user;obtaining at least one message from a message thread among users, wherein each of the obtained at least one message is in a format comprising at least one of text, audio, and image;obtaining, audio vector embeddings by encoding the obtained input text message and each of the obtained at least one message;obtaining lip vertices from the audio vector embeddings, and generating a series of face expression codes based on the audio vector embeddings and the obtained lip vertices;obtaining guide pose vectors from the audio vector embeddings, and generating a series of joint angle vectors based on the audio vector embeddings and the obtained guide pose vectors; andanimating an emoji based on the generated series of face expression codes and the generated series of joint angle vectors.2.The method as claimed in claim 1, wherein the obtaining the audio vector embeddings comprising:encoding, by a phoneme encoder of a multimodal embedding module, the obtained input text message into the audio vector embeddings;encoding, by a text encoder of the multimodal embedding module, the obtained at least one message in text format into the audio vector embeddings;encoding, by an image encoder of the multimodal embedding module, the obtained at least one message in image format into the audio vector embeddings; andencoding, by an audio encoder of the multimodal embedding module, the obtained at least one message in audio format into the audio vector embeddings.3.The method as claimed in claim 1 or claim 2, wherein the obtained at least one message in the image format includes at least one of: a static image, animated images, or video.4.The method as claimed in any one of claims 1 to 3, wherein the encoding the obtained at least one message in text and image format and the encoding the input text message further comprise:encoding, by the text encoder, the obtained at least one message in text format into context vector embeddings;encoding, by the image encoder, the obtained at least one message in image format into the context vector embeddings;encoding, by the phoneme encoder, the input text message into content vector embeddings; andcombining and encoding, by a text-to-speech encoder of the multimodal embedding module, the context vector embeddings and the content vector embeddings into the audio vector embeddings.5.The method as claimed in any one of claims 1 to 4, wherein the obtaining lip vertices further comprises generating, by a lip regressor model of a face motion module, the lip vertices based on the audio vector embeddings.6.The method as claimed in any one of claims 1 to 5, wherein the obtaining guide pose vectors further comprises generating, by a guide pose generator model of a body motion module, the guide pose vectors based on the audio vector embeddings and a collection of reference guide pose data.7.The method as claimed in any one of claims 1 to 6, further comprising adding noise vectors as additional input to the body motion module and generating the series of joint angle vectors in further variations.8.A system for providing context-driven emoji animation comprising:a user-input interface for inputting a text message by a user;at least one processor comprising processing circuitry; anda memory in which one or more instructions are stored,wherein the one or more instructions, when executed by the at least one processor individually or collectively, cause the system toobtain, via the user-input interface, an input text message;obtain at least one message from a message thread among users, wherein each of the obtained at least one message is in a format comprising at least one of text, audio, and image;obtain audio vector embeddings by encoding the obtained input text message and each of the obtained at least one message;obtain lip vertices from the audio vector embeddings, and generat a series of face expression codes based on the audio vector embeddings and the obtained lip vertices;obtain guide pose vectors from the audio vector embeddings, and generat a series of joint angle vectors based on the audio vector embeddings and the obtained guide pose vectors; andanimate an emoji based on the generated series of face expression codes and the generated series of joint angle vectors.9.The system of claim 8, wherein the one or more instructions, when executed by the at least one processor individually or collectively, cause the system toencode, by a phoneme encoder of a multimodal embedding module, the obtained input text message into the audio vector embeddings;encode, by a text encoder of the multimodal embedding module, the obtained at least one message in text format into the audio vector embeddings;encode, by an image encoder of the multimodal embedding module, the obtained at least one message in image format into the audio vector embeddings; andencode, by an audio encoder of the multimodal embedding module, the obtained at least one message in audio format into the audio vector embeddings.10.The system of claim 8 or claim 9, wherein the obtained at least one message in the image format includes at least one of: a static image, animated images, or video.11.The system of any one of claims 8 to 10, wherein the one or more instructions, when executed by the at least one processor individually or collectively, cause the system toencode, by the text encoder, the obtained at least one message in text format into context vector embeddings;encode, by the image encoder, the obtained at least one message in image format into the context vector embeddings; andencode, by the phoneme encoder, the input text message into content vector embeddings.12.The system of any one of claims 8 to 11, wherein the one or more instructions, when executed by the at least one processor individually or collectively, cause the system to combine and encode, by a text-to-speech encoder of the multimodal embedding module, the context vector embeddings and the content vector embeddings into the audio vector embeddings.13.The system of any one of claims 8 to 12, wherein the one or more instructions, when executed by the at least one processor individually or collectively, cause the system togenerate, by a lip regressor model of a face motion module, the lip vertices based on the audio vector embeddings.14.The system of any one of claims 8 to 13, wherein the one or more instructions, when executed by the at least one processor individually or collectively, cause the system togenerate, by a guide pose generator model of a body motion module, the guide pose vectors based on the audio vector embeddings and a collection of reference guide pose data.15.The system of any one of claims 8 to 14, wherein the one or more instructions, when executed by the at least one processor individually or collectively, cause the system toadd noise vectors as additional input to the body motion module for generating the series of joint angle vectors in further variations.

Images