Context-aware motion generation device and method using text and emotion embeddings

Abstract

The present invention relates to a device and method for generating context-aware motion using text and emotion embeddings. An electronic device according to one embodiment of the present invention includes a memory and a processor connected to the memory, and the processor receives input information including a text prompt instructing the content of a character's action and emotion information to be assigned to the action for generating three-dimensional motion from a user terminal, and vectorizes the received text prompt and the emotion information, respectively, to generate a text embedding for the content of the action and an emotion embedding that determines the nuance of the action, and generates a motion sequence based on the text embedding using a Generative AI model, but reflects the emotion embedding as a condition of the generation process to assign an emotional context to the character's action, and can generate the motion sequence with the assigned emotional context as final motion data.

Description

Technology Field

[0001] The present invention relates to a device and method for generating context-aware motion using text and emotion embeddings. Background Technology

[0003] Unless otherwise indicated in this specification, the contents described in this section are not prior art for the claims of this application, and are not to be recognized as prior art simply because they are included in this section.

[0004] With the recent rapid growth of the immersive content market, including the Metaverse, Virtual Reality (VR), and Augmented Reality (AR), the role of 3D characters (Avatars) and Digital Humans that act as proxies for or interact with users within virtual spaces is becoming increasingly important. In order for these Digital Humans to provide users with a deep sense of immersion, it is essential to go beyond simply mimicking the human appearance and implement natural and rich motions that are appropriate for the situation and context.

[0005] Conventional 3D character motion generation technology mainly involved professional animators manually setting keyframes or recording actors' performances using motion capture equipment. However, these methods had the disadvantage of requiring high costs and long production times, so recently, 'Text-to-Motion' technology, which automatically generates corresponding motion using artificial intelligence when a text prompt is entered, is being actively researched.

[0006] However, existing text-based motion generation models rely solely on the information within the text itself to generate motion, so there were limitations in quantitatively controlling subtle emotional nuances, context, or nuances of movement that text cannot capture. For example, even if the user inputs "sadly walking" by adding the adjective "sadly" to the text prompt "walking," existing models only express sadness in an implicit and ambiguous form based on the average patterns of learned data, making it difficult to adjust the intensity of sadness or precisely control specific nonverbal expressions (eye movement, head bowing, shoulder drooping, etc.) as intended by the user.

[0007] In addition, within text embeddings, the 'content' and 'style' of actions are intertwined, making it technically difficult to clearly separate or control them independently. As a result, the generated output frequently gives a mechanical and soulless (robotic) impression or displays unnatural emotional expressions that are inappropriate for the situation, making it unsuitable for application in game cinematics, movies, or virtual human services that require a high level of acting ability.

[0008] Therefore, there is an urgent need for a new type of motion generation device and method capable of generating high-quality motion with rich expressiveness by integrally controlling a character's facial expressions, gestures, and postures, while the user instructs actions via text and simultaneously inputs emotional or situational context information as separate explicit conditions. Prior art literature

[0010] Korean Registered Patent No. 10-2872821 (October 14, 2025) The problem to be solved

[0011] One embodiment of the present invention provides a device and method for generating context-aware motion using text and emotion embeddings.

[0012] The technical problems to be solved by the present invention are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0014] To achieve the above-mentioned purpose, an electronic device according to one embodiment of the present invention includes a memory and a processor connected to the memory, and the processor receives input information including a text prompt instructing the action content of a character and emotion information to be assigned to the action for generating three-dimensional motion from a user terminal, and vectorizes the received text prompt and the emotion information, respectively, to generate a text embedding for the action content and an emotion embedding that determines the nuance of the action, and generates a motion sequence based on the text embedding using a Generative AI model, while reflecting the emotion embedding as a condition of the generation process to assign an emotional context to the action of the character, and can generate the motion sequence with the assigned emotional context as final motion data.

[0015] At this time, the emotion information includes at least one of a discrete emotion label indicating one of a plurality of preset emotion categories, or a continuous emotion vector including valence and arousal dimensions, and the processor generates an emotion embedding corresponding to the emotion information and injects it into the diffusion process or attention mechanism of the generative artificial intelligence model, and can control the motion characteristics of the motion sequence using the injected emotion embedding.

[0016] At this time, the processor, upon receiving the input information, further receives a quantitative numerical value representing the intensity of the emotion information, adjusts an emotion weight that determines the influence of the emotion embedding on the generative artificial intelligence model according to the received intensity, and applies the adjusted emotion weight to the diffusion process or the attention mechanism to generate the motion sequence having a different depth of emotion expression for the same type of emotion information.

[0017] At this time, the processor can integrally generate motions for each body part of the character using the emotion embedding, derive a facial expression corresponding to the emotion information by modifying the vertex positions of the character's face mesh, derive a hand gesture corresponding to the emotion information by adjusting the rotation values ​​of the character's finger joints, and derive a center of gravity and gaze according to the emotion information by adjusting the rotation angles of the character's spine and head joints.

[0018] At this time, the generative artificial intelligence model is a Motion Diffusion Model (MDM) that includes a layer for concatenating or cross-attentioning the text embedding and the emotion embedding, and the processor can generate the motion sequence by applying the emotion embedding as a guide signal during a noise removal process using the motion diffusion model, and modifying detailed motion characteristics including tremor, speed, and / or range of motion according to the guide signal while maintaining the basic motion structure determined by the text embedding.

[0019] At this time, the processor may perform emotion matching feedback correction by calculating a Motion Agitation Index based on a Jerk value, which is the amount of acceleration change of each joint within the motion sequence, to verify whether the generated motion sequence physically matches the intention of the input emotion information, calculating the error between the calculated Motion Agitation Index and the numerical value of the Arousal dimension included in the input information, and readjusting the Temporal Scale of the motion sequence in a direction to reduce the error if the error exceeds a preset threshold range.

[0020] At this time, the above motion turbulence index is derived by the following mathematical formula,

[0021]

[0022] I_agitation represents the motion agitation index, T represents the total number of frames of the motion sequence, N represents the total number of joints of the character, t represents the frame time point, j represents the joint index, and P_j(t) may represent the 3D position vector of the j-th joint at time point t.

[0023] At this time, if the emotional information is an emotional category related to the body's spatial occupation, such as 'Confidence' or 'Depression', the processor can calculate the Spatial Occupancy Index of the motion sequence and control the degree of contraction and expansion of the posture.

[0024] At this time, the space occupancy index is derived through the average value of the Euclidean distance from the center of mass of the character to the end-effectors, and the processor can finely adjust the joint angles of the character so that the space occupancy index increases as the emotional information is positive valence, and the space occupancy index decreases as the emotional information is negative valence.

[0025] At this time, the above space occupancy index is derived by the following mathematical formula,

[0026]

[0027] I_spatial represents the spatial occupancy index, T represents the total number of frames of the motion sequence, M represents the total number of end-effectors of the character, t represents the frame time point, k represents the end-effector index, p_k(t) represents the 3D position vector of the k-th end-effector (hand, foot, or head) at time point t, and c(t) represents the 3D position vector of the character's center of mass at time point t. Effects of the invention

[0029] As such, according to one embodiment of the present invention, a context-aware motion generation device and method using text and emotion embeddings can be provided.

[0030] The effects obtainable from the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0032] Other aspects, features, and benefits of specific preferred embodiments of the present invention, as described above, will become more apparent from the following description in conjunction with the accompanying drawings. FIG. 1 is a conceptual diagram of a context-aware motion generation device using text and emotion embeddings according to one embodiment of the present invention. FIG. 2 is a block diagram of an electronic device according to one embodiment of the present invention. FIG. 3 is a diagram showing the derivation of final motion data according to one embodiment of the present invention. FIG. 4 is a diagram illustrating the concept of a motion turbulence index according to one embodiment of the present invention. FIG. 5 is a diagram showing a terminal joint and a center of gravity according to an embodiment of the present invention. FIG. 6 is a flowchart of a method for generating context-aware motion using text and emotion embeddings according to an embodiment of the present invention. It should be noted that in the drawings above, similar reference numbers are used to illustrate identical or similar elements, features, and structures. Specific details for implementing the invention

[0033] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0034] In describing the embodiments, technical details that are well known in the technical field to which the present invention belongs and are not directly related to the present invention are omitted. This is intended to convey the essence of the present invention more clearly without obscuring it by omitting unnecessary explanations.

[0035] For the same reason, some components in the attached drawings have been exaggerated, omitted, or schematically depicted. Additionally, the size of each component does not entirely reflect its actual dimensions. Identical or corresponding components in each drawing have been assigned the same reference numbers.

[0036] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0037] At this point, it will be understood that each block of the process flow diagrams and combinations of the flow diagrams can be executed by computer program instructions. Since these computer program instructions can be loaded into the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create means to perform the functions described in the flow diagram block(s). Since these computer program instructions can also be stored in computer-available or computer-readable memory that can be directed toward the computer or other programmable data processing equipment to implement the function in a specific way, the instructions stored in computer-available or computer-readable memory can also produce a manufactured item containing instruction means to perform the function described in the flow diagram block(s). Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in the flowchart block(s).

[0038] Additionally, each block may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). It should also be noted that in some alternative execution examples, the functions mentioned in the blocks may occur out of order. For instance, two blocks described in succession may actually be executed substantially simultaneously, or the blocks may be executed in reverse order according to their corresponding functions.

[0039] In this embodiment, the term "part" refers to a software or hardware component such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC), and the "part" performs certain roles. However, the meaning of "part" is not limited to software or hardware. The "part" may be configured to reside in an addressable storage medium or configured to run one or more processors. Accordingly, as an example, the "part" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts." In addition, the components and '~parts' may be implemented to play one or more CPUs within the device or secure multimedia card.

[0040] In describing the embodiments of the present invention in detail, the primary focus will be on examples of specific systems, but the main point claimed in this specification is applicable to other communication systems and services having a similar technical background without significantly departing from the scope disclosed in this specification, and this will be possible at the judgment of a person with skilled technical knowledge in the relevant technical field.

[0041] At this time, the user terminal described below may include a communication-capable desktop computer, laptop computer, notebook, smartphone, tablet PC, mobile phone, smart watch, smart glass, e-book reader, PMP (portable multimedia player), portable game console, navigation device, digital camera, DMB (digital multimedia broadcasting) player, digital audio recorder, digital audio player, digital video recorder, digital video player, PDA (Personal Digital Assistant), etc.

[0043] FIG. 1 is a conceptual diagram of a context-aware motion generation device using text and emotion embeddings according to one embodiment of the present invention, and FIG. 2 is a block diagram of an electronic device (100) according to one embodiment of the present invention.

[0044] An electronic device (100) according to one embodiment includes a processor (110) and a memory (120). The processor (110) can perform at least one of the methods described above. The memory (120) can store information related to the method described above or store a program in which the method described above is implemented. The memory (120) may be volatile memory or non-volatile memory. The memory (120) may be referred to as a 'database', 'storage unit', etc.

[0045] The processor (110) can execute a program and control the electronic device (100). The code of the program executed by the processor (110) can be stored in memory (120). The device (100) can be connected to an external device (e.g., a personal computer or a network) through an input / output device (not shown) and exchange data.

[0046] At this time, the processor (110) may receive input information including a text prompt instructing the action of a character and emotion information to be assigned to the action, for generating three-dimensional motion from a user terminal.

[0047] At this time, the emotion information may include at least one of a discrete emotion label indicating any one of a plurality of preset emotion categories, or a continuous emotion vector including valence and arousal dimensions.

[0048] Specifically, the text prompt refers to an action instruction entered by the user in the form of natural language, representing data that determines the basic form of the motion to be generated, the type of action, or the semantic content. For example, specific descriptions of actions such as "walk forward," "jump in place," or "wave hand to greet someone" may fall under this category.

[0049] Additionally, the aforementioned emotion information refers to auxiliary data that determines the mood, style, or psychological state to be assigned to the basic action indicated by the text prompt. Here, the discrete emotion labels are categories of emotions universally perceived by humans designated in the form of text or an index, and may include emotion classifications that are clearly distinguished, such as 'joy', 'sadness', 'anger', 'fear', 'neutrality', etc.

[0050] On the other hand, the aforementioned continuous emotion vector may be data that quantifies and expresses emotions as coordinates in a multidimensional space. For example, the Valence dimension refers to a scale indicating whether the emotion is positive (e.g., happiness, satisfaction) or negative (e.g., depression, dissatisfaction), and the Arousal dimension refers to a scale indicating whether the energy level or physical activity of the emotion is high (e.g., excitement, tension) or low (e.g., relaxation, lethargy). Through this, users can quantitatively distinguish and input subtle emotional nuances, such as "energetic and passionate joy (High Arousal & Positive Valence)" and "comfortable and calm joy (Low Arousal & Positive Valence)," going beyond the simple word "joy."

[0052] Additionally, referring to FIG. 3, the processor vectorizes the received text prompt and the emotion information, respectively, to generate text embeddings for the action content and emotion embeddings that determine the nuance of the action, and uses a Generative AI model to generate a motion sequence based on the text embeddings, while reflecting the emotion embeddings as a condition of the generation process to provide an emotional context to the action of the character, and can generate the motion sequence with the provided emotional context as final motion data.

[0053] Looking more closely, the processor can generate an emotion embedding corresponding to the emotion information and inject it into the diffusion process or attention mechanism of the generative artificial intelligence model, and control the motion characteristics of the motion sequence using the injected emotion embedding.

[0054] Specifically, the text embedding is a high-dimensional vector converted through a natural language processing model (e.g., CLIP, BERT, etc.) and serves as reference information for determining the 'skeleton structure' or 'semantic action' of motion. On the other hand, the emotion embedding is a vector that determines the 'style' or 'atmosphere' of motion and functions as nuance information applied over the skeleton formed by the text embedding.

[0055] In this case, the fact that the processor injects the emotion embedding into the diffusion process means that the emotion vector is included in the operation as a condition at every denoising step that recovers meaningful behavioral data from random noise.

[0056] Furthermore, when utilizing an attention mechanism, particularly cross-attention, the generative model refers not only to the text embeddings but also to the emotion embeddings when determining the joint positions of each frame, thereby dynamically determining the weights to focus on reflecting emotion in specific parts of the motion.

[0057] Accordingly, the motion characteristics controlled in this way may encompass physical properties of the motion, such as speed, stride, range of motion of the joints, or jitter of the body.

[0058] For example, if a user inputs the text "walking" and the emotion information "sadness," the processor can generate final motion data that maintains the walking act itself (text embedding) but modifies the stride length, increases the angle of upper body bending, and slows down the speed of the movement due to the influence of the emotion embedding.

[0059] Conversely, if emotional information such as 'Joy' is input, even the same 'walking' motion can be controlled in a way that involves swinging the arms widely (increasing range of motion) or jumping rhythmically (increasing speed and elasticity).

[0061] Additionally, upon receiving the input information, the processor further receives a quantitative numerical value representing the intensity of the emotion information, adjusts an emotion weight that determines the influence of the emotion embedding on the generative artificial intelligence model according to the received intensity, and applies the adjusted emotion weight to the diffusion process or the attention mechanism to generate the motion sequence having a different depth of emotion expression for the same type of emotion information.

[0062] Specifically, the intensity refers to a parameter that quantifies the strength of the emotional expression intended by the user (e.g., a real value between 0.0 and 1.0).

[0063] At this time, the processor can increase the Emotion Weight as the intensity value increases, thereby strengthening the influence of emotional information relative to text information during the computation process of the generative artificial intelligence model.

[0064] For example, in the case of a diffusion model, by reflecting the aforementioned emotion weights in the 'conditional guidance scale' applied at each step of generating an image by removing noise, the result can be guided to conform more strongly to the emotion conditions.

[0065] Accordingly, the depth of emotional expression regulated can refer to the degree of exaggeration of actions or the specificity of behaviors within the same emotional category.

[0066] For example, even if the same emotional information such as 'Sadness' is input, if the intensity is set low (e.g., 0.2), the processor can generate static movements (passive expressions) such as slightly lowering the character's head or slowing down the walking speed.

[0067] On the other hand, when the intensity is set high (e.g., 0.9), the processor can generate dynamic and intense movements (active expressions), such as the character covering their face or staggering and falling to the floor. This allows the user to go beyond simple emotional classification and precisely control the acting tone appropriate for the situation.

[0069] In addition, the processor can integrally generate motions for each body part of the character using the emotion embedding, derive a facial expression corresponding to the emotion information by modifying the vertex positions of the character's face mesh, derive a hand gesture corresponding to the emotion information by adjusting the rotation values ​​of the character's finger joints, and derive a center of gravity and gaze according to the emotion information by adjusting the rotation angles of the character's spine and head joints.

[0070] Specifically, this configuration is based on the observation that human emotional expression is not merely limited to facial expressions but manifests through the complex interaction of overall bodily movements, such as hand gestures, posture, and gaze. The processor utilizes the emotion embedding as a global control signal, but converts it into local control parameters tailored to the characteristics of each body part and applies them.

[0071] First, the deformation of the vertex positions of the face mesh directly controls the geometric structure of the 3D model. For example, when the emotion of 'joy' is input, the vertices around the corners of the mouth are moved upward and the vertices around the eyes are contracted to create a smile, or when the emotion of 'anger' is input, the vertices between the eyebrows are gathered around the center and the vertices of the eyebrows are moved downward to produce a frowning expression. This enables three-dimensional changes in facial expression that are difficult to achieve with texture mapping alone.

[0072] In addition, for hand gestures, the processor finely controls the multi-joint rotation values ​​of the fingers to complete the details of the emotion. For example, if the emotion of 'anxiety' is input, the processor can generate repetitive movements (fidgeting) of irregularly bending and straightening the finger joints, or in the case of 'anger,' it can generate a clenched fist motion by strongly rotating all finger joints inward.

[0073] Furthermore, the control of the spine and head joints determines the character's overall silhouette and non-verbal communication. For positive emotions such as 'confidence' or 'happiness,' the processor controls the center of gravity by straightening the spine joints and lifting the head joints so that the gaze is directed forward or upward. Conversely, for negative emotions such as 'depression' or 'frustration,' it realistically implements a withdrawn body posture according to the emotional state by rounding the spine and dropping the head so that the gaze is directed toward the floor. Since this control of specific body parts is not performed individually but is executed in temporal synchronization by the emotion embeddings, a unified emotional performance is possible as a result.

[0075] Additionally, the generative artificial intelligence model may be a Motion Diffusion Model (MDM) that includes a layer for concatenating or cross-attentioning the text embeddings and the emotion embeddings.

[0076] Additionally, the processor can generate the motion sequence by applying the emotion embedding as a guide signal during the noise removal process using the motion diffusion model, and modifying detailed motion characteristics including tremor, speed, and / or range of motion according to the guide signal while maintaining the basic motion structure determined by the text embedding.

[0077] Specifically, the motion diffusion model can be designed to process by fusing a text embedding that serves as a criterion for motion generation and an emotion embedding that determines the style of the motion.

[0078] Here, concatenation refers to a method of physically joining two vectors to form a single long input vector that is injected into the model; this has the effect of simplifying the structure while allowing both pieces of information to be considered simultaneously.

[0079] On the other hand, the Cross-Attention method is a dynamic reference method in which the model assigns greater weight to emotional information that is highly associated with the action frame currently being generated. Through this, beyond simply mixing information, it becomes possible to achieve sophisticated control, such as maximizing the influence of emotion in specific segments of an action (e.g., the moment a hand is reached).

[0080] In addition, the above guide signal can serve as a vector field that provides directionality during the reverse diffusion process of restoring motion from noise, ensuring that the generated result does not deviate from emotional conditions.

[0081] At this time, the processor can first establish a basic motion structure, such as 'walking' or 'running', that is, the movement trajectory of the entire skeleton and the sequence order of the joints, through text embedding. Then, detailed motion characteristics are added to the trajectory by applying the guide signal.

[0082] For example, if the emotion of 'fear' is input as a guide signal, the processor may implement tremor by adding fine high-frequency vibrations to the trajectory of the walking motion, or rapidly change the speed of the motion by adjusting the interval between frames. Alternatively, in the case of the emotion of 'withdrawal', passive movement is generated by reducing the range of motion by limiting the maximum angle at which the joints extend. That is, the present invention can generate high-quality motion that is suitable for the purpose and has emotional depth by independently modifying only the 'style' without compromising the 'content' of the motion.

[0084] FIG. 4 is a diagram illustrating the concept of a motion turbulence index according to one embodiment of the present invention.

[0085] Referring to FIG. 4, the processor can perform emotion matching feedback correction by calculating a Motion Agitation Index based on a Jerk value, which is the amount of acceleration change of each joint in the motion sequence, to verify whether the generated motion sequence physically matches the intention of the input emotion information, calculating the error between the calculated Motion Agitation Index and the numerical value of the Arousal dimension included in the input information, and readjusting the Temporal Scale of the motion sequence in a direction to reduce the error if the error exceeds a preset threshold range.

[0086] To examine it more specifically, the above motion turbulence index can be derived by the following mathematical formula 1.

[0087] [Mathematical Formula 1]

[0088]

[0089] At this time, I_agitation represents the motion agitation index, T represents the total number of frames of the motion sequence, N represents the total number of joints of the character, t represents the frame time point, j represents the joint index, and P_j(t) may represent the 3D position vector of the j-th joint at time point t.

[0090] Specifically, the above configuration is intended to quantitatively verify whether the generated result physically and accurately reflects the emotional energy level intended by the user, going beyond mere visual similarity.

[0091] Generally, velocity or acceleration represents the speed or force of an action, but jerk, which is the third derivative of the rate of change of a position vector over time, can refer to a physical quantity representing a sudden change in acceleration—that is, the jolting or abrupt shaking of an action.

[0092] Therefore, this value can be used as a key indicator to distinguish between smooth movements in calm or depressed emotional states and rough and unstable movements in angry or surprised emotional states.

[0093] In addition, if we look at the design structure of the above mathematical formula 1, the formula is designed to encompass the movement characteristics of the entire character, rather than being limited to specific joints or specific points in time.

[0094] First, a jerk vector is obtained by performing a third derivative operation on the position vector of each joint, and the magnitude (Norm) of the vector is calculated to extract only the intensity of the pure change regardless of the direction of movement.

[0095] Subsequently, for every time frame in which all joints and motions distributed throughout the body are played back, the corresponding values ​​are all summed, and the result is divided by the product of the total number of joints and the total number of frames to undergo a normalization process.

[0096] This is intended to prevent noise at specific moments or excessive movement in specific parts from distorting the overall index, and to derive the average turbulence throughout the entire motion sequence as a single scalar value.

[0097] As a result, the processor can determine whether the emotional expression is insufficient or excessive by comparing the physical index derived in this way with the arousal value entered by the user.

[0098] If the calculated agitation index is significantly lower than the target arousal level, the processor can reduce the temporal scale of the motion to correct the motion more quickly and sharply, and conversely, if the index is too high, it can perform feedback by expanding the temporal scale to calm the motion.

[0099] Through this verification and correction loop, the present invention has the advantage of automatically optimizing a user's abstract emotional intention into a high-quality animation that conforms to specific physical laws.

[0101] FIG. 5 is a diagram showing a terminal joint and a center of gravity according to an embodiment of the present invention.

[0102] Referring to FIG. 5, the processor can control the degree of contraction and expansion of the posture by calculating the spatial occupancy index of the motion sequence when the emotional information is an emotional category related to the body's spatial occupancy, such as 'Confidence' or 'Depression'.

[0103] At this time, the space occupancy index can be derived through the average value of the Euclidean distances from the center of mass of the character to the end-effectors.

[0104] In addition, the processor can finely adjust the joint angles of the character so that the space occupancy index increases as the emotional information is positive (Positive Valence) and decreases as the emotional information is negative (Negative Valence).

[0105] Looking at it in more detail, the above space occupancy index can be derived by the following mathematical formula 2.

[0106] [Mathematical Formula 2]

[0107]

[0108] At this time, I_spatial represents the spatial occupancy index, T represents the total number of frames of the motion sequence, M represents the total number of end-effectors of the character, t represents the frame time point, k represents the end-effector index, p_k(t) represents the 3D position vector of the k-th end-effector (hand, foot, or head) at time point t, and c(t) represents the 3D position vector of the center of mass of the character at time point t.

[0109] Specifically, the above configuration is intended to convert the psychologically well-known correlation between the body's openness and closeness and emotions into quantitative values ​​and apply them to motion generation.

[0110] Generally, when humans are in a confident or happy positive emotional state, they tend to expand the space occupied by their bodies, such as spreading their limbs wide and opening their chest, whereas when they are in a depressed or fearful negative emotional state, they tend to contract the space occupied, such as curling up their bodies and bringing their limbs toward their torso.

[0111] Therefore, the aforementioned space occupancy index is utilized as a key metric to analyze the geometric characteristics of this posture, determine, and control whether the generated motion visually and correctly represents the character's inner psychology.

[0112] At this time, if we look at the design structure of the above mathematical formula 2, the formula is designed to comprehensively measure how far each extremity extends from the center of the character's body.

[0113] First, for every frame, the Euclidean distance between the coordinates of the character's center of gravity (e.g., pelvis or navel) and the coordinates of extremity joints such as the hands, feet, and head is calculated. By using the center of gravity as the reference point, only the pure degree of pose spread can be extracted, regardless of the character's position in world coordinates.

[0114] Subsequently, the distances calculated for all terminal joints are summed and averaged by dividing by the total number of frames and joints, thereby deriving a single indicator of the average posture scale across the entire motion sequence rather than the temporary movement at a specific moment.

[0115] Consequently, the processor sets a target space occupancy index based on emotional information (positive / negative), and can finely correct the joint angle if the actual index of the generated motion falls short of or exceeds this index.

[0116] For example, if the spatial occupancy index is low when 'confidence' is input, the joints are rotated to straighten the shoulders and spread the arms; if the index is high when 'depression' is input, the correction is made to lower the head and bend the shoulders inward.

[0117] Through this, there is an advantage in that it can automatically optimize subtle nuances of posture that are difficult to specify with text prompts alone, thereby maximizing the authenticity and immersion of the character's performance.

[0119] FIG. 6 is a flowchart of a method for generating context-aware motion using text and emotion embeddings according to an embodiment of the present invention.

[0120] Referring to FIG. 6, a context-aware motion generation method using text and emotion embeddings according to one embodiment of the present invention may receive input information including a text prompt instructing the action of a character and emotion information to be assigned to the action from a user terminal for generating three-dimensional motion (S101).

[0121] In addition, a context-aware motion generation method using text and emotion embeddings according to one embodiment of the present invention can vectorize the received text prompt and the emotion information, respectively, to generate a text embedding for the action content and an emotion embedding that determines the nuance of the action (S103).

[0122] In addition, the context-aware motion generation method using text and emotion embeddings according to one embodiment of the present invention can generate a motion sequence based on the text embeddings by using a generative AI model (S105).

[0123] In addition, the context-aware motion generation method using text and emotion embeddings according to one embodiment of the present invention can provide an emotional context to the character's movement by reflecting the emotion embeddings as a condition of the generation process (S107).

[0124] In addition, the context-aware motion generation method using text and emotion embeddings according to one embodiment of the present invention can generate the motion sequence to which the emotional context is applied as final motion data (S109).

[0125] In addition, the context-aware motion generation method using text and emotion embeddings according to one embodiment of the present invention can be configured in the same way as the context-aware motion generation device using text and emotion embeddings disclosed in FIGS. 1 to 5.

[0127] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0128] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0129] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0130] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0131] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 In an electronic device, memory; and a processor connected to said memory; The processor comprises: receiving input information including a text prompt indicating the content of a character's action and emotion information to be assigned to the action for generating 3D motion from a user terminal; vectorizing the received text prompt and the emotion information, respectively, to generate a text embedding for the content of the action and an emotion embedding determining the nuance of the action; generating a motion sequence based on the text embedding using a Generative AI model, wherein the emotion embedding is reflected as a condition of the generation process to assign an emotional context to the character's action; and generating the motion sequence with the assigned emotional context as final motion data; wherein the emotion information includes at least one of a discrete emotion label indicating any one of a plurality of preset emotion categories, or a continuous emotion vector including valence and arousal dimensions; and wherein the processor [represents] an emotion corresponding to the emotion information An embedding is generated and injected into the diffusion process or attention mechanism of the generative artificial intelligence model, and the motion characteristics of the motion sequence are controlled using the injected emotion embedding; the processor, upon receiving the input information, further receives a quantitative numerical value representing the intensity of the emotion information, and adjusts an emotion weight that determines the influence of the emotion embedding on the generative artificial intelligence model according to the received intensity.By applying the adjusted emotion weights to the above diffusion process or the above attention mechanism, the above motion sequences are generated in which the depth of emotion expression differs for the same type of the above emotion information, and the processor integrally generates motions for each body part of the character using the emotion embedding, deriving a facial expression corresponding to the emotion information by modifying the vertex positions of the character's face mesh, deriving a hand gesture corresponding to the emotion information by adjusting the rotation values ​​of the character's finger joints, and deriving a center of gravity and gaze according to the emotion information by adjusting the rotation angles of the character's spine and head joints, and the generative AI model is a Motion Diffusion Model (MDM) that includes a layer for concatenating or cross-attentioning the text embedding and the emotion embedding, and the processor, in a noise removal process using the above Motion Diffusion Model, uses the emotion embedding as a guide signal The processor generates the motion sequence by applying a signal and modifying detailed motion characteristics, including tremor, speed, and / or range of motion, according to the guide signal while maintaining the basic motion structure determined by the text embedding, and the processor calculates a motion agitation index based on a jerk value, which is the amount of acceleration change of each joint within the motion sequence, in order to verify whether the generated motion sequence physically matches the intention of the input emotion information, and calculates the error between the calculated motion agitation index and the numerical value of the arousal dimension included in the input information.If the above error exceeds a preset threshold range, an emotion-matching feedback correction is performed to readjust the temporal scale of the motion sequence in a direction that reduces the above error, and the motion turbulence index is derived by the following mathematical formula, I_agitation represents the motion agitation index, T represents the total number of frames of the motion sequence, N represents the total number of joints of the character, t represents the frame time point, j represents the joint index, and P_j(t) represents the 3D position vector of the j-th joint at time point t; the processor, when the emotion information is an emotion category related to the spatial occupation of the body, such as 'Confidence' or 'Depression', calculates the Spatial Occupancy Index of the motion sequence to control the degree of contraction and expansion of the posture, the Spatial Occupancy Index is derived through the average value of the Euclidean distance from the character's Center of Mass to the End-effectors, and the processor controls the Spatial Occupancy Index so that it increases as the emotion information is positive (Positive Valence), and as it is negative (Negative Valence), the Spatial Occupancy Index The joint angles of the above character are finely adjusted to reduce them, and the space occupancy index is derived by the following mathematical formula, An electronic device characterized in that I_spatial represents the spatial occupancy index, T represents the total number of frames of the motion sequence, M represents the total number of end-effectors of the character, t represents the frame time point, k represents the end-effector index, p_k(t) represents the 3D position vector of the k-th end-effector (hand, foot, or head) at time point t, and c(t) represents the 3D position vector of the center of mass of the character at time point t. Claim 2 delete Claim 3 delete

Images