Artificial intelligence-based character motion generation method reflecting style of individual

The AI-based character motion generation system addresses the lack of individuality in character motion by using user behavior data and GANs to synthesize style-reflected motion, offering a cost-effective and efficient method for creating personalized character behavior in games and CG applications.

WO2024248274A1PCT designated stage expired Publication Date: 2024-12-05KOREA ELECTRONICS TECH INST
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/002518
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-30
Filing Date
2024-02-27
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing character motion in games and metaverse environments lacks individuality, as it is standardized, and current CG methods require time-consuming and costly motion capture equipment to reflect an actor's motion accurately.

Method used

An AI-based method and system that generates character motion by acquiring user behavior data, extracting behavioral style features, and using Generative Adversarial Networks (GAN) to synthesize motion features, allowing for the reflection of personal style in character behavior and appearance without the need for expensive motion capture equipment.

Benefits of technology

Enables the creation of character motion that accurately reflects individual characteristics in both appearance and behavior, reducing the time and cost associated with traditional motion capture methods while providing a high-speed, AI-driven solution for CG applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024002518_05122024_PF_FP_ABST
    Figure KR2024002518_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Provided is an artificial intelligence-based character motion generation method reflecting a style of an individual. The character motion generation method according to an embodiment of the present invention extracts a behavior style feature from behavior data of a user, extracts a motion feature from a motion to be performed by a character, generates a style feature from the extracted behavior style feature, and generates a motion reflecting a style from the extracted motion feature and the style feature. Accordingly, by generating a character motion on the basis of artificial intelligence by reflecting the style of an individual, characteristics of the individual can be reflected not only in the appearance of the character but also in the behavior of the character.
Need to check novelty before this filing date? Find Prior Art

Description

How to create AI-based character motions that reflect individual style

[0001] The present invention relates to an image processing technology utilizing artificial intelligence, and more specifically, to a method and system for generating character motion in a virtual space based on artificial intelligence.

[0002] Avatars appearing in existing games and metaverse environments can change their appearance and clothing according to the user's preference, but the character's motion appears in a stereotypical form.

[0003] In other words, characters appearing in existing games only move according to their predefined characteristics, completely disregarding individual traits. However, such characters cannot meet individual needs. Therefore, a method for automatically generating motions that reflect individual characteristics is needed.

[0004] Additionally, in order to create character motions in movies using existing CG (Computer Graphics), motion capture equipment must be used to capture the motions of actors, which has the problem of consuming a lot of time and money.

[0005] The present invention has been devised to solve the above problems, and the purpose of the present invention is to provide a method and system for generating character motion based on artificial intelligence that reflects an individual's style, as a means to reflect the individual's characteristics not only in the appearance of the character but also in the behavior.

[0006] A method for generating a character motion according to one embodiment of the present invention for achieving the above object includes the steps of: acquiring user behavior data; extracting behavior style features from the acquired behavior data; receiving a motion to be performed by the character; extracting motion features from the input motion; generating style features from the extracted behavior style features; and generating a motion reflecting a style from the extracted motion features and style features.

[0007] The user's behavior data may be joint information calculated from a sensor or joint information extracted from an image of the user.

[0008] The style feature generation step may modulate the size of the action style feature depending on the size of the motion feature.

[0009] The character motion generation method according to the present invention further includes a step of acquiring user non-motion data; a step of extracting non-motion style features from the acquired non-motion data; and the style feature generation step may be to modulate and synthesize the sizes of the motion style features and the non-motion style features according to the size of the motion features.

[0010] The non-behavioral data may be a user's facial image, and the non-behavioral style features may be features of at least one of the user's age, gender, race, emotion, personality, sensitivity, intention, intelligence, action ability, sensing ability, and reaction ability.

[0011] The method for generating character motion according to the present invention further includes a step of receiving a user's style element; and the style feature generation step may be to modulate and synthesize the sizes of action style features, non-action style features, and style elements according to the size of the motion feature.

[0012] The style element may be an input value for at least one of the user's age, gender, race, emotion, personality, sensibility, intention, intelligence, action ability, sensing ability, and reaction ability.

[0013] A character motion generation system that generates motions reflecting styles may be trained to evolve competitively with a discriminator for determining the authenticity of generated motions reflecting styles, according to a Generative Adversarial Networks (GAN) algorithm.

[0014] A character motion generation system that generates motions that reflect styles may be trained in a direction in which the loss between the behavioral style extracted from behavioral data and the behavioral style extracted from the motions that reflect styles is reduced, and the loss between the motion features extracted from the motions that the character will perform and the motion features extracted from the motions that reflect styles is reduced.

[0015] According to another aspect of the present invention, a character motion generation system is provided, comprising: a first acquisition module for acquiring user behavior data; a first extraction module for extracting behavior style features from the acquired behavior data; an input module for receiving a motion to be performed by a character; a second extraction module for extracting motion features from the input motion; a first generation module for generating style features from the extracted behavior style features; and a second generation module for generating motions reflecting a style from the extracted motion features and style features.

[0016] According to another aspect of the present invention, a method for generating a character motion is provided, comprising: a step of extracting a behavior style feature from a user's behavior data; a step of extracting a motion feature from a motion to be performed by a character; a step of generating a style feature from the extracted behavior style feature; and a step of generating a motion reflecting a style from the extracted motion feature and style feature.

[0017] According to another aspect of the present invention, a character motion generation system is provided, comprising: a first extraction module for extracting behavioral style features from user behavior data; a second extraction module for extracting motion features from motions to be performed by a character; a first generation module for generating style features from the extracted behavioral style features; and a second generation module for generating motions reflecting a style from the extracted motion features and style features.

[0018] As described above, according to embodiments of the present invention, by generating character motions based on artificial intelligence that reflect an individual's style, it is possible to reflect the individual's characteristics not only in the character's appearance but also in its behavior.

[0019] In addition, according to embodiments of the present invention, when creating character motions in movies or games using CG, it is possible to reflect the motions of actors to characters at high speed without expensive motion capture equipment.

[0020] FIG. 1 is a diagram illustrating the configuration of a character motion generation system according to one embodiment of the present invention;

[0021] Figure 2 is a diagram showing the structure of the behavior style extraction module.

[0022] Figure 3 is a diagram showing the structure of an input motion encoder.

[0023] Figure 4 is a drawing showing the structure of a style reflection motion generation module.

[0024] FIG. 5 is a drawing illustrating a style reflection motion result by a character motion generation system according to an embodiment of the present invention, and

[0025] FIG. 6 is a diagram illustrating the configuration of a character motion generation system according to another embodiment of the present invention.

[0026] Hereinafter, the present invention will be described in more detail with reference to the drawings.

[0027] An embodiment of the present invention proposes a method and system for generating AI-based character motions that reflect an individual's style. This technology reflects an individual's style in AI-based character motion generation, ensuring that not only the character's appearance but also its behavior reflects the individual's characteristics.

[0028] FIG. 1 is a diagram illustrating the configuration of a character motion generation system according to one embodiment of the present invention. As illustrated, the character motion generation system according to the embodiment of the present invention is configured to include a behavior style extraction module (110), a style feature modulation module (120), an input motion encoder (130), and a style reflection motion generation module (140).

[0029] The behavioral style extraction module (110) is a machine learning model that receives user behavioral data, extracts behavioral style features from the input behavioral data, and outputs them. Behavioral style refers not to the type of behavior (walking, running, jumping, throwing, etc.), but rather to the attitudes, emotions, etc. that appear in that type of behavior. The user's behavioral data can be acquired through joint information calculated from sensors, joint information extracted from a user's video, etc.

[0030] Figure 2 presents the structure of a behavioral style extraction module (110). However, the structure of Figure 2 is merely exemplary, and it is not excluded that the behavioral style extraction module (110) may be implemented using a machine learning model with a structure other than the presented structure.

[0031] The input motion encoder (130) is a machine learning model that receives motions to be performed by a character, extracts motion features from the input motions, and outputs them. Motion refers to a type of action, such as walking, running, jumping, or throwing.

[0032] Figure 3 presents the structure of an input motion encoder (130). However, the structure of Figure 3 is merely exemplary, and it is not excluded that the input motion encoder (130) may be implemented using a machine learning model of a structure other than the presented structure.

[0033] The style feature modulation module (120) modulates and outputs the behavior style extracted from the behavior style extraction module (110) according to the size of the motion feature extracted from the input motion encoder (130).

[0034] The style feature modulation module (120) can be implemented as a machine learning module implemented with a MLP (Multi-Layer Perceptron), but implementation through a machine learning model of other structures is not excluded.

[0035] The style-reflecting motion generation module (140) is a machine learning model that receives motion features extracted from an input motion encoder (130) and behavioral style features modulated by a style feature modulation module (120), and generates and outputs motions with style reflected from the input motion features and behavioral style features. At this time, the synthesis of motion features and behavioral style features can be performed using AdaIN (Adaptive Instance Normalization).

[0036] Figure 4 presents the structure of a style-reflecting motion generation module (140). However, the structure of Figure 4 is merely exemplary, and it is not excluded that the style-reflecting motion generation module (140) may be implemented using a machine learning model of a structure other than the presented structure.

[0037] The result of a style-reflecting motion by the character motion generation system presented in Fig. 1 is illustrated in Fig. 5. As shown, the motion to be performed by the character was 'Punch' and the style was 'Proud'. However, since the user's behavior style is 'Old', it can be confirmed that the character motion generation system generated the motion 'Punch' with the behavior style of 'Old'.

[0038] The action style extraction module (110), style feature modulation module (120), input motion encoder (130), and style reflection motion generation module (140) constituting the character motion generation system according to an embodiment of the present invention are capable of unsupervised learning and specifically use the following loss function.

[0039] First, the GAN (Generative Adversarial Networks) algorithm is used. Specifically, a discriminator is used to determine the authenticity of the style-reflecting motion generated by the style-reflecting motion generation module (140), and the model is trained to evolve competitively with the character motion generation system.

[0040] Second, the components (110 to 140) of the character motion generation system are trained in a direction that reduces the loss between the style extracted from the behavior style extraction module (110) and the style of the motion generated by the style reflection motion generation module (140).

[0041] Third, the components (110 to 140) of the character motion generation system are trained in a direction that reduces the loss between the motion features extracted from the input motion encoder (130) and the motion features generated by the style-reflecting motion generation module (140).

[0042] FIG. 6 is a diagram illustrating the configuration of a character motion generation system according to another embodiment of the present invention. As illustrated, the character motion generation system according to the embodiment of the present invention is configured to include a behavior style extraction module (110), a style feature modulation module (120), an input motion encoder (130), a style reflection motion generation module (140), a non-behavioral style extraction module (150), and a style element input module (160).

[0043] The behavior style extraction module (110) is a machine learning model that receives user behavior data, extracts behavior style features from the input behavior data, and outputs them. It can be implemented in the same way as the behavior style extraction module (110) of FIG. 1 described above.

[0044] The input motion encoder (130) is a machine learning model that receives a motion to be performed by a character, extracts motion features from the input motion, and outputs them. It can be implemented in the same way as the input motion encoder (130) of FIG. 1 described above.

[0045] The non-action style extraction module (150) is a machine learning model that receives the user's non-action data, extracts non-action style features from the input non-action data, and outputs them.

[0046] Non-behavioral style refers to information that expresses a user's characteristics beyond their actions, such as age, gender, race, emotions, personality, sensitivity, intentions, intelligence, behavioral abilities, sensing abilities, and reaction abilities. Non-behavioral data can be obtained and utilized by capturing and utilizing facial images (images, videos), etc.

[0047] The style element input module (160) is configured to directly input the user's style elements as values ​​from the user. The style elements refer to age, gender, race, emotions, etc., which are the same as the aforementioned non-action style.

[0048] The style feature modulation module (120) generates a user's style feature from the behavioral style extracted from the behavioral style extraction module (110), the non-behavioral style extracted by the non-behavioral style extraction module (150), and the style element input through the style element input module (160).

[0049] In this process, the style feature modulation module (120) modulates and synthesizes the action style, non-action style, and style element according to the size of the motion feature extracted from the input motion encoder (130).

[0050] The style-reflecting motion generation module (140) is a machine learning model that receives motion features extracted from an input motion encoder (130) and the user's style features generated by a style feature modulation module (120), and generates and outputs a motion with the style reflected from the input motion features and style features. At this time, the synthesis of the motion features and style features can be performed using AdaIN.

[0051] So far, a preferred embodiment of a method and system for generating artificial intelligence-based character motion that reflects the user's style has been described in detail.

[0052] In the embodiment of the present invention, character motion is generated based on artificial intelligence to reflect the user's style, so that not only the character's appearance but also its behavior can reflect the individual's characteristics.

[0053] In addition, in an embodiment of the present invention, when creating character motions in a movie or game using CG, it is possible to reflect the motions of an actor to a character at high speed without expensive motion capture equipment.

[0054] Meanwhile, it goes without saying that the technical idea of ​​the present invention can also be applied to a computer-readable recording medium containing a computer program that performs the functions of the device and method according to the present embodiment. In addition, the technical idea according to various embodiments of the present invention can be implemented in the form of computer-readable code recorded on a computer-readable recording medium. The computer-readable recording medium can be any data storage device that can be read by a computer and store data. For example, the computer-readable recording medium can be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, etc. In addition, the computer-readable code or program stored on the computer-readable recording medium can be transmitted through a network connected between computers.

[0055] In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications can be made by a person having ordinary skill in the art to which the present invention pertains without departing from the gist of the present invention as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present invention.

Claims

1. Step of acquiring user behavior data; A step of extracting behavioral style features from acquired behavioral data; A step for receiving input for the motion to be performed by the character; A step of extracting motion features from input motion; A step of generating style features from the extracted behavioral style features; A method for generating character motion, comprising: a step of generating motion reflecting a style from extracted motion features and style features.

2. In claim 1, User behavior data, A method for generating character motion, characterized in that the joint information is calculated from a sensor or joint information is extracted from a user's image.

3. In claim 1, The style feature generation step is: A method for generating character motion, characterized in that the size of a behavior style feature is modulated according to the size of a motion feature.

4. In claim 1, Step of acquiring user's non-movement data; further comprising a step of extracting non-movement style features from the acquired non-movement data; The style feature generation step is: A character motion generation method characterized by modulating and synthesizing the sizes of action style features and non-action style features according to the size of motion features.

5. In claim 4, Non-action data, It is a video of the user's face, The characteristics of the non-action style are: A method for generating character motion characterized by at least one characteristic of a user's age, gender, race, emotion, personality, sensitivity, intention, intelligence, action ability, sensing ability, and reaction ability.

6. In claim 4, further comprising a step of receiving a user's style element; The style feature generation step is: A method for generating character motion, characterized in that the sizes of action style features, non-action style features, and style elements are modulated and synthesized according to the sizes of motion features.

7. In claim 6, The style elements are, A method for generating character motion, characterized in that the input value is at least one of the user's age, gender, race, emotion, personality, sensitivity, intention, intelligence, action ability, sensing ability, and reaction ability.

8. In claim 1, A character motion generation system that creates motions that reflect the style, A method for generating character motion characterized in that it learns to evolve competitively with a discriminator for determining the authenticity of motions reflecting a generated style according to a GAN (Generative Adversarial Networks) algorithm.

9. In claim 1, A character motion generation system that creates motions that reflect the style, A method for generating character motion, characterized in that learning is performed in a direction in which the loss between the behavior style extracted from behavior data and the behavior style extracted from the motion reflecting the style is reduced, and the loss between the motion features extracted from the motion to be performed by the character and the motion features extracted from the motion reflecting the style is reduced.

10. A first acquisition module that acquires user behavior data; A first extraction module for extracting behavioral style features from acquired behavioral data; An input module that receives the motion to be performed by the character; A second extraction module that extracts motion features from the input motion; A first generation module for generating style features from the extracted behavioral style features; A character motion generation system comprising: a second generation module for generating motion reflecting a style from extracted motion features and style features; 11. Step of extracting behavioral style features from user behavior data; A step of extracting motion features from motions performed by a character; A step of generating style features from the extracted behavioral style features; A method for generating character motion, comprising: a step of generating motion reflecting a style from extracted motion features and style features.

12. A first extraction module that extracts behavioral style features from the user's behavioral data; A second extraction module that extracts motion features from motions performed by a character; A first generation module for generating style features from the extracted behavioral style features; A character motion generation system comprising: a second generation module for generating motion reflecting a style from extracted motion features and style features;

Citation Information

Patent Citations

  • System for making of new character reflect act of user in real space to character of cyber space

    KR1020170099635A

  • Hole location updating device and operating method of hole location updating device

    KR102279165B1

  • Metaverse character making system and method for image analysis data

    KR102493097B1

  • Metaverse character making system for using user tracking and image analysis

    KR102495136B1