System and method for generating motion of a 3D character using user characteristic metadata based on artificial intelligence

Abstract

The embodiments present a system and method for generating motion of a 3D character using user characteristic metadata based on artificial intelligence. A method according to one embodiment may include the steps of: obtaining a text prompt for generating motion of a 3D character and user characteristic metadata from a user terminal; the user characteristic metadata includes a gender value, an age value, and a region code value; generating a text condition vector through a text encoder based on the text prompt; generating a user characteristic context vector including a value related to gender characteristics, a value related to age characteristics, and a value related to region characteristics through a user characteristic encoder based on the user characteristic metadata; generating a biomechanical context vector including an allowable rotation range per joint, a joint type, and a bone length through a biomechanical encoder based on skeleton structure information; generating the skeleton structure information based on the text condition vector; inputting the text condition vector, the user characteristic context vector, and the biomechanical context vector into a diffusion-based motion generation model to generate a rotation sequence per joint; and applying the rotation sequence per joint to the skeleton structure information to generate 3D motion data and transmitting it to the user terminal.

Description

Technology Field

[0001] The embodiments of the present disclosure relate to a technology for generating motion of a three-dimensional character, and to a system and method for generating motion of a three-dimensional character using user characteristic metadata based on artificial intelligence. Background Technology

[0002] Motion generation technology for 3D characters plays a key role in various application fields such as games, animation, the metaverse, real-time streaming, digital human production, and humanoid robot control. With the recent increase in user-participatory content, the demand for natural and immersive motion generation that reflects the individual characteristics of users is rapidly expanding. However, existing motion generation technologies have limitations in that they are primarily trained based on generalized average movement patterns, and thus fail to adequately reflect individual differences such as user gender, age, body type, cultural background, and regional gesture characteristics.

[0003] For example, even when expressing the same action as “leaning against a wall,” the actual movement can vary from user to user due to differences in gesture frequency based on gender, age, and cultural background. Nevertheless, conventional natural language-based motion generation models either do not consider user characteristic information as input or reflect it only at the level of simple global embeddings. Consequently, this leads to discrepancies between individual users and their action representations, resulting in unnatural motion, reduced emotional delivery, and a degraded user experience.

[0004] Furthermore, existing technologies suffer from the problem that data bias can be directly reflected in the motion generation results, as motion datasets are often constructed with a bias toward specific population groups or standard body types. For example, training data collected primarily from users of a specific age group or cultural background fails to guarantee generalization to diverse user groups, and consequently, there was a possibility that movements differing from actual user characteristics would be generated.

[0005] Furthermore, conventional methods for reflecting user characteristics have been primarily limited to individual attributes, failing to account for complex movement differences across cultures. However, actual human movement expression is determined by the interaction of multidimensional factors, and unique motion styles can be formed depending on specific combinations of user characteristics. Nevertheless, existing models have failed to provide a systematic contrastive learning structure or user embedding generation mechanism capable of learning these correlational structures.

[0006] Accordingly, a system and method for generating motion of a 3D character using user characteristic metadata based on artificial intelligence are required. The problem to be solved

[0007] Embodiments of the present disclosure can provide a system and method for generating motion of a 3D character using user characteristic metadata based on artificial intelligence.

[0008] The technical problems to be solved in the embodiments are not limited to those mentioned above, and other unmentioned technical problems may be considered by those skilled in the art from the various embodiments described below. means of solving the problem

[0009] A method for a server to generate motion of a 3D character using user characteristic metadata based on artificial intelligence according to one embodiment, wherein the method comprises the steps of: obtaining a text prompt for generating motion of a 3D character and the user characteristic metadata from a user terminal; wherein the user characteristic metadata includes a gender value, an age value, and a region code value; generating a text condition vector through a text encoder based on the text prompt; generating a user characteristic context vector including a value related to gender characteristics, a value related to age characteristics, and a value related to region characteristics through a user characteristic encoder based on the user characteristic metadata; generating a biomechanical context vector including an allowable rotation range per joint, a joint type, and a bone length through a biomechanical encoder based on skeleton structure information; wherein the skeleton structure information is generated based on the text condition vector; inputting the text condition vector, the user characteristic context vector, and the biomechanical context vector into a diffusion-based motion generation model to generate a rotation sequence per joint; and transmitting 3D motion data generated by applying the rotation sequence per joint to the skeleton structure information to the user terminal. Effects of the invention

[0010] According to the embodiments, the present disclosure can automatically generate personalized 3D motion that reflects a motion style, range of motion, speed, and expression intensity suitable for user characteristics, even for the same text prompt, by reflecting user characteristic metadata obtained from a user terminal into a conditional update operation of a diffusion-based motion generation model. Accordingly, unlike conventional motion generation methods based on a single average value, it is possible to ensure motion diversity according to differences in user characteristics, and to enhance user experience and maximize content immersion.

[0011] According to the embodiments, the present disclosure learns motion representations corresponding to combinations of user characteristics through contrast learning loss between user characteristic metadata and motion style vectors, and optimizes model parameters so that motions with the same combination of user characteristics become similar and motions with different combinations of user characteristics are distinguished, thereby providing a highly reliable user characteristic-based motion generation effect that reflects actual movement differences according to gender, age, and regional culture. Accordingly, even if the text-based instructions are the same, differences in movement magnitude between children and adults, differences in gesture usage frequency by culture, and differences in balance maintenance characteristics by age group can be naturally reproduced.

[0012] According to the embodiments, the present disclosure can generate consistent behavior in which text semantics and user characteristics do not contradict each other by combining user characteristic context vectors with text condition vectors and spatiotemporal graph features on a cross-attention basis. Accordingly, it is possible to prevent unintended exaggerated movements, inaccurate emotional expressions, or style distortions, and to ensure semantic consistency between text, user characteristics, and motion features.

[0013] The effects obtainable from the embodiments are not limited to those mentioned above, and other unmentioned effects can be clearly derived and understood by a person skilled in the art based on the detailed description below. Brief explanation of the drawing

[0014] The accompanying drawings, included as part of the detailed description to aid in understanding the embodiments, provide various embodiments and explain the technical features of the various embodiments together with the detailed description. FIG. 1 is a diagram showing the configuration of an electronic device according to one embodiment. FIG. 2 is a diagram showing the configuration of a program according to one embodiment. FIG. 3 shows a flowchart of a method for a server to generate motion of a 3D character using user characteristic metadata based on artificial intelligence according to one embodiment. FIG. 4 shows the architecture of a diffusion-based motion generation model according to one embodiment. FIG. 5 is a block diagram showing the configuration of a diffusion-based motion generation model according to one embodiment. FIG. 6 is a block diagram showing the configuration of a server according to one embodiment. Specific details for implementing the invention

[0015] The following embodiments are combinations of the components and features of the embodiments in a specific form. Each component or feature may be considered optional unless otherwise explicitly stated. Each component or feature may be implemented in a form not combined with other components or features. Additionally, various embodiments may be constructed by combining some components and / or features. The order of operations described in various embodiments may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment.

[0016] In the description of the drawings, procedures or steps that could obscure the essence of the various embodiments were not described, nor were procedures or steps that can be understood by a person of ordinary knowledge in the relevant technical field described.

[0017] Throughout the specification, when a part is described as "comprising" or "including" a component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part," "...unit," and "module" as used in the specification refer to a unit that performs at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software. Additionally, "one (a or an)," "one," "the," and similar related terms may be used in the context describing various embodiments (particularly in the context of the following claims) in both singular and plural forms, unless otherwise indicated in the specification or clearly contradicted by the context.

[0018] Hereinafter, embodiments according to various examples will be described in detail with reference to the accompanying drawings. The detailed description disclosed below, together with the accompanying drawings, is intended to describe exemplary embodiments of various examples and is not intended to represent the only embodiment.

[0019] In addition, specific terms used in various embodiments are provided to aid in understanding the various embodiments, and the use of such specific terms may be modified in other forms within the scope of not departing from the technical concept of the various embodiments.

[0020] FIG. 1 is a diagram showing the configuration of an electronic device according to one embodiment.

[0021] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)). The electronic device (101) may be referred to as a client, terminal, or peer.

[0022] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0023] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models.

[0024] An artificial intelligence model can be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. Artificial intelligence models may include software structures, either additionally or as a substitute, in addition to hardware structures.

[0025] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0026] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0027] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0028] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0029] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0030] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0031] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensing device, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0032] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0033] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0034] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0035] The camera module (180) can capture still images and images. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0036] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0037] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0038] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0039] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0040] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0041] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0042] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0043] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0044] The server (108) is connected to an electronic device (101) and can provide services to the connected electronic device (101). Additionally, the server (108) may proceed with the membership registration process, store and manage various information of users who have registered as members accordingly, and provide various purchase and payment functions related to the service. Furthermore, the server (108) may share execution data of service applications running on each of multiple electronic devices (101) in real time so that services can be shared among users. In terms of hardware, this server (108) may have the same configuration as a conventional web server or service server. However, in terms of software, it may include program modules that perform various functions and are implemented through any language such as C, C++, Java, Python, Golang, Kotlin, etc. Additionally, the server (108) generally refers to a computer system that is connected to an unspecified number of clients and / or other servers through an open computer network such as the Internet, receives requests for task execution from clients or other servers, and derives and provides the results of the task, as well as computer software (server program) installed for this purpose. Furthermore, the server (108) should be understood as a broad concept that includes, in addition to the aforementioned server program, a series of application programs running on the server (108) and, in some cases, various databases (DB: Database, hereinafter referred to as "DB") built internally or externally. Accordingly, the server (108) classifies membership registration information and various information and data regarding games, stores them in the DB, and manages them; such DB can be implemented internally or externally of the server (108).Additionally, the server (108) can be implemented using various server programs provided for operating systems such as Windows, Linux, UNIX, and Macintosh on general server hardware. Representative examples include IIS (Internet Information Server) used in Windows environments and CERN, NCSA, APPACH, and TOMCAT used in UNIX environments to implement web services. Additionally, the server (108) may be linked with an authentication system and a payment system for user authentication of the service or purchase payment related to the service.

[0045] The first network (198) and the second network (199) refer to a connection structure capable of exchanging information between each node, such as terminals and servers, or a network connecting a server (108) and electronic devices (101, 104). The first network (198) and the second network (199) include, but are not limited to, the Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), 3G, 4G, LTE, 5G, Wi-Fi, etc. The first network (198) and the second network (199) may be closed types such as LAN, WAN, etc., but it is preferable that they be open types such as the Internet. The Internet refers to a global open computer first network (198) and second network (199) structure that provides protocols such as TCP / IP protocol, TCP, and UDP (user datagram protocol), and various services existing in the upper layer, namely HTTP (HyperText Transfer Protocol), Telnet, FTP (File Transfer Protocol), DNS (Domain Name System), SMTP (Simple Mail Transfer Protocol), SNMP (Simple Network Management Protocol), NFS (Network File Service), and NIS (Network Information Service).

[0046] A database may have a general data structure implemented in the storage space (hard disk or memory) of a computer system using a database management program (DBMS). A database may have a data storage form that allows for the free retrieval (extraction), deletion, editing, and addition of data. A database may be implemented to suit the purpose of an embodiment of the present disclosure using a relational database management system (RDBMS) such as Oracle, Informix, Sybase, and DB2, an object-oriented database management system (OODBMS) such as Gemston, Orion, and O2, and an XML native database such as Excelon, Tamino, and Sekaiju, and may have appropriate fields or elements to achieve its functions.

[0047] FIG. 2 is a diagram showing the configuration of a program according to one embodiment.

[0048] FIG. 2 is a block diagram (200) illustrating a program (140) according to various embodiments. According to one embodiment, the program (140) may include an operating system (142), middleware (144), or an application (146) executable on the operating system (142) for controlling one or more resources of an electronic device (101). The operating system (142) may include, for example, Android™, iOS™, Windows™, Symbian™, Tizen™, or Bada™. At least some of the programs (140) may be preloaded into the electronic device (101) at manufacturing time, for example, or downloaded or updated from an external electronic device (e.g., electronic device (102 or 104), or server (108)) when used by a user. All or part of the program (140) may include a neural network.

[0049] The operating system (142) can control the management (e.g., allocation or reclamation) of one or more system resources (e.g., processes, memory, or power) of the electronic device (101). The operating system (142) may additionally or substantially include one or more driver programs for driving other hardware devices of the electronic device (101), e.g., an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197).

[0050] Middleware (144) may provide various functions to an application (146) so that functions or information provided from one or more resources of an electronic device (101) can be used by the application (146). Middleware (144) may include, for example, an application manager (201), a window manager (203), a multimedia manager (205), a resource manager (207), a power manager (209), a database manager (211), a package manager (213), a connectivity manager (215), a notification manager (217), a location manager (219), a graphics manager (221), a security manager (223), a call manager (225), or a voice recognition manager (227).

[0051] The application manager (201) can, for example, manage the life cycle of the application (146). The window manager (203) can, for example, manage one or more GUI resources used on the screen. The multimedia manager (205) can, for example, identify one or more formats required for the playback of media files and perform encoding or decoding of the corresponding media files among the media files using a codec that matches the selected corresponding format. The resource manager (207) can, for example, manage the source code of the application (146) or the memory space of the memory (130). The power manager (209) can, for example, manage the capacity, temperature, or power of the battery (189) and, using the relevant information, determine or provide relevant information required for the operation of the electronic device (101). According to one embodiment, the power manager (209) can interact with the BIOS (basic input / output system) (not shown) of the electronic device (101).

[0052] The database manager (211) can, for example, create, search, or modify a database to be used by the application (146). The package manager (213) can, for example, manage the installation or update of the application distributed in the form of a package file. The connectivity manager (215) can, for example, manage a wireless or direct connection between the electronic device (101) and an external electronic device. The notification manager (217) can, for example, provide a function to notify the user of the occurrence of a specified event (e.g., an incoming call, a message, or an alarm). The location manager (219) can, for example, manage location information of the electronic device (101). The graphics manager (221) can, for example, manage one or more graphic effects or related user interfaces to be provided to the user.

[0053] The security manager (223) may, for example, provide system security or user authentication. The telephony manager (225) may, for example, manage voice call functions or video call functions provided by the electronic device (101). The voice recognition manager (227) may, for example, transmit user voice data to the server (108) and receive from the server (108) a command corresponding to a function to be performed on the electronic device (101) based on at least part of the voice data, or text data converted based on at least part of the voice data. According to one embodiment, the middleware (244) may dynamically delete some existing components or add new components. According to one embodiment, at least part of the middleware (144) may be included as part of the operating system (142) or implemented as separate software different from the operating system (142).

[0054] The application (146) may include, for example, a home (251), a dialer (253), an SMS / MMS (255), an IM (instant message) (257), a browser (259), a camera (261), an alarm (263), a contact (265), a voice recognition (267), an email (269), a calendar (271), a media player (273), an album (275), a watch (277), a health (279) (e.g., measuring biometric information such as exercise volume or blood sugar), or an environmental information (281) (e.g., measuring atmospheric pressure, humidity, or temperature information). According to one embodiment, the application (146) may further include an information exchange application (not shown) capable of supporting information exchange between the electronic device (101) and an external electronic device. The information exchange application may include, for example, a notification relay application configured to transmit information (e.g., a call, a message, or an alarm) designated to an external electronic device, or a device management application configured to manage the external electronic device. The notification relay application may transmit notification information corresponding to a designated event (e.g., receiving mail) generated in another application of the electronic device (101) (e.g., an email application (269)) to the external electronic device. Additionally or alternatively, the notification relay application may receive notification information from the external electronic device and provide it to the user of the electronic device (101).

[0055] A device management application can control the power (e.g., turn-on or turn-off) or function (e.g., brightness, resolution, or focus) of an external electronic device or a part of its components (e.g., a display module or camera module of the external electronic device) that communicates with the electronic device (101). The device management application can additionally or substantially support the installation, deletion, or updating of applications running on the external electronic device.

[0056] Throughout this specification, the terms neural network, neural network, and network function may be used interchangeably. A neural network may consist of a set of interconnected computational units, which may generally be referred to as “nodes.” These “nodes” may also be referred to as “neurons.” A neural network is composed of at least two nodes. The nodes (or neurons) constituting neural networks may be interconnected by one or more “links.”

[0057] In a neural network, two or more nodes connected via links can form a relative relationship between an input node and an output node. The concepts of input and output nodes are relative; any node in an output node relationship with respect to one node may be in an input node relationship with respect to another node, and vice versa. As previously mentioned, the input node versus output node relationship can be generated based on links. One or more output nodes may be connected to a single input node via links, and vice versa.

[0058] In a relationship between an input node and an output node connected through a single link, the value of the output node can be determined based on data input to the input node. Here, the nodes interconnecting the input node and the output node may have weights. The weights may be variable and may be varied by a user or an algorithm to enable the neural network to perform the desired function. Here, the edges or links interconnecting the input node and the output node have weights that can be variably applied by a user or an algorithm to enable the neural network to perform the desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node may determine its value based on the values ​​input to the input nodes connected to the output node and the weights set on the links corresponding to each input node.

[0059] As described above, a neural network is formed in which two or more nodes are interconnected through one or more links to create input-output node relationships within the network. The characteristics of a neural network can be determined by the number of nodes and links within the network, the relationships between the nodes and links, and the weight values ​​assigned to each link. For example, if two neural networks exist with the same number of nodes and links but different weight values ​​between the links, the two neural networks can be recognized as being different from each other.

[0060] FIG. 3 shows a flowchart of a method for a server to generate motion of a 3D character using user characteristic metadata based on artificial intelligence according to one embodiment. The embodiment of FIG. 3 can be combined with various embodiments of the present disclosure.

[0061] Referring to FIG. 3, in step S310, the server can obtain a text prompt for generating motion of a 3D character and user characteristic metadata from the user terminal.

[0062] The server may be a server that generates motion of a 3D character using artificial intelligence and provides the generated 3D motion data to a user terminal. For example, the server may acquire a text prompt from a user terminal, acquire skeleton structure information based on the result of encoding the text prompt, and generate biomechanical information based on the skeleton structure information. For example, the server may acquire user characteristic metadata from a user terminal and generate a user characteristic context vector as a result of encoding the user characteristic metadata. The server may input text-based conditions, user characteristic-based conditions, and biomechanical-based conditions into a diffusion-based motion generation model to generate a joint-by-joint rotation sequence, and transmit 3D motion data to the user terminal to which the generated joint-by-joint rotation sequence has been applied. The text prompt may be a sentence describing the motion of a 3D character entered by the user in natural language. For example, the text prompt may include at least one of the subject performing the motion, the type of motion, the attributes of the motion, the spatial direction of the motion, the part of the body to which it is executed, and a body shape or proportional representation. For example, the text prompt may include at least one sentence. At this time, the server may use the entered text prompt as input for generating a text condition vector to be utilized in a diffusion-based motion generation model. User characteristic metadata may be information representing the characteristics of a user utilizing a user terminal. For example, user characteristic metadata may include a gender value, an age value, and a region code value. The gender value is a value representing the user's gender and may be set, for example, as a value to identify a male user or a female user. The age value is a value representing the user's age and may be, for example, an integer value representing the age in years calculated from date of birth information entered by the user terminal. The region code value may be a code value representing the region where the user resides.For example, the regional code value may consist of a 2-digit number for country identification, a 3-digit number for identifying a metropolitan administrative area, and a 5-digit number for identifying a basic administrative area. In this case, the server may use the input user characteristic metadata as input for generating a user characteristic context vector to be utilized in a diffusion-based motion generation model. Skeleton structure information may be structural information representing the joint configuration and connectivity of a character. Biomechanical information may be information representing physical constraints applied to the joint movements of the skeleton. The diffusion-based motion generation model may be an artificial intelligence model that receives a specific condition vector as input and generates a joint-specific rotation sequence through a probabilistic transformation process. For example, the server may include the server (108) of FIG. 1.

[0063] The user terminal may be a user terminal that generates motion for a 3D character. For example, the user terminal may transmit user characteristic metadata and a text prompt containing a sentence describing the motion of the 3D character to the server, and may display the 3D motion data received from the server on the screen or save it for animation production. For example, an application for performing motion generation for a 3D character may be pre-installed on the user terminal. For example, the user terminal may transmit user characteristic metadata to the server along with a text prompt containing a sentence describing the motion of the 3D character through the pre-installed application, and may display the 3D motion data received from the server on the screen. Through this, motion for a specific character can be easily generated through a natural language-based intuitive interface, and the 3D motion data provided by the server can be utilized. For example, the user terminal may include the electronic device (101) of FIG. 1.

[0064] For example, a user terminal can pre-enter gender, age, and region code values ​​as user characteristic metadata through a pre-installed application. Additionally, for example, if a region code value is not entered from the user terminal through a pre-installed application, the user terminal's Internet Protocol address value is transmitted to the server, and the server can determine the region code value by referring to a pre-built IP-region mapping table based on the Internet Protocol address value.

[0065] In step S320, the server can generate a text condition vector through a text encoder based on the text prompt.

[0066] A text encoder may be an encoding module for converting sentence structure, semantic context, and action-related intent contained in a text prompt input from a user terminal into an AI-based high-dimensional vector representation. For example, the text encoder may include at least one of a plurality of pre-trained language encoders such as BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (Robustly Optimized BERT Pretraining Approach), T5 (Text-to-Text Transfer Transformer), and GPT (Generative Pre-trained Transformer). For example, the text encoder may produce embeddings that reflect contextual relationships between words within a sentence.

[0067] For example, a text encoder can divide a text prompt into word- or subword-unit tokens and, for each token, calculate a pre-trained word embedding and a positional embedding indicating the order of the tokens to construct an input embedding sequence in which the token order is preserved. For example, the input embedding sequence can be generated in the form of a vector matrix that simultaneously includes the semantic features and positional information of each token within the sentence.

[0068] For example, a text encoder can extract contextual representations by inputting an input embedding sequence into a multi-layer transformer encoding layer. The transformer encoding layer includes multiple self-attention and feed-forward operations and can calculate attention weights to model semantic correlations between tokens within a sentence. For instance, the self-attention operation can generate contextual representations that emphasize action semantics, attributes, intensity, and directionality by assigning higher weights to key words directly related to the action, such as “lift,” “quickly,” “left arm,” and “rotate.” For instance, the transformer encoding layer can generate a global semantic representation for the entire sentence based on the contextual representation of each token and can progressively generate representations through multiple layers.

[0069] For example, a text encoder can generate fixed-length text condition vectors used in the conditional operations of a motion generative model by integrating sentence-level representation vectors produced in the final layer using mean pooling or special token-based aggregation methods. Special token-based aggregation may be a method that uses the hidden state of a special token placed at the beginning or a specific position of the input sequence as a representative vector for the entire sentence. That is, the [CLS] token of BERT-type models, and the T5 / GPT-type model <s>Or EOS token, RoBERTa's <s>It means using the final hidden vector of a token, which is designed to aggregate sentence-level meaning like a token, directly as a sentence representation. The text condition vector may be a vector that reflects semantic information regarding the agent performing the action, the type of action (e.g., jump, turn, move), attributes of the action (e.g., fast, slow, smooth), spatial directionality of the action (e.g., forward, upward), the part of the action (e.g., arm, leg, torso), and body shape and proportional representations (e.g., large, small, long legs, etc.).

[0070] In step S330, the server can generate a user characteristic context vector containing values ​​related to gender characteristics, values ​​related to age characteristics, and values ​​related to region characteristics through a user characteristic encoder based on user characteristic metadata.

[0071] The user feature encoder may be a neural network-based encoding module that generates embedding vectors to numerically represent the user's movement style based on user feature metadata. The user feature context vector may be a conditional expression utilized as a condition vector in the cross-attention operation of a diffusion-based motion generation model, allowing different motion styles to be generated depending on the user feature even when the same text prompt is input. Values ​​related to gender features may be indicators for quantitatively representing gait and movement patterns according to gender. Values ​​related to gender features may include the stride length ratio, the average arm swing angle, and the average values ​​of lower limb joint rotation acceleration. For example, the stride length ratio is a value representing the average straight-line distance traveled along the ground during one step, which may range from approximately 0.90 m to 1.40 m, and the unit may be meters (m). Additionally, the average arm swing angle is a value representing the average angle of rotation of the upper and lower arms in the forward and backward directions during walking, which may range from approximately 15 degrees to 55 degrees, and the unit may be degrees. The average value of lower extremity joint rotational acceleration represents the average rate of increase in the rotational speed of the knee and hip joints during walking or body movement, approximately 20 degrees / second. 2 At 95 degrees / second 2 It can have a range of values, and the unit is degrees / second. 2 It could be.

[0072] Values ​​related to age characteristics may be indicators for quantitatively expressing differences in joint mobility and movement tendencies that appear with age changes. For example, values ​​related to age characteristics may include the rate of decrease in maximum joint flexion angle, average walking speed, and the amplitude of balance-maintaining sway. For example, the rate of decrease in maximum joint flexion angle is a value representing the ratio of decrease compared to the average maximum joint flexion angle of people in their 20s, and may range from approximately 0% to 35%, with the unit being percentage (%). Average walking speed is a value representing the average movement speed when performing a straight walk of 5m or more, and may range from approximately 0.6 m / s to 1.6 m / s, with the unit being meters per second (m / s). The amplitude of balance-maintaining sway is a value representing the amplitude of vibration in which the center of the body sways from side to side while maintaining a static posture, and may range from approximately 2 mm to 22 mm, with the unit being millimeters (mm).

[0073] Values ​​related to regional characteristics may serve as indicators to quantitatively represent movement expression patterns observed in the corresponding regional culture. For example, values ​​related to regional characteristics may include the frequency of gesture use, the average amplitude of upper limb movement, and the ratio of time spent maintaining a static posture. For example, the frequency of gesture use is a value representing the average frequency of hand gestures occurring during communication or emotional expression; it may range from approximately 0.8 times / second to 3.5 times / second, and the unit may be times / second. The average amplitude of upper limb movement is a value representing the average distance the hand moves relative to the center of the shoulder; it may range from approximately 10 cm to 65 cm, and the unit may be centimeters (cm). For example, the ratio of time spent maintaining a static posture is a value representing the time during which the body remains in a minimal or stationary state out of the total movement time; it may range from approximately 15% to 62%, and the unit may be percentage (%).

[0074] For example, a user feature encoder may include a feature-specific embedding layer, a feature combination layer, and a normalization layer. The feature-specific embedding layer can take a gender channel, an age channel, and a region code channel as inputs, respectively, to produce a gender embedding vector, an age embedding vector, and a region-based embedding vector. For example, the gender embedding sublayer can generate a gender embedding vector representing gender characteristics by multiplying the stride length ratio, average arm swing angle, and average lower limb joint rotation acceleration corresponding to the gender value with embedding parameters. The age embedding sublayer can generate an age embedding vector representing age characteristics by combining the maximum joint flexion angle reduction rate, average walking speed, and balance-maintaining swing amplitude corresponding to the age value with embedding parameters. The region-based embedding sublayer can generate a region-based embedding vector representing region characteristics by combining the gesture usage frequency, average upper limb movement amplitude, and static posture maintenance time ratio corresponding to the region code value with embedding parameters. The feature combining layer can perform the role of aligning gender embedding vectors, age embedding vectors, and region-based embedding vectors to the same dimension and then integrating them into a single intermediate user feature vector through vector combining operations. For example, the feature combining layer can generate an intermediate user feature vector that reflects the interrelationships of features by sequentially concatenating the gender embedding vector, age embedding vector, and region-based embedding vector, and then passing them through a non-linear activation function by applying a fully combined weight matrix and a bias vector. In this case, the fully combined weight matrix can be used as a parameter to learn how gender-based movement tendencies, age-based mobility stability, and region-based gesture patterns are combined with each other.The normalization layer can generate a user feature context vector by performing normalization operations based on mean and variance on the intermediate user feature vector generated by the feature combining layer, thereby adjusting the distribution of values ​​so that they are not concentrated in a specific range. In other words, the normalization layer can play a role in uniformly adjusting the magnitude and distribution of the vector so that each component of the user feature context vector can be reliably used in the cross-attention operation of a diffusion-based motion generation model.

[0075] For example, skeleton structure information can be generated based on text condition vectors.

[0076] According to one embodiment, the server can obtain skeleton configuration parameters including an active joint set and a character proportional parameter based on a text condition vector. Based on the skeleton configuration parameters, the server can generate skeleton structure information including joint list information, joint connection information, joint characteristic information, and joint rotation axis information using a preset basic skeleton template.

[0077] Skeleton configuration parameters may refer to default settings that define the joint configurations and hierarchical relationships necessary for generating a skeleton structure based on the semantic information of text condition vectors. For example, skeleton configuration parameters may include active joint sets and character proportion parameters.

[0078] For example, the server can extract action directives (e.g., "lift," "jump," "rotate," "run") and expressions related to body parts (e.g., "arm," "leg," "waist") from a text condition vector. The server can compare the extracted expressions with a preset action-joint mapping table to determine the joints required to perform the corresponding action as the set of activated joints. For example, the expression "lift the arm" can be mapped to the shoulder joint, elbow joint, and wrist joint in the action-joint mapping table. Additionally, for example, the server can predefine a set of keywords corresponding to action directives and a set of keywords corresponding to expressions related to body parts, and can pre-set reference keyword embeddings for each keyword set. The server can calculate a normalized inner product-based similarity value, such as cosine similarity, between each of the multiple reference keyword embeddings and the text condition vector, and select keywords corresponding to reference keyword embeddings whose similarity value is above a preset threshold as candidates for action directives and body part expressions. For example, if a keyword is detected whose similarity value to reference keyword embeddings corresponding to "lift," "jump," "rotate," or "run" exceeds a preset threshold, that word can be extracted as an action directive; similarly, if a keyword is detected whose similarity value to reference keyword embeddings corresponding to "arm," "leg," or "waist" exceeds a preset threshold, that word can be extracted as an expression related to a body part. Subsequently, the server can refer to an action-joint mapping table to determine a matching set of joints using the combination of the extracted action directive and the expression related to the body part as a key. The action-joint mapping table may be a data structure defined such that a list of joint identifiers required to perform the corresponding action corresponds to each combination of the action directive and the expression related to the body part.For example, the item corresponding to "raise arm" may store identifiers for the shoulder joint, elbow joint, and wrist joint, respectively, in the motion-joint mapping table. The server can determine the set of activated joints by repeatedly querying the motion-joint mapping table for each of the multiple motion directives extracted from the text condition vector and the expressions related to body parts, and by integrating the joint identifiers obtained from the query results through a union operation.

[0079] For example, the server can determine expressions related to body shape and proportions (e.g., "long arms," ​​"short legs," "small body type") included in the text condition vector to calculate character proportion parameters for adjusting the bone length of the basic skeleton template. Additionally, for example, the server can pre-set a set of keywords corresponding to body shape expressions and a set of keywords representing proportion strength, and pre-set reference keyword embeddings for each keyword set. The server can calculate a normalized inner product-based similarity value, such as cosine similarity, between each of the multiple reference keyword embeddings and the text condition vector, and by determining the reference keyword embeddings whose similarity value is above a pre-set threshold, select the body shape expression and proportion strength expression corresponding to those reference keyword embeddings, respectively. The server can generate a body shape proportion vector by combining the reference keyword embeddings corresponding to the selected body shape expression and proportion strength expression, and input the body shape proportion vector into a multi-layer perceptron-based scale factor calculation module to calculate scale factors for each of the multiple body parts and the overall scale factor. The server can set the scale factor for each of multiple body parts and the overall scale factor as character proportion parameters, and adjust the proportions of the skeleton by multiplying the scale factor by the base value corresponding to the length of each bone segment included in the basic skeleton template.

[0080] Skeleton structure information may be information representing the structure of a skeleton that reflects the semantic information of a text condition vector. For example, skeleton structure information may include joint list information to identify all joints constituting the skeleton, joint connection information indicating the connection relationships between joints, joint characteristic information indicating the characteristics of the degrees of freedom of movement of each joint, and joint rotation axis information indicating the possible rotational directions of each joint.

[0081] The basic skeleton template is data that standardizes the anatomical structure of the human body and may include a complete list of joint identifiers, connection relationships between joints (tree-based connection structures with parent and child connections and adjacent connection structures including branching structures), joint types and basic rotation axes of each joint, and length information for basic skeletal segments. For example, the basic skeleton template may be pre-configured on a server.

[0082] For example, joint list information may be a set of multiple joint identifiers to identify each joint constituting the skeleton. Each joint included in the joint list information represents a single joint point on the skeleton and can serve as a basic unit for forming a set of nodes when constructing a spatiotemporal graph. For example, joint list information for a character may include multiple joint identifiers such as the pelvis, multiple levels of spinal joints, thoracic joints, neck joints, head joints, left and right shoulder joints, left and right elbow joints, left and right wrist joints, left and right hip joints, left and right knee joints, and left and right ankle joints. For example, the server may acquire a pre-configured default skeleton template and set the entire list of joint identifiers stored in the default skeleton template as the joint list information. In this case, the set of active joints included in the skeleton configuration parameters may be used as information to set the joints in the joint list that are directly involved in performing actions. For example, the server may compare each joint identifier in the set of active joints with the joint list information to determine a matching joint item, and add an activation flag value to the joint list information for the determined joint item to indicate that it is directly involved in performing actions.

[0083] For example, joint connection information may be information that hierarchically represents the connection relationships between the joints constituting the skeleton. Joint connection information may refer to an extended connection structure that includes a hierarchical connection form in which the entire structure of the skeleton extends upward and downward directions relative to the root joint, while simultaneously encompassing all adjacent joint pairwise connection relationships (branch connections and sibling node connections) defined in the base skeleton template. This joint connection information can subsequently be used as reference information for generating spatial edges of the spatiotemporal graph. For example, the server may refer to the connection relationships between joints stored in the base skeleton template and, for each joint included in the joint list information, set the adjacent joints of that joint as joint connection information by reflecting all connection relationships set on the base skeleton template.

[0084] For example, joint connection information may include a joint identifier to identify each joint, a set of joint identifiers for adjacent joints directly connected to the corresponding joint, a value for the connection characteristic of each connection relationship, and a hierarchy level value indicating the hierarchical depth from the root joint. The set of joint identifiers for adjacent joints directly connected to the corresponding joint may include joint identifiers for joints located in a higher direction than the corresponding joint, joints located in a lower direction than the corresponding joint, and joints branching out at the same hierarchy depth. The hierarchy level value is an integer value representing a relative position within the entire skeleton, and may have a structure where the topmost joint of the skeleton is set to 0 and increases by 1 as one moves down to lower joints. For example, the depth value for the pelvic joint may be set to 0, the first vertebral joint to 1, and the shoulder joint to 3. The hierarchy level value has an integer range greater than or equal to 0 and may be defined within a range of 0 to 20 depending on the structure of the entire skeleton. Additionally, the joint connection information may include values ​​for the connection characteristic for each pair of joint identifiers. The value for the connection characteristic is used to identify the type of connection method between joints and can be set to one of a rotational center connection, a linear movement center connection, or a fixed connection. A rotational center connection indicates a structure where two joints perform relative rotation around one or more rotation axes, while a linear movement center connection indicates a structure where joints can move relatively only in a specific linear direction. A fixed connection indicates a structure where two joints do not perform relative rotation or movement. The value for this connection characteristic can be represented as a predefined string value or an integer value such as 0, 1, or 2.

[0085] At this time, the server can maintain a tree structure in which the connection paths necessary for performing actions are not broken by ensuring that the joints included in the active joint set and the joints connected to them as upper and lower levels are all included in the hierarchical structure. Additionally, the server can include character proportion parameters and length information for basic bone segments in the joint connection information. The length information for basic bone segments may include values ​​representing the length of the basic bone segment between joints for each pair of joint identifiers.

[0086] For example, joint characteristic information may be information representing the characteristics of the degrees of freedom of motion allowed by each joint. The server may query the joint-specific joint types stored in the base skeleton template and set the joint type corresponding to each joint identifier as joint characteristic information. The joint type may be set to one of a plurality of preset characteristic values. For example, the plurality of characteristic values ​​may include values ​​for single-axis rotational joints, universal joints, and multi-axis rotational joints, respectively. A single-axis rotational joint may refer to a joint that allows rotation in only one direction relative to a single axis of rotation. A universal joint may refer to a joint that allows independent rotational movement in two directions relative to two mutually orthogonal axes of rotation. A multi-axis rotational joint may refer to a joint that allows rotational movement in all directions of three-dimensional space relative to three axes of rotation. Joint characteristic information can subsequently be used as reference information to select the form of the constraint model to be applied when the biomechanical encoder calculates the allowable rotational range per joint.

[0087] For example, joint rotation axis information may include the number of axes through which each joint can rotate and a vector (axis-specific direction vector) representing the direction of each axis. In other words, joint rotation axis information may be information indicating the direction of the axes through which each joint can rotate. Joint rotation axis information is a value used to define a reference axis indicating which direction each joint performs rotational motion relative to in a specific coordinate system; depending on the joint type, either a single axis or multiple axes may be set. For example, the elbow joint, which is a single-axis rotational joint type, may be set to have a single axis vector representing the direction of the rotation axis where the forearm bends relative to the upper arm. In this case, the rotation axis may be set as a unit vector representing the flexion-extension direction in a 3D coordinate system based on the joint center. This single unit vector represents the precise rotational direction in the space where the joint can rotate and can subsequently be used as a reference axis to express the amount of joint rotation in degrees. The shoulder joint, which is a multi-axis rotational joint type, may have multiple axis vectors to represent horizontal rotation, vertical rotation, and rotation, respectively. At this time, multiple rotation axes can be defined as three-dimensional unit vectors corresponding to each degree of freedom. For example, the unit vector in the x-axis direction may represent the rotation direction corresponding to flexion / extension, the unit vector in the y-axis direction may represent the rotation direction corresponding to abduction / adduction, and the unit vector in the z-axis direction may represent the rotation direction corresponding to internal / external rotation. For example, the server may query the basic rotation axis information for each joint included in the basic skeleton template and, depending on the joint type of each joint included in the joint list information, set one or more rotation axis vectors corresponding to the joint as the joint rotation axis information of the skeleton structure information.

[0088] For example, the server can integrate joint list information, joint connection information, joint characteristic information, and joint rotation axis information to determine skeleton structure information based on skeleton configuration parameters.

[0089] In step S340, the server can generate a biomechanical context vector including per-joint allowable rotation range, joint type, and bone length through a biomechanical encoder based on skeleton structure information.

[0090] The biomechanical encoder may be a neural network-based encoding module for quantifying joint-level biomechanical constraints based on skeleton structure information. The biomechanical context vector is a vector that numerically represents the degrees of freedom, rotational limits per axis, and structural proportions of each joint, and can subsequently be used as a constraint to ensure that only physically valid motions are generated during the joint rotation sequence sampling process of a diffusion-based motion generation model.

[0091] For example, a biomechanical encoder may include a joint structure parsing layer, a constraint binding layer, a skeletal information generation layer, a structural dependency encoding layer, and a normalization layer. In this case, the joint structure parsing layer can generate joint-unit input tokens based on joint list information, joint connection information, joint characteristic information, and joint rotation axis information. Specifically, the joint structure parsing layer assigns a unique index to each joint identifier included in the joint list information and sets the joint index, joint identifier, and joint type for each joint in the identification fields of the input token. The joint structure parsing layer can refer to the joint connection information to extract parent joint identifiers and child joint identifiers for each joint and set these values ​​in the parent joint identifier field and child joint identifier field of the joint input token. Additionally, the joint structure parsing layer can set a hierarchy level value as a hierarchy level field for each joint. The joint structure parsing layer can include connection characteristic values ​​included in the joint connection information in the joint input token. The joint structure parsing layer can query the axis-specific direction vectors corresponding to each joint from the joint rotation axis information and record the number of rotation axes and the axis-specific direction vector index representing the direction of each axis in the rotation axis field of the joint input token. In this way, the joint input token may be a joint-unit representation including a joint index, joint identifier, joint type, parent joint identifier, child joint identifier, hierarchy level value, values ​​for connection characteristics for each of the parent joint identifier and child joint identifier, and the axis-specific direction vector index.

[0092] The constraint binding layer identifies the axes to which each joint can rotate based on the joint type and axis-specific direction vector index included in the joint input token, and can look up the axis-specific minimum and maximum rotation angles from the biomechanical constraint table and add them to the joint input token. For example, a joint determined to be a single-axis rotational joint may have its minimum and maximum rotation angles determined for one axis, a joint determined to be a universal joint may have its minimum and maximum rotation angles determined for each of the two axes, and a joint determined to be a multi-axis rotational joint may have its minimum and maximum rotation angles determined for each of the three axes.

[0093] Specifically, for example, the constraint binding layer can map axis-specific rotation constraint items by activating axis items that match the axis direction vector index based on the joint type and axis-specific direction vector index included in the joint input token. The constraint binding layer can look up the axis-specific minimum and maximum rotation angles in a predefined biomechanical constraint table and independently assign the minimum and maximum rotation angles to each rotation axis activated in the parsed joint input token. For example, for a joint determined to be a single-axis rotational joint, the minimum and maximum rotation angles can be calculated for one axis corresponding to the flexion and extension axis; for a universal joint, the minimum and maximum rotation angles can be calculated for the flexion and extension axis and the abduction and adduction axis, respectively; and for a multi-axis rotational joint, the minimum and maximum rotation angles can be independently calculated for all three axes (flexion and extension, abduction and adduction, pronation and supination). Additionally, if a joint is determined to be non-rotatable, all axis-specific rotation angle values ​​can be set to 0. In this way, the constraint coupling layer can add the calculated axis-specific rotation angle information to the joint-specific input tokens to generate joint-unit constraint tokens with assigned rotation constraint values, and transmit them to the skeletal information generation layer.

[0094] The skeletal information generation layer can calculate the skeletal length based on the upper and lower joints of each joint using the joint unit constraint token and add it to the joint input token. The skeletal information generation layer can determine upper-lower joint pairs by referencing the upper joint identifier and lower joint identifier of the joint unit constraint token. Subsequently, the skeletal information generation layer can determine the basic skeletal length value corresponding to the upper-lower joint pair from the length information for the basic skeletal segment included in the joint connection information, and calculate the skeletal length by multiplying the corresponding basic skeletal length value by the scale factor corresponding to the upper-lower joint pair from the character proportional parameter included in the joint connection information. For example, if the basic skeletal length between the upper arm and the forearm is defined as 1.0 units, and the scale factor calculated by the character proportional parameter is 1.15, the upper arm-forearm skeletal length can be adjusted to 1.15 units. The skeletal information generation layer can complete structural proportional information indicating how far the corresponding joint is from the upper joint by recording the calculated skeletal length in the skeletal length field of the joint unit constraint token. The skeletal information generation layer can pass joint-unit structure tokens with added skeletal lengths to the structural dependency encoding layer.

[0095] The structural dependency encoding layer can generate biomechanical feature vectors that reflect inter-joint structural dependencies based on joint-unit structure tokens. For example, the structural dependency encoding layer can calculate the effect of rotational constraints on the range of motion of surrounding joints by referencing values ​​for upper joint identifiers, lower joint identifiers, bone lengths, hierarchy level values, and connectivity characteristics included in the joint-unit structure tokens. For example, the structural dependency encoding layer can calculate rotational constraint propagation relationship values ​​by referencing upper joint identifiers and lower joint identifiers representing the connectivity relationship between upper and lower joints, and by comparing the minimum and maximum rotation angles by axis of the upper joint with the minimum and maximum rotation angles by axis of the lower joint based on upper-lower joint pairs. The rotational constraint propagation relationship value is a value normalized from 0 to 1 by dividing the width of the upper joint's range of rotation by axis by the width of the lower joint's range of rotation by axis; a value closer to 1 may indicate a structure where the rotational restriction of the upper joint more strongly constrains the range of motion of the lower joint. For example, if the axial rotation range of the upper joint is narrower than that of the lower joint, the ratio of the axial rotation width of the upper joint to the axial rotation width of the lower joint has a value close to 1, reflecting a structure where the limitation of the upper joint strongly influences the actual rotation range of the lower joint. Additionally, the structural dependency encoding layer can calculate a rotation axis alignment coefficient by comparing the axial vector of the upper joint with the axial vector of the lower joint. The rotation axis alignment coefficient is a value normalized from 0 to 1 by converting the inner product value, which ranges from -1 to 1 based on the cosine similarity of the two rotation axis vectors, into (u*v + 1) / 2; it can be set to a value closer to 1 as the two axes are closer to parallel, and a value closer to 0 as the two axes are closer to perpendicular.Here, u may be the rotation axis vector of the upper joint, and v may be the rotation axis vector of the lower joint. The rotation axis alignment coefficient can be used as an axis alignment-based weight to determine how much the rotational directionality of the upper joint propagates to the rotational characteristics of the lower joint. Additionally, the structural dependency encoding layer can calculate the relative distance-based influence of upper-lower joint pairs by referencing the bone lengths included in the joint-unit structure tokens. This reflects a structure where the longer the bone length between the upper and lower joints, the greater the impact the rotational constraint of the upper joint has on the actual rotational range of the lower joint; the corresponding relative distance-based influence can be normalized to a ratio value between 0 and 1. Furthermore, the structural dependency encoding layer can use the values ​​for the connectivity characteristics of the joint-unit structure tokens as a connection method-based influence, depending on whether the structural connection method between the upper and lower joints is a rotation-centered connection, a linear-movement-centered connection, or a fixed connection. For example, the structural dependency encoding layer can convert these categorical values ​​into continuous influence coefficients between 0 and 1 using a predefined connection influence mapping table. For example, for a fixed connection, the influence factor can be set to 0 because rotational constraints of the upper joint are not transmitted to the lower joint. For a linear movement-centered connection, the influence factor can be set to a low value such as 0.2 because the ratio of rotational constraint transmission is limited. For a rotation-centered connection, the influence factor can be set to 1.0 because rotational changes of the upper joint are directly reflected in the lower joint. Additionally, the structural dependency encoding layer can calculate a hierarchy-based influence that reflects positional importance within the skeleton by referencing the hierarchy level value assigned to each joint.Joints with low hierarchy level values ​​(e.g., pelvis) are located in a central structure that supports multiple sub-joints, so the constraint influence of such joints can be set to a value close to 1, and joints with high hierarchy level values ​​(e.g., wrist) correspond to an extremity structure, so the constraint influence of such joints can be set to a relatively low value. Hierarchy-based influence can be normalized in the range of 0 to 1.

[0096] In this way, rotational constraint propagation relationship values, rotational axis alignment coefficients, relative distance-based influence, connection method-based influence, and hierarchy-based influence can be combined with joint- and axis-based rotational constraint information to be converted into joint-unit biomechanical feature vectors. The generated biomechanical feature vectors can then be provided to a normalization layer and included as components of a biomechanical context vector.

[0097] The normalization layer can complete the biomechanical context vector by aligning the joint-unit biomechanical feature vectors generated by the structural dependency encoding layer to the same dimension and combining them according to the order of the joint list information.

[0098] Additionally, for example, the biomechanical encoder can receive joint list information, joint connection information, joint characteristic information, and joint rotation axis information as input during the learning process to optimize intrinsic parameters for predicting the biomechanical constraints of each joint. The biomechanical encoder can learn joint type embedding parameters to infer joint-specific rotation characteristics based on the joint type. By learning projection parameters that normalize joint rotation axis information to a reference coordinate system, the biomechanical encoder can adjust its internal representation structure to reliably predict axis-specific rotation constraints.

[0099] For example, a biomechanical encoder can improve the performance of predicting the allowable rotation range per axis by updating the parameters of the constraint binding layer by referencing the minimum and maximum rotation angle labels for each joint included in the training data. During the training process, rotational constraint regression parameters can be adjusted based on the joint type and the number of joint rotation axes to predict the rotation range for one axis for single-axis rotational joints, and independent minimum-maximum rotation angles for each rotation axis for multi-axis rotational joints. For example, if the training labels for a humanoid elbow joint are a rotation range from -5 degrees to 145 degrees, the biomechanical encoder can update the parameters so that the output rotation range of the joint converges to this value, and for joints with two rotation axes, such as a bird-like wing joint, it can learn multi-axis rotational constraints by reflecting the minimum-maximum rotation angle labels corresponding to each axis.

[0100] For example, the biomechanical encoder can utilize inter-joint distance labels calculated for each joint by multiplying a reference skeletal length by a character proportional parameter during the training process. For instance, the skeletal information generation layer can be trained to predict adjusted inter-joint skeletal lengths by receiving the connection structure between upper and lower joints and proportional scaling factors as input, and parameters can be updated through a distance-based loss function to minimize the difference from the actual proportional values. This process serves to ensure that the encoder stably reflects proportional constraints even in situations where the inter-joint distance structure varies across different body types.

[0101] For example, the structural dependency encoding layer can optimize parameters during the learning process that reflect rotation constraint propagation relationship values, rotation axis alignment coefficients, relative distance-based influence, connection method-based influence, and hierarchy-based influence. The structural dependency encoding layer can learn relationship-based parameters that allow the rotation constraint of each joint to be influenced by the structural characteristics of adjacent joints. For instance, learning can be performed by adjusting attention weights between parent and child nodes to reflect the structural relationship where the rotation constraint of the shoulder joint also influences the rotational range of the elbow joint.

[0102] As the learning process iterates, the biomechanical encoder can progressively optimize the integrated representation of biomechanical constraints, such as rotational thresholds per axis, rotational characteristics based on joint type, inter-joint distance proportionality, and joint connectivity dependency. Ultimately, the biomechanical encoder can be trained to generate a biomechanical context vector that takes only skeleton structure information as input and consistently predicts the allowable rotation range and bone length of each joint; this vector can then be provided as a constraint to ensure that the diffusion-based motion generation model does not exceed the physically feasible rotation range when generating joint rotation sequences.

[0103] For example, the biomechanical context vector may include structural constraint variables representing the structural relationship between the basic constraint variables and the joints. For example, it may include per-joint allowable range of rotation, joint type, and bone length. For example, structural constraint variables may include rotational constraint propagation relationship values, rotation axis alignment coefficients, relative distance-based influence, connection method-based influence, and hierarchy-based influence.

[0104] The allowable rotation range per joint is numerical information containing the minimum and maximum rotation angles set for each axis around which the joint can rotate; it may represent the biomechanically achievable rotation range of the joint as a continuous value in degrees or radians. The allowable rotation range per joint may include the rotation range for each axis supported by the joint, such as flexion and extension, abduction and adduction, and pronation and supination. The joint type is characteristic information defining the number of degrees of freedom of motion and the number of rotatable axes possessed by each joint; it can be used as a standard for the number of rotatable axes, the application method of the rotation model, and the method of calculating rotation constraints for the corresponding joint. The bone length is a value representing the distance between joints connecting an upper joint and a lower joint, and it may be a real value determined by multiplying the standard bone length defined in the basic skeleton template by a scale factor calculated from the character proportionality parameter. The bone length can be used as an item to quantify the relative distance proportionality between joints and the strength of structural dependency.

[0105] In step S350, the server can generate a joint-specific rotation sequence by inputting the text condition vector, user characteristic context vector, and biomechanics context vector into a diffusion-based motion generation model.

[0106] For example, a diffusion-based motion generation model may be a probabilistic conditional generation architecture that utilizes a spatiotemporal graph representation in which each joint of each frame is set as a node to simultaneously reflect skeleton structure information along the time axis. The diffusion-based motion generation model may include multiple graph transformer layers that perform conditional feature update operations based on text condition vectors and biomechanics-based adjustment operations based on biomechanics context vectors. In this case, multi-stage diffusion may be performed to converge a motion representation from a probabilistic noise distribution while performing scale and shift adjustments to adjust the feature representations of the graph transformer layers in the direction of biomechanical constraints.

[0107] For example, a diffusion-based motion generation model can generate a spatiotemporal graph based on skeleton structure information and a preset target number of frames. The target number of frames is a value representing the total frame length preset for generating 3D character motion, and can serve as reference information for determining the temporal resolution and duration of the motion to be generated. For example, a user terminal can transmit the target number of frames corresponding to the motion length specified by the user to a server, and the server can set the size of the time axis of the spatiotemporal graph based on the target number of frames.

[0108] For example, a diffusion-based motion generation model can generate a spacetime graph combining joint structure and time flow by setting each joint in each frame as a node of a spacetime graph according to the number of frames (T) and the number of joints (J), generating spatial edges connecting pairs of nodes within the same frame, and generating temporal edges between adjacent frames where nodes corresponding to the same joint exist.

[0109] Each node may be configured with a node feature vector representing the physical state of joint j at frame t. For example, the node feature vector may include a rotation value (e.g., a 4-dimensional quaternion) and a motion flag value to represent the rotation state of the corresponding joint. The motion flag value indicates whether the corresponding joint is in a motionable state in the current frame and can be set as a binary value of 0 or 1. That is, for example, for a single-axis rotational joint, the rotation value of the joint can be set by constructing an axis-angle rotation representation from the unit rotation axis and rotation angle, and then converting that value into a 4-dimensional quaternion value. For a multi-axis rotational joint, the rotation value of the joint can be set by converting the 3-dimensional rotation result, obtained by sequentially applying the rotation angles for each axis, into a 4-dimensional quaternion value.

[0110] A spatial direction edge is an edge that connects two adjacent joints within a single frame according to the structural connection relationship of the skeleton, and can be generated based on joint connection information included in the skeleton structure information. The joint connection information may include all adjacent joint pairs defined in the skeleton template, including not only connections between upper and lower joints, but also sibling relationship connections between joints belonging to the same hierarchy and cross-direction connections occurring in branching structures. For example, a spatial direction edge can be set in the form of connecting nodes (t, j1) and (t, j2) corresponding to adjacent joints j1 and j2 in frame t. A spatial direction edge can be expressed as a structural connection relationship that includes a distinguishing value for identifying the spatial direction edge, the node index (j1, j2) of the adjacent joint pair, a bone length value between the two joints, a value for connection characteristics, and the absolute value of the difference between the hierarchy level values ​​of the two joints. In this case, the distinguishing value for identifying the spatial direction edge may include a joint identifier for each of the two nodes. The skeletal length value can be set as an adjusted skeletal length by querying the reference skeletal length corresponding to adjacent joint pairs based on length information for the basic skeletal segment, and multiplying it by the scale factor for the corresponding part included in the character proportional parameter. For example, not only adjacent joint pairs such as shoulder-elbow, elbow-wrist, vertebra 1-vertebra 2, and hip-knee, but also connections occurring in branching structures, such as the skeleton's rib cage to left shoulder and rib cage to right shoulder, can be included as spatial edges. Spatial edges generated in this manner constitute a graph structure reflecting the spatial connectivity of the entire skeleton and can be utilized as foundational data for spatial message passing in diffusion-based motion generation models.

[0111] A time-direction edge is an edge connecting pairs of nodes corresponding to the same joint in frame t and frame t+1, and can be generated to reflect the characteristic that the joint state changes continuously over time. A time-direction edge can be defined as a time-series connection relationship that includes a distinguishing value to identify the time-direction edge, a previous frame node index, a subsequent frame node index, and a value representing the time interval between frames.

[0112] For example, a diffusion-based motion generation model can configure an initial state by setting the node feature vector corresponding to each node of the spatiotemporal graph as the initial noise value, so that noise injected during the forward diffusion process is gradually removed during the reverse diffusion process. Subsequently, for example, the diffusion-based motion generation model can perform a graph transformer-based conditional feature update operation at each diffusion step. The conditional feature update operation may be an operation that combines the structure of the spatiotemporal graph with text condition vectors and user characteristic context vectors to readjust the directionality in the feature space so that feature representations related to motion are refined step by step. Through this, the feature representation can be reconstructed so that feature patterns reflecting text semantics are gradually strengthened and noise components are weakened.

[0113] For example, a diffusion-based motion generation model may perform biomechanical-based adjustment operations on output feature vectors generated by graph transformer layers by utilizing biomechanical context vectors at each diffusion step. Biomechanical-based adjustment operations may be operations that constrain output feature vectors so that the generated feature representations converge to physically feasible joint movements by reflecting biomechanical conditions, such as permissible rotation ranges per joint, joint types, and proportional constraints based on joint lengths, into the feature space. For example, biomechanical-based adjustment operations may include scaling and shifting. The output feature vector is a high-dimensional representation calculated from each layer of the diffusion-based motion generation model and may be a value representing a latent representation of an intermediate state for restoring the rotational value of a specific joint at a specific time point.

[0114] For example, scaling and shifting can be performed on output feature vectors generated in the graph transformer layer of a diffusion-based motion generation model using biomechanical context vectors. Scaling may be an operation that adjusts the size (quantitative range) of the output feature vector of the graph transformer layer using axis-specific proportionality coefficients calculated from the biomechanical context vector. Shifting may be an operation that moves the reference point in the feature space by adding a correction coefficient corresponding to a reference rotation value per joint or a neutral pose to the output feature vector. The rotation sequence per joint may be a motion representation sequence composed of rotation values ​​(e.g., 4D quaternion values) in chronological order, corresponding to a preset target number of frames (T), for each joint constituting the skeleton of a 3D character.

[0115] For example, the allowable rotation range may include a minimum rotation angle and a maximum rotation angle per axis set for one or more rotation axes depending on the joint type of the joint.

[0116] According to one embodiment, the server can determine whether the rotation value of each joint included in the joint-specific rotation sequence generated by the diffusion-based motion generation model exceeds the allowable rotation range by comparing it with the minimum rotation angle and maximum rotation angle per axis of the joint.

[0117] For example, if there is a rotation value that exceeds the allowable rotation range, the server can correct the rotation sequence per joint by replacing the rotation value exceeding the allowable rotation range with the minimum rotation angle or maximum rotation angle per axis of the corresponding joint.

[0118] Specifically, for example, the server receives rotation values ​​for each joint included in a joint-specific rotation sequence output from a diffusion-based motion generation model in the form of quaternions, and calculates rotation angles for the x-axis, y-axis, and z-axis, respectively, by performing a standard quaternion-Euler transform operation to directly compare the rotation values ​​with the axis-specific rotation angles. The server can determine whether the rotation angle of each axis is within the allowable rotation range by comparing the transformed axis-specific rotation angles with the axis-specific minimum and maximum rotation angles included in the biomechanical feature vector. For example, if the rotation angle of a specific axis is smaller than the minimum rotation angle of that axis, it may be determined that the lower limit has been exceeded, and if it is larger than the maximum rotation angle, it may be determined that the upper limit has been exceeded.

[0119] For example, if there is an axis rotation angle that falls outside the allowable rotation range, the server can determine the corrected rotation angle by replacing the excess rotation angle with the limit value of the corresponding axis. For instance, if the rotation angle is smaller than the minimum rotation angle, it can be corrected by replacing it with the minimum rotation angle, and if it is larger than the maximum rotation angle, it can be corrected by replacing it with the maximum rotation angle. After determining the corrected x-axis, y-axis, and z-axis rotation angles by performing independent substitution operations for each axis, the server can generate a corrected quaternion by applying an Euler-quaternion conversion function to restore the corrected axis rotation angles back into a quaternion format. In this case, the corrected quaternion may be maintained without change if the original quaternion rotation value is within the allowable rotation range, and may be replaced with the corrected value only if it falls outside the allowable range.

[0120] By reflecting the corrected quaternion rotation values ​​into the joint rotation sequence of the corresponding frame, the server can generate a joint rotation sequence adjusted so that the joint rotation sequence generated by the diffusion-based motion generation model complies with the biomechanically acceptable rotation range. By repeatedly applying this correction process to all frames and all joints included in the entire sequence, the server can ensure that the entire 3D character motion is consistently maintained in a form that does not violate biomechanical constraints.

[0121] In step S360, the server can transmit 3D motion data generated by applying a joint-by-joint rotation sequence to skeleton structure information to the user terminal.

[0122] For example, the server can transmit 3D motion data to a user terminal by applying joint-specific rotation sequences to skeleton structure information through forward kinematics operations and skin deformation, thereby converting them into 3D character motion data. Here, forward kinematics operations are operations that calculate the joint position and orientation in the world coordinate system by accumulating rotation values ​​of each joint along a hierarchical structure from the root joint to the end joint, and may be operations for calculating the final position of each joint through the sequential application of transformation matrices that consider the joint hierarchy. Skin deformation is a deformation operation for displaced the character's surface mesh according to joint rotation, and may be calculated in a manner where a joint transformation matrix is ​​applied to each mesh vertex based on predefined skin weights for one or more joints. Skin deformation may be a deformation process designed to cause the mesh surface to naturally stretch or contract as the joints rotate.

[0123] For example, the server can receive a joint-by-joint rotation sequence calculated by a diffusion-based motion generation model as input, and first calculate the absolute position and absolute direction of each joint by performing a forward kinematics operation that cumulatively applies rotation values ​​at each frame based on joint connection information included in the skeleton structure information. Subsequently, based on the joint position and direction information calculated as a result of the forward kinematics operation, the server can generate motion data of a 3D character that reflects the displacement of the surface mesh corresponding to the joint movement by performing a mesh deformation operation that applies pre-stored skin weights to each vertex constituting the character mesh.

[0124] For example, the server may encode the generated motion data into a standard 3D file format (at least one of BVH, FBX, and GLB) or a proprietary format defined within the system, and convert it into transmission data for transmission to a user terminal. By transmitting the encoded transmission data to the user terminal, the server may provide the user terminal with the ability to play or render the 3D motion data.

[0125] FIG. 4 shows the architecture of a diffusion-based motion generation model according to one embodiment. FIG. 5 shows a block diagram of a diffusion-based motion generation model according to one embodiment. The embodiments of FIG. 4 and FIG. 5 can be combined with various embodiments of the present disclosure.

[0126] Referring to FIG. 4, the diffusion-based motion generation model (400) can receive motion semantics described by text prompts as input, combine them with motion representations pre-learned from a motion database, and then generate a 3D motion sequence according to a reverse diffusion process. For example, the diffusion-based motion generation model (400) can use text condition vectors extracted from text prompts as condition inputs and pre-learn motion patterns based on a number of learned motions collected from a motion database, thereby performing a probabilistic motion reconstruction operation that reflects the spatiotemporal motion structure corresponding to the input text. In the initial stage, the diffusion-based motion generation model (400) uses a sequence set as random noise as input to reconstruct the probabilistic distribution in the forward diffusion process, and then gradually converges the sequence in the direction from which the noise is removed by repeatedly performing the reverse diffusion process. Finally, the diffusion-based motion generation model (400) can generate a joint rotation sequence that reflects the semantic conditions of the text description and the learned motion distribution, and output renderable 3D motion data based thereon.

[0127] Referring to FIG. 5, the diffusion-based motion generation model (500) may be a neural network structure in which a plurality of graph transformer layers are sequentially stacked.

[0128] A diffusion-based motion generation model (500) can generate a sequence of joint-specific rotation values ​​by providing a text condition vector, a user characteristic context vector, a biomechanical context vector, and a spatiotemporal graph as input information to a plurality of graph transformer layers. For example, the diffusion-based motion generation model (500) can generate a spatiotemporal graph that reflects the frame-unit time structure and spatial connection relationships between joints based on skeleton structure information and a preset target number of frames, and can generate a meaningful sequence of joint rotation values ​​with noise removed in the inverse diffusion process by utilizing the text condition vector, the user characteristic context vector, and the biomechanical context vector as condition information in each graph transformer layer.

[0129] For example, each graph transformer layer can perform conditional feature update operations and biomechanics-based adjustment operations. In this case, the output feature vector of each graph transformer layer can be sequentially provided as input to the next graph transformer layer.

[0130] For example, a diffusion-based motion generation model (500) can initialize the node values ​​of a spatiotemporal graph with noise and, for each node of the spatiotemporal graph at each preset diffusion step through a plurality of graph transformer layers, perform conditional feature update operations including a time-axis self-attention operation based on time-direction edges, a spatial-attention operation based on spatial-direction edges, and a cross-attention operation using a text condition vector and a user feature context vector. Here, the node value may be a node feature vector representing the rotation state of joint j in frame t.

[0131] The temporal self-attention operation is a process that allows the node feature vector of the current frame to learn temporal correlations from the node feature vectors of previous and subsequent frames by reflecting temporal edges between nodes when nodes corresponding to the same joint exist in temporally adjacent frames. For example, a graph transformer layer can align the node feature vectors of frames t₁, t, and t+1 corresponding to the same joint j based on temporal edges, calculate temporal attention weights, and reflect them in the node feature vector of the current frame t. In this case, temporal edges can be used as information representing temporal dependencies to maintain motion continuity and natural movement changes. For example, the graph transformer layer can generate a query vector based on the node feature vector of joint j corresponding to frame t. The query vector can serve as a reference vector for selecting necessary information from temporally adjacent frames based on the joint state of the current frame. The graph transformer layer can generate two key vectors by transforming the node feature vector of the previous frame t-1 and the node feature vector of the subsequent frame t+1, respectively, corresponding to the same joint j. The key vectors can be comparison vectors used to determine how relevant each adjacent frame is to the current frame. Additionally, the graph transformer layer can generate two value vectors by transforming the node feature vector of the previous frame t-1 and the node feature vector of the subsequent frame t+1, respectively, corresponding to the same joint j. The value vectors can be information vectors used to update the joint features of the current frame after time-axis attention weights have been applied. The graph transformer layer can produce two attention scores by calculating the similarity between the query vector and the key vector generated from the previous frame, and calculating the similarity between the query vector and the key vector generated from the subsequent frame in the same manner.The two calculated attention scores can be normalized through a softmax operation and converted into two time-axis attention weights. The graph transformer layer can generate the first intermediate feature vector of frame t by multiplying the value vector generated in the previous frame by the first time-axis attention weight, multiplying the value vector generated in the subsequent frame by the second time-axis attention weight, and then adding the two results. Through such operations, the joint features of the current frame can be updated to reflect the temporal continuity of the joint features of the previous and subsequent frames.

[0132] Spatial attention operations may be operations that reflect the structural connectivity between parent-child and adjacent joints within the same frame, ensuring that the rotation value of the corresponding joint is updated in a direction consistent with the body structure. For example, a graph transformer layer can collect node feature vectors of joint j and its parent, child, and adjacent joints based on spatial edges in frame t, and calculate spatial attention weights between the node feature vectors to update the output feature vector so that structural correlations between joints are reflected. For example, the graph transformer layer can generate a query vector by receiving the node feature vector corresponding to joint j in frame t as input, and generate key and value vectors based on the node feature vectors corresponding to the adjacent joints connected to j in frame t. The graph transformer layer can generate a query vector based on the node feature vector of joint j corresponding to frame t. Here, the query vector may be a reference vector used to determine which adjacent joints the corresponding joint is structurally associated with within the same frame. The graph transformer layer can generate multiple key vectors corresponding to each adjacent joint by transforming the node feature vectors corresponding to the adjacent joints directly connected to joint j. The key vectors may serve as comparison vectors to determine how structurally the adjacent joint influences the motion direction or rotation amount of joint j. Additionally, the graph transformer layer can generate multiple value vectors by transforming the node feature vectors of the same adjacent joints. The value vectors may serve as information vectors to convey information reflecting structural correlations through spatial attention weights to the feature update of joint j. The graph transformer layer can calculate the similarity between the query vector and the multiple key vectors generated from each adjacent joint, thereby generating one attention score for each pair of adjacent joints.The calculated multiple attention scores can be normalized by a softmax operation and converted into spatial attention weights representing the influence of each adjacent joint. The graph transformer layer can generate a second intermediate feature vector for joint j by multiplying the value vector corresponding to each adjacent joint by the spatial attention weight and then performing a weighted sum over all adjacent joints. Through such operations, the output feature of joint j can be updated to reflect the relative importance of structurally valid adjacent joints within the same frame.

[0133] The cross-attention operation may be an attention-based combination operation designed to determine the semantic association between each joint node feature vector on the spatiotemporal graph, the text condition vector, and the user characteristic context vector, and to ensure that this relationship is reflected in the rotation value update process. For example, the graph transformer layer can generate a query vector based on the node feature vector corresponding to joint j in frame t of the spatiotemporal graph. The query vector may serve as a reference vector to determine which token within the text condition vector is semantically associated with joint j. The query vector serves as a criterion for performing semantic alignment by joint that reflects text meaning and user characteristics, and can act as an indicator to select the meaning of the action that the corresponding joint must perform from among the text tokens. The graph transformer layer can generate key vectors and value vectors by transforming the text condition vector and the user characteristic context vector, respectively. The key vector generated from the text condition vector may be a comparison vector representing the semantic constraints required for joint j, such as conditions related to the agent of the action, action attributes, directionality, and body part included in the text prompt. The key vector generated from the user characteristic context vector may be a vector for comparing the influence of gender characteristics, age characteristics, and region-based movement characteristics on the joint-specific representation method. Additionally, the value vector generated from the text condition vector and the user characteristic context vector may be an information vector for conveying motion representation information adjusted according to semantic conditions and user characteristics to the joint node feature update. The graph transformer layer can calculate the text condition-based attention score and the user characteristic-based attention score by calculating the similarity between the query vector and the key vector generated from the text condition vector, and the similarity between the query vector and the key vector generated from the user characteristic context vector, respectively.Multiple attention scores generated can be normalized through a softmax operation and converted into cross-attention weights representing the relative influence of text conditions and user characteristic information, respectively, during the feature update process of joint j. The graph transformer layer can generate a third intermediate feature vector of joint j by multiplying the cross-attention weights corresponding to the value vector generated from the text condition vector and the value vector generated from the user characteristic context vector, respectively, and then performing a weighted sum. Through such operations, the output feature vector of joint j can be updated to simultaneously reflect not only the motion meaning required by the input text prompt but also the movement style determined by the user characteristic metadata. Therefore, even with the same text prompt input, different motions can be generated depending on user characteristics, and cultural differences in movement observed in the real world can be reflected in the motion generation process.

[0134] For example, the graph transformer layer can update the output feature vector of joint j by merging the temporal attention result, the spatial attention result, and the cross-attention result into an output feature vector through a linear combination operation. That is, the graph transformer layer can update the output feature vector of joint j by merging the first intermediate feature vector, the second intermediate feature vector, and the third intermediate feature vector into an output feature vector through a linear combination operation.

[0135] For example, the graph transformer layer can perform scaling and shifting by calculating scale factors and shift factors for the output feature vector based on the biomechanical context vector and applying the scale factors and shift factors to the output feature vector. In this case, the scale factor refers to a ratio adjustment value applied independently to each dimensional element (channel) constituting the output feature vector, and may be a continuous value used to control the intensity of the output feature within the range of biomechanically acceptable joint rotation amounts. The scale factor can be set within a range of 0 to 1; values ​​closer to 0 strongly suppress the rotation pattern reflected by the corresponding channel, while values ​​closer to 1 maintain the rotation pattern at its original size. The shift factor refers to a linear offset applied commonly to the entire output feature vector, and is a value used to adjust the reference point so that the amount of rotation does not deviate excessively from the biomechanically acceptable reference center of rotation. The shift factor can be set within a continuous range from negative to positive, where a value of 0 means no reference point adjustment is made, a positive value means adjustment that moves the reference center of rotation in an increasing direction, and a negative value means adjustment that moves it in a decreasing direction. The magnitude of the shift factor can be set within a limited range so as not to exceed the acceptable range of fine adjustment from the basic neutral pose for each joint.

[0136] For example, a graph transformer layer can generate a channel-unit representation by determining each dimensional element constituting the output feature vector of joint j as an individual channel.

[0137] For example, a graph transformer layer can transform a biomechanical context vector through a combination of a fully coupled layer and a non-linear activation function to generate a channel-specific scale factor corresponding to the dimensionality of the output feature vector. The graph transformer layer can perform scaling by multiplying the calculated channel-specific scale factor by each channel value of the output feature vector. In this case, the scaling operation is applied in a manner that reduces or maintains the magnitude of the rotation pattern reflected by each channel of the output feature vector in accordance with biomechanical standards, and can be set so that adjustment in the direction of increase (enlargement) is not allowed. That is, by ensuring that the scale factor is not set to a value greater than 1, it is possible to prevent the calculated potential amount of rotation from being expanded into an excessive rotation that is biomechanically impossible.

[0138] For example, a graph transformer layer can transform a biomechanical context vector through a combination of a fully coupled layer and a non-linear activation function to generate channel-specific shift coefficients corresponding to the number of dimensions of the scaled output feature vector. The graph transformer layer can perform a shift adjustment operation by adding the calculated channel-specific shift coefficients to each channel value of the scaled output feature vector. For example, a multidimensional shift coefficient representing an acceptable neutral center of rotation at joint j can be calculated by referencing the joint type and base center of rotation value in the biomechanical context vector and reflecting the rotation constraint propagation relationship value and the hierarchy-based influence. The shift adjustment operation is performed by applying a linear offset to reposition the reference point of the rotation pattern reflected by each channel of the scaled output feature vector within a biomechanically valid center range, and shift coefficients that move the reference point in a direction outside the biomechanically acceptable range may be restricted from being calculated. Through such shift adjustment operations, the output feature vector of joint j can be corrected to maintain a stable center of rotation within an acceptable rotation space.

[0139] For example, a diffusion-based motion generation model (500) can generate a more refined output feature vector by removing residual noise corresponding to the current diffusion step from the output feature vector on which shift adjustment has been performed.

[0140] For example, a diffusion-based motion generation model (500) can determine the noise component remaining in the current output feature vector using a learned noise prediction function, and remove the determined noise component to generate an output feature vector that gradually converges to the rotation value for each joint.

[0141] For example, a diffusion-based motion generation model (500) can generate a gradually completed joint-by-joint rotation sequence from initial noise by repeating the process described above for a preset number of diffusion steps.

[0142] Specifically, for example, the diffusion-based motion generation model (500) can receive a shift-adjusted output feature vector and a step index value representing the current diffusion step as input, and perform a noise prediction operation to predict the residual noise component to be removed in the corresponding diffusion step. The diffusion-based motion generation model (500) can calculate the residual noise vector for each node through a neural network operation including a fully connected layer and a non-linear activation function by combining the diffusion step embedding, text condition vector, and biomechanical context vector with the output feature vector for each node on the spatiotemporal graph where the shift adjustment is completed. Here, the residual noise vector is a vector value representing the stochastic noise component remaining in the output feature vector in the current diffusion step, and may mean a value to be removed in a subsequent step.

[0143] The diffusion-based motion generation model (500) can generate an updated output feature vector by calculating the amount of noise removal by multiplying the calculated residual noise vector by a noise removal coefficient pre-set for each diffusion step, and by subtracting the amount of noise removal from the output feature vector to which shift adjustment is applied. At this time, the noise removal coefficient for each diffusion step is configured such that the amount of removal is set larger as the step index is larger and the amount of removal is set smaller as the step index is smaller, so that coarse noise removal is performed in the initial step and fine-tuned noise removal is performed as the step index progresses. If necessary, the diffusion-based motion generation model (500) can be configured to maintain probabilistic diversity by adding a small amount of Gaussian noise based on a variance value set according to the residual variance schedule for each diffusion step, so that various motion sequences can be generated even under the same conditions.

[0144] In this way, the diffusion-based motion generation model (500) receives a spatiotemporal graph representation set as random noise in the initial diffusion stage as input, and sequentially performs conditional feature update operations, scale adjustment operations, and shift adjustment operations through multiple graph transformer layers at each stage while decreasing the diffusion stage index from the initial stage value to the end stage value, and can perform noise removal operations at each stage. By performing the above iteration process as many times as a preset number of diffusion stages, the diffusion-based motion generation model (500) can gradually converge the output feature vector of the initial noise state to a joint-specific rotation value reflecting text conditions, user characteristics, and biomechanical constraints, and finally generate a joint-specific rotation sequence.

[0145] For example, the number of multiple graph transformer layers can be set to 8, and the number of preset diffusion steps can be set to 100. In this case, the conditional feature update operation, the scaling operation, and the shifting operation can each be performed 800 times, and the noise removal operation can be performed 100 times.

[0146] For example, multiple graph transformer layers are executed sequentially within a single diffusion step, and the output feature vector of the final graph transformer layer can be provided as the input to the next diffusion step as the result state of the same diffusion step.

[0147] According to one embodiment, a diffusion-based motion generation model can be trained based on a training motion data set comprising a plurality of motion data and training user feature metadata for each of the plurality of motion data.

[0148] For example, each of the multiple motion data may include data regarding frame-by-frame skeleton construction, joint-by-joint rotation sequences, and temporal motion patterns. User characteristic metadata for training may include gender values, age values, and region code values, respectively.

[0149] For example, among multiple motion data, a pair of motion data with the same combination of gender, age, and region code values ​​may be determined as a positive comparison pair, and a pair of motion data with different combinations of age and region code values ​​may be determined as a negative comparison pair. That is, a positive comparison pair may be a pair of two motion data having the same user characteristics, and a negative comparison pair may be a pair of two motion data having different user characteristics.

[0150] For example, for each of the positive comparison pairs and negative comparison pairs, the joint-specific rotation sequences included in the corresponding motion data are input into a motion style encoder, and a motion style vector can be calculated that includes the rate of change in rotation between frames, the ratio of the average rotation angle to the maximum rotation angle per joint, the cumulative sum of the change in posture between frames, and the joint range of motion representing the difference between the maximum and minimum values ​​of the rotation values.

[0151] Here, the motion style encoder may be a neural network-based embedding module for converting the temporal change pattern and spatial range of motion of a rotation sequence per joint into a feature vector. The motion style vector is a vector representing the characteristics of the motion and may include, for example, the rate of change of rotation between frames, the ratio of the average rotation angle to the maximum rotation angle per joint, the cumulative sum of the change in posture between frames, and the joint range of motion representing the difference between the maximum and minimum values ​​of the rotation values.

[0152] The inter-frame rotation rate is a value representing the average speed at which rotational values ​​change between two consecutive frames, and can serve as an indicator of motion dynamism or movement intensity. The ratio of the average rotation angle to the maximum rotation angle is a value indicating the extent to which a joint is utilized within its full range of motion, and can represent the openness or restriction of movement expression. The cumulative sum of inter-frame posture changes is a value representing the total amount of change in the body center and relative joint positions throughout the entire rotational sequence, and can reflect the mobility or instability of the motion. The joint range of motion, representing the difference between the maximum and minimum rotational values, is a value that quantifies the size of the rotational range of a single joint, and can indicate the range or stylistic characteristics of movement expression.

[0153] Specifically, for example, a motion style encoder may include a time-series feature extraction layer, a joint structure-based combining layer, a style feature compression layer, and a normalization layer.

[0154] The time series feature extraction layer may be a layer for extracting time axis change patterns inherent in the joint-specific rotation sequence. For example, the time series feature extraction layer may calculate the rate of change of rotation between frames by calculating the difference in rotation values ​​between frame t and frame t₀₁ for each joint, and generate a first intermediate feature vector that reflects the recurring movement rhythm, rotation speed increase and decrease patterns, and temporal acceleration characteristics by applying a one-dimensional convolution operation or a recurrent neural network-based feature transformation operation. In this case, the first intermediate feature vector may be a vector that represents the amount of temporal change of each joint while maintaining the frame length dimension.

[0155] A joint-unit coupling layer may be a layer for integrating temporal features extracted independently for each joint into a representation based on the overall joint movement pattern. For example, the joint-unit coupling layer can generate a second intermediate feature vector that reflects the movement intensity distribution between joints and the relative contribution of each joint by combining the first intermediate feature vectors of all joints corresponding to the same frame into a matrix form and performing a feature coupling operation based on a multilayer perceptron or linear transformation. In this case, the joint-unit coupling layer can perform the coupling operation based solely on rotation values ​​included in the input motion data and temporal features calculated based on those rotation values, without utilizing skeleton connectivity relationships or separate structural information.

[0156] The style feature compression layer may be a layer for summarizing joint-specific time-series features into a style representation of the motion as a whole. For example, the style feature compression layer may perform at least one of time-axis average pooling, time-axis max pooling, or multilayer perceptron-based feature reduction operations on a second intermediate feature vector to produce a fixed-dimensional style feature vector that reflects motion intensity, expression range, motion rhythm, and motion density. Additionally, the style feature compression layer may perform weighted combination operations by applying predefined or learned joint importance weights so that the features of joints that contribute significantly to motion style formation are relatively emphasized. In this case, the style feature vector may be a fixed-length vector independent of the number of input joints or frames.

[0157] A normalization layer may be a layer designed to ensure comparability between data by adjusting the value range of style feature vectors. For example, the normalization layer may output a motion style vector that has been corrected so that embedding distance-based similarity calculations can be performed stably by performing a standardization operation for each feature dimension so that the mean is 0 and the variance is 1, or by performing L2 normalization so that the total size of the style feature vector is 1.

[0158] The motion style vector generated in this way can be used as an expression including the rate of change in rotation between frames, the ratio of the average rotation angle to the maximum rotation angle per joint, the cumulative sum of the change in joint center posture between frames, and the joint range of motion representing the difference between the maximum and minimum rotation values, and can be utilized in the process of calculating the embedding distance with the user feature context vector, distinguishing positive comparison pairs and negative comparison pairs, and calculating contrast learning loss.

[0159] For example, user feature metadata for training can be input into a user feature encoder to generate a user feature context vector for training.

[0160] For example, the contrast learning loss can be determined such that the embedding distance between the training user feature context vector and the motion style vector decreases in the case of positive comparison pairs and increases in the case of negative comparison pairs.

[0161] The contrast learning loss can be set as a loss term that decreases for positive comparison pairs such that the sum of squared Euclidean distances between the user feature context vector and the motion style vector is less than or equal to a preset first margin value. Additionally, the contrast learning loss can be set as a loss term that increases for negative comparison pairs such that the sum of squared Euclidean distances is greater than or equal to a preset second margin value. Specifically, the contrast learning loss can be calculated as a margin-based contrastive loss function that includes a distance reduction term for positive comparison pairs and a distance expansion term for negative comparison pairs, wherein the margin value may be a threshold value to ensure the distinguishability between user features.

[0162] For example, the parameters of the user feature encoder and the diffusion-based motion generation model can be iteratively updated in a direction that minimizes the contrast learning loss. For instance, the gradient value of the contrast learning loss is propagated in the direction of the parameters of the user feature encoder and the diffusion-based motion generation model through backpropagation, thereby adjusting the weights so that the style representations of motion data with positive comparison pairs become closer to each other in the embedding space, and the style representations of motion data with negative comparison pairs become further apart. Through this, the user feature context vector can be optimized to more accurately derive the motion style corresponding to the actual user features.

[0163] Additionally, according to one embodiment, the server may receive a joint-by-joint rotation sequence of training motion data as input and further train a style classifier that determines a user characteristic category corresponding to each item among gender value, age value, and region code value. At this time, the classification loss of the style classifier may be backpropagated to a motion style encoder and a diffusion-based motion generation model to adjust weight parameters so that the motion style vector is consistent with the actual user characteristic metadata.

[0164] Specifically, for example, the server may normalize the frame-by-frame rotation values ​​of each joint in chronological order for the joint-by-joint rotation sequences included in the training motion data, and use the normalized rotation value sequences as input features for a style classifier. Here, the normalization operation may be a preprocessing operation to mitigate training instability caused by deviations in the joint-by-joint rotation range.

[0165] For example, the server can input a sequence of normalized rotation values ​​into the feature extraction layer of a style classifier to generate an intermediate representation vector containing motion style information such as joint usage ratios, changes in movement speed, stride patterns, and the direction of body center of gravity movement. The intermediate representation vector may be a vector in a high-dimensional feature space containing inherent movement characteristics based on gender, age, and region.

[0166] For example, the server can input an intermediate representation vector into a classification head to calculate prediction probabilities for multiple user characteristic categories corresponding to gender, age, and region code values, respectively. The calculated prediction probabilities may be a probability distribution indicating which user characteristic category the corresponding motion has a higher association with.

[0167] For example, the server can calculate a cross-entropy-based classification loss value based on the difference between the actual user feature metadata assigned to the training motion data and the predicted probability distribution. The classification loss value can be used to quantitatively evaluate the classification accuracy of the style classifier.

[0168] For example, the server can update the weight matrix, non-linear transformation parameters, and normalization parameters within the motion style encoder by providing the calculated classification loss as input to the backpropagation path. In this case, the backpropagation operation may be a learning process to reconstruct the representation space of the motion style vectors so that they can better separate user feature metadata.

[0169] For example, the server can pass the same classification loss to the condition embedding layer and graph transformer layer of the diffusion-based motion generation model to adjust weights, enabling the generation of motion that reflects user characteristics. In this case, the weight adjustment may be a learning procedure designed to generate different motion styles when user characteristic values ​​differ, even if the same text prompt is entered.

[0170] For example, the server can complete training so that motion style vectors reliably match actual user feature metadata by repeating the backpropagation operation and model weight adjustment process until the number of training iterations or the loss convergence condition is satisfied.

[0171] Additionally, according to one embodiment, the server calculates the amount of change in rotation value between two consecutive frames in a joint-specific rotation sequence, and if the amount of change in rotation value exceeds a time change threshold set for the corresponding joint, the corresponding frame interval may be determined as an interval in which temporal discontinuity exists.

[0172] For example, for a section where a temporal discontinuity exists, the server may generate a first condition vector containing joint-specific rotation values ​​of the frame immediately preceding the discontinuity and a second condition vector containing joint-specific rotation values ​​of the frame immediately following the discontinuity. The first condition vector may be used as condition information representing a joint state that must be maintained in the preceding time, and the second condition vector may be used as condition information representing a joint state that must be maintained in the succeeding time.

[0173] For example, the server can determine the number of intermediate frames to insert based on the length of discontinuous intervals and the magnitude of the change in rotation values, and generate a noise initial value sequence having a length corresponding to the determined number. The noise initial value sequence can be used as the initial input for a diffusion-based interpolation operation.

[0174] For example, the server can perform a conditional feature update operation on the noise initial sequence by providing the first condition vector and the second condition vector as cross-attention inputs to the graph transformer layer within the diffusion-based motion generation model at each diffusion step. The conditional feature update operation on the noise initial sequence may be an operation to align the latent feature vectors of intermediate frames so that they form a temporally natural path between the joint state of the preceding time point and the joint state of the subsequent time point. At this time, the diffusion-based motion generation model can remove residual noise corresponding to the diffusion step by applying a learned noise prediction function to the feature vectors for each intermediate frame to which the conditional feature update for the noise initial sequence has been applied. The noise removal operation may consist of an operation to gradually remove residual noise according to the diffusion schedule and converge to the final rotation value.

[0175] For example, a diffusion-based motion generation model can generate joint-specific rotation values ​​for intermediate frames to be inserted between boundary intervals by repeating conditional feature update operations and noise removal operations for a preset number of diffusion steps. For instance, the server can perform smoothing that maintains temporal continuity by inserting the generated intermediate frames into the corresponding sections of an existing joint-specific rotation sequence.

[0176] FIG. 6 is a block diagram showing the configuration of a server according to one embodiment. The embodiment of FIG. 6 can be combined with various embodiments of the present disclosure.

[0177] As illustrated in FIG. 6, the server (600) may include a processor (610), a communication unit (620), and memory (630). However, not all components illustrated in FIG. 6 are essential components of the server (600). The server (600) may be implemented with more components than those illustrated in FIG. 6, or with fewer components than those illustrated in FIG. 6. For example, according to some embodiments, the server (600) may further include a user input interface (not shown), an output unit (not shown), etc., in addition to the processor (610), the communication unit (620), and memory (630).

[0178] The processor (610) typically controls the overall operation of the server (600). The processor (610) may be equipped with one or more processors to control other components included in the server (600). For example, the processor (610) may control the communication unit (620) and the memory (630) overall by executing programs stored in the memory (630). Additionally, the processor (610) may perform the functions of the server (600) described in FIGS. 3 to 5 by executing programs stored in the memory (630).

[0179] The communication unit (620) may include one or more components that enable the server (600) to communicate with another device (not shown) and the server (not shown). The other device (not shown) may be a computing device such as the server (600) or a sensing device, but is not limited thereto. The communication unit (620) may receive user input from another electronic device or receive data stored in an external device from an external device via a network.

[0180] The memory (630) can store a program for processing and controlling the processor (610). For example, the memory (630) can store information input to the server or information received from another device via a network. Additionally, the memory (630) can store data generated by the processor (610). The memory (630) can also store information input to the server (600) or output from the server (600).

[0181] The memory (630) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk.

[0182] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0183] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0184] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0185] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0186] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.< / s> < / s>

Claims

Claim 1 A method for a server to generate motion of a 3D character using user characteristic metadata based on artificial intelligence comprises: a step of obtaining a text prompt for generating motion of a 3D character and said user characteristic metadata from a user terminal; a step of generating a text condition vector through a text encoder based on said text prompt, wherein said user characteristic metadata includes a gender value, an age value, and a region code value; a step of generating a user characteristic context vector including a value related to gender characteristics, a value related to age characteristics, and a value related to region characteristics through a user characteristic encoder based on said user characteristic metadata; a step of generating a biomechanical context vector including an allowable rotation range per joint, a joint type, and a bone length through a biomechanical encoder based on skeleton structure information; a step of generating said skeleton structure information based on said text condition vector, and inputting said text condition vector, said user characteristic context vector, and said biomechanical context vector into a diffusion-based motion generation model to generate a rotation sequence per joint;The method includes the step of transmitting three-dimensional motion data generated by applying the above joint-specific rotation sequence to the above skeleton structure information to the user terminal, wherein the diffusion-based motion generation model is trained based on a learning motion data set comprising a plurality of motion data and learning user characteristic metadata for each of the plurality of motion data, wherein among the plurality of motion data, a pair of motion data having the same combination of gender value, age value, and region code value is determined as a positive comparison pair, and a pair of motion data having different combinations is determined as a negative comparison pair, wherein for each of the positive comparison pair and the negative comparison pair, the joint-specific rotation sequence included in the corresponding motion data is input into a motion style encoder, and a motion style vector is calculated including the rotation change rate between frames, the ratio of the average rotation angle to the maximum rotation angle per joint, the cumulative sum of the posture change amount between frames, and the joint range of motion representing the difference between the maximum and minimum values ​​of the rotation value, wherein the learning user characteristic metadata is input into the user characteristic encoder, and a learning user characteristic context vector is generated, wherein the embedding distance between the learning user characteristic context vector and the motion style vector decreases in the case of the positive comparison pair, and the negative comparison pair A method in which, in the case of increasing contrast learning loss, the parameters of the user feature encoder and the diffusion-based motion generation model are repeatedly updated in a direction that minimizes the contrast learning loss. Claim 2 delete Claim 3 In claim 1, the diffusion-based motion generation model comprises a plurality of graph transformer layers, sets each joint in each frame as a node of a spatiotemporal graph based on the skeleton structure information and a preset target number of frames, generates a spatiotemporal graph by generating a spatial edge connecting pairs of nodes within the same frame, and generates a temporal edge between adjacent frames where nodes corresponding to the same joint exist, initializes the node values ​​of the spatiotemporal graph with noise, performs a conditional feature update operation for each node of the spatiotemporal graph at each preset diffusion step, the conditional feature update operation including a temporal axis self-attention operation based on the temporal edge, a spatial-attention operation based on the spatial edge, and a cross-attention operation using the text condition vector and the user feature context vector, and calculates channel-specific scale coefficients and channel-specific shift coefficients for the output feature vector of the graph transformer layer based on the biomechanical context vector, and A method for generating a joint-specific rotation sequence by applying a channel-specific scale factor and a channel-specific shift factor to an output feature vector to perform scale adjustment and shift adjustment, and repeating the conditional feature update operation, the scale adjustment and the shift adjustment together with a noise removal operation of each diffusion step a preset number of times.

Images