System and method for generating three-dimensional character motion using joint-level biomechanical range constraints based on artificial intelligence

Abstract

The embodiments present a system and method for generating motion of a 3D character using rotation range constraints based on joint-specific biomechanical information based on artificial intelligence. A method according to one embodiment receives a text prompt for generating motion of a 3D character from a user terminal, inputs the text prompt into a text encoder to generate a text condition vector, obtains skeleton configuration parameters including a character type, an active joint set, and a character proportional parameter based on the text condition vector, generates skeleton structure information including joint list information, joint connection information, joint characteristic information, and joint rotation axis information based on the skeleton configuration parameters, inputs the character proportional parameter and the skeleton structure information into a biomechanics encoder to generate a biomechanics context vector including an allowable rotation range per joint, a joint type, and a bone length, inputs the text condition vector and the biomechanics context vector into a diffusion-based motion generation model to generate a rotation sequence per joint, transmits 3D motion data generated by applying the rotation sequence per joint to the user terminal, and performs scale adjustment and shift adjustment on the output feature vector generated in the graph transformer layer of the diffusion-based motion generation model using the biomechanics context vector.

Description

Technology Field

[0001] The embodiments of the present disclosure relate to a technology for generating motion of a three-dimensional character, and to a system and method for generating motion of a three-dimensional character using rotation range constraints based on joint-specific biomechanical information based on artificial intelligence. Background Technology

[0002] 3D character motion generation technology plays a pivotal role in various industrial fields, such as games, movies, virtual reality (VR), metaverse content production, digital human creation, and robot motion planning. In particular, motion data that precisely reflects joint rotation ranges, joint structures, and biomechanical constraints is essential for character movements to be expressed naturally and physically plausibly. However, in actual industrial settings, it is very difficult to automatically generate natural and consistent 3D motion due to the diversity of character types, heterogeneity of joint degrees of freedom, differences in joint rotation ranges, and the wide range of text-based motion instructions based on user intent. These issues lead to negative problems, such as increased character animation production costs, degraded motion quality, and the generation of motion that is inconsistent with user intent.

[0003] Conventional character motion generation methods have relied on motion capture data-based retargeting, manual keyframe animation, and motion library-based blending. However, these methods were limited to specific motion patterns or failed to automatically reflect biomechanical constraints for each joint, resulting in unnatural rotation values ​​that did not match the character's body structure. In particular, the ability to automatically generate corresponding joint rotation sequences by interpreting natural language-based motion descriptions input by the user in real time has not been sufficiently implemented in conventional technology.

[0004] For example, the rotational degrees of freedom for each joint of a character skeleton vary depending on the joint type, and there are minimum and maximum allowable rotation ranges for each axis. However, existing systems often fail to accurately reflect these rotational limits for each joint; consequently, during the motion generation process, rotation angles frequently deviate into biomechanically impossible ranges, or discontinuous or abnormal movements occur that do not align with the joint axes. Furthermore, while skeleton structures possess a hierarchical structure between joints and the rotation of a specific joint affects the rotation of adjacent joints, existing technologies have failed to adequately consider this structural dependency, leading to issues with reduced motion consistency.

[0005] In addition, deep learning-based motion generation technology has recently been proposed, but since this technology relies primarily on data-based probabilistic pattern learning, it has limitations in that it is difficult to naturally reflect the body structure of a character or the biomechanical constraints of each joint. For example, when generating motion based only on text conditions, unrealistic motion may occur during the generation process, such as joint rotation values ​​exceeding the allowable rotation range or specific joints being excessively twisted.

[0006] Accordingly, there is a need for a system and method to generate motion of a 3D character using rotation range constraints based on joint-specific biomechanical information based on artificial intelligence. The problem to be solved

[0007] Embodiments of the present disclosure can provide a system and method for generating motion of a 3D character using rotation range constraints based on joint-specific biomechanical information based on artificial intelligence.

[0008] The technical problems to be solved in the embodiments are not limited to those mentioned above, and other unmentioned technical problems may be considered by those skilled in the art from the various embodiments described below. means of solving the problem

[0009] A method for a server to generate motion of a 3D character using artificial intelligence according to one embodiment comprises: receiving a text prompt for generating motion of a 3D character from a user terminal; inputting the text prompt into a text encoder to generate a text condition vector; obtaining skeleton configuration parameters including a character type, an active joint set, and a character proportional parameter based on the text condition vector; generating skeleton structure information including joint list information, joint connection information, joint characteristic information, and joint rotation axis information based on the skeleton configuration parameters; inputting the character proportional parameter and the skeleton structure information into a biomechanics encoder to generate a biomechanics context vector including an allowable rotation range per joint, a joint type, and a bone length; inputting the text condition vector and the biomechanics context vector into a diffusion-based motion generation model to generate a rotation sequence per joint; transmitting 3D motion data generated by applying the rotation sequence per joint to the user terminal; and scaling the output feature vector generated in the graph transformer layer of the diffusion-based motion generation model using the biomechanics context vector. Adjustments and shift adjustments can be performed. Effects of the invention

[0010] According to the embodiments, the present disclosure can directly reflect the meaning, intensity, and attribute information of an action instructed by a user in natural language into the motion generation process by reflecting a text condition vector generated from a text prompt into a conditional update operation of a diffusion-based motion generation model. Accordingly, unlike simple pattern-based motion generation methods, motions that accurately reflect the meaning of the text can be generated, and generative diversity of action instructions and flexibility for user-customized motion generation can be secured.

[0011] According to the embodiments, the present disclosure generates a biomechanical context vector by integrating a list of joints, a joint hierarchy, joint rotation axes, and joint types included in skeleton structure information with a per-joint allowable rotation range extracted from a biomechanical encoder, and injects this as condition information into each graph transformer layer of a diffusion-based motion generation model, thereby enabling the automatic generation of natural motion in which per-joint rotation values ​​do not exceed the physical allowable range of a single joint. Accordingly, it is possible to effectively prevent abnormal over-rotation, non-physical motion in the form of joint dislocation, and unnatural joint movements caused by joint axis misalignment, which frequently occurred in conventional neural network-based motion generation processes.

[0012] According to the embodiments, the present disclosure can provide the effect of maintaining biomechanical consistency throughout the motion generation process by performing adjustments to the output feature vector of a graph transformer layer using scale coefficients and shift coefficients calculated from a biomechanical context vector to selectively suppress feature channels that violate the physical constraints of the joints. Accordingly, even when text conditions require highly exaggerated movements, natural 3D motion that does not deviate from physical constraints can be stably output.

[0013] According to the embodiments, the present disclosure can provide a post-processing stability effect that simultaneously ensures the physical validity and visual naturalness of motion by automatically verifying whether a generated joint-by-joint rotation sequence exceeds a biomechanical tolerance, correcting rotation values ​​to within the tolerance if necessary, and applying temporal smoothing when the amount of change between frames is rapid.

[0014] The effects obtainable from the embodiments are not limited to those mentioned above, and other unmentioned effects can be clearly derived and understood by a person skilled in the art based on the detailed description below. Brief explanation of the drawing

[0015] The accompanying drawings, included as part of the detailed description to aid in understanding the embodiments, provide various embodiments and explain the technical features of the various embodiments together with the detailed description. FIG. 1 is a diagram showing the configuration of an electronic device according to one embodiment. FIG. 2 is a diagram showing the configuration of a program according to one embodiment. FIG. 3 shows a flowchart of a method for a server to generate motion of a 3D character using artificial intelligence according to one embodiment. FIG. 4 shows the architecture of a diffusion-based motion generation model according to one embodiment. FIG. 5 is a block diagram showing the configuration of a diffusion-based motion generation model according to one embodiment. FIG. 6 is a block diagram showing the configuration of a server according to one embodiment. Specific details for implementing the invention

[0016] The following embodiments are combinations of the components and features of the embodiments in a predetermined form. Each component or feature may be considered optional unless otherwise explicitly stated. Each component or feature may be implemented in a form not combined with other components or features. Additionally, various embodiments may be constructed by combining some components and / or features. The order of operations described in various embodiments may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment.

[0017] In the description of the drawings, procedures or steps that could obscure the essence of the various embodiments were not described, nor were procedures or steps that can be understood by a person of ordinary knowledge in the relevant technical field described.

[0018] Throughout the specification, when a part is described as "comprising" or "including" a component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part," "...unit," and "module" as used in the specification refer to a unit that performs at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software. Additionally, "one (a or an)," "one," "the," and similar related terms may be used in the context describing various embodiments (particularly in the context of the following claims) in both singular and plural forms, unless otherwise indicated in the specification or clearly contradicted by the context.

[0019] Hereinafter, embodiments according to various examples will be described in detail with reference to the accompanying drawings. The detailed description disclosed below, together with the accompanying drawings, is intended to describe exemplary embodiments of various examples and is not intended to represent the only embodiment.

[0020] In addition, specific terms used in various embodiments are provided to aid in understanding the various embodiments, and the use of such specific terms may be modified in other forms within the scope of not departing from the technical concept of the various embodiments.

[0021] FIG. 1 is a diagram showing the configuration of an electronic device according to one embodiment.

[0022] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)). The electronic device (101) may be referred to as a client, terminal, or peer.

[0023] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0024] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model.

[0025] An artificial intelligence model can be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. Artificial intelligence models may include software structures, either additionally or as a substitute, in addition to hardware structures.

[0026] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0027] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0028] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0029] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0030] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0031] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0032] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensing device, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0033] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0034] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0035] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0036] The camera module (180) can capture still images and images. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0037] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0038] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0039] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0040] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0041] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0042] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0043] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0044] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0045] The server (108) is connected to an electronic device (101) and can provide services to the connected electronic device (101). Additionally, the server (108) may proceed with the membership registration process, store and manage various information of users who have registered as members accordingly, and provide various purchase and payment functions related to the service. Furthermore, the server (108) may share execution data of service applications running on each of multiple electronic devices (101) in real time so that services can be shared among users. In terms of hardware, this server (108) may have the same configuration as a conventional web server or service server. However, in terms of software, it may include program modules that perform various functions and are implemented through any language such as C, C++, Java, Python, Golang, Kotlin, etc. Additionally, the server (108) generally refers to a computer system that is connected to an unspecified number of clients and / or other servers through an open computer network such as the Internet, receives requests for task execution from clients or other servers, and derives and provides the results of the task, as well as computer software (server program) installed for this purpose. Furthermore, the server (108) should be understood as a broad concept that includes, in addition to the aforementioned server program, a series of application programs running on the server (108) and, in some cases, various databases (DB: Database, hereinafter referred to as "DB") built internally or externally. Accordingly, the server (108) classifies membership registration information and various information and data regarding games, stores them in the DB, and manages them; such DB can be implemented internally or externally of the server (108).Additionally, the server (108) can be implemented using various server programs provided for operating systems such as Windows, Linux, UNIX, and Macintosh on general server hardware. Representative examples include IIS (Internet Information Server) used in Windows environments and CERN, NCSA, APPACH, and TOMCAT used in UNIX environments to implement web services. Additionally, the server (108) may be linked with an authentication system and a payment system for user authentication of the service or purchase payment related to the service.

[0046] The first network (198) and the second network (199) refer to a connection structure capable of exchanging information between each node, such as terminals and servers, or a network connecting a server (108) and electronic devices (101, 104). The first network (198) and the second network (199) include, but are not limited to, the Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), 3G, 4G, LTE, 5G, Wi-Fi, etc. The first network (198) and the second network (199) may be closed types such as LAN, WAN, etc., but it is preferable that they be open types such as the Internet. The Internet refers to a global open computer first network (198) and second network (199) structure that provides protocols such as TCP / IP protocol, TCP, and UDP (user datagram protocol), and various services existing in the upper layer, namely HTTP (HyperText Transfer Protocol), Telnet, FTP (File Transfer Protocol), DNS (Domain Name System), SMTP (Simple Mail Transfer Protocol), SNMP (Simple Network Management Protocol), NFS (Network File Service), and NIS (Network Information Service).

[0047] A database may have a general data structure implemented in the storage space (hard disk or memory) of a computer system using a database management program (DBMS). A database may have a data storage form that allows for the free retrieval (extraction), deletion, editing, and addition of data. A database may be implemented to suit the purpose of an embodiment of the present disclosure using a relational database management system (RDBMS) such as Oracle, Informix, Sybase, and DB2, an object-oriented database management system (OODBMS) such as Gemston, Orion, and O2, and an XML native database such as Excelon, Tamino, and Sekaiju, and may have appropriate fields or elements to achieve its functions.

[0048] FIG. 2 is a diagram showing the configuration of a program according to one embodiment.

[0049] FIG. 2 is a block diagram (200) illustrating a program (140) according to various embodiments. According to one embodiment, the program (140) may include an operating system (142), middleware (144), or an application (146) executable on the operating system (142) for controlling one or more resources of an electronic device (101). The operating system (142) may include, for example, Android™, iOS™, Windows™, Symbian™, Tizen™, or Bada™. At least some of the programs (140) may be preloaded into the electronic device (101) at manufacturing time, for example, or downloaded or updated from an external electronic device (e.g., electronic device (102 or 104), or server (108)) when used by a user. All or part of the program (140) may include a neural network.

[0050] The operating system (142) can control the management (e.g., allocation or reclamation) of one or more system resources (e.g., processes, memory, or power) of the electronic device (101). The operating system (142) may additionally or substantially include one or more driver programs for driving other hardware devices of the electronic device (101), e.g., an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197).

[0051] Middleware (144) may provide various functions to an application (146) so that functions or information provided from one or more resources of an electronic device (101) can be used by the application (146). Middleware (144) may include, for example, an application manager (201), a window manager (203), a multimedia manager (205), a resource manager (207), a power manager (209), a database manager (211), a package manager (213), a connectivity manager (215), a notification manager (217), a location manager (219), a graphics manager (221), a security manager (223), a call manager (225), or a voice recognition manager (227).

[0052] The application manager (201) can, for example, manage the life cycle of the application (146). The window manager (203) can, for example, manage one or more GUI resources used on the screen. The multimedia manager (205) can, for example, identify one or more formats required for the playback of media files and perform encoding or decoding of the corresponding media files among the media files using a codec that matches the selected corresponding format. The resource manager (207) can, for example, manage the source code of the application (146) or the memory space of the memory (130). The power manager (209) can, for example, manage the capacity, temperature, or power of the battery (189) and, using the relevant information, determine or provide relevant information required for the operation of the electronic device (101). According to one embodiment, the power manager (209) can interact with the BIOS (basic input / output system) (not shown) of the electronic device (101).

[0053] The database manager (211) can, for example, create, search, or modify a database to be used by the application (146). The package manager (213) can, for example, manage the installation or update of the application distributed in the form of a package file. The connectivity manager (215) can, for example, manage a wireless or direct connection between the electronic device (101) and an external electronic device. The notification manager (217) can, for example, provide a function to notify the user of the occurrence of a specified event (e.g., an incoming call, a message, or an alarm). The location manager (219) can, for example, manage location information of the electronic device (101). The graphics manager (221) can, for example, manage one or more graphic effects or related user interfaces to be provided to the user.

[0054] The security manager (223) may, for example, provide system security or user authentication. The telephony manager (225) may, for example, manage voice call functions or video call functions provided by the electronic device (101). The voice recognition manager (227) may, for example, transmit user voice data to the server (108) and receive from the server (108) a command corresponding to a function to be performed on the electronic device (101) based on at least part of the voice data, or text data converted based on at least part of the voice data. According to one embodiment, the middleware (244) may dynamically delete some existing components or add new components. According to one embodiment, at least part of the middleware (144) may be included as part of the operating system (142) or implemented as separate software different from the operating system (142).

[0055] The application (146) may include, for example, a home (251), a dialer (253), an SMS / MMS (255), an IM (instant message) (257), a browser (259), a camera (261), an alarm (263), a contact (265), a voice recognition (267), an email (269), a calendar (271), a media player (273), an album (275), a watch (277), a health (279) (e.g., measuring biometric information such as exercise volume or blood sugar), or an environmental information (281) (e.g., measuring atmospheric pressure, humidity, or temperature information). According to one embodiment, the application (146) may further include an information exchange application (not shown) capable of supporting information exchange between the electronic device (101) and an external electronic device. The information exchange application may include, for example, a notification relay application configured to transmit information (e.g., a call, a message, or an alarm) designated to an external electronic device, or a device management application configured to manage the external electronic device. The notification relay application may transmit notification information corresponding to a designated event (e.g., receiving mail) generated in another application of the electronic device (101) (e.g., an email application (269)) to the external electronic device. Additionally or alternatively, the notification relay application may receive notification information from the external electronic device and provide it to the user of the electronic device (101).

[0056] A device management application can control the power (e.g., turn-on or turn-off) or function (e.g., brightness, resolution, or focus) of an external electronic device or a part of its components (e.g., a display module or camera module of the external electronic device) that communicates with the electronic device (101). The device management application can additionally or substantially support the installation, deletion, or updating of applications running on the external electronic device.

[0057] Throughout this specification, the terms neural network, neural network, and network function may be used interchangeably. A neural network may consist of a set of interconnected computational units, which may generally be referred to as “nodes.” These “nodes” may also be referred to as “neurons.” A neural network is composed of at least two nodes. The nodes (or neurons) constituting neural networks may be interconnected by one or more “links.”

[0058] In a neural network, two or more nodes connected via links can form a relative relationship between an input node and an output node. The concepts of input and output nodes are relative; any node in an output node relationship with respect to one node may be in an input node relationship with respect to another node, and vice versa. As previously mentioned, the input node versus output node relationship can be generated based on links. One or more output nodes may be connected to a single input node via links, and vice versa.

[0059] In a relationship between an input node and an output node connected through a single link, the value of the output node can be determined based on data input to the input node. Here, the nodes interconnecting the input node and the output node may have weights. The weights may be variable and may be varied by a user or an algorithm to enable the neural network to perform the desired function. Here, the edges or links interconnecting the input node and the output node have weights that can be variably applied by a user or an algorithm to enable the neural network to perform the desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node may determine its value based on the values ​​input to the input nodes connected to the output node and the weights set on the links corresponding to each input node.

[0060] As described above, a neural network is formed in which two or more nodes are interconnected through one or more links to create input-output node relationships within the network. The characteristics of a neural network can be determined by the number of nodes and links within the network, the relationships between the nodes and links, and the weight values ​​assigned to each link. For example, if two neural networks exist with the same number of nodes and links but different weight values ​​between the links, the two neural networks can be recognized as being different from each other.

[0061] FIG. 3 shows a flowchart of a method for a server to generate motion of a 3D character using artificial intelligence according to one embodiment. The embodiment of FIG. 3 can be combined with various embodiments of the present disclosure.

[0062] Referring to FIG. 3, in step S310, the server can receive a text prompt from a user terminal for generating motion of a 3D character.

[0063] The server may be a server that generates motion of a 3D character using artificial intelligence and provides the generated 3D motion data to a user terminal. For example, the server may receive a text prompt from a user terminal, obtain skeleton structure information based on the result of encoding the text prompt, and generate biomechanical information based on the skeleton structure information. The server may input text-based conditions and biomechanical-based conditions into a diffusion-based motion generation model to generate a joint-by-joint rotation sequence, and transmit 3D motion data to the user terminal to which the generated joint-by-joint rotation sequence has been applied. The text prompt may be a sentence describing the movement of a 3D character entered by the user in natural language. The skeleton structure information may be structural information representing the joint configuration and connection relationships of the character. The biomechanical information may be information representing physical constraints applied to the joint movements of the skeleton. The diffusion-based motion generation model may be an artificial intelligence model that receives a specific condition vector as input and generates a joint-by-joint rotation sequence through a probabilistic transformation process. For example, the server may include the server (108) of FIG. 1.

[0064] A user terminal may be a user terminal that generates motion of a 3D character. For example, the user terminal may transmit a text prompt containing a sentence describing the motion of a 3D character to a server, and may display 3D motion data received from the server on a screen or save it for animation production. For example, an application for performing motion generation of a 3D character may be pre-installed on the user terminal. For example, the user terminal may transmit a text prompt containing a sentence describing the motion of a 3D character to a server through a pre-installed application, and may display 3D motion data received from the server on a screen. Through this, motion for a specific character type can be easily generated through a natural language-based intuitive interface, and 3D motion data provided by the server can be utilized. For example, the user terminal may include the electronic device (101) of FIG. 1. For example, the text prompt may include at least one of the subject performing the motion, the type of motion, the attributes of the motion, the spatial direction of the motion, the execution part, and the body shape or proportional representation. For example, the text prompt may include at least one sentence. At this time, the server can use the entered text prompt as input for generating a text condition vector to be utilized in a diffusion-based motion generation model.

[0065] In step S320, the server can input a text prompt into a text encoder to generate a text condition vector.

[0066] A text encoder may be an encoding module for converting sentence structure, semantic context, and action-related intent contained in a text prompt input from a user terminal into an AI-based high-dimensional vector representation. For example, the text encoder may include at least one of a plurality of pre-trained language encoders such as BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (Robustly Optimized BERT Pretraining Approach), T5 (Text-to-Text Transfer Transformer), and GPT (Generative Pre-trained Transformer). For example, the text encoder may produce embeddings that reflect contextual relationships between words within a sentence.

[0067] For example, a text encoder can divide a text prompt into word- or subword-unit tokens and, for each token, calculate a pre-trained word embedding and a positional embedding indicating the order of the tokens to construct an input embedding sequence in which the token order is preserved. For example, the input embedding sequence can be generated in the form of a vector matrix that simultaneously includes the semantic features and positional information of each token within the sentence.

[0068] For example, a text encoder can extract contextual representations by inputting an input embedding sequence into a multi-layer transformer encoding layer. The transformer encoding layer includes multiple self-attention and feed-forward operations and can calculate attention weights to model semantic correlations between tokens within a sentence. For instance, the self-attention operation can generate contextual representations that emphasize action semantics, attributes, intensity, and directionality by assigning higher weights to key words directly related to the action, such as “lift,” “quickly,” “left arm,” and “rotate.” For instance, the transformer encoding layer can generate a global semantic representation for the entire sentence based on the contextual representation of each token and can progressively generate representations through multiple layers.

[0069] For example, a text encoder can generate fixed-length text condition vectors used in the conditional operations of a motion generative model by integrating sentence-level representation vectors produced in the final layer using mean pooling or special token-based aggregation methods. Special token-based aggregation may be a method that uses the hidden state of a special token placed at the beginning or a specific position of the input sequence as a representative vector for the entire sentence. That is, the [CLS] token of BERT-type models, and the T5 / GPT-type model <s>Or EOS token, RoBERTa's <s>It means using the final hidden vector of a token, which is designed to aggregate sentence-level meaning like a token, directly as a sentence representation. The text condition vector may be a vector that reflects semantic information regarding the agent performing the action, the type of action (e.g., jump, turn, move), the attributes of the action (e.g., fast, slow, smooth), the spatial direction of the action (e.g., forward, upward), the part of the action (e.g., arm, leg, torso), and body shape and proportional representations (e.g., large, small, long, etc.).

[0070] In step S330, the server can obtain skeleton configuration parameters including character type, set of active joints, and character proportional parameters based on a text condition vector.

[0071] Skeleton configuration parameters may refer to default settings that define the character type, joint configuration, and hierarchical relationships necessary for generating a skeleton structure based on the semantic information of the text condition vector. For example, skeleton configuration parameters may include the character type, set of active joints, and character proportion parameters.

[0072] For example, the server can determine one of multiple types through a pre-trained character type classification module based on text condition vectors. For instance, the character type may include a value representing one of the following: Humanoid, Quadruped, Avian, Aquatic, Arthropod, Creature, or Mechanical. For instance, if the text condition vector includes expressions such as "a person" or "a male," it may be determined as Humanoid; if it includes expressions such as "a dog runs" or "a horse jumps," it may be determined as Quadruped; and if it includes expressions such as "a robot arm" or "a mechanical arm rotates," it may be determined as Mechanical. The character type can serve as a high-level parameter that determines the type of base skeleton template to be referenced in a subsequent step. Additionally, the server provides a text condition vector to the input layer of the character type classification module, and the character type classification module can convert the text condition vector into a character type-specific feature space through neural network operations including multiple fully connected layers and non-linear activation functions, and then calculate a score value for each character type in the final output layer. The server can calculate a probability distribution for each character type by applying a softmax operation to the score value, and determine the character type with the maximum probability in the calculated probability distribution as the character type corresponding to the text meaning. Here, the character type classification module may be a module trained using a supervised learning method with a training dataset consisting of pre-collected text prompts and corresponding character type labels.

[0073] For example, the server can extract action directives (e.g., "lift," "jump," "rotate," "run") and expressions related to body parts (e.g., "arm," "leg," "waist," "wing") from text condition vectors. The server can compare the extracted expressions with a pre-configured action-joint mapping table corresponding to a character type to determine the joints required to perform the corresponding action as the set of active joints. For example, the expression "lift arm" can be mapped to the shoulder joint, elbow joint, and wrist joint in a pre-configured action-joint mapping table corresponding to a humanoid, and the expression "spread wing" can be mapped to the scapular joint and the primary and secondary joints of the wing in a pre-configured action-joint mapping table corresponding to a bird-like type. Additionally, for example, the server can pre-define a set of keywords corresponding to action directives and a set of keywords corresponding to expressions related to body parts, and pre-configure reference keyword embeddings for each set of keywords. The server calculates a normalized inner product-based similarity value, such as cosine similarity, between each of multiple reference keyword embeddings and a text condition vector, and can select keywords corresponding to reference keyword embeddings whose similarity value is above a preset threshold as candidate action directives and candidate body part expressions. For example, if a keyword is detected with a similarity value above a preset threshold to reference keyword embeddings corresponding to "lift," "jump," "rotate," and "run," the corresponding word can be extracted as an action directive; and if a keyword is detected with a similarity value above a preset threshold to reference keyword embeddings corresponding to "arm," "leg," "waist," and "wing," the corresponding word can be extracted as a body part-related expression. Subsequently, the server can refer to an action-joint mapping table corresponding to the character type and determine a matching set of joints using the combination of the extracted action directive and the body part-related expression as a key.A motion-joint mapping table may be a data structure defined such that a list of joint identifiers required to perform a corresponding motion corresponds to a combination of character type, motion directive, and expressions related to body parts. For example, the item corresponding to "raise arm" may store the shoulder joint, elbow joint, and wrist joint identifiers from a preset motion-joint mapping table corresponding to the humanoid type, and the item corresponding to "spread wing" may store the scapular joint, wing primary joint, and wing secondary joint identifiers from a preset motion-joint mapping table corresponding to the bird type. The server may determine the set of activated joints by repeatedly querying the motion-joint mapping table for each of the multiple motion directives and expressions related to body parts extracted from the text condition vector, and by integrating the joint identifiers obtained as a result of the query through a union operation.

[0074] For example, the server can determine expressions related to body shape and proportions included in the text condition vector (e.g., "long arms," ​​"short legs," "large robot," "dwarf body") to calculate character proportion parameters for adjusting the bone length of the basic skeleton template. Additionally, for example, the server can pre-set a set of keywords corresponding to body shape expressions and a set of keywords representing proportion strength, and pre-set reference keyword embeddings for each keyword set. The server can calculate a normalized inner product-based similarity value, such as cosine similarity, between each of the multiple reference keyword embeddings and the text condition vector, and by determining the reference keyword embeddings whose similarity value is greater than or equal to a pre-set threshold, select the body shape expression and proportion strength expression corresponding to those reference keyword embeddings, respectively. The server generates a body proportion vector by combining reference keyword embeddings corresponding to the selected body shape expression and proportional strength expression, and inputs this body proportion vector, along with a value representing the character type, into a multilayer perceptron-based scale factor calculation module to calculate scale factors for each of multiple body parts and the overall scale factor. The server sets the scale factors for each of the multiple body parts and the overall scale factor as character proportion parameters and can adjust the proportions of the skeleton by multiplying the scale factor by a base value corresponding to the length of each bone segment included in the basic skeleton template. At this time, the multiple body parts may be set differently depending on the character type. That is, the value representing the character type may be an input value intended to reflect the fact that the calculation rules for the scale factor vary depending on the different morphological characteristics of the character structure, such as humanoid, quadrupedal, avian, aquatic, or mechanical types.

[0075] In step S340, the server can generate skeleton structure information including joint list information, joint connection information, joint characteristic information, and joint rotation axis information based on skeleton configuration parameters.

[0076] Skeleton structure information may be information representing the structure of a skeleton that reflects the semantic information of a text condition vector. For example, skeleton structure information may include joint list information to identify all joints constituting the skeleton, joint connection information indicating the connection relationships between joints, joint characteristic information indicating the characteristics of the degrees of freedom of movement of each joint, and joint rotation axis information indicating the possible rotational directions of each joint.

[0077] Specifically, for example, the server may obtain a basic skeleton template corresponding to a character type from among a plurality of basic skeleton templates based on the character type included in the skeleton configuration parameters. The basic skeleton template is data that standardizes the anatomical or mechanical structure of the corresponding character type, and may include a list of all joint identifiers, connection relationships between joints (tree-based connection structures with parent and child connections and adjacent connection structures including branching structures), joint type and basic rotation axis information for each joint, and length information for basic bone segments. For example, the basic skeleton template may have different structures depending on the character type.

[0078] For example, joint list information may be a set of multiple joint identifiers to identify each joint constituting the skeleton. Each joint included in the joint list information represents a single joint point on the skeleton and can serve as a basic unit for forming a set of nodes when constructing a spatiotemporal graph. For example, joint list information for a humanoid character may include multiple joint identifiers such as the pelvis, multiple levels of spinal joints, thoracic joints, neck joints, head joints, left and right shoulder joints, left and right elbow joints, left and right wrist joints, left and right hip joints, left and right knee joints, and left and right ankle joints. For example, the server may refer to the character type included in the skeleton configuration parameters to obtain a basic skeleton template corresponding to that character type, and set the entire list of joint identifiers stored in the basic skeleton template as the joint list information. In this case, the set of active joints included in the skeleton configuration parameters may be used as information to set the joints directly involved in performing actions in the joint list. For example, the server can determine a matching joint item by comparing each joint identifier of the set of active joints with the joint list information, and add an activation flag value to the joint list information indicating that the determined joint item is directly involved in performing an action.

[0079] For example, joint connection information may be information that hierarchically represents the connection relationships between the joints constituting the skeleton. Joint connection information may refer to an extended connection structure that includes a hierarchical connection form in which the entire structure of the skeleton extends in upward and downward directions relative to the root joint, while simultaneously encompassing all connection relationships between adjacent joint pairs (branch connections and connections between sibling nodes) defined in the base skeleton template. This joint connection information can subsequently be used as reference information for generating spatial edges of the spatiotemporal graph. For example, the server may refer to the connection relationships between joints stored in the base skeleton template corresponding to the character type, and for each joint included in the joint list information, reflect all connection relationships set on the base skeleton template to set the adjacent joints of that joint as joint connection information.

[0080] For example, joint connection information may include a joint identifier to identify each joint, a set of joint identifiers for adjacent joints directly connected to the corresponding joint, a value for the connection characteristic of each connection relationship, and a hierarchy level value indicating the hierarchical depth from the root joint. The set of joint identifiers for adjacent joints directly connected to the corresponding joint may include joint identifiers for joints located in a higher direction than the corresponding joint, joints located in a lower direction than the corresponding joint, and joints branching out at the same hierarchy depth. The hierarchy level value is an integer value representing a relative position within the entire skeleton, and may have a structure where the topmost joint of the skeleton is set to 0 and increases by 1 as one moves down to lower joints. For example, the depth value for the pelvic joint may be set to 0, the first vertebral joint to 1, and the shoulder joint to 3. The hierarchy level value has an integer range greater than or equal to 0 and may be defined within a range of 0 to 20 depending on the structure of the entire skeleton. Additionally, the joint connection information may include values ​​for the connection characteristic for each pair of joint identifiers. The value for the connection characteristic is used to identify the type of connection method between joints and can be set to one of a rotational center connection, a linear movement center connection, or a fixed connection. A rotational center connection indicates a structure where two joints perform relative rotation around one or more rotation axes, while a linear movement center connection indicates a structure where joints can move relatively only in a specific linear direction. A fixed connection indicates a structure where two joints do not perform relative rotation or movement. The value for this connection characteristic can be represented as a predefined string value or an integer value such as 0, 1, or 2.

[0081] At this time, the server can maintain a tree structure in which the connection paths necessary for performing actions are not broken by ensuring that the joints included in the active joint set and the joints connected to them as upper and lower levels are all included in the hierarchical structure. Additionally, the server can include character proportion parameters and length information for basic bone segments in the joint connection information. The length information for basic bone segments may include values ​​representing the length of the basic bone segment between joints for each pair of joint identifiers.

[0082] For example, joint characteristic information may be information representing the characteristics of the degrees of freedom of motion allowed by each joint. The server may query the joint-specific joint types stored in the base skeleton template and set the joint type corresponding to each joint identifier as joint characteristic information. The joint type may be set to one of a plurality of preset characteristic values. For example, the plurality of characteristic values ​​may include values ​​for single-axis rotational joints, universal joints, multi-axis rotational joints, and sliding joints, respectively. A single-axis rotational joint may refer to a joint that allows rotation in only one direction relative to a single axis of rotation. A universal joint may refer to a joint that allows independent rotational movement in two directions relative to two mutually orthogonal axes of rotation. A multi-axis rotational joint may refer to a joint that allows rotational movement in all directions of three-dimensional space relative to three axes of rotation. A sliding joint may refer to a joint that allows movement in only one linear direction. Joint characteristic information can subsequently be used as reference information to select the form of the constraint model to be applied when the biomechanical encoder calculates the allowable rotation range per joint.

[0083] For example, joint rotation axis information may include the number of axes through which each joint can rotate and a vector (axis-specific direction vector) representing the direction of each axis. In other words, joint rotation axis information may be information indicating the direction of the axes through which each joint can rotate. Joint rotation axis information is a value used to define a reference axis indicating which direction each joint performs rotational motion relative to in a specific coordinate system; depending on the joint type, either a single axis or multiple axes may be set. For example, the elbow joint, which is a single-axis rotational joint type, may be set to have a single axis vector representing the direction of the rotation axis where the forearm bends relative to the upper arm. In this case, the rotation axis may be set as a unit vector representing the flexion-extension direction in a 3D coordinate system based on the joint center. This single unit vector represents the precise rotational direction in the space where the joint can rotate and can subsequently be used as a reference axis to express the amount of joint rotation in degrees. The shoulder joint, which is a multi-axis rotational joint type, may have multiple axis vectors to represent horizontal rotation, vertical rotation, and rotation, respectively. At this time, multiple rotation axes can be defined as three-dimensional unit vectors corresponding to each degree of freedom. For example, the unit vector in the x-axis direction may represent the rotation direction corresponding to flexion / extension, the unit vector in the y-axis direction may represent the rotation direction corresponding to abduction / adduction, and the unit vector in the z-axis direction may represent the rotation direction corresponding to internal / external rotation. For example, the server may query the basic rotation axis information for each joint included in the basic skeleton template corresponding to the character type, and set one or more rotation axis vectors corresponding to the joint according to the joint type of each joint included in the joint list information as the joint rotation axis information of the skeleton structure information.

[0084] For example, the server can integrate joint list information, joint connection information, joint characteristic information, and joint rotation axis information to determine skeleton structure information based on skeleton configuration parameters.

[0085] In step S350, the server can input skeleton structure information into a biomechanical encoder to generate a biomechanical context vector including per-joint allowable rotation range, joint type, and bone length.

[0086] The biomechanical encoder may be a neural network-based encoding module for quantifying joint-level biomechanical constraints based on skeleton structure information. The biomechanical context vector is a vector that numerically represents the degrees of freedom, rotational limits per axis, and structural proportions of each joint, and can subsequently be used as a constraint to ensure that only physically valid motions are generated during the joint rotation sequence sampling process of a diffusion-based motion generation model.

[0087] For example, a biomechanical encoder may include a joint structure parsing layer, a constraint binding layer, a skeletal information generation layer, a structural dependency encoding layer, and a normalization layer. In this case, the joint structure parsing layer can generate joint-unit input tokens based on joint list information, joint connection information, joint characteristic information, and joint rotation axis information. Specifically, the joint structure parsing layer assigns a unique index to each joint identifier included in the joint list information and sets the joint index, joint identifier, and joint type for each joint in the identification fields of the input token. The joint structure parsing layer can refer to the joint connection information to extract parent joint identifiers and child joint identifiers for each joint and set these values ​​in the parent joint identifier field and child joint identifier field of the joint input token. Additionally, the joint structure parsing layer can set a hierarchy level value as a hierarchy level field for each joint. The joint structure parsing layer can include connection characteristic values ​​included in the joint connection information in the joint input token. The joint structure parsing layer can query the axis-specific direction vectors corresponding to each joint from the joint rotation axis information and record the number of rotation axes and the axis-specific direction vector index representing the direction of each axis in the rotation axis field of the joint input token. In this way, the joint input token may be a joint-unit representation including a joint index, joint identifier, joint type, parent joint identifier, child joint identifier, hierarchy level value, values ​​for connection characteristics for each of the parent joint identifier and child joint identifier, and the axis-specific direction vector index.

[0088] The constraint binding layer can identify the axes to which each joint can rotate based on the joint type and axis-specific direction vector index included in the joint input token, and look up the axis-specific minimum and maximum rotation angles from the character type-specific biomechanical constraint table and add them to the joint input token. For example, a joint determined to be a single-axis rotational joint may be determined by the minimum and maximum rotation angles for one axis, a joint determined to be a universal joint may be determined by the minimum and maximum rotation angles for each of the two axes, and a joint determined to be a multi-axis rotational joint may be determined by the minimum and maximum rotation angles for each of the three axes. In this case, for example, a sliding joint may be determined by an empty value, i.e., a value of 0, because it is a joint that cannot rotate.

[0089] Specifically, for example, the constraint binding layer can map axis-specific rotation constraint items by activating axis items that match the axis direction vector index based on the joint type and axis-specific direction vector index included in the joint input token. The constraint binding layer can look up the axis-specific minimum and maximum rotation angles in a predefined biomechanical constraint table using a key value combining the joint type and the character type, and can independently assign the minimum and maximum rotation angles to each rotation axis activated in the parsed joint input token. For example, for a joint determined to be a single-axis rotational joint, the minimum and maximum rotation angles can be calculated for one axis corresponding to the flexion and extension axis; for a universal joint, the minimum and maximum rotation angles can be calculated for the flexion and extension axis and the abduction and adduction axis, respectively; and for a multi-axis rotational joint, the minimum and maximum rotation angles can be independently calculated for all three axes (flexion and extension, abduction and adduction, pronation and supination). Additionally, if a joint is determined to be non-rotatable, all axis-specific rotation angle values ​​can be set to 0. In this way, the constraint coupling layer can add the calculated axis-specific rotation angle information to the joint-specific input tokens to generate joint-unit constraint tokens with assigned rotation constraint values, and transmit them to the skeletal information generation layer.

[0090] The skeletal information generation layer can calculate the skeletal length based on the upper and lower joints of each joint using the joint unit constraint token and add it to the joint input token. The skeletal information generation layer can determine upper-lower joint pairs by referencing the upper joint identifier and lower joint identifier of the joint unit constraint token. Subsequently, the skeletal information generation layer can determine the basic skeletal length value corresponding to the upper-lower joint pair from the length information for the basic skeletal segment included in the joint connection information, and calculate the skeletal length by multiplying the corresponding basic skeletal length value by the scale factor corresponding to the upper-lower joint pair from the character proportional parameter included in the joint connection information. For example, if the basic skeletal length between the upper arm and the forearm is defined as 1.0 units, and the scale factor calculated by the character proportional parameter is 1.15, the upper arm-forearm skeletal length can be adjusted to 1.15 units. The skeletal information generation layer can complete structural proportional information indicating how far the corresponding joint is from the upper joint by recording the calculated skeletal length in the skeletal length field of the joint unit constraint token. The skeletal information generation layer can pass joint-unit structure tokens with added skeletal lengths to the structural dependency encoding layer.

[0091] The structural dependency encoding layer can generate biomechanical feature vectors that reflect inter-joint structural dependencies based on joint-unit structure tokens. For example, the structural dependency encoding layer can calculate the effect of rotational constraints on the range of motion of surrounding joints by referencing values ​​for upper joint identifiers, lower joint identifiers, bone lengths, hierarchy level values, and connectivity characteristics included in the joint-unit structure tokens. For example, the structural dependency encoding layer can calculate rotational constraint propagation relationship values ​​by referencing upper joint identifiers and lower joint identifiers representing the connectivity relationship between upper and lower joints, and by comparing the minimum and maximum rotation angles by axis of the upper joint with the minimum and maximum rotation angles by axis of the lower joint based on upper-lower joint pairs. The rotational constraint propagation relationship value is a value normalized from 0 to 1 by dividing the width of the upper joint's range of rotation by axis by the width of the lower joint's range of rotation by axis; a value closer to 1 may indicate a structure where the rotational restriction of the upper joint more strongly constrains the range of motion of the lower joint. For example, if the axial rotation range of the upper joint is narrower than that of the lower joint, the ratio of the axial rotation width of the upper joint to the axial rotation width of the lower joint has a value close to 1, reflecting a structure where the limitation of the upper joint strongly influences the actual rotation range of the lower joint. Additionally, the structural dependency encoding layer can calculate a rotation axis alignment coefficient by comparing the axial vector of the upper joint with the axial vector of the lower joint. The rotation axis alignment coefficient is a value normalized from 0 to 1 by converting the inner product value, which ranges from -1 to 1 based on the cosine similarity of the two rotation axis vectors, into (u*v + 1) / 2; it can be set to a value closer to 1 as the two axes are closer to parallel, and a value closer to 0 as the two axes are closer to perpendicular.Here, u may be the rotation axis vector of the upper joint, and v may be the rotation axis vector of the lower joint. The rotation axis alignment coefficient can be used as an axis alignment-based weight to determine how much the rotational directionality of the upper joint propagates to the rotational characteristics of the lower joint. Additionally, the structural dependency encoding layer can calculate the relative distance-based influence of upper-lower joint pairs by referencing the bone lengths included in the joint-unit structure tokens. This reflects a structure where the longer the bone length between the upper and lower joints, the greater the impact the rotational constraint of the upper joint has on the actual rotational range of the lower joint; the corresponding relative distance-based influence can be normalized to a ratio value between 0 and 1. Furthermore, the structural dependency encoding layer can use the values ​​for the connectivity characteristics of the joint-unit structure tokens as a connection method-based influence, depending on whether the structural connection method between the upper and lower joints is a rotation-centered connection, a linear-movement-centered connection, or a fixed connection. For example, the structural dependency encoding layer can convert these categorical values ​​into continuous influence coefficients between 0 and 1 using a predefined connection influence mapping table. For example, for a fixed connection, the influence factor can be set to 0 because rotational constraints of the upper joint are not transmitted to the lower joint. For a linear movement-centered connection, the influence factor can be set to a low value such as 0.2 because the ratio of rotational constraint transmission is limited. For a rotation-centered connection, the influence factor can be set to 1.0 because rotational changes of the upper joint are directly reflected in the lower joint. Additionally, the structural dependency encoding layer can calculate a hierarchy-based influence that reflects positional importance within the skeleton by referencing the hierarchy level value assigned to each joint.Joints with low hierarchy level values ​​(e.g., pelvis) are located in a central structure that supports multiple sub-joints, so the constraint influence of such joints can be set to a value close to 1, and joints with high hierarchy level values ​​(e.g., wrist) correspond to an extremity structure, so the constraint influence of such joints can be set to a relatively low value. Hierarchy-based influence can be normalized in the range of 0 to 1.

[0092] In this way, rotational constraint propagation relationship values, rotational axis alignment coefficients, relative distance-based influence, connection method-based influence, and hierarchy-based influence can be combined with joint- and axis-based rotational constraint information to be converted into joint-unit biomechanical feature vectors. The generated biomechanical feature vectors can then be provided to a normalization layer and included as components of a biomechanical context vector.

[0093] The normalization layer can complete the biomechanical context vector by aligning the joint-unit biomechanical feature vectors generated by the structural dependency encoding layer to the same dimension and combining them according to the order of the joint list information.

[0094] Additionally, for example, the biomechanical encoder can receive joint list information, joint connection information, joint characteristic information, and joint rotation axis information as input during the learning process to optimize intrinsic parameters for predicting the biomechanical constraints of each joint. The biomechanical encoder can learn joint type embedding parameters to infer joint-specific rotation characteristics based on the joint type. By learning projection parameters that normalize joint rotation axis information to a reference coordinate system, the biomechanical encoder can adjust its internal representation structure to reliably predict axis-specific rotation constraints.

[0095] For example, a biomechanical encoder can improve the performance of predicting the allowable rotation range per axis by updating the parameters of the constraint binding layer by referencing the minimum and maximum rotation angle labels for each joint included in the training data. During the training process, rotational constraint regression parameters can be adjusted based on the joint type and the number of joint rotation axes to predict the rotation range for one axis for single-axis rotational joints, and independent minimum-maximum rotation angles for each rotation axis for multi-axis rotational joints. For example, if the training labels for a humanoid elbow joint are a rotation range from -5 degrees to 145 degrees, the biomechanical encoder can update the parameters so that the output rotation range of the joint converges to this value, and for joints with two rotation axes, such as a bird-like wing joint, it can learn multi-axis rotational constraints by reflecting the minimum-maximum rotation angle labels corresponding to each axis.

[0096] For example, the biomechanical encoder can utilize inter-joint distance labels calculated for each joint by multiplying the reference skeletal length by the character proportional parameter during the training process. For instance, the skeletal information generation layer can be trained to predict adjusted inter-joint skeletal lengths by receiving the connection structure between upper and lower joints and proportional scaling factors as input, and the parameters can be updated through a distance-based loss function to minimize the difference from the actual proportional values. This process serves to ensure that the encoder stably reflects proportional constraints even in situations where the inter-joint distance structure varies across different character types or body shapes.

[0097] For example, the structural dependency encoding layer can optimize parameters during the learning process that reflect rotation constraint propagation relationship values, rotation axis alignment coefficients, relative distance-based influence, connection method-based influence, and hierarchy-based influence. The structural dependency encoding layer can learn relationship-based parameters that allow the rotation constraint of each joint to be influenced by the structural characteristics of adjacent joints. For instance, learning can be performed by adjusting attention weights between parent and child nodes to reflect the structural relationship where the rotation constraint of the shoulder joint also influences the rotational range of the elbow joint.

[0098] As the learning process iterates, the biomechanical encoder can progressively optimize the integrated representation of biomechanical constraints, such as rotational thresholds per axis, rotational characteristics based on joint type, inter-joint distance proportionality, and joint connectivity dependency. Ultimately, the biomechanical encoder can be trained to generate a biomechanical context vector that takes only skeleton structure information as input and consistently predicts the allowable rotation range and bone length of each joint; this vector can then be provided as a constraint to ensure that the diffusion-based motion generation model does not exceed the physically feasible rotation range when generating joint rotation sequences.

[0099] For example, the biomechanical context vector may include structural constraint variables representing the structural relationship between the basic constraint variables and the joints. For example, it may include per-joint allowable range of rotation, joint type, and bone length. For example, structural constraint variables may include rotational constraint propagation relationship values, rotation axis alignment coefficients, relative distance-based influence, connection method-based influence, and hierarchy-based influence.

[0100] The allowable rotation range per joint is numerical information containing the minimum and maximum rotation angles set for each axis around which the joint can rotate; it may represent the biomechanically achievable rotation range of the joint as a continuous value in degrees or radians. The allowable rotation range per joint may include the rotation ranges for each axis supported by the joint, such as flexion and extension, abduction and adduction, and pronation and supination. The joint type is characteristic information defining the number of degrees of freedom and rotational axes possessed by each joint; it can serve as a standard for the number of rotational axes, the application method of the rotation model, and the method of calculating rotational constraints for the corresponding joint. The bone length is a value representing the distance between joints connecting an upper joint and a lower joint, and it may be a real value determined by multiplying the standard bone length defined in the basic skeleton template by a scale factor calculated from the character proportionality parameter. The bone length can be used as an item to quantify the relative distance proportionality between joints and the strength of structural dependency.

[0101] In step S360, the server can generate a joint-specific rotation sequence by inputting the text condition vector and the biomechanical context vector into a diffusion-based motion generation model.

[0102] For example, a diffusion-based motion generation model may be a probabilistic conditional generation architecture that utilizes a spatiotemporal graph representation in which each joint of each frame is set as a node to simultaneously reflect skeleton structure information along the time axis. The diffusion-based motion generation model may include multiple graph transformer layers that perform conditional feature update operations based on text condition vectors and biomechanics-based adjustment operations based on biomechanics context vectors. In this case, multi-stage diffusion may be performed to converge a motion representation from a probabilistic noise distribution while performing scale and shift adjustments to adjust the feature representations of the graph transformer layers in the direction of biomechanical constraints.

[0103] For example, a diffusion-based motion generation model can generate a spatiotemporal graph based on skeleton structure information and a preset target number of frames. The target number of frames is a value representing the total frame length preset for generating 3D character motion, and can serve as reference information for determining the temporal resolution and duration of the motion to be generated. For example, a user terminal can transmit the target number of frames corresponding to the motion length specified by the user to a server, and the server can set the size of the time axis of the spatiotemporal graph based on the target number of frames.

[0104] For example, a diffusion-based motion generation model can generate a spacetime graph combining joint structure and time flow by setting each joint in each frame as a node of a spacetime graph according to the number of frames (T) and the number of joints (J), generating spatial edges connecting pairs of nodes within the same frame, and generating temporal edges between adjacent frames where nodes corresponding to the same joint exist.

[0105] Each node may be configured with a node feature vector representing the physical state of joint j at frame t. For example, the node feature vector may include a rotation value (e.g., a 4-dimensional quaternion) and a motion flag value to represent the rotation state of the corresponding joint. The motion flag value indicates whether the corresponding joint is in a motionable state in the current frame and can be set as a binary value of 0 or 1. That is, for example, for a single-axis rotational joint, the rotation value of the joint can be set by constructing an axis-angle rotation representation from the unit rotation axis and rotation angle, and then converting that value into a 4-dimensional quaternion value. For a multi-axis rotational joint, the rotation value of the joint can be set by converting the 3-dimensional rotation result, obtained by sequentially applying the rotation angles for each axis, into a 4-dimensional quaternion value.

[0106] A spatial directional edge is an edge that connects two adjacent joints within a single frame according to the structural connectivity of the skeleton, and can be generated based on joint connectivity information included in the skeleton structure information. The joint connectivity information may include all adjacent joint pairs defined in the skeleton template, including not only connections between upper and lower joints, but also sibling relationship connections between joints belonging to the same hierarchy and cross-directional connections occurring in branching structures. For example, a spatial directional edge can be defined in the form of connecting nodes (t, j1) and (t, j2) corresponding to adjacent joints j1 and j2 in frame t. A spatial directional edge can be expressed as a structural connectivity relationship that includes a distinguishing value for identifying the spatial directional edge, the node index (j1, j2) of the adjacent joint pair, a bone length value between the two joints, a value for connectivity characteristics, and the absolute value of the difference between the hierarchy level values ​​of the two joints. In this case, the distinguishing value for identifying the spatial directional edge may include a joint identifier for each of the two nodes. The skeletal length value can be set as an adjusted skeletal length by querying the reference skeletal length corresponding to adjacent joint pairs based on length information for the basic skeletal segment, and multiplying it by the scale factor for the corresponding part included in the character proportional parameter. For example, not only adjacent joint pairs such as shoulder-elbow, elbow-wrist, vertebra 1-vertebra 2, and hip-knee, but also connections occurring in branching structures, such as the skeleton's rib cage-left shoulder and rib cage-right shoulder, can be included as spatial edges. Spatial edges generated in this manner constitute a graph structure reflecting the spatial connectivity of the entire skeleton and can be utilized as foundational data for spatial message passing in diffusion-based motion generation models.

[0107] A time-direction edge is an edge connecting pairs of nodes corresponding to the same joint in frame t and frame t+1, and can be generated to reflect the characteristic that the joint state changes continuously over time. A time-direction edge can be defined as a time-series connection relationship that includes a distinguishing value to identify the time-direction edge, a previous frame node index, a subsequent frame node index, and a value representing the time interval between frames.

[0108] For example, a diffusion-based motion generation model can configure an initial state by setting the node feature vector corresponding to each node of the spatiotemporal graph as the initial noise value, so that noise injected during the forward diffusion process is gradually removed during the reverse diffusion process. Subsequently, for example, the diffusion-based motion generation model can perform a graph transformer-based conditional feature update operation at each diffusion step. The conditional feature update operation may be an operation that readjusts the directionality in the feature space by combining the structure of the spatiotemporal graph with semantic information inherent in the text condition vector, thereby allowing the feature representation related to motion to be refined stepwise. Through this, the feature representation can be reconstructed so that feature patterns reflecting text semantics are gradually strengthened and noise components are weakened.

[0109] For example, a diffusion-based motion generation model may perform biomechanical-based adjustment operations on output feature vectors generated by graph transformer layers by utilizing biomechanical context vectors at each diffusion step. Biomechanical-based adjustment operations may be operations that constrain output feature vectors so that the generated feature representations converge to physically feasible joint movements by reflecting biomechanical conditions, such as permissible rotation ranges per joint, joint types, and proportional constraints based on joint lengths, into the feature space. For example, biomechanical-based adjustment operations may include scaling and shifting. The output feature vector is a high-dimensional representation calculated from each layer of the diffusion-based motion generation model and may be a value representing a latent representation of an intermediate state for restoring the rotational value of a specific joint at a specific time point.

[0110] For example, scaling and shifting can be performed on output feature vectors generated in the graph transformer layer of a diffusion-based motion generation model using biomechanical context vectors. Scaling may be an operation that adjusts the size (quantitative range) of the output feature vector of the graph transformer layer using axis-specific proportionality coefficients calculated from the biomechanical context vector. Shifting may be an operation that moves the reference point in the feature space by adding a correction coefficient corresponding to a reference rotation value per joint or a neutral pose to the output feature vector. The rotation sequence per joint may be a motion representation sequence composed of rotation values ​​(e.g., 4D quaternion values) in chronological order, corresponding to a preset target number of frames (T), for each joint constituting the skeleton of a 3D character.

[0111] For example, the allowable rotation range may include a minimum rotation angle and a maximum rotation angle per axis set for one or more rotation axes depending on the joint type of the joint.

[0112] According to one embodiment, the server can determine whether the rotation value of each joint included in the joint-specific rotation sequence generated by the diffusion-based motion generation model exceeds the allowable rotation range by comparing it with the minimum rotation angle and maximum rotation angle per axis of the joint.

[0113] For example, if there is a rotation value that exceeds the allowable rotation range, the server can correct the rotation sequence per joint by replacing the rotation value exceeding the allowable rotation range with the minimum rotation angle or maximum rotation angle per axis of the corresponding joint.

[0114] Specifically, for example, the server receives rotation values ​​for each joint included in a joint-specific rotation sequence output from a diffusion-based motion generation model in the form of quaternions, and calculates rotation angles for the x-axis, y-axis, and z-axis, respectively, by performing a standard quaternion-Euler transform operation to directly compare the rotation values ​​with the axis-specific rotation angles. The server can determine whether the rotation angle of each axis is within the allowable rotation range by comparing the transformed axis-specific rotation angles with the axis-specific minimum and maximum rotation angles included in the biomechanical feature vector. For example, if the rotation angle of a specific axis is smaller than the minimum rotation angle of that axis, it may be determined that the lower limit has been exceeded, and if it is larger than the maximum rotation angle, it may be determined that the upper limit has been exceeded.

[0115] For example, if there is an axis rotation angle that falls outside the allowable rotation range, the server can determine the corrected rotation angle by replacing the excess rotation angle with the limit value of the corresponding axis. For instance, if the rotation angle is smaller than the minimum rotation angle, it can be corrected by replacing it with the minimum rotation angle, and if it is larger than the maximum rotation angle, it can be corrected by replacing it with the maximum rotation angle. After determining the corrected x-axis, y-axis, and z-axis rotation angles by performing independent substitution operations for each axis, the server can generate a corrected quaternion by applying an Euler-quaternion conversion function to restore the corrected axis rotation angles back into a quaternion format. In this case, the corrected quaternion may be maintained without change if the original quaternion rotation value is within the allowable rotation range, and may be replaced with the corrected value only if it falls outside the allowable range.

[0116] By reflecting the corrected quaternion rotation values ​​into the joint rotation sequence of the corresponding frame, the server can generate a joint rotation sequence adjusted so that the joint rotation sequence generated by the diffusion-based motion generation model complies with the biomechanically acceptable rotation range. By repeatedly applying this correction process to all frames and all joints included in the entire sequence, the server can ensure that the entire 3D character motion is consistently maintained in a form that does not violate biomechanical constraints.

[0117] In step S370, the server can transmit 3D motion data generated by applying a joint-by-joint rotation sequence to skeleton structure information to the user terminal.

[0118] For example, the server can transmit 3D motion data to a user terminal by applying joint-specific rotation sequences to skeleton structure information through forward kinematics operations and skin deformation, thereby converting them into 3D character motion data. Here, forward kinematics operations are operations that calculate the joint position and orientation in the world coordinate system by accumulating rotation values ​​of each joint along a hierarchical structure from the root joint to the end joint, and may be operations for calculating the final position of each joint through the sequential application of transformation matrices that consider the joint hierarchy. Skin deformation is a deformation operation for displaced the character's surface mesh according to joint rotation, and may be calculated in a manner where a joint transformation matrix is ​​applied to each mesh vertex based on predefined skin weights for one or more joints. Skin deformation may be a deformation process designed to cause the mesh surface to naturally stretch or contract as the joints rotate.

[0119] For example, the server can receive a joint-by-joint rotation sequence calculated by a diffusion-based motion generation model as input, and first calculate the absolute position and absolute direction of each joint by performing a forward kinematics operation that cumulatively applies rotation values ​​at each frame based on joint connection information included in the skeleton structure information. Subsequently, based on the joint position and direction information calculated as a result of the forward kinematics operation, the server can generate motion data of a 3D character that reflects the displacement of the surface mesh corresponding to the joint movement by performing a mesh deformation operation that applies pre-stored skin weights to each vertex constituting the character mesh.

[0120] For example, the server may encode the generated motion data into a standard 3D file format (at least one of BVH, FBX, and GLB) or a proprietary format defined within the system, and convert it into transmission data for transmission to a user terminal. By transmitting the encoded transmission data to the user terminal, the server may provide the user terminal with the ability to play or render the 3D motion data.

[0121] FIG. 4 shows the architecture of a diffusion-based motion generation model according to one embodiment. FIG. 5 shows a block diagram of a diffusion-based motion generation model according to one embodiment. The embodiments of FIG. 4 and FIG. 5 can be combined with various embodiments of the present disclosure.

[0122] Referring to FIG. 4, the diffusion-based motion generation model (400) can receive motion semantics described by text prompts as input, combine them with motion representations pre-learned from a motion database, and then generate a 3D motion sequence according to a reverse diffusion process. For example, the diffusion-based motion generation model (400) can use text condition vectors extracted from text prompts as condition inputs and pre-learn motion patterns based on a number of learned motions collected from a motion database, thereby performing a probabilistic motion reconstruction operation that reflects the spatiotemporal motion structure corresponding to the input text. In the initial stage, the diffusion-based motion generation model (400) uses a sequence set as random noise as input to reconstruct the probabilistic distribution in the forward diffusion process, and then gradually converges the sequence in the direction from which the noise is removed by repeatedly performing the reverse diffusion process. Finally, the diffusion-based motion generation model (400) can generate a joint rotation sequence that reflects the semantic conditions of the text description and the learned motion distribution, and output renderable 3D motion data based thereon.

[0123] Referring to FIG. 5, the diffusion-based motion generation model (500) may be a neural network structure in which a plurality of graph transformer layers are sequentially stacked.

[0124] A diffusion-based motion generation model (500) can generate a sequence of joint-specific rotation values ​​by providing a text condition vector, a biomechanical context vector, and a spatiotemporal graph as input information to a plurality of graph transformer layers. For example, the diffusion-based motion generation model (500) can generate a spatiotemporal graph that reflects the frame-unit temporal structure and spatial connection relationships between joints based on skeleton structure information and a preset target number of frames, and can generate a meaningful sequence of joint rotation values ​​with noise removed in the inverse diffusion process by utilizing the text condition vector and the biomechanical context vector as condition information in each graph transformer layer.

[0125] For example, each graph transformer layer can perform conditional feature update operations and biomechanics-based adjustment operations. In this case, the output feature vector of each graph transformer layer can be sequentially provided as input to the next graph transformer layer.

[0126] For example, a diffusion-based motion generation model (500) can initialize the node values ​​of a spatiotemporal graph with noise and, for each node of the spatiotemporal graph at each preset diffusion step through a plurality of graph transformer layers, perform a conditional feature update operation including a time-axis self-attention operation based on time-direction edges, a spatial-attention operation based on spatial-direction edges, and a cross-attention operation using a text condition vector. Here, the node value may be a node feature vector representing the rotation state of joint j in frame t.

[0127] The temporal self-attention operation is a process that allows the node feature vector of the current frame to learn temporal correlations from the node feature vectors of previous and subsequent frames by reflecting temporal edges between nodes when nodes corresponding to the same joint exist in temporally adjacent frames. For example, a graph transformer layer can align the node feature vectors of frames t₁, t, and t+1 corresponding to the same joint j based on temporal edges, calculate temporal attention weights, and reflect them in the node feature vector of the current frame t. In this case, temporal edges can be used as information representing temporal dependencies to maintain motion continuity and natural movement changes. For example, the graph transformer layer can generate a query vector based on the node feature vector of joint j corresponding to frame t. The query vector can serve as a reference vector for selecting necessary information from temporally adjacent frames based on the joint state of the current frame. The graph transformer layer can generate two key vectors by transforming the node feature vector of the previous frame t-1 and the node feature vector of the subsequent frame t+1, respectively, corresponding to the same joint j. The key vectors can be comparison vectors used to determine how relevant each adjacent frame is to the current frame. Additionally, the graph transformer layer can generate two value vectors by transforming the node feature vector of the previous frame t-1 and the node feature vector of the subsequent frame t+1, respectively, corresponding to the same joint j. The value vectors can be information vectors used to update the joint features of the current frame after time-axis attention weights have been applied. The graph transformer layer can produce two attention scores by calculating the similarity between the query vector and the key vector generated from the previous frame, and calculating the similarity between the query vector and the key vector generated from the subsequent frame in the same manner.The two calculated attention scores can be normalized through a softmax operation and converted into two time-axis attention weights. The graph transformer layer can generate the first intermediate feature vector of frame t by multiplying the value vector generated in the previous frame by the first time-axis attention weight, multiplying the value vector generated in the subsequent frame by the second time-axis attention weight, and then adding the two results. Through such operations, the joint features of the current frame can be updated to reflect the temporal continuity of the joint features of the previous and subsequent frames.

[0128] Spatial attention operations may be operations that reflect the structural connectivity between parent-child and adjacent joints within the same frame, ensuring that the rotation value of the corresponding joint is updated in a direction consistent with the body structure. For example, a graph transformer layer can collect node feature vectors of joint j and its parent, child, and adjacent joints based on spatial edges in frame t, and calculate spatial attention weights between the node feature vectors to update the output feature vector so that structural correlations between joints are reflected. For example, the graph transformer layer can generate a query vector by receiving the node feature vector corresponding to joint j in frame t as input, and generate key and value vectors based on the node feature vectors corresponding to the adjacent joints connected to j in frame t. The graph transformer layer can generate a query vector based on the node feature vector of joint j corresponding to frame t. Here, the query vector may be a reference vector used to determine which adjacent joints the corresponding joint is structurally associated with within the same frame. The graph transformer layer can generate multiple key vectors corresponding to each adjacent joint by transforming the node feature vectors corresponding to the adjacent joints directly connected to joint j. The key vectors may serve as comparison vectors to determine how structurally the adjacent joint influences the motion direction or rotation amount of joint j. Additionally, the graph transformer layer can generate multiple value vectors by transforming the node feature vectors of the same adjacent joints. The value vectors may serve as information vectors to convey information reflecting structural correlations through spatial attention weights to the feature update of joint j. The graph transformer layer can calculate the similarity between the query vector and the multiple key vectors generated from each adjacent joint, thereby generating one attention score for each pair of adjacent joints.The calculated multiple attention scores can be normalized by a softmax operation and converted into spatial attention weights representing the influence of each adjacent joint. The graph transformer layer can generate a second intermediate feature vector for joint j by multiplying the value vector corresponding to each adjacent joint by the spatial attention weight and then performing a weighted sum over all adjacent joints. Through such operations, the output feature of joint j can be updated to reflect the relative importance of structurally valid adjacent joints within the same frame.

[0129] The cross-attention operation is an attention-based association operation designed to correlate multiple token embedding vectors constituting the text condition vector with the feature vectors of each joint node in the spatiotemporal graph. For example, a graph transformer layer can generate a query vector based on the node feature vector corresponding to joint j in frame t of the spatiotemporal graph. The query vector can serve as a reference vector to determine which token within the text condition vector is semantically associated with joint j. The query vector serves as a criterion for performing semantic alignment by joint that reflects text meaning, and can act as an indicator to select the meaning of the action that the corresponding joint must perform from among the text tokens. The graph transformer layer can generate multiple key vectors corresponding to each text token by transforming the token embeddings constituting the text condition vector. The key vectors can serve as comparison vectors to determine the extent to which the text token matches the meaning related to the action or body part that joint j must perform. For example, given the text "[raises the arm]", the token embeddings of "[arm]", "[lift]", and "[raise]" are converted into key vectors that can serve as criteria for evaluating semantic agreement with joint j. Additionally, the graph transformer layer can generate value vectors based on each token embedding constituting the text condition vector. The value vector may be an information vector intended to convey the semantic information contained in the text tokens to the third intermediate feature vector of joint j. The graph transformer layer can calculate the similarity between the query vector generated for joint j and the key vector generated from each token of the text condition vector, respectively, to produce a single cross-attention score per text token. The multiple calculated cross-attention scores can be normalized through a softmax operation and converted into cross-attention weights that represent the extent to which each text token contributes to determining the action semantics of joint j.The Graph Transformer layer can generate a third feature vector for joint j by multiplying the value vector corresponding to each text token by cross-attention weights and then performing a weighted sum over all text tokens. Through this operation, the output feature of joint j can be updated to reflect the motion instruction semantics and body part-related information contained in the text condition vector. Therefore, the cross-attention operation can perform the role of semantically aligning text-based instructions with spatiotemporal graph-based joint representations, ensuring that the motion indicated in the text is accurately reflected as joint-specific rotation values ​​during the motion generation process.

[0130] For example, the graph transformer layer can update the output feature vector of joint j by merging the temporal attention result, the spatial attention result, and the cross-attention result into an output feature vector through a linear combination operation. That is, the graph transformer layer can update the output feature vector of joint j by merging the first intermediate feature vector, the second intermediate feature vector, and the third intermediate feature vector into an output feature vector through a linear combination operation.

[0131] For example, the graph transformer layer can perform scaling and shifting by calculating scale factors and shift factors for the output feature vector based on the biomechanical context vector and applying the scale factors and shift factors to the output feature vector. In this case, the scale factor refers to a ratio adjustment value applied independently to each dimensional element (channel) constituting the output feature vector, and may be a continuous value used to control the intensity of the output feature within the range of biomechanically acceptable joint rotation amounts. The scale factor can be set within a range of 0 to 1; values ​​closer to 0 strongly suppress the rotation pattern reflected by the corresponding channel, while values ​​closer to 1 maintain the rotation pattern at its original size. The shift factor refers to a linear offset applied commonly to the entire output feature vector, and is a value used to adjust the reference point so that the amount of rotation does not deviate excessively from the biomechanically acceptable reference center of rotation. The shift factor can be set within a continuous range from negative to positive, where a value of 0 means no reference point adjustment is made, a positive value means adjustment that moves the reference center of rotation in an increasing direction, and a negative value means adjustment that moves it in a decreasing direction. The magnitude of the shift factor can be set within a limited range so as not to exceed the acceptable range of fine adjustment from the basic neutral pose for each joint.

[0132] For example, a graph transformer layer can generate a channel-unit representation by determining each dimensional element constituting the output feature vector of joint j as an individual channel.

[0133] For example, a graph transformer layer can transform a biomechanical context vector through a combination of a fully coupled layer and a non-linear activation function to generate a channel-specific scale factor corresponding to the number of dimensions of the output feature vector. The graph transformer layer can perform scaling by multiplying the calculated channel-specific scale factor by each channel value of the output feature vector. In this case, the scaling operation is applied in a way that reduces or maintains the magnitude of the rotation pattern reflected by each channel of the output feature vector in accordance with biomechanical standards, and can be set so that adjustment in the direction of increase (enlargement) is not allowed. That is, by ensuring that the scale factor is not set to a value greater than 1, it is possible to prevent the calculated potential amount of rotation from being expanded into an excessive rotation that is biomechanically impossible.

[0134] For example, a graph transformer layer can transform a biomechanical context vector through a combination of a fully coupled layer and a non-linear activation function to generate channel-specific shift coefficients corresponding to the number of dimensions of the scaled output feature vector. The graph transformer layer can perform a shift adjustment operation by adding the calculated channel-specific shift coefficients to each channel value of the scaled output feature vector. For example, a multidimensional shift coefficient representing an acceptable neutral center of rotation at joint j can be calculated by referencing the joint type and base center of rotation value in the biomechanical context vector and reflecting the rotation constraint propagation relationship value and the hierarchy-based influence. The shift adjustment operation is performed by applying a linear offset to reposition the reference point of the rotation pattern reflected by each channel of the scaled output feature vector within a biomechanically valid center range, and shift coefficients that move the reference point in a direction outside the biomechanically acceptable range may be restricted from being calculated. Through such shift adjustment operations, the output feature vector of joint j can be corrected to maintain a stable center of rotation within an acceptable rotation space.

[0135] For example, a diffusion-based motion generation model (500) can generate a more refined output feature vector by removing residual noise corresponding to the current diffusion step from the output feature vector on which shift adjustment has been performed.

[0136] For example, a diffusion-based motion generation model (500) can determine the noise component remaining in the current output feature vector using a learned noise prediction function, and remove the determined noise component to generate an output feature vector that gradually converges to the rotation value for each joint.

[0137] For example, a diffusion-based motion generation model (500) can generate a gradually completed joint-by-joint rotation sequence from initial noise by repeating the process described above for a preset number of diffusion steps.

[0138] Specifically, for example, a diffusion-based motion generation model (500) can receive a shift-adjusted output feature vector and a step index value representing the current diffusion step as input, and perform a noise prediction operation to predict the residual noise component to be removed in the corresponding diffusion step. The diffusion-based motion generation model (500) can calculate a residual noise vector for each node by combining a diffusion step embedding, a text condition vector, and a biomechanical context vector with the output feature vector for each node on the spatiotemporal graph where the shift adjustment is completed, through a neural network operation including a fully connected layer and a non-linear activation function. Here, the residual noise vector is a vector value representing the stochastic noise component remaining in the output feature vector in the current diffusion step, and may mean a value to be removed in a subsequent step.

[0139] The diffusion-based motion generation model (500) can generate an updated output feature vector by calculating the amount of noise removal by multiplying the calculated residual noise vector by a noise removal coefficient pre-set for each diffusion step, and by subtracting the amount of noise removal from the output feature vector to which shift adjustment is applied. At this time, the noise removal coefficient for each diffusion step is configured such that the amount of removal is set larger as the step index is larger and the amount of removal is set smaller as the step index is smaller, so that coarse noise removal is performed in the initial step and fine-tuned noise removal is performed as the step index progresses. If necessary, the diffusion-based motion generation model (500) can be configured to maintain probabilistic diversity by adding a small amount of Gaussian noise based on a variance value set according to the residual variance schedule for each diffusion step, so that various motion sequences can be generated even under the same conditions.

[0140] In this way, the diffusion-based motion generation model (500) receives a spatiotemporal graph representation set as random noise in the initial diffusion stage as input, and sequentially performs conditional feature update operations, scale adjustment operations, and shift adjustment operations through multiple graph transformer layers at each stage while decreasing the diffusion stage index from the initial stage value to the end stage value, and can perform noise removal operations at each stage. By performing the above iteration process as many times as a preset number of diffusion stages, the diffusion-based motion generation model (500) can gradually converge the output feature vector of the initial noise state to a joint-specific rotation value reflecting text conditions and biomechanical constraints, and finally generate a joint-specific rotation sequence.

[0141] For example, the number of multiple graph transformer layers can be set to 8, and the number of preset diffusion steps can be set to 100. In this case, the conditional feature update operation, the scaling operation, and the shifting operation can each be performed 800 times, and the noise removal operation can be performed 100 times.

[0142] For example, multiple graph transformer layers are executed sequentially within a single diffusion step, and the output feature vector of the final graph transformer layer can be provided as the input to the next diffusion step as the result state of the same diffusion step.

[0143] Additionally, according to one embodiment, the server calculates the amount of change in rotation value between two consecutive frames in a joint-by-joint rotation sequence, and if the amount of change in rotation value exceeds a time change threshold set for the corresponding joint, the corresponding frame interval may be determined as an interval where temporal discontinuity exists. For example, time change thresholds may be pre-set for multiple joints for each character type.

[0144] For example, for a section where a temporal discontinuity exists, the server may generate a first condition vector containing joint-specific rotation values ​​of the frame immediately preceding the discontinuity and a second condition vector containing joint-specific rotation values ​​of the frame immediately following the discontinuity. The first condition vector may be used as condition information representing a joint state that must be maintained in the preceding time, and the second condition vector may be used as condition information representing a joint state that must be maintained in the succeeding time.

[0145] For example, the server can determine the number of intermediate frames to insert based on the length of discontinuous intervals and the magnitude of the change in rotation values, and generate a noise initial value sequence having a length corresponding to the determined number. The noise initial value sequence can be used as the initial input for a diffusion-based interpolation operation.

[0146] For example, the server can perform a conditional feature update operation on the noise initial sequence by providing the first condition vector and the second condition vector as cross-attention inputs to the graph transformer layer within the diffusion-based motion generation model at each diffusion step. The conditional feature update operation on the noise initial sequence may be an operation to align the latent feature vectors of intermediate frames so that they form a temporally natural path between the joint state of the preceding time point and the joint state of the subsequent time point. At this time, the diffusion-based motion generation model can remove residual noise corresponding to the diffusion step by applying a learned noise prediction function to the feature vectors for each intermediate frame to which the conditional feature update for the noise initial sequence has been applied. The noise removal operation may consist of an operation to gradually remove residual noise according to the diffusion schedule and converge to the final rotation value.

[0147] For example, a diffusion-based motion generation model can generate joint-specific rotation values ​​for intermediate frames to be inserted between boundary intervals by repeating conditional feature update operations and noise removal operations for a preset number of diffusion steps. For instance, the server can perform smoothing that maintains temporal continuity by inserting the generated intermediate frames into the corresponding sections of an existing joint-specific rotation sequence.

[0148] For example, the number of multiple graph transformer layers can be set to 8, and the number of preset diffusion steps can be set to 100.

[0149] For example, multiple graph transformer layers are executed sequentially within a single diffusion step, and the output feature vector of the final graph transformer layer can be provided as the input to the next diffusion step as the result state of the same diffusion step.

[0150] FIG. 6 is a block diagram showing the configuration of a server according to one embodiment. The embodiment of FIG. 6 can be combined with various embodiments of the present disclosure.

[0151] As illustrated in FIG. 6, the server (600) may include a processor (610), a communication unit (620), and memory (630). However, not all components illustrated in FIG. 6 are essential components of the server (600). The server (600) may be implemented with more components than those illustrated in FIG. 6, or with fewer components than those illustrated in FIG. 6. For example, according to some embodiments, the server (600) may further include a user input interface (not shown), an output unit (not shown), etc., in addition to the processor (610), the communication unit (620), and memory (630).

[0152] The processor (610) typically controls the overall operation of the server (600). The processor (610) may be equipped with one or more processors to control other components included in the server (600). For example, the processor (610) may control the communication unit (620) and the memory (630) overall by executing programs stored in the memory (630). Additionally, the processor (610) may perform the functions of the server (600) described in FIGS. 3 to 6 by executing programs stored in the memory (630).

[0153] The communication unit (620) may include one or more components that enable the server (600) to communicate with another device (not shown) and the server (not shown). The other device (not shown) may be a computing device such as the server (600) or a sensing device, but is not limited thereto. The communication unit (620) may receive user input from another electronic device or receive data stored in an external device from an external device via a network.

[0154] The memory (630) can store a program for processing and controlling the processor (610). For example, the memory (630) can store information input to the server or information received from another device via a network. Additionally, the memory (630) can store data generated by the processor (610). The memory (630) can also store information input to the server (600) or output from the server (600).

[0155] The memory (630) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk.

[0156] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0157] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0158] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0159] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0160] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.< / s> < / s>

Claims

Claim 1 A method for a server to generate motion of a 3D character using artificial intelligence comprises: receiving a text prompt for generating motion of a 3D character from a user terminal; inputting the text prompt into a text encoder to generate a text condition vector; obtaining skeleton configuration parameters including a character type, an active joint set, and a character proportional parameter based on the text condition vector; generating skeleton structure information including joint list information, joint connection information, joint characteristic information, and joint rotation axis information based on the skeleton configuration parameters; inputting the skeleton structure information into a biomechanical encoder to generate a biomechanical context vector including an allowable rotation range per joint, a joint type, and a bone length; and inputting the text condition vector and the biomechanical context vector into a diffusion-based motion generation model to generate a rotation sequence per joint. The method includes the step of transmitting three-dimensional motion data generated by applying the above joint-specific rotation sequence to the above skeleton structure information to the user terminal, wherein scale adjustment and shift adjustment are performed on the output feature vector generated in the graph transformer layer of the diffusion-based motion generation model using the above biomechanical context vector, and the diffusion-based motion generation model includes a plurality of graph transformer layers, and based on the above skeleton structure information and a preset target number of frames, each joint in each frame is set as a node of a spatiotemporal graph, a spatial edge connecting pairs of nodes within the same frame is generated, and a temporal edge is generated between adjacent frames where nodes corresponding to the same joint exist, thereby generating a spatiotemporal graph, and the node values ​​of the spatiotemporal graph are initialized with noise.For each pre-set diffusion step, for each node of the spatiotemporal graph, a conditional feature update operation is performed including a time-axis self-attention operation based on the time-direction edge, a spatial-attention operation based on the spatial-direction edge, and a cross-attention operation using the text condition vector; a channel-specific scale factor and a channel-specific shift factor for the output feature vector are calculated based on the biomechanical context vector, and the scale adjustment and shift adjustment are performed by applying the channel-specific scale factor and the channel-specific shift factor to the output feature vector; the conditional feature update operation, the scale adjustment, and the shift adjustment are repeated a pre-set number of times along with the noise removal operation of each diffusion step to generate the joint-specific rotation sequence; the allowable rotation range includes a minimum rotation angle and a maximum rotation angle for one or more rotation axes set according to the joint type of the corresponding joint, and the rotation value of each joint included in the joint-specific rotation sequence generated by the diffusion-based motion generation model is [corresponding] A method for determining whether the rotational value exceeds the allowable rotational range by comparing it with the minimum rotational angle and maximum rotational angle of each axis of the joint, and, if a rotational value exceeding the allowable rotational range exists, correcting the rotational sequence of each joint by replacing the rotational value exceeding the allowable rotational range with the minimum rotational angle or maximum rotational angle of each axis of the joint. Claim 2 delete Claim 3 delete

Images