Style-based motion conversion device and method therefor

Abstract

The present disclosure relates to a style-based motion conversion device and a method therefor. The device can extract a content feature from content motion data, including motion of an object, by using a content feature extraction model, extract a style feature from style information by using a style feature extraction model; and generate a style motion on the basis of the content feature and the style feature by using a style generation model.

Description

Style-based motion conversion device and method thereof

[0001] The present disclosure relates to a style-based motion conversion device.

[0002] With the advancement of generative artificial intelligence, the technology to create text or two-dimensional images is becoming popular and is expanding to create three-dimensional objects.

[0003] However, the technology to create three-dimensional objects or movements as they are is still at an insufficient level, and various technologies are being researched for this purpose.

[0004] In particular, 3D objects and movements are essential elements for realistic character animation in the film and game industries, but there is a problem that it is very difficult to obtain various styles of human movements using motion capture alone.

[0005] The purpose of the embodiments disclosed in the present disclosure is to provide a style-based motion conversion device.

[0006] In addition, the embodiment disclosed in the present disclosure aims to provide a style-based motion conversion device capable of extracting content features from content motion and style features from style information, and then generating style motion based on the content features but reflecting the style features.

[0007] In addition, the embodiment disclosed in the present disclosure aims to provide a style-based motion conversion device capable of extracting style features using a large-scale language model and a VLP model.

[0008] The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0009] According to an embodiment of the present disclosure for solving the above-described problem, an electronic device includes a memory storing at least one process for generating a style motion; and a processor performing an operation related to the at least one process, wherein the processor extracts the content feature from content motion data including the motion of an object using a first model for extracting content features, extracts the style feature from style information using a second model for extracting style features, and generates a style motion in which the content feature and the style feature are reflected using a third model for generating a style, wherein the style information includes at least one of text, voice, image, and motion, and the style feature can be acquired from text and image included in the style information using a VLP model (Vision-Language Pre-training model).

[0010] At this time, the processor may convert a first text of a preset length or longer among the texts into at least one second text in the form of words expressing the characteristics and emotions of an object included in the first text using a large-scale language model (LLM), input the at least one second text into the VLP model, and obtain the style feature as an output value of the VLP model.

[0011] Additionally, if the style information relates to an in-game character, the processor may input data including a background description of the character into the VLP model.

[0012] In addition, the processor can obtain a style distribution for a feature space based on text included in the style information using the VLP model, and obtain the style feature by sampling a style vector from the obtained style distribution.

[0013] In addition, the processor inputs a first style feature obtained from text and image included in the style information, a second style feature obtained from voice included in the style information, and a third style feature obtained from motion included in the style information into a linear layer, respectively, and controls the vector sizes of the first style feature, the second style feature, and the third style feature to be the same, and trains the second model by reducing the vector distance of the first style feature, the second style feature, and the third style feature.

[0014] In addition, the processor may extract a first content feature from first content motion data including a first motion of a first object, extract a second content feature from second content motion data including a second motion of a second object, extract a first style feature from first style information of the first object, extract a second style feature from second style information of the second object, generate a first style motion based on the first content feature and the first style feature, generate a second style motion based on the second content feature and the second style feature, and train the first model and the third model by a method of reducing a vector distance between the first content motion data and the first style motion, and train the first model and the third model by a method of reducing a vector distance between the second content motion data and the second style motion. At this time, the processor may generate a third style motion based on the second content feature and the first style feature, extract a third style feature from the third style motion, extract a third content feature from the third style motion, generate a fourth style motion based on the first content feature and the third style feature, generate a fifth style motion based on the third content feature and the second style feature, train the first model and the third model by a method of reducing a vector distance between the first content motion data and the fourth style motion, and train the first model and the third model by a method of reducing a vector distance between the second content motion data and the fifth style motion.

[0015] In addition, the processor may use an encoder model when extracting the content features, generate the style motion so that the extracted style features are applied while removing the remaining style in the content features using AdaIN technology, perform at least one upsampling on the extracted content features whose size is reduced by using the encoder model, and apply the extracted style features.

[0016] In addition, a method for generating a style-based motion performed by a processor of an electronic device according to an embodiment of the present disclosure for solving the above-described problem may include: extracting content features from content motion data including motion of an object using a first model for extracting content features; extracting style features from style information using a second model for extracting style features; generating a style motion in which the content features and the style features are reflected using a third model for generating a style, wherein the style information includes at least one of text, voice, image, and motion; and obtaining style features from text and image included in the style information using a VLP model (Vision-Language Pre-training model).

[0017] In addition, a computer program stored in a computer-readable recording medium for executing a method for implementing the present disclosure may be further provided.

[0018] In addition, a computer-readable recording medium recording a computer program for executing a method for implementing the present disclosure may be further provided.

[0019] According to the above-described problem solving means of the present disclosure, an effect of providing a style-based motion conversion device is provided.

[0020] In addition, according to the above-described problem solving means of the present disclosure, it is possible to extract content features from content motion and style features from style information, and then generate a style motion based on the content features but reflecting the style features.

[0021] In addition, the aforementioned problem solving means of the present disclosure provides the effect of being able to extract style features using a large-scale language model and a VLP model.

[0022] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0023] FIG. 1 is a schematic diagram of a style-based motion conversion system according to an embodiment of the present disclosure.

[0024] FIG. 2 is a block diagram of a style-based motion conversion device according to an embodiment of the present disclosure.

[0025] FIG. 3 is a flowchart of a style-based motion conversion method according to an embodiment of the present disclosure.

[0026] Figure 4 is a diagram illustrating a process for extracting style features from style information.

[0027] Figure 5 is a diagram illustrating the use of an encoder model as a motion feature extraction model and a style feature extraction model for 3D motion.

[0028] Figure 6 is a diagram explaining measuring the vector distance between each extracted style feature in Figure 4.

[0029] Figure 7 is a diagram illustrating the generation of style-added content motion by repeatedly performing Convolution, AdaIN, and Upsampling.

[0030] FIG. 8 is a diagram illustrating a method for training an artificial intelligence model used in a style-based motion conversion device according to an embodiment of the present disclosure.

[0031] Throughout this disclosure, the same reference numerals denote the same components. This disclosure does not describe all elements of the embodiments, and any content that is common in the technical field to which this disclosure pertains or that overlaps between embodiments is omitted. The terms "part, module, element, block" used in the specification may be implemented in software or hardware, and depending on the embodiments, multiple "parts, modules, elements, blocks" may be implemented as a single component, or a single "part, module, element, block" may include multiple components.

[0032] Throughout the specification, when a part is said to be "connected" to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection via a wireless communication network.

[0033] Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0034] Throughout the specification, when we say that an element is "on" another element, this includes not only cases where the element is in contact with the other element, but also cases where another element exists between the two elements.

[0035] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0036] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0037] The identification codes for each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

[0038] The operating principle and embodiments of the present disclosure are described below with reference to the attached drawings.

[0039] As used herein, the term "electronic device according to the present disclosure" encompasses a variety of devices capable of performing computational processing and providing results to a user. For example, an electronic device according to the present disclosure may include a computer, a server device, and a portable terminal, or may be any one of them.

[0040] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0041] The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

[0042] The above portable terminal may include, for example, all kinds of handheld-based wireless communication devices such as PCS, GSM, PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, smart phones, etc., as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMD).

[0043] The artificial intelligence-related functions according to the present disclosure are operated through a processor and a storage unit. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, an AP, a Digital Signal Processor (DSP), a graphics-only processor such as a GPU or a Vision Processing Unit (VPU), or an artificial intelligence-only processor such as an NPU. One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in the storage unit. Alternatively, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0044] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned by a learning algorithm using a plurality of learning data, thereby creating the predefined operation rules or artificial intelligence models set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0045] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weights, and performs neural network operations through operations between the operation results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network.

[0046] According to an exemplary embodiment of the present disclosure, a processor can implement artificial intelligence. Artificial intelligence refers to a machine learning method based on an artificial neural network that mimics human neurons (biological neurons) to enable machines to learn. Artificial intelligence methodologies can be categorized into supervised learning, where input and output data are provided together as training data, thereby determining the solution (output data) to a problem (input data); unsupervised learning, where only input data is provided without output data, so that the solution (output data) to a problem (input data) is not determined; and reinforcement learning, where a reward is provided from an external environment each time an action is taken in the current state, and learning proceeds in a direction that maximizes this reward. Furthermore, artificial intelligence methodologies can be categorized by the architecture of the learning model. The architectures of widely used deep learning technologies can be categorized into convolutional neural networks, recurrent neural networks, transformers, and generative adversarial networks.

[0047] The device may include an artificial intelligence model. The artificial intelligence model may be a single artificial intelligence model or may be implemented as multiple artificial intelligence models. The artificial intelligence model may be composed of a neural network (or artificial neural network) and may include statistical learning algorithms that mimic biological neurons in machine learning and cognitive science. A neural network may refer to a model in general that has problem-solving capabilities by changing the binding strength of synapses through learning, formed by artificial neurons (nodes) that form a network by combining synapses. The neurons of the neural network may include a combination of weights or biases. The neural network may include one or more layers composed of one or more neurons or nodes. For example, the device may include an input layer, a hidden layer, and an output layer. The neural network constituting the device can infer a desired outcome from an arbitrary input by changing the weights of the neurons through learning.

[0048] The processor can create a neural network, train (or learn) a neural network, perform a calculation based on received input data, generate an information signal based on the calculation result, or retrain the neural network. The models of the neural network can include various types of models such as CNN, R-CNN, RPN, RNN, S-DNN, S-SDNN, Deconvolution Network, DBN, RBM, Fully Convolutional Network, LSTM Network, Classification Network, etc., such as GoogleNet, AlexNet, VGG Network, etc., but are not limited thereto. The processor can include one or more processors for performing calculations according to the models of the neural network. For example, the neural network can include a deep neural network.

[0049] Neural networks include CNN, RNN, perceptron, multilayer perceptron, Feed Forward (FF), Radial Basis Network (RBF), Deep Feed Forward (DFF), Long Short Term Memory (LSTM), Gated Recurrent Unit (GRU), Auto Encoder (AE), Variational Auto Encoder (VAE), Denoising Auto Encoder (DAE), Sparse Auto Encoder (SAE), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), GAN (Generative Adversarial Network), LSM (Liquid State Machine), ELM (Extreme Learning) Machine), ESN (Echo State Network), DRN (Deep Residual Network), DNC (Differentiable Network) It will be understood by those skilled in the art that the neural network may include any neural network, including but not limited to a Neural Computer (NN), a Neural Turning Machine (NTM), a Capsule Network (CN), a Kohonen Network (KN), and an Attention Network (AN).

[0050] According to an exemplary embodiment of the present disclosure, the processor may be configured to perform a process for generating a CNN (Convolution Neural Network) such as GoogleNet, AlexNet, VGG Network, Region with Convolution Neural Network (R-CNN), Region Proposal Network (RPN), Recurrent Neural Network (RNN), Stacking-based deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restrcted Boltzman Machine (RBM), Fully Convolutional Network, Long Short-Term Memory (LSTM) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC / QA for natural language processing, Text Analysis, Dialog System, GPT-3, GPT-4, Visual Analytics for vision processing, Visual Understanding, Video Synthesis, ResNet for data intelligence, Anomaly Detection, Prediction, Time-Series Forecasting, Various artificial intelligence structures and algorithms, including optimization, recommendation, and data creation, can be utilized, but are not limited thereto. Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0051] FIG. 1 is a schematic diagram of a style-based motion conversion system (10) according to an embodiment of the present disclosure.

[0052] Referring to FIG. 1, a style-based motion conversion system (10) according to an embodiment of the present disclosure generates style motion by performing the following process.

[0053] The motion conversion device inputs content motion data into a first model (131) that extracts content features to extract content features.

[0054] The motion conversion device inputs style information into a second model (132) that extracts style features to extract style features.

[0055] The motion conversion device extracts style motion by inputting content features and style features into a third model (133) that generates a style.

[0056] The style-based motion conversion system (10) according to the embodiment of the present disclosure can generate style motion based on content features and style features by performing such a process.

[0057] Below, a detailed example of how each process is performed is described with reference to other drawings.

[0058] FIG. 2 is a block diagram of a style-based motion conversion device according to an embodiment of the present disclosure.

[0059] Referring to FIG. 2, an electronic device (100) for converting motion based on style according to an embodiment of the present disclosure includes a processor (110), a communication unit (120), a memory (130), an extraction unit (140), an input unit (150), an output unit (160), and a rendering unit (170).

[0060] However, in some embodiments, the electronic device (100) may include fewer or more components than those illustrated in FIG. 2.

[0061] An electronic device (100) according to an embodiment of the present disclosure is configured to include a server device and can operate as a style-based motion conversion server.

[0062] The processor (110) may be implemented as a storage unit that stores data regarding an algorithm for controlling the operation of components within the device or a program that reproduces the algorithm, and at least one processor (110) that performs the aforementioned operation using the data stored in the storage unit. In this case, the storage unit and the processor (110) may each be implemented as separate chips. Alternatively, the storage unit and the processor (110) may be implemented as a single chip.

[0063] In addition, the processor (110) can control any one or a combination of the components described above to implement various embodiments according to the present disclosure described in the drawings below on the device.

[0064] In addition to operations related to the above-described application, the processor (110) can typically control the overall operation of the device. The processor (110) can process signals, data, information, etc. input or output through the components described above, or run application programs stored in the storage unit, thereby providing or processing appropriate information or functions to the user.

[0065] In addition, the processor (110) may control at least some of the components of the device to run an application program stored in the storage unit. Furthermore, the processor (110) may operate at least two or more of the components included in the device in combination to run the application program.

[0066] The processor (110) may be implemented as one or more. Hereinafter, even if the processor (110) is expressed as singular, it may be considered as plural. The processor (110) may control the configurations of the electronic device (100). The processor (110) may refer to a data processing device built into hardware that has a physically structured circuit to perform a function expressed by a code or command included in a program. As such, the processor (110) is an example of a data processing device built into hardware, and may encompass processing devices such as a microprocessor, a central processing unit (CPU), a processor (110) core, a multiprocessor, an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA), but the scope of the present invention is not limited thereto. The processor (110) may separately include a learning processor (110) for performing artificial intelligence operations, or may include a learning processor (110) on its own.

[0067] In various embodiments, the processor (110) may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). At least a portion of the processor (110) may be hardware, access memory (130), and perform functions related to instructions stored in the memory (130).

[0068] The communication unit (120) may include one or more modules that connect the electronic device (100) to one or more networks.

[0069] The communication unit (120) may include one or more components that enable communication with an external device, and may include, for example, at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

[0070] The wired communication module may include various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as a Universal Serial Bus (USB), a High Definition Multimedia Interface (HDMI), a Digital Visual Interface (DVI), RS-232 (recommended standard 232), power line communication, or plain old telephone service (POTS).

[0071] The wireless communication module may include a wireless communication module that supports various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G, in addition to a WiFi module and a Wireless Broadband module.

[0072] The wireless communication module may include a wireless communication interface including an antenna and a transmitter for transmitting communication signals. Furthermore, the wireless communication module may further include a signal conversion module that modulates a digital control signal output from the processor (110) through the wireless communication interface into an analog wireless signal under the control of the processor (110).

[0073] The short-range communication module is for short-range communication, and can support short-range communication using at least one of Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

[0074] The communication unit (120) may also use the name of the communication interface.

[0075] The communication interface can establish communication between the electronic device (100) and an external device. For example, the communication interface can communicate with the external device via wireless communication (e.g., Wi-Fi (Wireless Fidelity), Bluetooth, NFC (Near Field Communication), MST (Magnetic Stripe Transmission), etc.) or wired communication.

[0076] The memory (130) can store data supporting various functions of the device. The memory (130) can store a plurality of application programs (or applications) running on the device, data for the operation of the device, and commands. At least some of these application programs may exist for the basic functions of the device. Meanwhile, the application programs can be stored in the memory (130), installed on the device, and driven to perform operations (or functions) by the processor (110).

[0077] The memory (130) can store data supporting various functions of the device and programs for the operation of the processor (110), input / output data (e.g., music files, still images, moving images, etc.) can be stored, and a plurality of application programs (or applications) run on the device, data for the operation of the device, and commands can be stored. At least some of these application programs can be downloaded from an external server via wireless communication.

[0078] The memory (130) may include at least one type of storage medium among a flash memory (130) type, a hard disk type, an SSD (Solid State Disk type), an SDD (Silicon Disk Drive) type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory (130), a magnetic disk, and an optical disk. In addition, the memory (130) may be a database that is separate from the device but is connected by wire or wirelessly.

[0079] The memory (130) may be electrically connected to the processor (110) and may store at least one code executed by the processor (110). The memory (130) may collectively refer to various types of storage devices. The memory (130) may store information necessary for performing operations using artificial intelligence, machine learning, and artificial neural networks.

[0080] The memory (130) can store various learning models. The learning models stored in the memory (130) can infer result values ​​for new input data other than learning data, and the inferred values ​​can be used as a basis for judgment to perform a certain action. The learning models stored in the memory (130) can perform learning based on label information, and various backpropagation algorithms can be applied so that the loss function has a target value to increase the accuracy of learning.

[0081] Additionally, the memory (130) may have multiple processes for the electronic device (100).

[0082] The memory (130) can store at least one artificial intelligence model for converting motion based on style.

[0083] In the embodiment of the present disclosure, various models can be applied as artificial intelligence models, such as a content feature extraction model (first model), a style feature extraction model (second model), a style generation model (third model), a vision-language pre-training model (VLP model), a large-scale language model (LLM), and an encoder model.

[0084] In the examples below, the content feature extraction model refers to the first model, the style feature extraction model refers to the second model, and the style generation model refers to the third model.

[0085] The extraction unit (140) can extract target features from various types of received data.

[0086] In detail, the extraction unit (140) can extract style features from style information, the extraction unit (140) can extract content features from content motion data, and various artificial intelligence models can be used in the extraction process.

[0087] The input unit (150) is for inputting video information (or signal), audio information (or signal), data, or information input from a user, and may include at least one camera, at least one microphone, and at least one user input unit (150). Voice data or image data collected by the input unit (150) may be analyzed and processed into a user control command.

[0088] The input unit (150) is for receiving information from the user. When information is input through the input unit (150), the processor (110) can control the operation of the device to correspond to the input information. The input unit (150) may include hardware-type physical keys (e.g., buttons located on at least one of the front, rear, and side of the device, dome switches, jog wheels, jog switches, etc.) and software-type touch keys. As an example, the touch keys may be formed of virtual keys, soft keys, or visual keys displayed on a touchscreen-type display unit through software processing, or may be formed of touch keys placed on a part other than the touchscreen. Meanwhile, the virtual keys or visual keys may be displayed on the touchscreen in various forms, and may be formed of, for example, graphics, text, icons, videos, or a combination thereof.

[0089] The output unit (160) is for generating output related to visual, auditory, or tactile sensations, and may include at least one of a display unit, an audio output unit (160), a haptic module, and an optical output unit (160). The display unit may be formed as a layer structure with a touch sensor or formed as an integral part, thereby implementing a touch screen. This touch screen may function as a user input unit (150) that provides an input interface between the device and a user, and at the same time, provide an output interface between the device and the user.

[0090] The display unit displays (outputs) information processed by this device. For example, the display unit may display execution screen information of an application program (e.g., an application) running on this device, or UI (User Interface) or GUI (Graphical User Interface) information based on such execution screen information.

[0091] The display can display various contents (e.g., text, images, videos, icons, symbols, etc.). For example, the display can display an image corresponding to at least one image data included in the application program. In various embodiments, when the electronic device (100) adopts VR mode, the display can separate one image into two images corresponding to the user's left and right eyes and display them. In various embodiments, the display can include a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display.

[0092] The audio output unit (160) can output audio data received through the communication unit (120) or stored in the storage unit, or output audio signals related to functions performed by the device. The audio output unit (160) can include a receiver, a speaker, a buzzer, etc.

[0093] The rendering unit (170) can render images and videos, and can generate style motion videos based on content features and style features. For example, the rendering unit (170) can generate style motion videos by rendering videos based on content features while reflecting style features.

[0094] FIG. 3 is a flowchart of a style-based motion conversion method according to an embodiment of the present disclosure.

[0095] Referring to FIG. 3, the operation process of a style-based motion conversion device, method, and program according to an embodiment of the present disclosure will be described.

[0096] The processor (110) extracts content features from content motion data. (S100)

[0097] Content motion data is data that contains the motion of an object (character).

[0098] In detail, the processor (110) can extract content features from content motion data using a content feature extraction model that has learned a method for extracting content features.

[0099] The processor (110) extracts style features from style information. (S200)

[0100] The processor (110) generates a style motion so that style features are reflected based on content features. (S300)

[0101] The processor (110) can generate style motion based on content features and style features.

[0102] In the embodiment of the present disclosure, style information is not limited to a single format, such as text format, and may be configured in various ways.

[0103] Accordingly, when style information is received, the processor (110) can analyze the received style information to determine the data format included in the style information, and extract style features from the style information using an appropriate model based on the determined result.

[0104] In an embodiment of the present disclosure, the style information may include at least one of text data, voice data, image data, and motion data, and may further include other types of data in addition to the above-described data according to the embodiment.

[0105] In some embodiments, the electronic device (100) may analyze long texts, such as dialogue (script) between characters in order to determine the style of the characters appearing in a movie or animation.

[0106] In addition, when the processor (110) uses the voice of a character speaking in an animation or movie as input, it can analyze the voice data to identify the character's emotions and intentions and extract the characteristics of the style.

[0107] When the style information includes text data or image data, the processor (110) can analyze the data included in the style information using a VLP model (Vision-Language Pre-training model) to obtain style features as output.

[0108] Figure 4 is a diagram illustrating a process for extracting style features from style information.

[0109] Referring to FIG. 4, a process for extracting style features (450) according to a data format included in style information by an electronic device (100) is described.

[0110] The processor (110) can extract style features (450) from content motion data including images, videos, and text through a process (410) using a VLP model (135).

[0111] In one embodiment, the processor (110) can directly input style information into the VLP model (135) to extract style features (450) for data containing images, videos, and short text data.

[0112] In the embodiment of the present disclosure, the VLP model (135) is a model pre-trained with large-scale image-text data to enable interaction between images and text, and is widely available. Representative models include Contrastive Language-Image Pre-training (CLIP) and Bootstrapping Language-Image Pre-training (BLIP). Through this, the processor (110) can extract similar feature vectors based on similar content between text and images.

[0113] At this time, short text data may be word-type data containing 1 to 2 words.

[0114] The processor (110) converts first text data (e.g., long text data, sentence-type text data, conversational text data) that does not correspond to the short text data above into at least one second text data in the form of words expressing the characteristics and emotions of objects included in the first text using a large language model (134, LLM: Large language model).

[0115] For example, in order to describe a person's background, tendencies, etc., there are cases where many sentences must be input. In the case of such long text, the electronic device (100) can use a large-scale language model (134) such as GPT, BERT, LaMDA, or LLaMA to convert the text into word-type text that well expresses the personality, emotions, etc. contained in the text, and use it as input for the VLP model (135).

[0116] And, the processor (110) can input at least one second text data into the VLP model (135) to extract style features (450).

[0117] The processor (110) can extract style features (450) by performing a process (420) of performing data processing and extracting voice features when voice data is included in the style information.

[0118] At this time, the processor (110) performs data processing processes such as resampling to match the speed of voice data, filtering to obtain only data that satisfies a specific criterion, and Fourier transform to decompose into a sum of periodic functions with various frequencies, and then extracts voice features using Mel-Frequency Cepstral Coefficients (MFCCs), Melspectrograms, etc.

[0119] The processor (110) can perform a process (430) of extracting style features (450) using a motion feature extraction model (136) and a style feature extraction model (132) when 3D motion data is included in the style information.

[0120] Figure 5 is a diagram illustrating the use of an encoder model as a motion feature extraction model and a style feature extraction model for 3D motion.

[0121] Figure 6 is a diagram explaining measuring the vector distance between each extracted style feature in Figure 4.

[0122] Referring to FIG. 5, the processor (110) can use an encoder model as a motion feature extraction model (136) and a style feature extraction model (132).

[0123] This is a neural network that trains representation learning and feature learning in the form of unsupervised learning, as shown in Fig. 5. The Latent Vector is a latent variable, and the Encoder (630) corresponds to the input side (610), and the Decoder (640) corresponds to the output side (620).

[0124] At this time, the encoder acts as a kind of feature extractor, and the decoder is responsible for restoring the compressed data.

[0125] At this time, the style features extracted through the process (410, 420, 430) as in Fig. 4 exist in the form of a specific distribution for each style in the feature space (440) expressing all styles.

[0126] The processor (110) can extract style features (450) by sampling style feature vectors from the distribution of the corresponding style.

[0127] Referring to FIG. 6 following FIG. 4, the processor (110) inputs each result extracted through the execution of each process (410, 420, 430) of FIG. 4 into a linear layer to calculate a vector distance value of each extracted result (510, 520, 530, 540). At this time, the closer the calculated distance value is, the more the artificial intelligence model used in the electronic device (100) is properly trained and outputs the correct result. That is, the electronic device (100) can train the artificial intelligence model so that the distances of the feature vectors (510, 520, 530, 540) for the styles extracted through each process (410, 420, 430) become closer.

[0128] At this time, the VLP model (135) is a model that has been pre-trained with large-scale image-text data to enable interaction between images and text, so 2D images can be processed immediately, but videos cannot be processed immediately.

[0129] The reason why a Linear Layer exists at the end of each process (410, 420, 430) is to ensure that the sizes of the feature vectors (outputs) for different input methods are the same.

[0130] Accordingly, the processor (110) can regard each frame of the video as a 2D image, pass each frame through the VLP, and use the average of the resulting features.

[0131] The electronic device (100) will select and use one of various style input methods. Therefore, the style features extracted through all style input methods must be identical or nearly identical.

[0132] For example, the extracted features should be almost similar when the text "as if drunk" is entered, when drunk voice data is entered, and when 3D motion data containing drunken movements is entered.

[0133] Figure 7 is a diagram illustrating the generation of style-added content motion by repeatedly performing Convolution, AdaIN, and Upsampling.

[0134] In an embodiment of the present disclosure, the style generation model generates style motion by combining content motion features (710) and style features (720).

[0135] In an embodiment of the present disclosure, the electronic device (100) can build a model using the AdaIN method to well combine (reflect) style features (720) with content features (710).

[0136] AdaIN can be said to be a method of removing style from a feature vector containing content and then filling it back with the style desired by the creator.

[0137] At this time, the content features are received through the encoder, so their size is reduced. Therefore, the processor (110) must perform an upsampling process to increase the size to a size that expresses motion. Consequently, the processor (110) can perform process (730) by performing upsampling through the model, as shown in FIG. 7, to gradually increase the size while continuously reflecting the style through AdaIN.

[0138] And, the processor (110) can perform a process (730) as in FIG. 7 to finally generate content motion (740) with added style features.

[0139] In the above-described embodiment, the AdaIN method is used as an example to reflect style features, but it is not limited thereto, and various algorithms and models can be applied to reflect style features.

[0140] FIG. 8 is a diagram illustrating a method for training an artificial intelligence model used in a style-based motion conversion device according to an embodiment of the present disclosure.

[0141] Referring to FIG. 8, a method for training an artificial intelligence model by an electronic device (100) according to an embodiment of the present disclosure will be described in more detail.

[0142] The electronic device (100) can use the following learning methods.

[0143] The processor (110) first constructs an input pair expressing the same style.

[0144] For example, an input pair is a text data "like drunk" and data containing a drunken walking motion. The processor (110) inputs these input pairs into each model and extracts a feature vector for each.

[0145] Then, the processor (110) inputs a loss function between two feature vectors and backpropagates it to the model. The processor (110) performs training in this manner, ensuring that the distance between two feature vectors representing the same style becomes very close. Consequently, the AI ​​model trained in this manner outputs almost identical results for feature vectors representing the same style, regardless of the input method.

[0146] At this time, the first object is a zombie, the second object is a robot style, the first motion is a walking motion, and the second motion is a sitting motion.

[0147] The processor (110) extracts a first content feature from first content data (811) including a first motion of a first object using a content feature extraction model (812).

[0148] The processor (110) extracts a first style feature from the first style information (815) of the first object using a style feature extraction model (816).

[0149] The processor (110) can generate a first style motion (814) based on the first content feature and the first style feature using the style motion generation model (813).

[0150] For example, the processor (110) inputs the first content feature and the first style feature into the style motion generation model (813) to generate a first style motion (814) in which the first style feature is reflected based on the first content feature.

[0151] If the artificial intelligence model is fully trained, the distance between the vector values ​​of the first content motion data (811) and the first style motion (814) should be almost the same.

[0152] In one embodiment, the processor (110) trains the model so that the distance between the vector values ​​of the first content motion data (811) and the first style motion (814) is reduced.

[0153] The processor (110) extracts second content features from second content motion data (831) including the second motion of the second object using a content feature extraction model (832).

[0154] The processor (110) extracts second style features from second style information (834) of the second object using a style feature extraction model (835).

[0155] Then, the processor (110) inputs the second content feature and the second style feature into the style motion generation model (833) to generate a second style motion (836) in which the second style feature is reflected based on the second content feature.

[0156] If the artificial intelligence model is fully trained, the second content motion data (831) and the second style motion (836) should be almost identical.

[0157] In one embodiment, the processor (110) trains the model so that the distance between the vector values ​​of the second content motion data (831) and the second style motion (836) is reduced.

[0158] The processor (110) can generate a third style motion (819) based on the first content feature and the second style feature using the style motion generation model (818).

[0159] For example, the processor (110) generates a third style motion (819) based on the second content feature so that the first style feature is reflected using the style motion generation model (818).

[0160] The processor (110) extracts a third style feature from the third style motion (819) using a style feature extraction model (820).

[0161] The processor (110) uses a style motion generation model (817) to generate a fourth style motion (830) based on the first content feature so that the third style feature is reflected.

[0162] If the artificial intelligence model is fully trained, the first content motion data (811) and the fourth style motion (830) should be almost identical.

[0163] In one embodiment, the processor (110) trains the model so that the distance between the vector values ​​of the first content motion data (811) and the fourth style motion (830) is reduced.

[0164] The processor (110) extracts third content features from the third style motion (819) using the content feature extraction model (837).

[0165] The processor (110) generates a fifth style motion (840) based on the third content feature so that the second style feature is reflected using the style motion generation model (838).

[0166] If the artificial intelligence model is fully trained, the second content motion data (831) and the fifth style motion (840) should be almost identical.

[0167] In one embodiment, the processor (110) trains the model so that the distance between the vector values ​​of the second content motion data (831) and the fifth style motion (840) is reduced.

[0168] In Fig. 8, the content feature extraction models (812, 832, 837) may all be the same model.

[0169] In Fig. 8, the style feature extraction models (816, 835, 820) may all be the same model.

[0170] In Fig. 8, the style motion generation models (813, 817, 818, 833, 838) may all be the same model.

[0171] In an embodiment of the present disclosure, the processor (110) trains at least one model so that the vector value and the vector distance are reduced as described above. At this time, any model that is a target of training may be a model that is required for the electronic device (100) to perform a style-based motion conversion process. For example, the processor (110) may train at least one model among a content feature extraction model, a style feature extraction model, a style generation model, a large-scale language model, a motion feature extraction model, and a VLP model.

[0172] The method according to one embodiment of the present disclosure described above can be implemented as a program (or application) and stored in a medium to be executed in combination with a hardware server.

[0173] The above-described program may include codes coded in a computer language, such as C, C++, JAVA, or machine language, that can be read by the processor (CPU) of the computer through the device interface of the computer, so that the computer reads the program and executes the methods implemented as a program. Such codes may include functional codes related to functions that define functions necessary for executing the methods, and may include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. In addition, such codes may further include memory reference-related codes regarding which location (address address) of the internal or external memory of the computer should reference additional information or media necessary for the processor of the computer to execute the functions. In addition, if the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the functions, the code may further include communication-related code regarding how to communicate with any other computer or server located remotely using the communication module of the computer, and what information or media to send and receive during communication.

[0174] The above storage medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the program can be stored in various recording media on various servers that the computer can access or in various recording media on the user's computer. In addition, the medium can be distributed across network-connected computer systems, so that computer-readable code can be stored in a distributed manner.

[0175] The steps of a method or algorithm described in connection with the embodiments of the present disclosure may be implemented directly in hardware, implemented as a software module executed by hardware, or implemented by a combination thereof. The software module may reside in a random access memory (RAM), a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer-readable recording medium well known in the art to which the present disclosure pertains.

[0176] While the embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will appreciate that the present disclosure can be implemented in other specific forms without altering the technical spirit or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. Memory storing at least one process for generating style motion; and A processor comprising: a processor for performing an operation related to at least one process; The above processor, Extracting content features from content motion data containing object motion using a first model that extracts content features, Extracting the style features from the style information using a second model that extracts style features, A style motion is generated by using a third model that generates a style, wherein the content features and the style features are reflected, and the style information includes at least one of text, voice, video, and motion. A method characterized in that style features are acquired from text and images included in the style information using a VLP model (Vision-Language Pre-training model). Electronic devices.

2. In paragraph 1, The above processor, Among the above texts, a first text having a preset length or longer is converted into at least one second text in the form of words expressing the characteristics and emotions of objects included in the first text using a large-scale language model (LLM), characterized in that the at least one second text is input into the VLP model, and the style feature is obtained as an output value of the VLP model. Electronic devices.

3. In paragraph 1, The above processor, If the above style information is about a character in the game, data including a background description of the character is input together with the VLP model. Electronic devices.

4. In paragraph 1, The above processor, Using the above VLP model, a style distribution for a feature space is obtained based on the text included in the above style information, Characterized in that the style feature is obtained by sampling a style vector from the style distribution obtained above. Electronic devices.

5. In paragraph 1, The above processor, A first style feature obtained from text and images included in the style information, a second style feature obtained from voice included in the style information, and a third style feature obtained from motion included in the style information are input into a linear layer, respectively, and the vector sizes of the first style feature, the second style feature, and the third style feature are controlled to be the same. Characterized in that the second model is trained by a method of reducing the vector distance of the first style feature, the second style feature, and the third style feature. Electronic devices.

6. In paragraph 1, The above processor, Extracting first content features from first content motion data including the first motion of the first object, Extracting second content features from second content motion data that includes the second motion of the second object, Extracting a first style feature from the first style information of the first object, Extracting second style features from the second style information of the second object, Generate a first style motion based on the first content feature and the first style feature, Generate a second style motion based on the second content feature and the second style feature, Train the first model and the third model by reducing the vector distance between the first content motion data and the first style motion, Characterized in that the first model and the third model are trained by a method of reducing the vector distance between the second content motion data and the second style motion. Electronic devices.

7. In paragraph 6, The above processor, Generate a third style motion based on the second content feature and the first style feature, Extracting third style features from the above third style motion, Extracting third content features from the third style motion, Generate a fourth style motion based on the first content feature and the third style feature, Generate a fifth style motion based on the third content feature and the second style feature, Train the first model and the third model by reducing the vector distance between the first content motion data and the fourth style motion, Characterized in that the first model and the third model are trained by a method of reducing the vector distance between the second content motion data and the fifth style motion. Electronic devices.

8. In paragraph 1, The above processor, When extracting the above content features, an encoder model is used, The style motion is generated so that the extracted style features are applied while removing the remaining style in the content features using AdaIN technology, characterized in that at least one upsampling is performed on the extracted content features whose size is reduced by using the encoder model, and the extracted style features are applied. Electronic devices.

9. A method for generating style-based motion performed by a processor of an electronic device, A step of extracting content features from content motion data including object motion using a first model that extracts content features; A step of extracting style features from style information using a second model that extracts style features; A step of generating a style motion reflecting the content features and the style features using a third model for generating a style, wherein the style information includes at least one of text, voice, video, and motion; and A method characterized by comprising a step of obtaining style features from text and images included in the style information using a VLP model (Vision-Language Pre-training model). A method for generating style-based motion.

10. A computer-readable recording medium, coupled with a computer device as hardware, storing a computer program for performing the style-based motion generation method of claim 9.

Images