3D motion capture method and system

Abstract

The present disclosure relates to a 3D motion capture method executed by at least one processor, the 3D motion capture method including: receiving an input image including a human object; detecting a human region, a hand region, and a face region from at least a portion of the input image using a first machine learning model; recognizing 3D motion information from a human sub-image including the human region, a hand sub-image including the hand region, and a face sub-image including the face region; and generating a character image including a character that replicates the body motion, hand motion, and face motion of the human object included in the input image by rendering based on the 3D motion information.

Description

[Technical Field]

[0001] The present disclosure relates to a 3D motion capture method and system, and more particularly to a method and system for recognizing 3D motion information of a human object included in an image in order to generate a character image including a character that replicates the body motion, hand motion, and facial motion of the human object included in the image. [Background technology]

[0002] As interest in virtual spaces increases, so does interest in characters that represent users in virtual spaces. Many service providers are making great efforts to realize virtual 3D characters that behave in the same way as humans in the real world.

[0003] Motion capture is a technique that uses footage of actual animal or human movements to create a digital character that moves in the same way as the animal or human in the video. 3D motion capture often involves complex calculations, which can result in significant delays when performing motion capture. Furthermore, when performing motion capture in real time, calculations must be completed within a short time frame, which can result in unnatural character movements or the character being unable to accurately replicate the detailed movements and facial expressions of the human subject. Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides a 3D motion capture method, a computer-readable non-transitory recording medium having instructions recorded thereon, and an apparatus (system) for solving the above problems. [Means for solving the problem]

[0005] The present disclosure may be embodied in numerous ways, including as a method, an apparatus (system), or a computer-readable non-transitory recording medium having instructions recorded thereon.

[0006] According to one embodiment of the present disclosure, a 3D motion capture method executed by at least one processor includes: receiving an input image including a human object; detecting a human region, a hand region, and a face region from at least a portion of the input image using a first machine learning model; and recognizing 3D motion information from a human sub-image including the human region, a hand sub-image including the hand region, and a face sub-image including the face region; and generating a character image including a character that mimics the body motion, hand motion, and face motion of the human object included in the input image by rendering based on the 3D motion information.

[0007] According to one embodiment of the present disclosure, the step of receiving an input image includes a step of sequentially receiving a plurality of frame images including a human object, and as each of the plurality of frame images is sequentially received, a step of detecting and a step of recognizing are performed in real time for each of the plurality of frame images, and a plurality of character images including a character that imitates the body motion, hand motion, and facial motion of the human object included in each of the plurality of frame images are generated by rendering based on 3D motion information.

[0008] According to one embodiment of the present disclosure, the first machine learning model includes a hand classifier, and the detecting step includes classifying the detected hand region into at least one of a left hand region, a right hand region, or both hand regions using the hand classifier.

[0009] According to one embodiment of the present disclosure, the recognizing step includes: recognizing body motion information from a human sub-image including a human region using a second machine learning model; recognizing hand motion information from a hand sub-image including a hand region using a third machine learning model; and recognizing face motion information from a face sub-image including a face region using a fourth machine learning model.

[0010] According to one embodiment of the present disclosure, the second machine learning model is a model configured to output body model parameters and camera parameters based on the human sub-image, and the body motion information includes body 3D joint angle information, body 3D joint position information, and body 3D mesh information obtained based on the body model parameters.

[0011] According to one embodiment of the present disclosure, the second machine learning model is a model configured to further output body 2D joint estimation information and a body part segmentation map based on the human sub-image.

[0012] According to one embodiment of the present disclosure, the method further includes classifying the detected hand region into at least one of a left hand region, a right hand region, or both hand regions using the body 3D joint position information, the body 2D joint estimation information, and the body part segmentation map.

[0013] According to one embodiment of the present disclosure, the first machine learning model is a model configured to detect human regions, hand regions, and face regions from at least a portion of an input image and recognize body motion information, and the second machine learning model is the same model as the first machine learning model.

[0014] According to one embodiment of the present disclosure, the step of receiving an input image includes a step of sequentially receiving multiple frame images of the same size, and as each of the multiple frame images is sequentially received, a detection step and a 3D motion information recognition step are performed in real time for each of the multiple frame images, and in the detection step and 3D motion information recognition step for a specific frame image among the multiple frame images, at least a portion of the specific frame image is applied as an input to a first machine learning model, and if a human region is detected in a previous frame image of the specific frame image, at least a portion of the specific frame image applied as an input to the first machine learning model includes an area at the same position as the human region detected in the previous frame image, and if a human region is not detected in the previous frame image of the specific frame image, it is the entire specific frame image.

[0015] According to one embodiment of the present disclosure, the third machine learning model includes a hand classifier, and the step of recognizing the hand motion information includes classifying the detected hand region into at least one of a left hand region, a right hand region, or both hand regions using the hand classifier.

[0016] According to one embodiment of the present disclosure, the third machine learning model is a model configured to output hand pose parameters based on the hand sub-images, and the hand motion information includes first wrist angle information, hand 3D joint angle information, hand 3D mesh information, and camera parameters obtained based on the hand pose parameters.

[0017] According to one embodiment of the present disclosure, the body motion information includes second wrist angle information acquired based on the output of a second machine learning model, and the step of recognizing the 3D motion information further includes a step of correcting the second wrist angle information acquired based on the output of the second machine learning model using the first wrist angle information acquired based on the hand pose parameters.

[0018] According to one embodiment of the present disclosure, the body motion information includes elbow angle information and shoulder angle information acquired based on the output of the second machine learning model, and the step of recognizing the 3D motion information further includes a step of correcting the elbow angle information and shoulder angle information acquired based on the output of the second machine learning model using the first wrist angle information.

[0019] According to one embodiment of the present disclosure, the fourth machine learning model is a model configured to output 3D face model parameters, 3D facial landmark information, blendshapes, and camera parameters based on the facial sub-images.

[0020] According to one embodiment of the present disclosure, the facial motion information includes head pose information, facial shape information, and facial expression information, and the step of recognizing the facial motion information includes a step of classifying and recognizing the head pose information, facial shape information, and facial expression information from the facial sub-image using a fourth machine learning model.

[0021] According to one embodiment of the present disclosure, the facial expression information includes gaze information and tongue movement information.

[0022] According to one embodiment of the present disclosure, the body motion information includes head angle information obtained based on the output of a second machine learning model, and the step of recognizing the 3D motion information further includes a step of correcting the head angle information obtained based on the output of the second machine learning model using head pose information.

[0023] According to one embodiment of the present disclosure, the 3D motion information includes joint angle information, and the step of recognizing the 3D motion information includes the step of correcting the joint angle information using at least one of a body proportion of a human object included in the input image, a body proportion of a character, or joint angle restriction information related to a character.

[0024] A computer-readable non-transitory recording medium having instructions recorded thereon for executing a 3D motion capture method on a computer according to one embodiment of the present disclosure is provided.

[0025] A motion capture system according to an embodiment of the present disclosure includes a memory and at least one processor coupled to the memory and configured to execute at least one computer-readable program stored in the memory, the at least one program receiving an input image including a human object, detecting a human region, a hand region, and a face region from at least a portion of the input image using a first machine learning model, recognizing 3D motion information from a human sub-image including the human region, a hand sub-image including the hand region, and a face sub-image including the face region, and generating a character image including a character that mimics the body motion, hand motion, and face motion of the human object included in the input image by rendering based on the 3D motion information. [Effects of the Invention]

[0026] According to some embodiments of the present disclosure, it is possible to individually recognize motion information for each target object using a plurality of models of different target objects. Also, it is possible to individually recognize motion information for each target object based on a plurality of sub-images in which each target object occupies a large proportion. Therefore, it is possible to extract detailed motion information from an image in a short time.

[0027] According to some embodiments of the present disclosure, object detection and body motion recognition can be performed simultaneously to perform motion capture within a short time. Also, by using the detection result for a previous frame image for the detection for the next frame image, the amount of calculation or time required for motion capture for continuously received images can be reduced.

[0028] According to some embodiments of the present disclosure, motion information acquired based on the output of a particular model can be used to correct or replace motion information acquired based on the output of another model, thereby generating a character image that more accurately and naturally replicates the motion of a human object.

[0029] According to some embodiments of the present disclosure, head pose information, face shape information, and facial expression information can be separately recognized. In addition, the facial expression information can include gaze information and tongue movement information. Therefore, a character image that more precisely replicates the facial expression of a human object can be generated.

[0030] The effects of the present disclosure are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those having ordinary skill in the art to which the present disclosure pertains (referred to as "persons skilled in the art") from the description in the claims. [Brief explanation of the drawings]

[0031] Embodiments of the present disclosure are described below with reference to the accompanying drawings, in which like reference numerals indicate like elements, but are not limited to the drawings, in which:

[0032] [Figure 1] FIG. 1 illustrates an example utilizing a 3D motion capture method according to one embodiment of the present disclosure. [Figure 2] FIG. 2 is a block diagram showing the internal configuration of a motion capture system 200 according to an embodiment of the present disclosure. [Figure 3] FIG. 1 is a schematic diagram illustrating an example of a 3D motion capture method according to one embodiment of the present disclosure. [Figure 4] 1 is a diagram illustrating an example of an input image including a human object according to an embodiment of the present disclosure. [Figure 5] FIG. 10 is a diagram illustrating an example of a motion capture system detecting a human region, a hand region, and a face region from an input image according to one embodiment of the present disclosure. [Figure 6]1A and 1B are diagrams illustrating an example of a motion capture system recognizing body motion information from a human sub-image including a human region according to an embodiment of the present disclosure. [Figure 7] FIG. 1 illustrates an example of a motion capture system using the same model to perform detection and body motion information recognition according to an embodiment of the present disclosure. [Figure 8] FIG. 1 illustrates an example of a motion capture system using the same model to perform detection and body motion information recognition according to an embodiment of the present disclosure. [Figure 9] FIG. 10 illustrates an example of a motion capture system recognizing hand motion information from a hand sub-image including a hand region, according to one embodiment of the present disclosure. [Figure 10] FIG. 1 illustrates an example of a motion capture system recognizing facial motion information from a facial sub-image including a facial region, according to one embodiment of the present disclosure. [Figure 11] FIG. 10 illustrates an example of performing rendering based on motion information according to an embodiment of the present disclosure. [Figure 12] 10A and 10B are diagrams illustrating examples of character images including a character that replicates the motion of a human object according to an embodiment of the present disclosure. [Figure 13] 1 is a flowchart illustrating an example of a 3D motion capture method according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0033] Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, specific descriptions of well-known functions and configurations will be omitted if they may unnecessarily obscure the gist of the present disclosure.

[0034] In the accompanying drawings, identical or corresponding components are denoted by the same reference numerals. In addition, in the following description of the embodiments, duplicated descriptions of identical or corresponding components may be omitted. However, the omission of a description of a component does not mean that such a component is not included in a certain embodiment.

[0035] The advantages and features of the disclosed embodiments, as well as methods for achieving them, will become apparent from the following examples, taken in conjunction with the accompanying drawings. However, the present disclosure is not limited to the following examples, and may be embodied in various different forms. The present examples are provided solely for the purpose of completing the disclosure and fully conveying the scope of the invention to those skilled in the art.

[0036] The terms used in this specification will be briefly explained, and the disclosed embodiments will be specifically described. The terms used in this specification are currently commonly used and generally selected in consideration of the function of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedent cases, the emergence of new technologies, etc. In addition, in certain cases, the applicant may arbitrarily select terms, and in such cases, their meanings will be described in detail in the description section of this disclosure. Therefore, the terms used in this disclosure should be defined based on the meanings of the terms and the overall content of this disclosure, rather than simply by their names.

[0037] In this specification, the singular includes the plural unless the context clearly dictates otherwise. Furthermore, the plural includes the singular unless the context clearly dictates otherwise. Throughout the specification, when a part includes a certain element, this does not mean that it excludes other elements, but that it may further include other elements, unless otherwise specified.

[0038] Furthermore, the terms "module" and "module" used in this specification refer to software or hardware components, each of which performs a certain function. However, the terms "module" and "module" are not limited to software or hardware. A "module" or "module" may be configured to reside on an addressable storage medium or to execute one or more processors. Thus, as an example, a "module" or "module" may include components such as software components, object-oriented software components, class components, and task components, as well as at least one of processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. Components and "modules" or "modules" may be combined into fewer components and "modules" or "modules" or further separated into additional components and "modules" or "modules" in terms of the functions provided therein.

[0039] According to one embodiment of the present disclosure, a "module" or a "unit" may be embodied by a processor and memory. "Processor" should be broadly interpreted to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, "processor" may refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field-programmable gate array (FPGA), etc. "Processor" may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or any other such configuration. Also, "memory" should be broadly interpreted to include any electronic component capable of storing electronic information. "Memory" may refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage devices, registers, etc. Memory is said to be in electronic communication with a processor if the processor can read information from or store information in the memory. Memory that is integrated into a processor is in electronic communication with the processor.

[0040] In the present disclosure, a "system" may include at least one of a server device and a cloud device, but is not limited thereto. For example, a system may be configured with one or more server devices. As another example, a system may be configured with one or more cloud devices. As yet another example, a system may be configured and operated by a server device and a cloud device together.

[0041] In this disclosure, a "machine learning model" may include any model used to infer an answer to a given input. According to one embodiment, the machine learning model may include an artificial neural network model including an input layer, multiple hidden layers, and an output layer, where each layer may include multiple nodes. Also, in this disclosure, a machine learning model may refer to an artificial neural network model, and an artificial neural network model may refer to a machine learning model.

[0042] In the present disclosure, "each of a plurality of A's" or "each of a plurality of A's" can refer to each of all components included in the plurality of A's, or can refer to each of some of the components included in the plurality of A's.

[0043] In the present disclosure, a "person" or a "person object" may include not only a person but also any individual that can move in the real world or the virtual world, such as an animal. In some embodiments of the present disclosure, a "person" or a "person object" may refer to an area in an image where a person or a person object is captured or an area in which a person or a person object is detected from an image.

[0044] In some embodiments of the present disclosure, an "image" may include not only one image but multiple images, for example, an image may include a video including multiple frame images.

[0045] FIG. 1 is a diagram illustrating an example of using a 3D motion capture method according to an embodiment of the present disclosure. According to one embodiment, a user can capture a human object 112 (e.g., the user) moving using a user terminal (e.g., a first user terminal 110). The user terminal can include, but is not limited to, a mobile device (e.g., a smartphone, a tablet PC, etc.), a webcam device with processing capabilities (e.g., a webcam device including a processor chip, etc.), a desktop PC, etc. In the example of FIG. 1, the human object 112 included in the captured image is shown as an illustration, but this is for illustrative purposes only, and the captured image in an actual implementation may be an image of an actual person performing a specific motion.

[0046] The user terminal can recognize motion information of the human object 112 included in the captured image. For example, the motion information may be 3D motion information and may include body motion information, hand motion information, and facial motion information. According to one embodiment, the body motion information, hand motion information, and facial motion information may be recognized using separate models. Rendering based on the recognized motion information can generate a character image 122 including a character that replicates the body motion, hand motion, and facial motion of the human object 112 included in the image.

[0047] A user terminal (e.g., a second user terminal 120) can output the generated character image 122. In the example of FIG. 1, the first user terminal 110 captures the human object 112 and the second user terminal 120 outputs the character image 122. However, this is for convenience of explanation and is not limiting. At least one of the operations of capturing the human object 112, recognizing motion information, rendering, and outputting the character image 122 can be performed on the same device, or each operation can be performed on a separate device. For example, an image or video captured by a first device can be transmitted to a second device, and motion information recognition for the image or video can be performed by the second device. The recognized motion information can be transmitted to a third device and rendered by the third device. In one embodiment, when an image including the human object 112 is captured and the generated character image 122 is output using the same user terminal, the captured image and the character image 122 that mimics the motion of the human object 112 included in the image can be simultaneously output on a split screen of the user terminal display.

[0048] Furthermore, the user terminal may provide a user interface (e.g., GUI) (not shown) that can utilize the character image 122. For example, the generated character image 122 may be utilized in various fields that require motion capture, such as streaming broadcasting, video production, animation production, game motion production, and webtoon production, and the user terminal may provide an appropriate user interface that suits the purpose.

[0049] In some embodiments of the present disclosure, at least one device that performs at least one task for implementing the motion capture method of the present disclosure (e.g., at least one task such as photographing a human object 112, recognizing motion information, correcting motion information, rendering, and outputting a character image 122) can be referred to as a motion capture system.

[0050] 2 is a block diagram illustrating the internal configuration of a motion capture system 200 according to one embodiment of the present disclosure. The motion capture system 200 may include a memory 210, a processor 220, a communication module 230, and an input / output interface 240. The motion capture system 200 may be configured to communicate information and / or data over a network using the communication module 230.

[0051] The memory 210 may include any non-transitory computer-readable recording medium. According to one embodiment, the memory 210 may include a permanent mass storage device such as a read only memory (ROM), a disk drive, a solid state drive (SSD), or a flash memory. As another example, a permanent mass storage device such as a ROM, an SSD, a flash memory, or a disk drive may be a separate permanent storage device distinct from the memory and may be included in the motion capture system 200. The memory 210 may also store an operating system and at least one program code (e.g., code for object detection, motion recognition, etc., that is incorporated into and operated by the motion capture system 200).

[0052] Such software components may be loaded from a computer-readable recording medium separate from the memory 210. Such a separate computer-readable recording medium may include a recording medium directly connectable to the motion capture system 200, such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, or a memory card. As another example, the software components may be loaded into the memory 210 via the communication module 230, which is not a computer-readable recording medium. For example, at least one program may be loaded into the memory 210 based on a computer program (e.g., a program for object detection, motion recognition, etc.) embedded in a file provided via the communication module 230 by a developer or a file distribution system that distributes application embedding files.

[0053] The processor 220 may be configured to process computer program instructions by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to a user terminal (not shown) or other external systems via the memory 210 or the communication module 230. For example, the processor 220 may receive an input image including a human object and detect a human region, a hand region, and a face region from at least a portion of the input image using a first machine learning model. The processor 220 may also recognize 3D motion information from a human sub-image including a human region, a hand sub-image including a hand region, and a face sub-image including a face region. According to one embodiment, the processor 220 may sequentially receive a plurality of frame images including a human object and perform detection and motion information recognition for each of the plurality of frame images as the plurality of frame images are sequentially received.

[0054] The communication module 230 can provide a configuration or function for the motion capture system 200 and an external device to communicate with each other via a network, and can provide a configuration or function for the motion capture system 200 to communicate with an external system (such as a separate cloud system, for example). As an example, control signals, instructions, data, etc. provided under the control of the processor 220 of the motion capture system 200 can be transmitted to the external device and / or external system via the communication module of the external device and / or external system via the communication module 230 and the network.

[0055] Furthermore, the input / output interface 240 of the motion capture system 200 may be a means for interfacing with an input or output device (not shown) that may be coupled to or included in the motion capture system 200. While the input / output interface 240 is shown in FIG. 2 as an element configured separately from the processor 220, this is not limiting, and the input / output interface 240 may also be configured to be included in the processor 220. The motion capture system 200 may include more components than those shown in FIG. 2. However, it is not necessary to explicitly show most of the prior art components.

[0056] FIG. 3 is a schematic diagram illustrating an example of a 3D motion capture method according to one embodiment of the present disclosure.

[0057] According to one embodiment, a motion capture system may receive an input image 310 including a human object and extract motion information 330 by performing motion recognition 320 on the input image 310 .

[0058] For example, the motion capture system may first use detection model 321 to detect human, hand, and face regions from at least a portion of input image 310, as will be described in more detail below with reference to FIG.

[0059] The motion capture system can then recognize 320 motion information from the human sub-image containing the detected human region, the hand sub-image containing the detected hand region, and the face sub-image containing the detected face region.

[0060] For example, the motion capture system may recognize body motion information from a human sub-image using a body motion recognition model 322. The motion capture system may also recognize hand motion information from a hand sub-image using a hand motion recognition model 323. The motion capture system may also recognize facial motion information from a facial sub-image using a facial motion recognition model 324. By individually recognizing motion information for each target object based on a plurality of sub-images in which each target object occupies a large proportion using a plurality of models for different target objects, detailed motion information 330 can be extracted from an image in a short time. Specific examples of how the motion capture system performs motion recognition 320 will be described in detail below with reference to FIGS. 6 to 10.

[0061] According to one embodiment, the motion capture system may correct and / or supplement at least a portion of the recognized motion information using an exception handling algorithm 325. For example, the exception handling algorithm 325 may be an algorithm that corrects and / or supplements motion information so that the character naturally assumes a default motion (e.g., a default pose) or maintains the motion or pose of the previous frame in the generated character image 350 when a specific body part (e.g., the back of a hand) of a human object in the input image 310 is out of the screen, the human object moves quickly and blurs, the right / left hand classification fails, or there is a problem with detection and / or motion recognition for other reasons. Furthermore, the motion capture system may apply a smoothing algorithm to at least a portion of the recognized motion information (e.g., joint angle information) to avoid abrupt pose changes and realize natural motion during exception handling.

[0062] If the body proportions of the human object included in the input image 310 are different from those of the target character included in the character image 350, applying the motion information of the human object in the image 310 to the character as is may result in an inaccurate character replicating the motion of the human object. Therefore, according to one embodiment, the motion capture system may correct at least a portion of the motion information using a retargeting algorithm 326.

[0063] For example, when a human object in an image claps its hands, if the joint angles of the human object are applied to a character with a different body proportion, the character's hands may not touch. Therefore, the motion capture system may correct the joint angle information using at least one of the body proportions of the human object or the character included in the input image 310 so that the character can properly replicate the motion of the human object.

[0064] As another example, if the character's head is too large, when the human object in the input image 310 raises its arm, the character's arm in the generated character image 350 may penetrate the head, resulting in inter-penetration. To prevent this inter-penetration, the motion capture system can correct joint angle information using joint angle restriction information associated with the character.

[0065] Through the above-described series of motion recognition 320 processes, motion information 330 can be extracted from the input image 310. Rendering 340 can be performed based on the extracted motion information 330, thereby generating a character image 350 including a character that replicates the movement of a human object included in the input image 310. A specific example of performing rendering 340 based on the motion information 330 will be described in detail below with reference to FIG. 11.

[0066] 4 is a diagram illustrating an example of an input image including a human object according to an embodiment of the present disclosure. In the example of FIG. 4, the human object included in the input image is displayed as an illustration, but this is for illustrative purposes only. In actual implementation, the input image may be an image of an actual person performing a specific motion.

[0067] The motion capture system can receive input images including a human object, for example, images or videos captured in real time via a camera, or by loading or decoding pre-recorded images or videos.

[0068] According to one embodiment, a motion capture system can sequentially receive a plurality of frame images 410, 420, 430 including a human object.

[0069] For example, the user can capture an image in which the human subject looks straight into the camera, makes a specific face, starts a motion with one hand clenched in a fist and the other hand with all fingers extended, sequentially bends the fingers of the extended hand, and ends with all fingers of the other hand folded in a clenched fist pose.

[0070] Therefore, a first frame image 410 in which the human object is standing facing the camera, making a specific face, making a fist with one hand clenched and the other hand with all fingers extended, a second frame image 420 in which the human object is standing facing the camera, making a specific face, making a fist with one hand clenched and the other hand with some fingers bent and the other fingers extended, and a third frame image 430 in which the human object is standing facing the camera, making a specific face, and making a fist with both hands clenched can be captured sequentially.

[0071] The motion capture system may sequentially receive the first frame image 410, the second frame image 420, and the third frame image 430. For example, the motion capture system may sequentially receive the first frame image through the third frame image 410, 420, and 430, which are captured in real time. As another example, the motion capture system may sequentially receive the first frame image through the third frame image 410, 420, and 430 by loading or decoding pre-captured images. In one embodiment, the motion capture system may also receive metadata along with the images, including frame numbers indicating the order of the first frame image through the third frame image 410, 420, and 430.

[0072] The motion capture system can perform a motion information recognition process (described below) on each of the plurality of frame images 410, 420, and 430 in real time as each of the plurality of frame images 410, 420, and 430 is sequentially received.

[0073] FIG. 5 is a diagram illustrating an example of a motion capture system detecting a human region 412, hand regions 414_1 and 414_2, and a face region 416 from an input image according to an embodiment of the present disclosure.

[0074] According to one embodiment, the motion capture system can detect a human region 412, hand regions 414_1 and 414_2, and a face region 416 from a first frame image 410 using a detection model 321. In one embodiment, the detection model 321 may be a machine learning model (e.g., a network based on deep learning). For example, the detection model 321 may be a machine learning model configured to output, based on an input image, a bounding box 412 including a human region, bounding boxes 414_1 and 414_2 including hand regions, and a bounding box 416 including a face region included in the input image.

[0075] Furthermore, the detection model 321 may include a hand classifier. In such an embodiment, the motion capture system may use the detection model 321 including the hand classifier to classify the detected hand regions 414_1, 414_2 into at least one of a left hand region, a right hand region, or both hand regions. For example, the detection model 321 may be a model configured to further output whether each of the bounding boxes 414_1, 414_2 including the hand regions includes a left hand, a right hand, or both hands.

[0076] 6 is a diagram illustrating an example in which a motion capture system recognizes body motion information 620 from a human sub-image 610 including a human region according to an embodiment of the present disclosure. The motion capture system can recognize body motion information 620 from the human sub-image 610 including the human region using a body motion recognition model 322. The body motion recognition model 322 may be a machine learning model (e.g., a network based on deep learning). Furthermore, the human sub-image 610 may be a sub-image obtained by loosely cropping the first frame image to include the human region detected by the detection model.

[0077] According to an embodiment, the body motion information 620 may include body 3D joint angle information, body 3D joint position information, body 3D mesh information, camera parameters, etc. Here, the joints may include, but are not limited to, the head, neck, shoulders, elbows, wrists, phalanges, pelvis, knees, ankles, phalanges, etc. According to an embodiment, the body motion recognition model 322 may be a model configured to output body model parameters (e.g., Skinned Multi-Person Linear Model Parameters; SMPL parameters) and camera parameters based on the human sub-image 610. Furthermore, at least a portion of the body motion information 620 may be acquired based on the body model parameters and / or camera parameters output by the body motion recognition model 322. Furthermore, the motion capture system can obtain joint coordinate information from the image by reprojecting at least a portion of the recognized body motion information 620 (e.g., 3D joint angle information, body 3D joint position information, etc.) as an image.

[0078] Furthermore, the body motion information 620 may further include 2D joint estimation information and a human part segmentation map. For example, the body motion recognition model 322 may be a model configured to further output body 2D joint estimation information and a human part segmentation map based on the human sub-image 610. In such an embodiment, the motion capture system may classify a hand region detected from the detection model into at least one of a left hand region, a right hand region, or both hand regions using the body 3D joint position information, the body 2D joint estimation information, and the body part segmentation map.

[0079] According to one embodiment, the motion capture system may apply a smoothing algorithm to at least a portion of the body motion information 620 (e.g., 3D joint angle information, camera parameters, etc.) to prevent jittering when the body motion information 620 is applied to a character.

[0080] 7 and 8 are diagrams illustrating an example in which a motion capture system according to an embodiment of the present disclosure performs detection and body motion information recognition using the same model. According to an embodiment, the motion capture system can simultaneously perform detection and body motion information recognition using the same machine learning model, instead of using separate models for each. For example, the motion capture system can use a detection and body motion recognition model 700 to detect the human region 412, hand regions 414_1 and 414_2, and face region 416 from the first frame image 410 and recognize the body motion information 620. Here, the detection and body motion recognition model 700 may be a machine learning model configured to detect the human region 412, hand regions 414_1 and 414_2, and face region 416 from at least a portion of the input image and recognize the body motion information 620. For example, the detection and body motion recognition model 700 may be, but is not limited to, a deep learning-based network.

[0081] When a motion capture system sequentially receives multiple frame images, the input of the detection and body motion recognition model 700 can be determined based on the detection results from the previous frame image. For example, the motion capture system can apply at least a portion 810 of the second frame image as input to the detection and body motion recognition model 700 to detect a human region 812, hand regions 814_1 and 814_2, and a face region 816 in the second frame image and recognize body motion information 820. Here, the image applied as input to the detection and body motion recognition model 700 can be determined based on whether the human region 412 was detected in the first frame image 410, which is the frame image prior to the second frame image.

[0082] For example, if a human region 412 is detected in the first frame image 410, the motion capture system can apply a sub-image 810 (e.g., an image obtained by loosely cropping the area of ​​the second frame image at the same position as the human region detected in the first frame image 410) that includes the area of ​​the second frame image at the same position as the human region detected in the first frame image 410 as an input to the detection and body motion recognition model 700. In contrast, if a human region 412 is not detected in the first frame image 410, the motion capture system can apply the entire second frame image as an input to the detection and body motion recognition model 700. This configuration allows the motion capture system to perform motion capture at a faster speed for continuously received images.

[0083] 9 is a diagram illustrating an example in which a motion capture system according to an embodiment of the present disclosure recognizes hand motion information 920 from hand sub-images 910_1 and 910_2 including hand regions. The motion capture system can recognize hand motion information 920 from hand sub-images 910_1 and 910_2 including hand regions using a hand motion recognition model 323. The hand motion recognition model 323 may be a machine learning model (e.g., a network based on deep learning).

[0084] According to one embodiment, the hand sub-images 910_1 and 910_2 may be sub-images obtained by loosely cropping the first frame image to include hand regions detected from the detection model (or the detection and body motion recognition model). In the example of Fig. 9, the first hand sub-image 910_1 including one hand and the second hand sub-image 910_2 including the other hand are applied as separate inputs, but this is not limiting. For example, a single hand sub-image including both hands (e.g., a hand sub-image including both hands at the moment when a human subject is clapping) may be applied as an input.

[0085] According to one embodiment, the hand motion information 920 may include first wrist angle information, hand 3D joint angle information, hand 3D mesh information, and camera parameters. In one embodiment, the hand motion recognition model 323 may be a model configured to output hand pose parameters (e.g., parameters representing a 3D hand pose, for example, hand Model with Articulated and Non-rigid defOrmations parameters (MANO parameters)) based on the hand sub-images 910_1 and 910_2, and at least a portion of the hand motion information 920 may be acquired based on the hand pose parameters output by the hand motion recognition model 323.

[0086] According to one embodiment, the body motion information may include second wrist angle information, elbow angle information, and shoulder angle information acquired based on the output of the body motion recognition model (or detection and body motion recognition model). The first wrist angle information may be more accurate than the second wrist angle information because it is acquired based on hand sub-images 910_1 and 910_2, which have a large proportion of hands in the image, and based on the output (hand pose parameters) of the hand motion recognition model 323, which recognizes only the hand motion information 920. Therefore, in this embodiment, the motion capture system may use the first wrist angle information to correct or replace the second wrist angle information included in the body motion information. Additionally or alternatively, the motion capture system may use the more accurate first wrist angle information to correct the elbow angle information and shoulder angle information included in the body motion information, thereby achieving a natural pose in which the body and hands are well-matched to the human anatomy.

[0087] Further, the hand motion recognition model 323 may include a hand classifier. In such an embodiment, the motion capture system may use the hand motion recognition model 323 including the hand classifier to determine whether the hand sub-images 910_1, 910_2 include hands or classify the hands included in the hand sub-images 910_1, 910_2 as at least one of a left hand, a right hand, or both hands.

[0088] According to one embodiment, the motion capture system may apply a smoothing algorithm to at least a portion of the hand motion information 920 (e.g., hand 3D joint angle information, wrist angle information, etc.) to prevent slight trembling when the hand motion information 920 is applied to a character.

[0089] 10 is a diagram illustrating an example of a motion capture system recognizing facial motion information 1020 from a facial sub-image 1010 including a facial region, according to one embodiment of the present disclosure. The motion capture system can recognize the facial motion information 1020 from the facial sub-image 1010 including a facial region using a facial motion recognition model 324. The facial motion recognition model 324 may be a machine learning model (e.g., a network based on deep learning).

[0090] According to one embodiment, the facial sub-image 1010 may be a sub-image obtained by loosely cropping the first frame image to include a facial region detected by the detection model (or the detection and body motion recognition model). According to one embodiment, the motion capture system may perform facial alignment pre-processing on the facial sub-image 1010. For example, the motion capture system may receive multiple facial landmarks (e.g., eye corners, eyebrows, nose, mouth, and chin) as input and perform the facial alignment pre-processing. The image on which the facial alignment pre-processing has been performed may be applied as an input to the facial motion recognition model 324, or alternatively, the facial motion recognition model 324 may perform the facial alignment pre-processing.

[0091] The facial motion information 1020 may include head pose information, face shape information, and facial expression information. Here, the facial expression information may include gaze information and tongue movement information. For example, the facial motion recognition model 324 may be a model configured to output 3D face model (e.g., a 3D Morphable Model (3DMM) such as FaceScape) parameters, 3D facial landmark information, blend shapes (e.g., blend shapes based on facial action units (FACS) that allow a 3D character to express facial expressions), and camera parameters based on the facial sub-image 1010, and at least a portion of the facial motion information 1020 may be acquired based on the output of the facial motion recognition model 324. In one embodiment, the facial motion recognition model 324 may be a model trained to separately recognize head pose information, face shape information, and facial expression information. Therefore, the motion capture system can separately recognize head pose information, face shape information, and facial expression information.

[0092] According to one embodiment, the body motion information may include second head pose information acquired based on the output of the body motion recognition model (or the detection and body motion recognition model). The first head pose information may be more accurate than the second head pose information because it is acquired based on the face sub-image 1010, which has a large facial component in the image, and based on the output of the face motion recognition model 324, which recognizes only the face motion information 1020. Therefore, in this embodiment, the motion capture system may use the first head pose information to correct or replace the second head pose information included in the body motion information.

[0093] According to one embodiment, the motion capture system may apply a smoothing algorithm to at least a portion of the facial motion information 1020 (e.g., 3D facial landmark information, head pose information, facial shape information, facial expression information, etc.) to prevent micro-tremors from occurring when the facial motion information 1020 is applied to a character.

[0094] 11 is a diagram illustrating an example of performing rendering 340 based on motion information 1110 according to one embodiment of the present disclosure. Rendering 340 can be performed on the same device as motion recognition or on a different device.

[0095] According to one embodiment, rendering 340 can be performed on the same device as motion recognition. For example, if motion recognition and rendering 340 are performed in the same device in the same program (or the same process), separate communication may not be necessary. Alternatively, if motion recognition and rendering 340 are performed in the same device but by separate programs (or separate processes), an Inter Process Communication (IPC) communication method can be used to communicate motion information 1110.

[0096] Alternatively, rendering 340 may be performed by a device different from motion recognition. For example, motion recognition may be performed by a first device, and rendering 340 may be performed by a second device. In this case, motion information 1110 may be transmitted to the second device using any wired or wireless communication method between devices (e.g., USB communication, Wi-Fi wireless communication, socket communication using a LAN line, etc.).

[0097] When the motion information 1110 is transmitted to another device, program, or process using any communication method, according to an embodiment, the motion information 1110 may be transmitted in parallel using multiple threads or multiple processes for efficient transmission. Additionally or alternatively, metadata including a frame number indicating the order of the motion information 1110 may be transmitted together with the motion information 1110. Additionally or alternatively, the motion information 1110 may be encrypted before transmission so that other users cannot access the motion information 1110. If the motion information 1110 is transmitted encrypted, the device (or program) performing the rendering 340 may decrypt the motion information 1110 before performing the rendering 340.

[0098] According to one embodiment, a character image 1120 including a character that replicates the body motion, hand motion, and facial motion of a human object included in a first frame image can be generated by rendering 340 based on the motion information 1110. For example, a 3D character that performs the same motion as a human object can be generated by rendering 340 based on the motion information 1110 received in real time using a 3D engine (e.g., Unity, Unreal Engine, etc.). The rendering 340 can be performed in a manner in which the character is fixed within the rendering engine and the camera moves, or in a manner in which the camera is fixed and the character moves.

[0099] According to one embodiment, rendering 340 may be performed to reflect not only the character's motion but also its position and size in space. Furthermore, the character (or character image 1120) may be enlarged or reduced by dynamically adjusting the field of view (e.g., FoV value) depending on the orientation (e.g., portrait or landscape) of the display on which the character image 1120 is output.

[0100] Furthermore, a background scene for the character image 1120 can be generated by the rendering 340. For example, the background scene may be the same as or similar to the background scene included in the first frame image. As another example, the background scene may be different from the background scene included in the first frame image and may be a basic background scene or a background scene selected by the user.

[0101] 12 is a diagram illustrating an example of a character image including a character that mimics the motion of a human object according to an embodiment of the present disclosure. Through the motion recognition process and rendering process as illustrated above, a first character image 1210 including a character that mimics the motion of a human object included in a first frame image can be generated.

[0102] In one embodiment, when a motion capture system sequentially receives a plurality of frame images including a human object, it can sequentially generate a plurality of character images for each of the plurality of frame images. For example, when the motion capture system sequentially receives a first frame image, a second frame image, and a third frame image, it can sequentially generate a first character image 1210 including a character that imitates the motion of the human object included in the first frame image, a second character image 1220 including a character that imitates the motion of the human object included in the second frame image, and a third character image 1230 including a character that imitates the motion of the human object included in the third frame image.

[0103] According to one embodiment, the generated character image may be output to a user terminal. For example, a first character image 1210, a second character image 1220, and a third character image 1230 may be sequentially output to the user terminal. The user terminal to which the character image is output may be the same as or different from the user terminal that captures the image including the human object. In one embodiment, when an image including the human object is captured and the generated character image is output using the same user terminal, the captured image and the character image that mimics the motion of the human object included in the image may be simultaneously output on a split screen of the display of the user terminal.

[0104] 13 is a flowchart illustrating an example of a 3D motion capture method 1300 according to one embodiment of the present disclosure. The method 1300 may begin with at least one processor (e.g., at least one processor of a motion capture system) receiving an input image including a human object (S1310).

[0105] The processor may then use the first machine learning model to detect human regions, hand regions, and face regions from at least a portion of the input image (S1320).

[0106] According to one embodiment, the first machine learning model used for detection can include a hand classifier, and the processor can use the hand classifier to classify the detected hand region into at least one of a left hand region, a right hand region, or both hand regions.

[0107] The processor can then recognize 3D motion information from the human sub-image including the detected human region, the hand sub-image including the detected hand region, and the face sub-image including the detected face region (S1330).

[0108] For example, the processor may use a second machine learning model to recognize body motion information from a human sub-image that includes a human region, a third machine learning model to recognize hand motion information from a hand sub-image that includes a hand region, and a fourth machine learning model to recognize face motion information from a face sub-image that includes a face region.

[0109] According to an embodiment, the second machine learning model used for recognizing the body motion information may be a model configured to output body model parameters and camera parameters based on the human sub-image. The body motion information may include body 3D joint angle information, body 3D joint position information, and body 3D mesh information, at least a portion of which may be acquired based on the body model parameters output by the second machine learning model.

[0110] Additionally, the second machine learning model may be configured to further output body 2D joint estimation information and a body part segmentation map based on the human sub-image. In such an embodiment, the processor may classify the detected hand region into at least one of a left hand region, a right hand region, or both hand regions using the body 3D joint position information, the body 2D joint estimation information, and the body part segmentation map.

[0111] Additionally or alternatively, the processor may perform the detection (S1320) and the body motion information recognition simultaneously using the same machine learning model. For example, the first machine learning model may be a model configured to detect a human region, a hand region, and a face region from at least a portion of the input image and recognize body motion information. The processor may perform the detection (S1320) and the body motion information recognition simultaneously using the first machine learning model configured in this manner.

[0112] According to an embodiment, the third machine learning model used for recognizing hand motion information may be a model configured to output hand pose parameters based on the hand sub-images. The hand motion information may include first wrist angle information, hand 3D joint angle information, hand 3D mesh information, and camera parameters, and at least a portion of such information may be acquired based on the hand pose parameters output by the third machine learning model.

[0113] According to one embodiment, the body motion information may include second wrist angle information, elbow angle information, and shoulder angle information acquired based on the output of the second machine learning model. In this embodiment, the processor may correct the second wrist angle information, elbow angle information, and shoulder angle information included in the body motion information (e.g., information acquired based on the output of the second machine learning model) using the first wrist angle information acquired based on the hand pose parameters.

[0114] In one embodiment, the third machine learning model used for recognizing hand motion information may include a hand classifier, in which case the processor may classify the detected hand region into at least one of a left hand region, a right hand region, or both hand regions using the hand classifier included in the third machine learning model.

[0115] According to one embodiment, the fourth machine learning model used for recognizing facial motion information may be a model configured to output 3D facial model parameters, 3D facial landmark information, blend shapes, and camera parameters based on the facial sub-images. The facial motion information may include head pose information, face shape information, and facial expression information, and at least a portion of such information may be acquired based on the output of the fourth machine learning model. In one embodiment, when recognizing the facial motion information based on the facial sub-images using the fourth machine learning model, the processor may separately recognize the head pose information, face shape information, and facial expression information. Here, the facial expression information may include gaze information and tongue movement information.

[0116] According to one embodiment, the body motion information may include head angle information acquired based on the output of the second machine learning model. In such an embodiment, the processor may correct the head angle information (head angle information acquired based on the output of the second machine learning model) included in the body motion information using the head pose information (e.g., head pose information acquired based on the output of the fourth machine learning model) included in the facial motion information.

[0117] According to an embodiment, 3D motion information (body motion information, hand motion information, facial motion information, etc.) may include joint angle information, and the processor may correct the joint angle information included in the 3D motion information using at least one of the body proportions of a human object included in the input image, the body proportions of a character, or joint angle restriction information related to a character.

[0118] Rendering can be performed based on the 3D motion information recognized (and / or corrected) by the above-described method, thereby generating a character image including a character that mimics the body motion, hand motion, and facial motion of a human object contained in an input image.

[0119] According to one embodiment, in the receiving step (S1310), the processor may sequentially receive a plurality of frame images including a human object. That is, the processor may continuously receive a series of images (videos). As each of the plurality of frame images is sequentially received, the processor may perform the above-described motion recognition (e.g., detection (S1320) and recognition (S1330)) for each of the plurality of frame images in real time. Thus, a series of character images (i.e., character videos) including a character that replicates the movements (body motion, hand motion, and facial motion) of the human object included in the series of images (i.e., videos) may be generated.

[0120] In this manner, in an embodiment in which a plurality of frame images are sequentially received, when the processor simultaneously performs detection (S1320) and body motion information recognition using a first machine learning model, the input of the first machine learning model may be determined based on the detection results from a previous frame image. For example, to perform detection and body motion information recognition for a specific frame image, at least a portion of the specific frame image may be applied as an input to the first machine learning model. Here, the image to be applied as an input to the first machine learning model may be determined based on whether a human region is detected in the previous frame image. If a human region is detected in the previous frame image, at least a portion of the specific frame image applied as an input to the first machine learning model may include a region located at the same position as the human region detected in the previous frame image. Conversely, if a human region is not detected in the previous frame image, the entire specific frame image may be applied as an input to the first machine learning model.

[0121] 13 and the above description are merely exemplary embodiments of the present disclosure, and may be embodied in other ways. For example, in some embodiments, one or more steps may be added, deleted, or modified, or the order of the steps may be changed.

[0122] The above-described method may be provided as a computer program stored on a computer-readable recording medium for execution by a computer. The medium may permanently store a computer-executable program or may temporarily store the program for execution or download. The medium may be various recording or storage means in the form of a single piece of hardware or multiple pieces of hardware combined together. It is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Other examples of media include recording or storage media managed by app stores that distribute applications and other sites or servers that provide or distribute various software.

[0123] The methods, operations, or techniques of the present disclosure may be implemented by a variety of means. For example, such techniques may be implemented by hardware, firmware, software, or a combination thereof. Those skilled in the art will appreciate that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the present disclosure may also be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented by hardware or software depends on the particular application and design requirements imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementations should not be interpreted as departing from the scope of the present disclosure.

[0124] In a hardware implementation, the processing units used to perform the techniques may be embodied within one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described in this disclosure, computers, or combinations thereof.

[0125] Accordingly, the various illustrative logic blocks, modules, and circuits described in connection with this disclosure may be embodied or performed by a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate and transistor logic, discrete hardware components, or any combination designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be embodied by a combination of computing devices, such as a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

[0126] In a firmware and / or software implementation, the techniques may be embodied by instructions stored on a computer-readable medium such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage device, etc. The instructions may be executed by one or more processors and may cause the processor(s) to perform particular aspects of the functions described in this disclosure.

[0127] If embodied in software, the techniques may be stored on or transmitted via a computer-readable medium as one or more instructions or code. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available medium accessible by a computer. By way of non-limiting example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium accessible by a computer that can be used to transport or store desired program code in the form of instructions or data structures. Additionally, any connection may be appropriately effected by the computer-readable medium.

[0128] For example, if software is transmitted from a website, server, or other remote source via coaxial cable, fiber optic cable, twisted wire, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted wire, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. As used herein, "disk" and "disc" include CDs, laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs. Here, disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0129] A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium may be coupled to the processor such that the processor can read information from, and record information to, the storage medium. Alternatively, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. Alternatively, the processor and the storage medium may reside as discrete components in a user terminal.

[0130] Although the above-described embodiments have been described as utilizing aspects of the presently disclosed subject matter on one or more stand-alone computer systems, the present disclosure is not limited thereto and may be implemented in connection with any computing environment, such as a network or distributed computing environment. Additionally, aspects of the presently disclosed subject matter may be implemented on multiple processing chips or devices, and storage may be affected similarly across multiple devices. Such devices may include PCs, network servers, and portable devices.

[0131] Although the present disclosure has been described in connection with several embodiments herein, various modifications and changes that can be understood by those skilled in the art to which the present disclosure pertains can be made without departing from the scope of the present disclosure, and such modifications and changes must be considered to fall within the scope of the claims appended hereto.

Claims

1. 1. A 3D motion capture method executed by at least one processor, comprising: receiving an input image including a human object; Detecting a human region, a hand region, and a face region from at least a portion of the input image using a first machine learning model; Recognizing 3D motion information from the human sub-image including the human region, the hand sub-image including the hand region, and the face sub-image including the face region; Including, A 3D motion capture method for generating a character image including a character that replicates the body motion, hand motion, and facial motion of the human object included in the input image by rendering based on the 3D motion information.

2. The step of receiving an input image comprises: sequentially receiving a plurality of frame images including the human object; performing the detecting step and the recognizing step in real time for each of the plurality of frame images as the frame images are sequentially received; 2. The 3D motion capture method of claim 1, further comprising generating a plurality of character images including a character that replicates the body motion, hand motion, and facial motion of the human object included in each of the plurality of frame images by rendering based on the 3D motion information.

3. the first machine learning model includes a hand classifier; The detecting step includes: The 3D motion capture method of claim 1 , further comprising classifying the detected hand regions into at least one of a left hand region, a right hand region, or both hand regions using the hand classifier.

4. The step of recognizing includes: recognizing body motion information from the human sub-image including the human region using a second machine learning model; recognizing hand motion information from the hand sub-images including the hand regions using a third machine learning model; recognizing facial motion information from the facial sub-image including the facial region using a fourth machine learning model; 2. The 3D motion capture method of claim 1, comprising:

5. the second machine learning model is configured to output body model parameters and camera parameters based on the human sub-image; The 3D motion capture method of claim 4 , wherein the body motion information includes body 3D joint angle information, body 3D joint position information, and body 3D mesh information acquired based on the body model parameters.

6. The 3D motion capture method of claim 5 , wherein the second machine learning model is configured to further output body 2D joint estimation information and a body part segmentation map based on the human sub-image.

7. 7. The 3D motion capture method of claim 6, further comprising classifying the detected hand region into at least one of a left hand region, a right hand region, or both hand regions using the body 3D joint position information, the body 2D joint estimation information, and the body part segmentation map.

8. the first machine learning model is configured to detect the human region, the hand region, and the face region from at least a portion of the input image and recognize the body motion information; The 3D motion capture method of claim 4 , wherein the second machine learning model is the same model as the first machine learning model.

9. The step of receiving an input image comprises: sequentially receiving a plurality of frame images of the same size; performing the detecting step and the recognizing step for the 3D motion information in real time for each of the plurality of frame images as the frame images are sequentially received; applying at least a portion of a specific frame image among the plurality of frame images as an input to the first machine learning model in the detecting and recognizing 3D motion information for the specific frame image; At least a portion of the specific frame image applied as an input to the first machine learning model includes: If a human region is detected in a frame image previous to the specific frame image, the specific frame image includes a region at the same position as the human region detected in the previous frame image; The 3D motion capture method of claim 8 , wherein if no human region is detected in a frame image preceding the specific frame image, the entire specific frame image is captured.

10. the third machine learning model includes a hand classifier; The step of recognizing hand motion information includes: The 3D motion capture method of claim 4 , further comprising classifying the detected hand regions into at least one of a left hand region, a right hand region, or both hand regions using the hand classifier.

11. the third machine learning model is configured to output hand pose parameters based on the hand sub-images; The 3D motion capture method of claim 4 , wherein the hand motion information includes first wrist angle information, hand 3D joint angle information, hand 3D mesh information, and camera parameters acquired based on the hand pose parameters.

12. the body motion information includes second wrist angle information obtained based on an output of the second machine learning model; The step of recognizing 3D motion information includes:

12. The 3D motion capture method of claim 11, further comprising: correcting the second wrist angle information acquired based on an output of the second machine learning model using first wrist angle information acquired based on the hand pose parameters.

13. The body motion information includes elbow angle information and shoulder angle information acquired based on the output of the second machine learning model, The step of recognizing 3D motion information includes: The 3D motion capture method of claim 11 , further comprising: correcting the elbow angle information and the shoulder angle information acquired based on an output of the second machine learning model using the first wrist angle information.

14. 5. The 3D motion capture method of claim 4, wherein the fourth machine learning model is configured to output 3D face model parameters, 3D facial landmark information, blend shapes, and camera parameters based on the facial sub-images.

15. The facial motion information includes head pose information, face shape information, and facial expression information, The step of recognizing facial motion information includes: The 3D motion capture method of claim 4 , further comprising: classifying and recognizing the head pose information, the face shape information, and the facial expression information from the face sub-images using the fourth machine learning model.

16. The 3D motion capture method of claim 15 , wherein the facial expression information includes gaze information and tongue movement information.

17. the body motion information includes head angle information obtained based on the output of the second machine learning model; The step of recognizing 3D motion information includes: The 3D motion capture method of claim 15 , further comprising: correcting the head angle information acquired based on the output of the second machine learning model using the head pose information.

18. the 3D motion information includes joint angle information; The step of recognizing 3D motion information includes:

2. The 3D motion capture method of claim 1, further comprising: correcting the joint angle information using at least one of a body proportion of a human object included in the input image, a body proportion of the character, or joint angle restriction information related to the character.

19. A computer-readable non-transitory recording medium having recorded thereon instructions for executing the method of claim 1 on a computer.

20. A motion capture system, Memory and at least one processor coupled to the memory and configured to execute at least one computer-readable program contained in the memory; Including, The at least one program receiving an input image including a human object; detecting a human region, a hand region, and a face region from at least a portion of the input image using a first machine learning model; and recognizing 3D motion information from a human sub-image including the human region, a hand sub-image including the hand region, and a face sub-image including the face region; A motion capture system that generates a character image including a character that replicates the body motion, hand motion, and facial motion of the human object included in the input image by rendering based on the 3D motion information.

Images