Method for modeling automatically rigged 3D character based on artificial intelligence technology and apparatus for the same

Abstract

The present disclosure relates to a method for modeling an automatically rigged 3D character based on artificial intelligence technology and an apparatus for the same. A method for generating an automatically rigged 3D character according to one embodiment of the present disclosure may include: acquiring monocular video data for a specific target; generating a 3D representation of the specific target including external shape information of the specific target; generating a reference 3D mesh including rigging information using the 3D representation; and applying one or more of pose information or physically based rendering information to the reference 3D mesh to generate a 3D character model corresponding to the specific target.

Description

Technology Field

[0001] The present disclosure relates to 3D (3-dimensional) character modeling, and specifically to a method for modeling a 3D character that is automatically rigged based on artificial intelligence technology and has physically based rendering information, and an apparatus for the same. Background Technology

[0002] 3D space modeling is being applied in various services. Modeling 3D space requires modeling not only the background space but also the objects contained within that space. When multiple objects exist within a space, significant costs and resources can be consumed for modeling both the space and each individual object. Therefore, automated modeling methods for spaces and objects are being developed and researched.

[0003] In particular, to automatically model human-shaped 3D characters corresponding to user characters and NPCs (non-player characters), methods based on 3D scanning, multi-view / multi-view data, and complex processing are being applied. To solve this problem, there is a need for a method to model 3D characters using artificial intelligence technology based on monocular / single-view video data that can be easily acquired.

[0004] Furthermore, when a human figure is simply modeled as a 3D character, joint information is not rigged, so in order to solve the problem of being unable to express the 3D character in the form of a moving animation, a new 3D character modeling method including rigging information is required to enable animation generation using artificial intelligence technology based on monocular / single-viewpoint video data. The problem to be solved

[0005] The present disclosure aims to provide a method and apparatus for modeling a 3D character that is automatically rigged based on monocular video data.

[0006] The present disclosure aims to provide a method and apparatus for modeling a 3D character that is automatically rigged without using a pre-provided 3D mesh.

[0007] The present disclosure aims to provide a method and apparatus for modeling a 3D character based on monocular video data while simultaneously generating physically based rendering information.

[0008] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs from the description below. means of solving the problem

[0009] A method for generating an automatically rigged 3D character according to one embodiment of the present disclosure may include: acquiring monocular video data for a specific target; generating a 3D representation of the specific target including external shape information of the specific target; generating a reference 3D mesh including rigging information using the 3D representation; and applying one or more of pose information or physically based rendering information to the reference 3D mesh to generate a 3D character model corresponding to the specific target.

[0010] An apparatus for generating an automatically rigged 3D character according to an additional embodiment of the present disclosure may include one or more transceivers; one or more memories; and one or more processors. The one or more processors may be configured to perform preprocessing of monocular video data for a specific target using monocular video data obtained through the one or more transceivers; generate a 3D representation of the specific target including external shape information of the specific target using the preprocessed result and store it in the one or more memories; generate a reference 3D mesh including rigging information using the 3D representation and store it in the one or more memories; and apply one or more of pose information or physically based rendering information to the reference 3D mesh to generate a 3D character model corresponding to the specific target and store it in the one or more memories.

[0011] A processing unit according to an additional embodiment of the present disclosure may include one or more processors; and one or more local memories operably connected to the one or more processors. The one or more local memories may store instructions for performing operations, based on execution by the one or more processors, to acquire monocular video data for a specific target; to generate a 3D representation of the specific target including shape information of the specific target; to generate a reference 3D mesh including rigging information using the 3D representation; and to apply one or more of pose information or physically based rendering information to the reference 3D mesh to generate a 3D character model corresponding to the specific target.

[0012] One or more non-transitory computer-readable media according to additional embodiments of the present disclosure may store one or more commands. The one or more commands may be executed by one or more processors to control a device to perform the following operations: acquiring monocular video data for a specific target; generating a 3D representation of the specific target including shape information of the specific target; generating a reference 3D mesh including rigging information using the 3D representation; and applying one or more of pose information or physically based rendering information to the reference 3D mesh to generate a 3D character model corresponding to the specific target.

[0013] In some embodiments of the present disclosure, through preprocessing of monocular video data for the specific target, one or more of segmentation information for the specific target, background information excluding the specific target, joint information of the specific target, pose information of the specific target, or position information of the specific target may be generated.

[0014] In some embodiments of the present disclosure, the preprocessing may include generating an image sequence in which the monocular video data is separated by frame.

[0015] In some embodiments of the present disclosure, the 3D representation may be learned or inferred through a deep neural network using one or more of the information generated through the preprocessing.

[0016] In some embodiments of the present disclosure, volume rendering and a 3D mesh may be applied to the 3D representation.

[0017] In some embodiments of the present disclosure, the reference 3D mesh may be generated based on pre-learning using a standard template model containing human joint information.

[0018] In some embodiments of the present disclosure, the rigging information of the reference 3D mesh may be represented based on linear blending skinning. Additionally, the linear blending skinning may be applied based on one or more elements of the external skin information, posture information, facial expression information, or clothing information of the specific object.

[0019] In some embodiments of the present disclosure, the pose information may be applied to the reference 3D mesh to generate a pose 3D mesh.

[0020] In some embodiments of the present disclosure, the physically based rendering information may be applied to the pose 3D mesh to generate the 3D character model.

[0021] In some embodiments of the present disclosure, the physics-based rendering information may be estimated from monocular video data for the specific target.

[0022] In some embodiments of the present disclosure, the physically based rendering information may include optical information indicating the reflection of light on the surface material of a 3D object.

[0023] In some embodiments of the present disclosure, the physically based rendering information may include one or more elements of base color, roughness, reflected light, opacity, luminescence, refraction, or distortion, distinguished according to a photorealistic or non-photorealistic approach.

[0024] In some embodiments of the present disclosure, when the 3D character model is generated during the inference process, the 3D character model may be output using the learned model.

[0025] In some embodiments of the present disclosure, when the 3D character model is generated during the training process, an error based on a loss calculated by comparing the 3D character model with the specific target may be backpropagated to the step of generating the 3D representation. Additionally, the trained model may be updated using the backpropagated error.

[0026] In some embodiments of the present disclosure, comparing the 3D character model with the specific target may include comparing an image rendered of a state learned using the 3D character model with a preprocessed image of the monocular video data for the specific target.

[0027] In some embodiments of the present disclosure, the loss may be calculated for one or more of a normal map, a depth map, or physically based rendering information.

[0028] In some embodiments of the present disclosure, one or more new pose information may be applied to the 3D character model to generate a 3D dynamic animation of the 3D character model.

[0029] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure. Effects of the invention

[0030] According to the present disclosure, a method for modeling a 3D character that is automatically rigged based on monocular video data and an apparatus for the same may be provided.

[0031] According to the present disclosure, a method for modeling a 3D character that is automatically rigged without using a pre-provided 3D mesh and an apparatus for the same may be provided.

[0032] According to the present disclosure, a method and apparatus for modeling a 3D character based on monocular video data and generating physically based rendering information at the same time may be provided.

[0033] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure pertains from the description below. Brief explanation of the drawing

[0034] FIG. 1 shows an example of a 3D character model with rigging applied to which the present disclosure can be applied. FIG. 2 is a drawing for explaining an example of a 3D character modeling method capable of automatic rigging according to the present disclosure. FIG. 3 shows an example of an image sequence obtained from a monocular RGB video input in the present disclosure. Figure 4 shows an example of person segmentation information and background information corresponding to the result of applying preprocessing to the image sequence of Figure 3. FIG. 5 is a drawing showing an example of a rendering of a reference 3D mesh generated according to the present disclosure. FIG. 6 is a drawing showing an example of a rendering of a pose 3D mesh generated according to the present disclosure. FIG. 7 is a drawing showing examples of physically based rendering information according to the present disclosure. FIG. 8 shows an example of rendering of a 3D mesh having physically based rendering information according to the present disclosure. FIG. 9 shows an example in which a part corresponding to a person is separated from input data according to the present disclosure. FIG. 10 shows examples of various pose information applied to an automatically rigged 3D character according to the present disclosure. FIG. 11 is a drawing for explaining an example of an automatically rigged 3D character modeling method according to the present disclosure. FIG. 12 shows an example of a block diagram of an automatically rigged 3D character modeling device according to the present disclosure. Specific details for implementing the invention

[0035] The present disclosure is subject to various modifications and may have various embodiments; therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the present disclosure. Similar reference numerals in the drawings refer to the same or similar functions across various aspects. The shapes and sizes of components in the drawings may be exaggerated for clearer explanation. The detailed description of exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments as examples. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that various embodiments are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present disclosure in relation to one embodiment. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the following detailed description is not intended to be taken in a limiting sense, and the scope of exemplary embodiments is limited only by the appended claims, together with all equivalents to those claimed therein, provided they are properly described.

[0036] In this disclosure, terms such as first, second, etc. may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of this disclosure, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0037] Where any component of the present disclosure is described as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to the other component, or that there may be other components in between. On the other hand, where any component is described as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0038] The components shown in the embodiments of the present disclosure are depicted independently to represent different characteristic functions and do not imply that each component consists of separate hardware or a single software unit. That is, each component is listed and included as a separate component for convenience of explanation; however, at least two of the components may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separated embodiments of each component are included within the scope of the rights of the present disclosure as long as they do not depart from the essence of the present disclosure.

[0039] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit this disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this disclosure, terms such as "comprising" or "having" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. That is, the description in this disclosure that a specific configuration "comprising" does not exclude configurations other than that configuration, but means that additional configurations may be included within the scope of the practice or technical concept of this disclosure.

[0040] Some components of the present disclosure may not be essential components performing an essential function in the present disclosure, but may be optional components merely for enhancing performance. The present disclosure may be implemented by including only the components essential to embody the essence of the present disclosure, excluding components used merely for enhancing performance, and a structure including only the essential components, excluding optional components used merely for enhancing performance, is also included within the scope of the rights of the present disclosure.

[0041] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In describing the embodiments of this specification, if it is determined that a detailed description of related known configurations or functions may obscure the gist of this specification, such detailed description is omitted, and the same reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.

[0042] In this disclosure, various examples are described of how to generate an automatically rigged 3D character capable of producing animation by receiving monocular RGB (red green blue) video input using artificial intelligence technology.

[0043] For example, to create 3D spaces for various services such as virtual reality and games, it is important not only to model the background space but also to model the key objects that fill that space. These key objects can include characters corresponding to users of the services, non-player characters (NPCs) provided by the services, and many different types of objects used by users. Modeling such diverse spaces and objects consumes significant costs and resources. Therefore, to address this issue, methods for automatically modeling various spaces and objects into 3D data are being developed and researched.

[0044] Among these, human-shaped characters can be broadly classified into characters corresponding to users who directly use the service and NPCs that are automatically implemented in the service. Although various methods exist for generating 3D data for human-shaped characters, existing methods require 3D scanning or multi-view / multi-view video data and complex processes. This disclosure describes a method for receiving general monocular / single-view video data (e.g., RGB video data) that can be easily acquired using artificial intelligence technology as input, modeling a 3D character based on this data, and automatically generating rigging information and physically based rendering information to enable animation generation. In particular, the examples in this disclosure can be differentiated from existing technologies in that they do not require pre-provided 3D data such as 3D point clouds and meshes, but instead generate an automatically rigged 3D character having physically based rendering information based on monocular / single-view video data.

[0045] Examples of the present disclosure may include a learning process for generating a 3D character based on artificial intelligence and an inference process for generating a 3D character. Through this, a monocular RGB video is received as input, a 3D character of a target person is generated from the video data, skeleton information of the character is automatically rigged, and physically based rendering information is generated.

[0046] FIG. 1 shows an example of a 3D character model with rigging applied to which the present disclosure can be applied.

[0047] Generally, modeled 3D characters mostly contain only 3D information, so the generation of statically rendered images can be easily performed.

[0048] However, creating 3D animations using 3D characters can be broadly divided into two methods: generating static 3D characters over time and rendering each as an image, and generating dynamic 3D characters to move over time and rendering them as images. Generally, the latter method is more efficient than the former in terms of cost, so it is mainly adopted. Therefore, when dynamically modeling animation-capable 3D characters, many modelers perform additional rigging work by attaching a skeleton, which is bone information, to the 3D character to make the 3D character move in accordance with movement.

[0049] Rigging can also be referred to as skeletal animation. Rigging refers to the process of modeling a 3D character by creating and placing a skeleton, assigning joints to the character, and forming a skeletal structure from which these elements combine to enable movement. Rigging is commonly applied to produce 3D animations using 3D characters. If rigging is applied to a 3D character, moving the bones, joints, and skeleton—which are linked to the character's external information—allows the character to perform natural movements accordingly.

[0050] FIG. 2 is a drawing for explaining an example of a 3D character modeling method capable of automatic rigging according to the present disclosure.

[0051] First, when a specific person to be the target for creating a 3D character is determined, monocular RGB video data containing the specific person's appearance information can be acquired and received as input (S210). Next, the received monocular RGB video can be preprocessed (S220). Using the preprocessed information, a 3D representation of the target person can be generated (S230).

[0052] A reference 3D mesh having rigging information can be generated using the information of the 3D representation generated as above (S240). In addition, a pose 3D mesh with a pose applied to the rigging information of the reference 3D mesh can be generated using the preprocessed information (S250).

[0053] In addition, physically based rendering information can be generated to render a specific person in a realistic manner on a 3D modeling engine and platform (S260).

[0054] Next, it can be determined whether the above processes are a learning process for generating a 3D character model or a generation and inference process (S270). If it is a generation and inference process, a 3D character model capable of automatic rigging can be output using the learned model (S280). If it is a learning process, the learned state can be rendered as an image using the learned model, the error can be compared with the preprocessed input information, the loss can be calculated, the error can be backpropagated to the 3D representation process, and the learned model can be updated (S290).

[0055] Various examples of the present disclosure will be described in detail below with reference to FIG. 2.

[0056] With the recent advancement of AI-based computer vision technology, it is now possible to estimate 3D data using a single image. However, generating 3D data from a single 2D image in which much information is missing is not easy. Furthermore, it is difficult to generate 3D data by plausibly inferring information about unseen parts of a 2D image using information from a trained model. Generally, 3D data can be generated relatively accurately only by transforming it using basic 3D information or by using images and videos with various viewpoints capable of restoring 3D information.

[0057] The input data required to generate such accurate 3D data can generally be acquired from the following devices and environments. First, there are devices capable of directly acquiring 3D data, such as 3D scanners or computed tomography (CT) scanners that can process 3D information. Next, there is a method of acquiring 2D image or video data using a series of large-scale platforms consisting of pre-fabricated structures, cameras attached to the structures, and synchronization devices. After acquisition, an additional process of converting the acquired data into 3D data is required.

[0058] These existing methods can generate relatively accurate 3D data, but they consume a lot of resources and require complex processes to acquire data to be used as input. Therefore, unlike existing methods, in the present disclosure, when sufficient full-body information of a subject can be collected using monocular RGB video that can be acquired with readily available mobile devices and handheld devices, such monocular RGB video can be used as input. (S210)

[0059] The preprocessing process (S220) may include a process of separating the input monocular RGB video into individual frames to generate an image sequence. The information generated through the preprocessing process may include information used in subsequent processes. For example, segmentation information of the target person, background information excluding the person, joint information of the person, posture information of the person, the position of the person in a specific coordinate system, and internal and external information of the camera used for shooting may be generated through the preprocessing process and used in subsequent processes.

[0060] FIG. 3 shows an example of an image sequence obtained from a monocular RGB video input in the present disclosure.

[0061] Figure 4 shows an example of person segmentation information and background information corresponding to the result of applying preprocessing to the image sequence of Figure 3.

[0062] In the example of Fig. 3, the input monocular RGB video is shown with images one by one over time. Fig. 4 separates and displays the segmentation information regarding the person and the background information for each of the images obtained from the RGB video input in Fig. 3. In Fig. 4, the parts marked in white represent the segmentation information regarding the person that serves as the target for 3D character modeling. In Fig. 4, the parts marked in black represent the background information that does not include the person.

[0063] Next, the process of generating a 3D representation by receiving preprocessed information as input (S230) is described. Methods for learning 3D representations can be classified in various ways depending on the perspective. Generally, they can be divided into extrinsic methods that extract 3D information from learned 3D representations by directly using provided 3D information, and intrinsic methods that learn 3D representations within a Deep Neural Network (DNN) during training.

[0064] The present disclosure assumes that 3D representations are learned and inferred using an intrinsic method. Additionally, while 3D representations can be displayed in various ways, the present disclosure assumes a hybrid method that uses both volume rendering and 3D meshes. Representing 3D in this manner has the advantage of possessing information on points, lines, and surfaces, rather than in the form of a 3D point cloud. Furthermore, since 3D contour information for an object can be roughly sampled in 3D space using a volume rendering method, processing speed and efficiency can be secured.

[0065] Furthermore, if a Scratch method that learns from scratch is used for learning 3D representations, the quality of the final 3D character generated for the target person may be poor. Therefore, in this disclosure, pre-training can be performed using a standard template model that represents a person in 3D. Then, training for the 3D character to be generated can be carried out using preprocessed information from the pre-trained model. Using such pre-trained information reduces the opportunity to learn information that differs from the person, thereby preventing the generation of errors or artifacts. There are various methods developed and proposed in many 3D modeling engines, platforms, and research fields regarding 3D standard template models for people. In this disclosure, it is assumed that the SMPL-X (Skinned Multi-Person Linear Model expressive) model is used as the 3D standard template model for people. Of course, other template models or methods for people may also be adopted and used as 3D representation methods.

[0066] The 3D representation represents the 3D external shape information of the person being represented. Using the 3D representation generated in this way, a reference 3D mesh based on a standard template model can be generated (S240). Since the generated reference 3D mesh has joint information of the standard template model, various movements can be applied. Therefore, the reference 3D mesh must have or adopt a method to represent the joint information of the 3D mesh at the joint locations so that it can perform various movements as a person.

[0067] In this disclosure, it is assumed that linear blend skinning (LBS) is applied as a method to represent joint information or rigging information in a reference 3D mesh. In order to generate a pose 3D mesh by reflecting pose information in the reference 3D mesh, natural deformation of the joint information is possible. The linear blend skinning method applied in this disclosure is a method that reflects various skinning weights as a diffused linear sum. For example, skinning weights possessed by SMPL-X, one of the 3D standard template models, can be applied. The linear blend skinning method can consider various elements, which broadly include human external skin information, pose information, facial expression information, and clothing information. Specifically, it can be applied using a method that considers various elements such as skin tone and roughness, clothing material, wrinkles, and hairstyle.

[0068] FIG. 5 is a drawing showing an example of a rendering of a reference 3D mesh generated according to the present disclosure.

[0069] The example in FIG. 5 corresponds to an example in which one of the intermediate results randomly generated during the process of learning a reference 3D mesh is rendered and visualized as an image. The rigged 3D character provided as a result of the present disclosure retains the joint information of the standard template model referenced during the generation of the reference 3D mesh. For example, when SMPL-X is used as the standard template model, the number of joints is 55. Pose information can be applied to the joint information, and the character can be automatically rigged according to the generation method of the pose 3D mesh subsequently generated.

[0070] Next, the pose information extracted during the preprocessing process can be applied to the generated reference 3D mesh to create a pose 3D mesh (S250). Since SMPL-X was adopted as the standard template model in the example above, methods capable of extracting SMPL-X information can be adopted and used during the preprocessing process.

[0071] FIG. 6 is a drawing showing an example of a rendering of a pose 3D mesh generated according to the present disclosure.

[0072] The example in Fig. 6 corresponds to an exemplary image visualizing an arbitrary intermediate result during the training process for generating a pose 3D mesh. Unlike the example in Fig. 5, in Fig. 6, a human pose can be seen applied to the reference 3D mesh. Furthermore, it can be seen that the pose 3D mesh can assume a different pose from the reference 3D mesh to which linear blending skinning is applied, and that the 3D character mesh is naturally generated even when the joints move.

[0073] The 3D character generated according to the example described above possesses geometric information and connectivity information, so the target person can be represented as three-dimensional information. Meanwhile, optical information including object and light-induced texture information is also reflected in the input monocular RGB video. Accordingly, in the present disclosure, physically based rendering information is comprehensively generated (S260), and based thereon, a 3D character that can be reused in a 3D modeling engine and platform is created. Although there are methodological differences for detailed implementation, physically based rendering information may correspond to a computer graphics (CG) representation method that simulates how light reflection on the surface material of a 3D object physically reacts in a 3D modeling engine and platform.

[0074] Physically based rendering can be broadly classified into photorealistic and non-photorealistic methods. Representative distinguishing factors are divided into base color, roughness, and reflected light regarding the surface material of the 3D object to be represented; additionally, base color (or albedo), opacity, luminescence, refraction, and distortion may also be used as additional factors. In the present disclosure, physically based rendering information can be estimated from input monocular RGB video to generate a 3D character of a target person. For example, physically based rendering information from input monocular RGB video can be estimated during training, inference, and generation to generate a 3D character. The estimated physically based rendering information can be edited in 3D modeling engines and platforms, and can be modified to the physically based rendering information of the 3D character to be generated. The modified physically based rendering information can also be utilized in virtual reality and augmented reality services in different environments.

[0075] Even if a linear blending skinning method is applied to the aforementioned basic 3D mesh, it is not easy to achieve a natural representation of various human hairstyles, hair, and clothing. Therefore, physically based rendering information plays a role in helping the generated 3D character be rendered naturally.

[0076] FIG. 7 is a drawing showing examples of physically based rendering information according to the present disclosure.

[0077] The example of FIG. 7 corresponds to an example in which an intermediate result corresponding to a point in time as in FIG. 6 is visualized during the learning process to generate physically based rendering information for a person in a monocular RGB video input according to the present disclosure. For example, as physically based rendering information, a basic color map, a normal map (e.g., a map in which the normal (or normal) vector information of an object surface is expressed as RGB color values), a roughness map, etc. may be generated.

[0078] FIG. 8 shows an example of rendering of a 3D mesh having physically based rendering information according to the present disclosure.

[0079] The example in Fig. 8 corresponds to an example of rendering a 3D character model generated by assuming the example of the pose 3D mesh in Fig. 6 and the example of physically based rendering information in Fig. 7.

[0080] FIG. 9 shows an example in which a part corresponding to a person is separated from input data according to the present disclosure.

[0081] The example in Fig. 9 shows a scene in which a person who is the actual subject appears in the input monocular RGB video. By comparing the examples in Fig. 8 and Fig. 9, it can be seen that the 3D mesh representing the person's pose and physically based rendering information, generated using the input image from the monocular RGB video, is substantially identical to the shape and texture of the person in the actual input RGB video.

[0082] Next, it can be determined whether the aforementioned processes are learning processes for 3D character modeling or generation and inference processes (S270).

[0083] If it is a generation or inference process, an automatic rigging-capable 3D character modeling result generated according to the above process may be output (S280). The output for the person in the input monocular RGB video may include a reference 3D mesh with rigging applied and physically based rendering information. Based on this reference 3D mesh and physically based rendering information, a rigged 3D character model capable of applying various poses may be generated.

[0084] If it is a learning process, the generative model can be trained (S290). In the present disclosure, through the processes described above, the loss / error can be calculated by comparing the rendering image of a rigged 3D mesh with the generated physically based rendering information shown in FIG. 8 and the preprocessed ground truth (GT) of the input image shown in FIG. 9. Additionally, the calculated loss / error can be backpropagated to the process of generating a 3D representation (S230), and the generative model can be updated and trained accordingly. Furthermore, the generative model can be trained together by backpropagating the error after calculating the loss of the normal map value estimated from the input image and the result of creating a normal map, which is mainly used to represent 3D information, using the generated 3D modeling information. Since the normal map used in this process is one of the methods for representing 3D information, it is also possible to apply the aforementioned physically based rendering information, etc., to training in a similar manner.

[0085] Finally, a rigged 3D character model with physically based rendering information generated by inputting a person appearing in monocular RGB video can be created. By inputting new pose information to the rigged 3D character finally created through these processes, various 3D dynamic animations can be produced.

[0086] FIG. 10 shows examples of various pose information applied to an automatically rigged 3D character according to the present disclosure.

[0087] The example in Fig. 10 shows the results of applying a dining pose, a sitting pose, and a fencing pose, respectively, to automatically rigged joints. As illustrated in Fig. 10, it can be seen that a natural appearance is expressed when a new pose is applied to the automatically rigged 3D character model generated in the corresponding disclosure.

[0088] As described above, according to the examples of the present disclosure, the problem of being unable to express a moving person in a dynamic form because joint information is not rigged when a person is simply modeled as a 3D character can be solved. Specifically, according to the examples of the present disclosure, a 3D character including rigging can be automatically generated by simply receiving a standard RGB video of a person to be targeted with a monocular camera.

[0089] FIG. 11 is a drawing for explaining an example of an automatically rigged 3D character modeling method according to the present disclosure.

[0090] In step S1110, the device can acquire monocular video data for a specific target.

[0091] In some examples, through preprocessing of monocular video data for a specific target, segmentation information for the specific target, background information excluding the specific target, joint information of the specific target, pose information of the specific target, and / or location information of the specific target can be generated.

[0092] In some examples, this preprocessing may include generating image sequences separated frame by frame from monocular video data.

[0093] In step S1110, the device can generate a 3D representation of a specific target.

[0094] In some examples, the 3D representation may include external shape information of a specific object.

[0095] In some examples, a 3D representation can be learned or inferred through a deep neural network using information generated through preprocessing (e.g., segmentation information for a specific object, background information excluding a specific object, joint information of a specific object, pose information of a specific object, and / or position information of a specific object).

[0096] In some examples, volume rendering and 3D meshes may be applied to 3D representations. Accordingly, 3D representations may contain information about points, lines, and faces, and their contour information may be sampled in a volume rendering manner.

[0097] In step S1130, the device can generate a reference 3D mesh containing rigging information using the 3D representation generated in step S1120.

[0098] In some examples, the reference 3D mesh can be generated based on pre-learning using a standard template model (e.g., SMPL-X model) containing human joint information.

[0099] In some examples, the rigging information of a reference 3D mesh can be represented based on linear blend skinning. For example, linear blend skinning can be applied based on one or more elements of the external skin information, pose information, facial expression information, or clothing information of a specific object.

[0100] In step S1140, the device can apply pose information and / or physically based rendering information to the reference 3D mesh generated in step S1130 to generate a 3D character model corresponding to a specific target.

[0101] In some examples, pose information may be applied to a reference 3D mesh to generate a pose 3D mesh, and physically based rendering information may be applied to the pose 3D mesh to generate a 3D character model. Alternatively, a 3D character model may be generated by applying pose information after physically based rendering information is applied to the reference 3D mesh, or by applying both pose information and physically based rendering information together.

[0102] In some examples, physically based rendering information can be estimated from monocular video data of a specific target. For example, physically based rendering information can also be estimated / generated during the aforementioned preprocessing process.

[0103] In some examples, physically based rendering information may include optical information representing the reflection of light on the surface material of a 3D object. For example, physically based rendering information may include information distinguished according to a photorealistic or non-photorealistic approach. For example, physically based rendering information may include information on one or more elements of base color, roughness, reflected light, opacity, luminescence, refraction, or distortion.

[0104] When a 3D character model generated according to the examples described above is generated during the inference process, the 3D character model can be output using the learned model. One or more new pose information may be applied to this 3D character model to generate a 3D dynamic animation of the 3D character model.

[0105] If a 3D character model generated according to the examples described above is generated during the training process, an error based on a loss calculated by comparing the 3D character model with the specific target (e.g., comparing an image rendered of a state learned using the 3D character model with a preprocessed image of monocular video data for the specific target) can be backpropagated to the step of generating a 3D representation of step S1120. For example, such a loss can be calculated for a normal map, a depth map, and / or physically based rendering information. As the error is backpropagated, the trained model can be updated.

[0106] FIG. 12 shows an example of a block diagram of an automatically rigged 3D character modeling device according to the present disclosure.

[0107] An automatic rigged 3D character modeling device (1200) may include one or more processors (1210), one or more memories (1220), one or more transceivers (1230), one or more user interfaces (1240), etc. The memory (1220) may be included in the processor (1210) or may be configured separately. The memory (1220) may store instructions that cause the device (1200) to perform an operation when executed by the processor (1210). The transceiver (1230) may transmit and / or receive signals, data, etc. that the device (1200) exchanges with other entities. The user interface (1240) may receive user input regarding the device (1200) or provide the output of the device (1200) to the user. Among the components of the device (1200), components other than the processor (1210) and memory (1220) may not be included depending on the case, and other components not shown in FIG. 12 may be included in the device (1200).

[0108] The processor (1210) may be configured to enable the device (1200) to perform the operation of the device according to various examples of the present disclosure. Although not illustrated in FIG. 12, the processor (1210) may be configured as a set of modules that perform each function. The modules may be configured in the form of hardware and / or software.

[0109] For example, the processor (1210) may be configured to perform preprocessing of monocular video data for a specific target using monocular video data obtained through a transceiver (1230). Additionally, the processor (1210) may use the preprocessed result to generate a 3D representation including external shape information of the specific target and store it in memory (1220). Additionally, the processor (1210) may be configured to generate a reference 3D mesh including rigging information using the stored 3D representation and store it in memory (1220). Additionally, the processor (1210) may be configured to apply pose information and / or physically based rendering information to the reference 3D mesh to generate a 3D character model corresponding to the specific target and store it in memory (1220). Additionally, the processor (1210) may be configured to generate 3D dynamic animation data with one or more pose information applied to a 3D character model, store it in memory (1220), or output it through a user interface (1240).

[0110] The components described in the exemplary embodiments of the present disclosure may be implemented by hardware elements. For example, the hardware elements may include at least one of a digital signal processor (DSP), a processor, a controller, an application-specific integrated circuit (ASIC), a programmable logic element such as an FPGA, a GPU, other electronic devices, or a combination thereof. At least some of the functions or processes described in the exemplary embodiments of the present disclosure may be implemented in software, and the software may be recorded on a recording medium. The components, functions, and processes described in the exemplary embodiments may be implemented by a combination of hardware and software.

[0111] A method according to one embodiment of the present disclosure may be implemented as a program that can be executed by a computer, and said computer program may be recorded on various recording media such as magnetic storage media, optical reading media, digital storage media, etc.

[0112] The various technologies described in this disclosure may be implemented as digital electronic circuits or computer hardware, firmware, software, or a combination thereof. The technologies may be implemented as computer program products, namely, computer programs tangibly implemented on information media or computer programs (e.g., machine-readable storage devices (e.g., computer-readable media) or data processing devices), or as computer programs implemented as signals processed by or propagated to perform operations of data processing devices (e.g., programmable processors, computers, or a plurality of computers).

[0113] Computer program(s) may be written in any form of programming language, including compiled or interpreted languages, and may be distributed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. Computer programs may be executed by a single computer, or by multiple computers separated at one site or multiple sites and interconnected by a communication network.

[0114] Examples of processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and one or more processors of a digital computer. Generally, a processor receives instructions and data from read-only memory or random access memory, or both. Components of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Additionally, the computer may include one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or may be connected to said mass storage devices to receive and / or transmit data. Examples of information media suitable for implementing computer program instructions and data include semiconductor memory devices (magnetic media such as hard disks, floppy disks, and magnetic tapes), optical media such as compact disc read-only memory (CD-ROM) and digital video discs (DVD), magneto-optical media such as floptical disks, and Read Only Memory (ROM), Random Access Memory (RAM), flash memory, Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), and other known computer-readable media. Processors and memory may be complemented or integrated by special-purpose logic circuits.

[0115] A processor may execute an operating system (OS) and one or more software applications running on the OS. A processor unit may also access, store, manipulate, process, and generate data in response to software execution. For simplification, a processor unit has been described in the singular; however, a person of ordinary skill in the art will understand that a processor unit may include multiple processing elements and / or various types of processing elements. For example, a processor unit may include multiple processors or a processor and a controller. It may also constitute different processing structures, such as parallel processors. Furthermore, a computer-readable medium means any medium accessible by a computer and may include both computer storage media and transmission media.

[0116] The present disclosure includes detailed descriptions of various detailed embodiments, but such details are not intended to limit the invention or claims proposed in the present disclosure and should be understood as describing the features of specific exemplary embodiments.

[0117] Features individually described in exemplary embodiments in this disclosure may be implemented by a single exemplary embodiment. Conversely, various features described with respect to a single exemplary embodiment in this disclosure may be implemented by a combination of multiple exemplary embodiments or a suitable sub-combination. Furthermore, in this disclosure, said features may operate by a specific combination and may be described as said combination first claimed, but in some cases, one or more features may be excluded from the claimed combination, or the claimed combination may be changed into a sub-combination or a modified form of a sub-combination.

[0118] Likewise, even if operations are described in a specific order in the drawings, it should not be understood that it is necessary to execute the operations in a specific sequence or order, or that all operations must be performed, in order to obtain the desired result. In certain cases, multitasking and parallel processing may be useful. Furthermore, it should not be understood that the various device components in the exemplary embodiments of all embodiments must be separated, and the aforementioned program components and devices may be packaged into a single software product or multiple software products.

[0119] The exemplary embodiments disclosed in this specification are merely illustrative and are not intended to limit the scope of this disclosure. Those skilled in the art will recognize that various modifications to the exemplary embodiments may be made without departing from the spirit and scope of the claims and their equivalents.

[0120] Accordingly, the present disclosure shall be deemed to include all other substitutions, modifications, and changes falling within the scope of the following claims.

Claims

Claim 1 A method for generating an automatically rigged 3D character, wherein the method comprises: acquiring monocular video data for a specific target; generating a 3D representation of the specific target including shape information of the specific target; generating a reference 3D mesh including rigging information using the 3D representation; and generating a 3D character model corresponding to the specific target by applying one or more of pose information or physically based rendering information to the reference 3D mesh. Claim 2 A method according to claim 1, wherein one or more of segmentation information for the specific target, background information excluding the specific target, joint information of the specific target, pose information of the specific target, or position information of the specific target are generated through preprocessing of monocular video data for the specific target. Claim 3 In claim 2, the method comprises the preprocessing of generating an image sequence in which the monocular video data is separated frame by frame. Claim 4 A method according to claim 2, wherein the 3D representation is learned or inferred through a deep neural network using one or more of the information generated through the preprocessing. Claim 5 A method according to claim 1, wherein volume rendering and a 3D mesh are applied to the 3D representation. Claim 6 A method according to claim 1, wherein the reference 3D mesh is generated based on pre-learning using a standard template model containing human joint information. Claim 7 In claim 6, the rigging information of the reference 3D mesh is represented based on linear blending skinning, and the linear blending skinning is applied based on one or more elements of the external skin information, posture information, facial expression information, or clothing information of the specific object. Claim 8 A method according to claim 1, wherein the pose information is applied to the reference 3D mesh to generate a pose 3D mesh. Claim 9 In claim 8, a method in which the physically based rendering information is applied to the pose 3D mesh to generate the 3D character model. Claim 10 A method according to claim 1, wherein the physical-based rendering information is estimated from monocular video data for the specific target. Claim 11 A method according to claim 1, wherein the physically based rendering information includes optical information representing the reflection of light on the surface material of a 3D object. Claim 12 A method according to claim 1, wherein the physically based rendering information comprises one or more elements of basic color, roughness, reflected light, opacity, luminescence, refraction, or distortion, distinguished according to a photorealistic approach or a non-photorealistic approach. Claim 13 A method according to claim 1, wherein when the 3D character model is generated during the inference process, the 3D character model is output using the learned model. Claim 14 A method according to claim 1, wherein, when the 3D character model is generated during the learning process, an error based on a loss calculated by comparing the 3D character model with the specific target is backpropagated to the step of generating the 3D representation, and the learned model is updated using the backpropagated error. Claim 15 A method according to claim 14, wherein comparing the 3D character model with the specific target comprises comparing an image rendered of a state learned using the 3D character model with a preprocessed image of the monocular video data for the specific target. Claim 16 A method according to claim 14, wherein the loss is calculated for one or more of a normal map, a depth map, or physically based rendering information. Claim 17 A method according to claim 1, wherein one or more new pose information is applied to the 3D character model to generate a 3D dynamic animation of the 3D character model. Claim 18 In a device for generating an automatically rigged 3D character, the device comprises: one or more transceivers; one or more memories; and one or more processors, wherein the one or more processors are: Using monocular video data obtained through the above one or more transmitters / receivers, preprocessing of monocular video data for a specific target is performed; Using the above-mentioned preprocessed result, a 3D representation of the specific object including external shape information of the specific object is generated and stored in the one or more memories; Generating a reference 3D mesh including rigging information using the above 3D representation and storing it in one or more memories; and A device configured to apply one or more of detail information or physically based rendering information to the reference 3D mesh to generate a 3D character model corresponding to the specific target and store it in one or more memories. Claim 19 One or more processors; and based on being operably connected to the one or more processors and executed by the one or more processors, The operation of acquiring monocular video data for a specific target; An operation to generate a 3D representation of the specific object including external shape information of the specific object; The operation of generating a reference 3D mesh including rigging information using the above 3D representation; and A processing unit comprising one or more local memories storing commands for performing an operation to generate a 3D character model corresponding to a specific target by applying one or more of posture information or physically based rendering information to the reference 3D mesh. Claim 20 One or more non-transitory computer-readable media storing one or more instructions, wherein the one or more instructions are executed by one or more processors, The operation of acquiring monocular video data for a specific target; An operation to generate a 3D representation of the specific object including external shape information of the specific object; The operation of generating a reference 3D mesh including rigging information using the above 3D representation; and A computer-readable medium that controls a device to perform an operation of generating a 3D character model corresponding to a specific target by applying one or more of detail information or physically based rendering information to the reference 3D mesh.

Images