Method and apparatus for generating avatars based on guided diffusion model considering body contact and interaction

Abstract

A method and apparatus for generating an avatar based on an induced diffusion model considering physical contact and interaction according to one embodiment may include the steps of: inputting a partner avatar representation and an interaction label into an avatar generation model; generating a denoising sample representation from which noise has been removed by performing a denoising process of the avatar generation model based on the input partner avatar representation and interaction label and a noise sample; generating a target sample representation by modifying the denoising sample representation based on guidance that considers a contact objective set based on a contact prediction area set; generating a partially denoising avatar representation by performing a diffusion process of the avatar generation model from the target sample representation; and generating a target avatar representation corresponding to the interaction label by repeating the generation of the denoising sample representation and the generation of the partially denoising avatar representation.

Description

Technology Field

[0001] Below, technology for generating a 3D human that interacts with a partner based on contact is provided. Background Technology

[0003] 3D modeling refers to the creation or representation of objects in the real world within a virtual three-dimensional space. In the past, 3D modeling primarily involved humans directly drawing objects or manipulating basic shapes to transform them into desired objects; however, currently, methods are used to automatically generate 3D models by scanning actual objects using sensors, cameras, or detectors.

[0004] When this 3D modeling technology is applied to the field of robotics, robots can act based on the situation they understand as they recognize and understand 3D modeled objects. Furthermore, there is a growing demand for 3D modeling technology that can be utilized in public sectors requiring problem-solving and management in areas such as construction, transportation, energy, and the environment, as well as in private sectors requiring game / broadcast / film content production and VR / AR services.

[0005] The background technology described above is possessed or acquired by the inventor in the process of deriving the content of the disclosure of the present application, and cannot necessarily be considered as prior art disclosed to the general public prior to the filing of this application. Prior art literature

[0007] Korean Published Patent Application No. 10-2024-0013613 (published January 30, 2024) discloses a method for generating AI human 3D motion using only images and a recording medium thereof. Korean Registered Patent Application No. 10-2382419 (published March 30, 2022) discloses a method and apparatus for generating a 3D joint point regression model. The problem to be solved

[0008] A method and device for generating an avatar based on an induced diffusion model considering physical contact and interaction according to one embodiment can predict potential contact areas between people and provide guidance on diffusion.

[0009] A method and device for generating an avatar based on an induced diffusion model considering physical contact and interaction according to one embodiment can generate a three-dimensional avatar based on a contact area according to the interaction of an input partner avatar representation.

[0010] A method and apparatus for generating an avatar based on an induced diffusion model considering physical contact and interaction according to one embodiment can indirectly optimize a denoised sample representation rather than directly optimizing a partially denoised avatar representation that is the result of a reverse process.

[0011] A method and apparatus for generating an avatar based on an induced diffusion model considering physical contact and interaction according to one embodiment can estimate a contact probability map for each region between a given partner avatar representation and a denoising sample representation using a contact prediction module configured based on a feature vector converter.

[0012] However, technical challenges are not limited to the technical challenges described above, and other technical challenges may exist. means of solving the problem

[0014] A method for generating an avatar based on an induced diffusion model considering physical contact and interaction according to one embodiment may include the steps of: inputting a partner avatar representation and an interaction label into an avatar generation model; generating a denoising sample representation from which noise has been removed by performing a denoising process of the avatar generation model based on the input partner avatar representation and interaction label and a noise sample; generating a target sample representation by modifying the denoising sample representation based on guidance that considers a contact objective set based on a contact prediction area set; generating a partially denoising avatar representation by performing a diffusion process of the avatar generation model from the target sample representation; and generating a target avatar representation corresponding to the interaction label by repeating the generation of the denoising sample representation and the generation of the partially denoising avatar representation.

[0015] The step of generating the target avatar expression comprises: generating a denoising sample expression of the next cycle based on the partner avatar expression and interaction label and the partially denoising avatar expression; generating a target sample expression of the next cycle based on the generated denoising sample expression of the next cycle; generating a partially denoising avatar expression of the next cycle based on the target sample expression of the next cycle; and generating the target avatar expression by repeating the generation of the denoising sample expression of the next cycle and the partially denoising avatar expression of the next cycle, wherein for the last iteration, the target avatar expression may be the partially denoising avatar expression of the next cycle.

[0016] The step of generating the target sample representation may include: generating a set of contact prediction regions by inputting the denoising sample representation, the partner avatar representation, and the interaction label into a contact prediction module of the avatar generation model; setting a contact goal that promotes physical contact between the denoising sample representation and the partner avatar representation based on the generated set of contact prediction regions; calculating the guidance using a contact goal-based objective function for the partner avatar representation, the interaction label, and the denoising sample representation; and generating the target sample representation based on an element-wise product between the denoising sample representation, the calculated guidance, and a scaling vector for the guidance.

[0017] The step of setting the contact goal may include setting the contact goal based on the sum of the chamfer distances between two sets of vertices of regions belonging to the contact prediction region set.

[0018] The step of generating the contact prediction area set may include: converting the bodies of the denoising sample representation and the partner avatar representation into mesh surfaces; dividing the bodies of the denoising sample representation and the partner avatar representation into a predetermined number of regions based on vertex indices on the mesh surfaces of the denoising sample representation and the partner avatar representation; and generating the contact prediction area set by predicting a contact probability map for each region between the denoising sample representation and the partner avatar representation for each divided region.

[0019] The step of generating the contact prediction region set may include: inputting a center location for each region of the denoising sample representation and the partner avatar representation into a feature vector converter to calculate a feature vector for each region; inputting the calculated feature vector for each region into the contact prediction module to estimate a contact probability map for each region that indicates the probability of contact occurring between the region of the denoising sample representation and the region of the partner avatar representation; and generating the contact prediction region set including a pair of contact prediction regions that correspond to a probability greater than or equal to a predetermined threshold in the contact probability map for each region.

[0020] The step of generating the above contact prediction region set may further include the step of training the contact prediction module based on a binary cross-entropy objective function that minimizes the loss between the ground truth contact probability map and the estimated contact probability map before estimating the contact probability map for each region.

[0021] The above method may further include the step of training a noise prediction model that performs a denoising process of the avatar generation model before inputting the partner avatar representation and interaction labels into the avatar generation model.

[0022] The step of training the noise prediction model may include training the noise prediction model by minimizing the loss between the true value noise and the predicted noise, which are separated into one or more parts according to the avatar representation characteristics of the training data.

[0023] An avatar generation device based on an induced diffusion model considering physical contact and interaction according to one embodiment may include a processor that inputs a partner avatar representation and an interaction label into an avatar generation model, generates a denoising sample representation with noise removed by performing a denoising process of the avatar generation model based on the input partner avatar representation and interaction label and a noise sample, generates a target sample representation by modifying the denoising sample representation based on guidance considering a contact objective set based on a contact prediction area set, generates a partially denoising avatar representation by performing a diffusion process of the avatar generation model from the target sample representation, and generates a target avatar representation corresponding to the interaction label by repeating the generation of the denoising sample representation and the generation of the partially denoising avatar representation.

[0024] The processor generates a denoising sample representation of the next cycle based on the partner avatar representation and interaction label and the partially denoising avatar representation, generates a target sample representation of the next cycle based on the generated denoising sample representation of the next cycle, generates a partially denoising avatar representation of the next cycle based on the target sample representation of the next cycle, and generates the target avatar representation by repeating the generation of the denoising sample representation of the next cycle and the partially denoising avatar representation of the next cycle, and for the last iteration, the target avatar representation may be the partially denoising avatar representation of the next cycle.

[0025] The processor can generate a set of contact prediction regions by inputting the denoising sample representation, the partner avatar representation, and the interaction label into a contact prediction module of the avatar generation model, set a contact goal that promotes physical contact between the denoising sample representation and the partner avatar representation based on the generated set of contact prediction regions, calculate the guidance using an objective function based on the contact goal for the partner avatar representation, the interaction label, and the denoising sample representation, and generate the target sample representation based on an element-wise product between the denoising sample representation, the calculated guidance, and a scaling vector for the guidance.

[0026] The processor can set the contact target based on the sum of the chamfer distances between two sets of vertices of regions belonging to the contact prediction region set.

[0027] The processor can convert the bodies of the denoising sample representation and the partner avatar representation into a mesh surface, divide the bodies of the denoising sample representation and the partner avatar representation into a predetermined number of regions based on vertex indices on the mesh surface of the denoising sample representation and the partner avatar representation, and generate the contact prediction region set by predicting a contact probability map for each region between the denoising sample representation and the partner avatar representation for each divided region.

[0028] The processor calculates a feature vector for each region by inputting a center location for each region of the denoising sample representation and the partner avatar representation into a feature vector converter, and inputs the calculated feature vector for each region into a contact prediction module to estimate a contact probability map for each region that indicates the probability of contact occurring between the region of the denoising sample representation and the region of the partner avatar representation, and can generate a set of contact prediction regions including pairs of contact prediction regions that correspond to a probability greater than or equal to a predetermined threshold in the contact probability map for each region.

[0029] The processor can train the contact prediction module based on a binary cross-entropy objective function that minimizes the loss between the ground truth contact probability map and the estimated contact probability map before estimating the contact probability map for each region.

[0030] The processor can train a noise prediction model that performs a denoising process of the avatar generation model before inputting the partner avatar representation and interaction labels into the avatar generation model.

[0031] The processor can train the noise prediction model by minimizing the loss between the true value noise and the predicted noise, which are separated into one or more parts according to the avatar representation characteristics of the training data. Effects of the invention

[0033] A method and device for generating an avatar based on an induced diffusion model considering physical contact and interaction according to one embodiment can predict potential contact areas between people and provide guidance on diffusion to generate dynamic and non-rigid motion based on interaction between people without initialization and additional networks.

[0034] A method and device for generating an avatar based on an induced diffusion model considering physical contact and interaction according to one embodiment can generate a three-dimensional avatar based on a contact area according to the interaction of an input partner avatar representation, so that the generated avatar can be realistically represented to naturally interact with a given partner while physically contacting them.

[0035] A method and apparatus for generating an avatar based on an induced diffusion model considering body contact and interaction according to one embodiment can more effectively estimate the contact area by indirectly optimizing the denoising sample representation rather than directly optimizing the partially denoised avatar representation resulting from the reverse process.

[0036] A method and apparatus for generating avatars based on an induced diffusion model considering physical contact and interaction according to one embodiment can generate various high-quality interactive avatars by adaptively providing potential and dynamic contact areas and avoiding collisions between diffusion directions and guiding directions by using a contact prediction module configured based on a feature vector converter to estimate a contact probability map for each region between a given partner avatar representation and a denoising sample representation. Brief explanation of the drawing

[0038] FIG. 1 illustrates a block diagram of an avatar generation device based on an inductive diffusion model that considers body contact and interaction according to one embodiment. FIG. 2 schematically illustrates the process of generating an avatar in an induced diffusion model-based avatar generation device that considers body contact and interaction according to one embodiment. FIG. 3 specifically illustrates the process of generating an avatar in an induced diffusion model-based avatar generation device that considers physical contact and interaction according to one embodiment. FIG. 4 exemplarily illustrates the structure of a contact prediction model within an avatar generation model of an avatar generation device based on an induced diffusion model that considers body contact and interaction according to one embodiment. FIG. 5 illustrates a flowchart of an avatar generation method based on an induced diffusion model considering body contact and interaction according to one embodiment. FIG. 6 exemplarily shows an avatar generated by an avatar generation device and method based on an induced diffusion model considering body contact and interaction according to one embodiment, and an avatar generated by a different method. FIG. 7 exemplarily illustrates an avatar generated based on a contact area with a partner avatar by a different method than the device and method for generating an avatar based on an induced diffusion model considering body contact and interaction according to one embodiment. Specific details for implementing the invention

[0039] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments.

[0040] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0041] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.

[0042] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0043] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0044] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0045] The need for generative modeling of 3D shapes is rapidly increasing due to high demand in various fields such as computer vision, graphics, robotics, and content creation. 3D shapes can be generated using various methods, such as Generative Adversarial Networks (GANs), Variational Auto-Encoders (VAEs), normalization flows, and auto-regression models.

[0046] Meanwhile, human behavior often stems from various interactions rather than independent actions such as walking or running. For example, humans continuously interact with various objects in their surroundings and engage in daily communication with others. Among these, person-to-person interaction can be a crucial element in fulfilling various aspects of life, such as building social relationships, exchanging information, and sharing emotions. However, current research on the creation of 3D people (e.g., avatars) focuses on generating 3D human behavior based on sound, scenes, text, and previous motion inputs, without person-to-person interaction. Furthermore, research on physical interaction focuses on generating 3D forms based on hand-object interactions between a person and an object, rather than person-to-person interaction.

[0047] More specifically, the recognition and prediction of human behavior are crucial for understanding human behavior, leading to active research in the field of computer vision. Human behavior generation can be achieved by utilizing VAE architectures that learn latent spaces to generate human motion based on given action labels or text descriptions containing multiple movements. However, VAE-based methods have the disadvantage of being prone to backward collapse, which limits the diversity of human motion. To address this issue, methods based on diffusion models can generate human motion with superior performance. Nevertheless, these diffusion-based methods have the drawback of generating only single motions and failing to produce motions based on interactions.

[0048] In addition, interaction prediction technology based on physical contact can generate a 3D shape of a person grasping an object by utilizing the proximity between the person and the object. However, since such interaction prediction technology is for interactions between people and objects, it has the problem of not being suitable for dynamic and non-rigid interactions between people. Although there is a technology to generate 3D people based on interactions between people, it requires initialization of the generative model, which necessitates an additional network, resulting in high implementation costs and increased complexity.

[0049] Accordingly, the method and device for generating an avatar based on an induced diffusion model considering physical contact and interaction according to one embodiment can predict potential contact areas between people and provide guidance on diffusion to generate dynamic and non-rigid motion based on interaction between people without initialization and additional networks.

[0050] FIG. 1 illustrates a block diagram of an avatar generation device based on an inductive diffusion model that considers body contact and interaction according to one embodiment.

[0051] FIG. 2 schematically illustrates the process of generating an avatar in an induced diffusion model-based avatar generation device that considers body contact and interaction according to one embodiment.

[0052] The device (100) may include a processor (110) and a memory (120).

[0053] A processor (110) inputs a partner avatar representation (211) and an interaction label (212) into an avatar generation model (e.g., ContactGen) (220), generates a denoising sample representation with noise removed by performing a denoising process of the avatar generation model (220) based on the input partner avatar representation (211) and interaction label (212) and a noise sample, generates a target sample representation by modifying the denoising sample representation based on guidance that considers a contact objective set based on a set of contact prediction regions, generates a partially denoising avatar representation by performing a diffusion process of the avatar generation model (220) from the target sample representation, and generates a target avatar representation (231) corresponding to the interaction label by repeating the generation of the denoising sample representation and the generation of the partially denoising avatar representation. A detailed description of how to generate a target avatar representation (231) is given later in FIG. 3.

[0054] On the other hand, since humans are dynamic rather than rigid, numerous combinations of poses and shapes can be generated based on this. Furthermore, compared to objects (or scenes) with clearly defined purposes, the area of ​​contact between humans can vary depending on the type of interaction. For example, when two people greet each other, the body parts in contact may differ depending on whether they shake hands, hug, or give a bisou.

[0055] Accordingly, an avatar generation device (100) based on an inductive diffusion model that considers physical contact and interaction according to one embodiment can first generate a partner avatar representation (211) in three dimensions. Since there are already many three-dimensional generation models with excellent performance, one of them (e.g., SMPL-X) can be used to generate a person (e.g., partner avatar representation (211)) with a specific intention (e.g., hand position and posture, etc.) in advance. This partner avatar representation (211) can be generated by the device (100), but is not limited thereto; it can be generated by a device of another configuration including a three-dimensional generation model, and then the device (100) can receive the generated partner avatar representation (211) via communication. The fact that the partner avatar representation (211) is generated first may mean that a partner avatar representation (211) with a specific intention is generated so that, even in the case of a complex interaction such as a greeting, the interaction is limited to a certain range such as a handshake. Accordingly, the processor (110) can generate a realistic 3D person that interacts naturally with a given partner by physically making contact based on the contact area resulting from the interaction of a partner avatar representation (211) with a specific intention. For example, if there is an interaction called "Push," the partner avatar representation (211) is in a posture of turning its back and leaving, so the processor (110) can generate a target avatar representation (231) in the form of pushing the back of the partner avatar representation (211) based on the posture of the partner avatar representation (211) and the contact area resulting from the interaction.

[0056] The avatar generation model (220) may include a forward process and a reverse process based on a denoising diffusion model (DDM). The denoising diffusion model can be trained by gradually damaging data through the diffusion process of the forward process and removing noise from the damaged data through the denoising process. In this way, the avatar generation model (220) can be trained to predict and remove noise from noisy avatar parameters based on a fully connected network (FCN), and can generate a new avatar by gradually and repeatedly removing noise from random noise based on input partner avatar representations and interaction labels.

[0057] To explain the aforementioned process in more detail, the forward process of the avatar generation model (220) can be performed by diffusion using Equations 1 and 2, which are expressed as a Markov chain. The true value target avatar representation ( ) can be gradually damaged by Gaussian noise and eventually become a standard Gaussian distribution.

[0058]

[0059]

[0060] In mathematical formula 1, can represent noise scheduling parameters. Noise scheduling parameters can be set by the user as hyperparameters. In addition, the forward process samples at an arbitrary time step (e.g., period) t in a closed form as shown in Equation 3 ( ) can be allowed.

[0061]

[0062] In mathematical formula 3, And, am.

[0063] In the reverse process, denoising can be performed using mathematical formulas 4 and 5, which are expressed as a Markov chain. Accordingly, the processor (110) through the reverse process, the partner avatar representation ( ) and interaction labels( By predicting noise based on the condition ), gradually sampling ( ) can be restored.

[0064]

[0065]

[0066] In mathematical formulas 4 and 5, represents the standard Gaussian distribution, and Is Noise prediction network parameterized by ( Represents the mean derived from ), can represent time step-dependent variance. As the processor (110) repeats this denoising process, the target avatar representation ( Can generate ).

[0067] FIG. 3 specifically illustrates the process of generating an avatar in an induced diffusion model-based avatar generation device that considers physical contact and interaction according to one embodiment.

[0068] Target Avatar Representation ( To explain the process of generating )(324) in more detail, the processor (110) inputs the partner avatar representation (311)( ) and interaction labels (312)( ) and noise samples( Based on )(321), the avatar generation model (30) (e.g., denoising model (33)) Denoising sample representation with noise removed by performing the denoising process ( It can generate ). Also, noise samples ( ) may be, but is not limited to, a sample of random noise having a standard Gaussian distribution, and the processor (110) uses an avatar generation model (30) to create a partner avatar representation (311)( ) and interaction labels (312)( It may be a sample in which noise with a standard Gaussian distribution is added to a result generated solely by ). Also, the denoising model (33)( ) is a noise prediction model( It can be based on ), and such a noise prediction model ( The method of training ) is described later through mathematical formula 14.

[0069] The processor (110) (e.g., in the interaction optimization process (32)) denoising sample representation ( based on guidance that considers a contact target set based on a set of contact prediction regions) By modifying ) target sample representation( Can generate ). Denoising sample representation( ) and target sample representation( ) is a set of parameters and may mean a visualization of a three-dimensional shape corresponding to the set of parameters through an avatar generation model (30). Accordingly, a denoising sample representation ( Modifying ) is the denoising sample representation ( It can be considered that the parameters of ) have been modified. Target sample representation( Detailed information on generating ) will be described later using mathematical formula 9. The processor (110) generates a target sample representation ( By performing the diffusion process (31) of the avatar generation model (30) from ) a partially denoised avatar representation ( )(322) can be generated. The cycle of the forward process can proceed from 1 (one) to T, and the cycle of the reverse process can proceed from T to 1 (one).

[0070] Then, the processor (110) performs the reverse process again, this time the partner avatar representation (311)( ) and interaction labels (312)( ) and partially denoised avatar representation( Denoising sample representation of the next period (t) based on )(322) Can generate ). Denoising sample representation of the next period (t) ) can be generated using mathematical formula 6.

[0071]

[0072] Additionally, the processor (110) has a denoising sample representation of the next generated period (t) ( Based on ), the target sample representation of the next period (t) Generate ) and the target sample representation of the next period (t) Partially denoised avatar representation of the next period (t-1) based on ) )(323) can be generated. The processor (110) can generate a target avatar representation ( )(324) can be generated. For example, for the last iteration (1), the target avatar representation ( ) is the partially denoised avatar representation of the next cycle ( It can be ). Also, the initial noise sample ( )(321) is formed in an unrealistic pose or angle. Accordingly, the processor (110) repeats the denoising process and the diffusion process to produce a partially denoised avatar representation ( )(322) and ( Target avatar representation via )(323)( As )(324) is generated, the avatar representation can gradually form realistic poses or angles.

[0073] An avatar generation device (100) according to one embodiment is of Equation 4 In conditional sampling at, a specific goal ( satisfying ) It is possible to generate a guided (e.g., guided) target avatar representation that samples using. First, the guided sampling distribution ( ) can be expressed as Equations 7 and 8 using the form of a Markov chain.

[0074]

[0075]

[0076] In mathematical formula 8, Is and specific goals ( A goal-recognized denoised person prediction function based on ) can be represented. In addition, is a target sample representation ( generated using an interaction optimization process (32) It can be. It can be generated using mathematical formula 9 and mathematical formula 10.

[0077]

[0078]

[0079] λ is guidance( It represents a scaling vector for ), and ⊙ can represent the element-wise product. Accordingly, the target sample representation ( ) is guidance( Specific goal( by ) ) can be considered. Guidance( ) can be generated based on a contact prediction region where contact is predicted by interaction. A specific goal ( ) will be discussed later in Equation 12.

[0080] First of all, the processor (110) (e.g., through the interaction optimization process (32)) denoising sample representation ( ), partner avatar representation( ) and interaction labels( By inputting ) into the contact prediction module of the avatar generation model, the contact prediction area set ( ) can generate. The processor (110) generates a set of contact prediction regions ( Denoising sample representation based on ) ) and partner avatar representation( Contact targets that facilitate physical contact between ) Set ) and partner avatar representation( ), interaction labels( ) and denoising sample representation( Contact target for ) Guidance using an objective function based on ) (e.g., Equation 10) ( ) can be calculated. The processor (110) can calculate the denoising sample representation ( ) and calculated guidance( ) and guidance( Based on the element-wise product (⊙) (e.g., Equation 9) between the scaling vector (λ) for ) the target sample representation ( Can generate ).

[0081] Accordingly, the processor (110) has a partially denoised avatar representation that is the result of the reverse process ( Rather than directly optimizing ) indirectly denoising sample representation( By optimizing ), the contact area can be estimated more effectively.

[0082] FIG. 4 exemplarily illustrates the structure of a contact prediction model within an avatar generation model of an avatar generation device based on an induced diffusion model that considers body contact and interaction according to one embodiment.

[0083] Using a contact prediction model, a set of contact prediction areas ( In generating ), first, the processor (110) inputs a denoising sample representation ( ) and partner avatar representation( Center locations visualization can be performed on ). In performing center locations visualization, the processor (110) denoising sample representation ( ) and partner avatar representation( The body of ) can be converted into a mesh surface. The processor (110) converts the body of the denoising sample representation and the partner avatar representation based on vertex indices on the mesh surface of the denoising sample representation and the partner avatar representation into a predetermined number ( It can be divided into regions of )(e.g., 75). The processor (110) creates a region-specific contact probability map ( between the denoising sample representation and the partner avatar representation for each divided region) By predicting ) a set of contact prediction regions (e.g., potential regions)( Can generate ).

[0084] More specifically, the processor (110) can calculate feature vectors for each region by inputting a predetermined number of center locations for each region of the denoising sample representation and the partner avatar representation into a feature vector converter. The feature vector converter may be configured with a structure in which one or more self-attention and cross-attention intersect. The processor (110) uses a contact prediction module configured based on this feature vector converter to calculate a given partner avatar representation ( ) and denoising sample representation( By estimating the contact probability map for each region between ), potential and dynamic contact areas can be adaptively provided, and by avoiding collisions between the diffusion direction and the guiding direction, various high-quality interactive avatars can be generated.

[0085] The processor (110) inputs the calculated feature vector for each region into the contact prediction module to obtain a denoising sample representation ( The domain of ) and partner avatar representation( Contact probability map for each region representing the probability of contact occurring between the regions of ) ) can be estimated. Subsequently, the processor (110) can estimate the contact probability map for each region ( A predetermined threshold value at ) A set of contact prediction regions containing pairs of contact prediction regions corresponding to a probability of )(e.g., 0.5) or higher ( Can generate ).

[0086] Additionally, the processor (110) estimates the ground truth contact probability map ( ) and estimated contact probability map( A contact prediction module can be trained based on a binary cross-entropy objective function that minimizes the loss between ). The binary cross-entropy objective function can be expressed by Equation 11.

[0087]

[0088] In mathematical formula 11, is a denoising sample representation ( ) and partner avatar representation( It can represent the probability that contact occurs between the i-th region and the j-th region of ).

[0089] In addition, the processor (110) has a contact goal (which is an optimization goal) Using ) to provide direction in terms of physical interaction and denoising sample representation( ) and partner avatar representation( By facilitating physical contact between ), it can aid in the diffusion process and enable natural interaction. The processor (110) has a set of contact prediction areas ( Two sets of vertices of regions belonging to ) Chamfer distance between ) Contact target ( based on the sum of ) You can set the contact goal ( ) can be expressed by mathematical formula 12 and mathematical formula 13.

[0090]

[0091]

[0092] In mathematical formulas 12 and 13, and are each of the regions ( ),( Denoising sample representations belonging to ) ) and partner avatar representation( It can represent a set of vertices between ).

[0093] The processor (110) represents the partner avatar in the avatar creation model (30). )(311) and interaction labels( A noise prediction model that performs the denoising process of the avatar generation model before inputting )(312) (e.g., a noise prediction network ( )) can be trained. The processor (110) can train ground truth noise (which is separated into one or more parts according to the avatar representation characteristics of the training data) ) and predicted noise( By minimizing the loss between ) noise prediction model( ) can be trained. For example, the processor (110) can train a noise prediction model (represented by Equation 14) ) true value noise( ) and predicted noise( Training can be performed using the Mean Squared Error (MSE) between ).

[0094]

[0095] In mathematical equation 14, true value noise ( ) and predicted noise( ) are true value noise( ) and predicted noise( Representing a part of ), silver The dimension of can be represented. For example, the processor (110) predicts noise ( Four parts based on characteristics such as transformation, rotation, body pose, and hand pose ( It can be divided into ). Accordingly, the processor (110) can train a model focused on each person's characteristics with equal weights.

[0096] According to various embodiments, the processor (110) can execute computer-readable code (e.g., software) stored in memory (120) and instructions triggered by the processor (110). The processor (110) may be a data processing device implemented in hardware having a circuit having a physical structure for executing desired operations. Desired operations may include, for example, code or instructions included in a program. The data processing device implemented in hardware may include, for example, a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an Application-Specific Integrated Circuit (ASIC), or a Field Programmable Gate Array (FPGA).

[0097] The memory (120) can store an avatar generation model (220), a training data set required for training the avatar generation model (220), hyperparameters required for the avatar generation model (220), and parameters generated by training the avatar generation model (220).

[0098] FIG. 5 illustrates a flowchart of an avatar generation method based on an induced diffusion model considering body contact and interaction according to one embodiment.

[0099] In step (510), the processor can input partner avatar representations and interaction labels into the avatar generation model.

[0100] In step (520), the processor can generate a denoising sample representation with noise removed by performing a denoising process on the avatar generation model based on the input partner avatar representation and interaction label and the noise sample.

[0101] In step (530), the processor can generate a target sample representation by modifying the denoising sample representation based on guidance that considers a contact objective set based on a set of contact prediction regions.

[0102] In step (540), the processor can generate a partially denoised avatar representation by performing a diffusion process of the avatar generation model from the target sample representation.

[0103] In step (550), the processor can generate a target avatar representation corresponding to an interaction label by repeating the generation of a denoising sample representation and the generation of a partially denoising avatar representation.

[0104] In addition, the processor can train a noise prediction model that performs a denoising process on the avatar generation model before inputting partner avatar representations and interaction labels into the avatar generation model. For example, the processor can train the noise prediction model by minimizing the loss between the true value noise separated into one or more parts and the predicted noise based on the avatar representation characteristics of the training data.

[0105] Additionally, in generating the target avatar representation, the processor may generate the denoised sample representation of the next cycle based on the partner avatar representation and interaction labels and the partially denoised avatar representation. The processor may generate the target sample representation of the next cycle based on the generated denoised sample representation of the next cycle. The processor may generate the partially denoised avatar representation of the next cycle based on the target sample representation of the next cycle. The processor may generate the target avatar representation by repeating the generation of the denoised sample representation of the next cycle and the partially denoised avatar representation of the next cycle. For the last iteration, the target avatar representation may be the partially denoised avatar representation of the next cycle.

[0106] In generating the target sample representation, first, the processor can generate a set of contact prediction regions by inputting the denoising sample representation, the partner avatar representation, and interaction labels into the contact prediction module of the avatar generation model. In generating the set of contact prediction regions, the processor can convert the bodies of the denoising sample representation and the partner avatar representation into mesh surfaces. Additionally, the bodies of the denoising sample representation and the partner avatar representation can be divided into a predetermined number of regions based on vertex indices on the mesh surfaces of the denoising sample representation and the partner avatar representation.

[0107] Additionally, the processor can generate a set of contact prediction regions by predicting a contact probability map for each region between the denoising sample representation and the partner avatar representation for each divided region. More specifically, the processor can calculate a feature vector for each region by inputting the center location for each region of the denoising sample representation and the partner avatar representation into a feature vector transformer. The processor can estimate a contact probability map for each region, which indicates the probability of contact occurring between the region of the denoising sample representation and the region of the partner avatar representation, by inputting the calculated feature vector for each region into a contact prediction module. The processor can generate a set of contact prediction regions that includes pairs of contact prediction regions corresponding to probabilities above a predetermined threshold in each region's contact probability map.

[0108] In addition, the processor can train a contact prediction module based on a binary cross-entropy objective function that minimizes the loss between the ground truth contact probability map and the estimated contact probability map before estimating the contact probability map for each region.

[0109] The processor can set a contact goal that facilitates physical contact between the denoising sample representation and the partner avatar representation based on a generated set of contact prediction regions. For example, the processor can set a contact goal based on the sum of the chamfer distances between two sets of vertices of regions belonging to the set of contact prediction regions.

[0110] Algorithm 1 is an algorithm for training an avatar generation model, which can be described in detail in Table 1 below.

[0111]

[0112] In the first line, the processor based on the training data Target avatar representation that is a true value within the distribution ( ), partner avatar representation( ) and interaction labels( You can select ).

[0113] In the second line, the processor can select a value of t to use with uniform probability from values ​​1 to T.

[0114] In the 3rd line, the processor is noise that follows a multidimensional Gaussian distribution with a mean of 0 and a covariance matrix of the identity matrix (I). ) can be defined.

[0115] In the 4th line, the processor [represents] the target avatar representation selected in the 1st line ( ) and the noise defined on the 3rd line( t-th noisy sampling based on ) Can generate ).

[0116] In the 5th line, the processor is a noise prediction model ( Sampling selected from the 4th line using ) ), t selected in the 2nd line, partner avatar representation selected in the 1st line( ) and interaction labels( Predicted noise based on ) Can generate ).

[0117] In the 6th line, the processor uses the initially input learning rate ( ) and the noise defined on the 3rd line( ), predicted noise generated on the 5th line( Based on the gradient descent method that minimizes the loss function between ), the parameters ( Can update ).

[0118] The processor performs the aforementioned processes on a noise prediction model ( This can be repeated until it converges.

[0119] Algorithm 2 is an algorithm that generates a target avatar representation of an avatar generation model trained by Algorithm 1, and can be described in detail in Table 2 below.

[0120]

[0121] In the first line, the processor takes a noise sample following a multivariate Gaussian distribution with a mean of 0 and a covariance matrix of identity (I). ) can be defined.

[0122] In the second line, the processor can indicate that it performs from time step T to 1.

[0123] In the 3rd line, the processor, as described above in Equation 6, the partner avatar representation at time step t ( ), interaction labels( ) and partially denoised avatar representation( Denoising sample representation based on ) Can generate ).

[0124] In the 4th line, the processor represents the partner avatar as described above in Equation 10 ( ), interaction labels( ) and denoising sample representation( Contact target for ) Guidance using an objective function based on ) (e.g., gradient descent) ( ) can be calculated.

[0125] In the 5th line, the processor, as described above in Equation 9, the denoising sample representation ( ) and calculated guidance( ) and guidance( Based on the element-wise product (⊙) between the scaling vector (λ) for ) target sample representation ( Can generate ).

[0126] In the 6th line, the processor is the target sample representation ( ) and partially denoised avatar representation( Partially denoised avatar representation based on ) ) can be generated. This may mean that the diffusion process can be performed without the noise that is previously imposed at every diffusion step. Additionally, the processor can repeat the process from the 3rd line to the 6th line according to the time step in the 2nd line. In the last step (e.g., repetition)(1), the process of the 6th line is omitted so that the target sample representation generated in the 5th line ( ) target avatar representation( It can be output as ).

[0127] FIG. 6 exemplarily shows an avatar generated by an avatar generation device and method based on an induced diffusion model considering body contact and interaction according to one embodiment, and an avatar generated by a different method.

[0128] (610) represents avatars generated by an avatar generation device and method according to one embodiment. An avatar generation device and method according to one embodiment can generate different target avatar representations even in the same interaction depending on the partner avatar representation.

[0129] (620) is a method of generating two people in the form of a three-dimensional skeleton based on interaction, and (630) is a method of generating human movement based on contact between an object and a person.

[0130] For ease of comparison, when the value for one person is set identically, (620) and (630) also show incorrect results compared to (610), where the avatar is generated in an inappropriate pose and penetrates another avatar.

[0131] A quantitative comparison of (610), (620), and (630) can be explained in detail in Table 3 below.

[0132]

[0133] (610), (620), and (630) can evaluate quality using Frechet Human Interaction Distance (FHID) and Top-1 Accuracy, and evaluate physical feasibility using contact score and non-collision score. Table 3 evaluates the results by adjusting for similar results, as (620) and (630) do not generate interactive 3D avatars based on physical contact. The evaluation results show that (610) has superior results compared to (620) and (630) in terms of FHID, Top-1 Accuracy, and non-collision score.

[0134] FIG. 7 exemplarily illustrates an avatar generated based on a contact area with a partner avatar by a different method than the device and method for generating an avatar based on an induced diffusion model considering body contact and interaction according to one embodiment.

[0135] (710) represents an avatar (711) generated based on a contact area by an avatar generation device and method according to one embodiment.

[0136] (720) represents an avatar (721) generated based on a contact area in the manner of (630) in FIG. 6. Since (720) estimates the contact probability per vertex rather than per vertex of the contact area, its ability to capture dynamic interactions with people may be inferior to (710).

[0137] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0138] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0139] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0140] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0141] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based thereon. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0142] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A method for generating an avatar that considers physical contact and interaction by a processor comprises: inputting a partner avatar representation and interaction labels into an avatar generation model; generating a denoising sample representation from which noise has been removed by performing a denoising process of the avatar generation model based on the input partner avatar representation and interaction labels and a noise sample; generating a target sample representation by modifying the denoising sample representation based on guidance that considers a contact objective set based on a set of contact prediction regions; and generating a partially denoised avatar representation by performing a diffusion process of the avatar generation model from the target sample representation. A method comprising the step of generating a target avatar representation corresponding to the interaction label by repeating the generation of the denoising sample representation and the generation of the partially denoising avatar representation, wherein the step of generating the target sample representation comprises: generating a contact prediction region set by inputting the denoising sample representation, the partner avatar representation, and the interaction label into a contact prediction module of the avatar generation model; setting a contact goal that promotes physical contact between the denoising sample representation and the partner avatar representation based on the sum of the chamfer distances between two vertex sets of regions belonging to the generated contact prediction region set; calculating the guidance using an objective function based on the contact goal for the partner avatar representation, the interaction label, and the denoising sample representation; and generating the target sample representation based on an element-wise product between the denoising sample representation, the calculated guidance, and a scaling vector for the guidance. Claim 2 The method according to claim 1, wherein the step of generating the target avatar representation comprises: generating a denoising sample representation of the next cycle based on the partner avatar representation and interaction label and the partially denoising avatar representation; generating a target sample representation of the next cycle based on the generated denoising sample representation of the next cycle; generating a partially denoising avatar representation of the next cycle based on the target sample representation of the next cycle; and generating the target avatar representation by repeating the generation of the denoising sample representation of the next cycle and the partially denoising avatar representation of the next cycle, wherein, for the last iteration, the target avatar representation is the partially denoising avatar representation of the next cycle. Claim 3 delete Claim 4 delete Claim 5 A method according to claim 1, wherein the step of generating the contact prediction area set comprises: converting the body of the denoising sample representation and the partner avatar representation into a mesh surface; dividing the body of the denoising sample representation and the partner avatar representation into a predetermined number of regions based on vertex indices on the mesh surface of the denoising sample representation and the partner avatar representation; and generating the contact prediction area set by predicting a contact probability map for each region between the denoising sample representation and the partner avatar representation for each divided region. Claim 6 In claim 5, the step of generating the contact prediction area set comprises: inputting a center position for each area of ​​the denoising sample representation and the partner avatar representation into a feature vector converter to calculate a feature vector for each area; inputting the calculated feature vector for each area into a contact prediction module to estimate a contact probability map for each area representing the probability of contact occurring between the area of ​​the denoising sample representation and the area of ​​the partner avatar representation; and generating the contact prediction area set including a contact prediction area pair corresponding to a probability greater than or equal to a predetermined threshold in the contact probability map for each area. Claim 7 In claim 6, a method further comprising the step of training the contact prediction module based on a binary cross-entropy objective function that minimizes the loss between the ground truth contact probability map and the estimated contact probability map, prior to estimating the contact probability map for each region. Claim 8 A method according to claim 1, further comprising the step of training a noise prediction model that performs a denoising process of the avatar generation model before inputting the partner avatar representation and interaction labels into the avatar generation model. Claim 9 In claim 8, the step of training the noise prediction model comprises the step of training the noise prediction model by minimizing the loss between the true value noise separated into one or more parts and the predicted noise according to the avatar representation characteristics of the training data. Claim 10 The processor comprises a processor that inputs a partner avatar representation and an interaction label into an avatar generation model, generates a denoising sample representation with noise removed by performing a denoising process of the avatar generation model based on the input partner avatar representation and interaction label and a noise sample, generates a target sample representation by modifying the denoising sample representation based on guidance considering a contact objective set based on a contact prediction region set, generates a partially denoising avatar representation by performing a diffusion process of the avatar generation model from the target sample representation, and generates a target avatar representation corresponding to the interaction label by repeating the generation of the denoising sample representation and the generation of the partially denoising avatar representation, wherein the processor generates a contact prediction region set by inputting the denoising sample representation, the partner avatar representation, and the interaction label into a contact prediction module of the avatar generation model, and based on the sum of the chamfer distances between two vertex sets of regions belonging to the generated contact prediction region set, the denoising sample representation and the partner A device for setting a contact goal that facilitates physical contact between avatar representations, calculating the guidance using a contact goal-based objective function for the partner avatar representation, the interaction label, and the denoising sample representation, and generating the target sample representation based on an element-wise product between the denoising sample representation, the calculated guidance, and a scaling vector for the guidance. Claim 11 In claim 10, the processor comprises the steps of generating a denoising sample representation of the next cycle based on the partner avatar representation and interaction label and the partially denoising avatar representation, generating a target sample representation of the next cycle based on the generated denoising sample representation of the next cycle, generating a partially denoising avatar representation of the next cycle based on the target sample representation of the next cycle, and generating the target avatar representation by repeating the generation of the denoising sample representation of the next cycle and the partially denoising avatar representation of the next cycle, wherein for the last iteration, the target avatar representation is the partially denoising avatar representation of the next cycle. Claim 12 delete Claim 13 delete Claim 14 In claim 10, the processor converts the body of the denoising sample representation and the partner avatar representation into a mesh surface, divides the body of the denoising sample representation and the partner avatar representation into a predetermined number of regions based on vertex indices on the mesh surface of the denoising sample representation and the partner avatar representation, and generates the contact prediction region set by predicting a contact probability map for each region between the denoising sample representation and the partner avatar representation for each divided region. Claim 15 In claim 14, the processor calculates a feature vector for each region by inputting a center location for each region of the denoising sample representation and the partner avatar representation into a feature vector converter, and inputs the calculated feature vector for each region into a contact prediction module to estimate a contact probability map for each region indicating the probability of contact occurring between the region of the denoising sample representation and the region of the partner avatar representation, and generates a set of contact prediction regions including a pair of contact prediction regions corresponding to a probability greater than or equal to a predetermined threshold in the contact probability map for each region. Claim 16 In claim 15, the device, wherein the processor trains the contact prediction module based on a binary cross-entropy objective function that minimizes the loss between the ground truth contact probability map and the estimated contact probability map before estimating the contact probability map for each region. Claim 17 In claim 10, the processor is a device that trains a noise prediction model that performs a denoising process of the avatar generation model before inputting the partner avatar representation and interaction labels into the avatar generation model. Claim 18 In claim 17, the processor is a device that trains the noise prediction model by minimizing the loss between the true value noise and the predicted noise, separated into one or more parts according to the avatar representation characteristics of the training data.

Images