Generating 3D human-object interactions using neural language models

Abstract

A method for generating 3D human-object interactions using a neural language model is disclosed. [Solution] This method involves receiving input including interaction locations and text prompts. A text-to-image generation model is applied to the text prompts. Initial difference parameters of a 3D parametric model representing a person in the interaction image are determined, and multiple 2D views of the 3D environment are extracted. A sequence of prompts is prepared based on the multiple 2D views, and a neural language model is applied to the sequence of prompts. A loss function is initialized based on a set of body parts and a set of locations. The initial difference parameters are iteratively updated based on the loss function to obtain the final difference parameters. The 3D parametric model is rendered in the 3D environment based on the final difference parameters.

Description

[Technical Field]

[0001] The embodiments described herein relate to the generation of 3D human-object interactions using a neural language model (NLM). [Background technology]

[0002] With advancements in artificial intelligence (AI), numerous machine learning models have been developed for a variety of applications. Recently, there has been a significant increase in focus on generating 3D human-object interactions within specific environmental scenes. This area is now central to mainstream text-to-image processing technologies.

[0003] The generation of 3D human-object interactions includes both common and rare interactions. Various techniques have been developed for this purpose. One such technique involves a learning-based method that relies on ground truth data, such as 3D scene and 3D motion data of human interactions. However, a major drawback of this approach is its inability to generate diverse sets of human-scene interactions, particularly low-frequency ones. Furthermore, the high cost of data collection limits the availability of the data needed to learn and generate these rare interactions, thereby constraining the diversity of actions and scenes that can be effectively modeled and reproduced.

[0004] Another technique uses a zero-shot synthesis method to generate a diverse set of human-scene interactions. This method uses natural language descriptions and coarse point locations to guide a stable diffusion restoration process, creating human figures within scene images based on text. These human figures can then be converted into 3D human models. While existing text-to-image models can generate common interactions without relying on extensive datasets, such models may struggle to generate rare interactions due to dataset limitations. This limitation stems from the significant influence of the input images and the objects they contain on the restoration process.

[0005] The subject matter claimed in this disclosure is not limited to embodiments that solve any defects or that operate only in the environment described above. Rather, this background is provided solely to illustrate one exemplary technical field in which some of the embodiments described in this disclosure may be implemented. [Overview of the project] [Means for solving the problem]

[0006] According to one aspect of one embodiment, a system is provided. The system may include memory and a processor. The memory may be configured to store a set of instructions. The processor may be coupled to the memory and execute instructions for performing a set of actions. The processor may receive input including interaction locations in a three-dimensional (3D) environment and text prompts describing interactions between a person and objects. The processor may further generate interaction images by applying a text-to-image generation model to the text prompts and determine initial difference parameters for a 3D parametric model representing a person in the interaction images. The processor may extract multiple two-dimensional (2D) views of the 3D environment from multiple viewpoints around the interaction locations and prepare a sequence of prompts based on the multiple 2D views and the text prompts. The processor may further apply a neural language model (NLM) to the sequence of prompts to identify a set of body parts of a 3D parametric model that must be in contact with a set of locations in the 3D environment to achieve the interaction. The processor may further initialize the loss function based on a set of body parts and a set of positions, and may iteratively update the initial difference parameters based on the loss function to obtain the final difference parameters of the 3D parametric model. Based on the final difference parameters, the processor may finally render the 3D parametric model in a 3D environment.

[0007] The objectives and advantages of the embodiments are realized and achieved, at least, by the elements, features, and combinations specifically indicated in the claims.

[0008] The general description above and the detailed description below are given as examples and are for illustrative purposes only and do not limit the claimed invention. [Brief explanation of the drawing]

[0009] The exemplary embodiments are described and explained with additional specificity and detail by using the accompanying drawings.

[0010] [Figure 1] Figure 2 is a block diagram illustrating an exemplary network environment for 3D human-object interaction generation using a neural language model.

[0011] [Figure 2] Figure 2 is a block diagram illustrating the exemplary system of Figure 1 for 3D human-object interaction generation using a neural language model.

[0012] [Figure 3] Figure 2 is a diagram illustrating an exemplary execution pipeline for determining initial difference parameters of a 3D parametric model.

[0013] [Figure 4] Figure 2 is a diagram illustrating an exemplary execution pipeline for 3D human-object interaction generation using a neural language model.

[0014] [Figure 5A] This is one of the diagrams illustrating an exemplary scenario for human-object interaction generation and different body part selection using a neural language model. [Figure 5B] This is one of the diagrams illustrating an exemplary scenario for human-object interaction generation and different body part selection using a neural language model. [Figure 5C] This is one of the diagrams illustrating an exemplary scenario for human-object interaction generation and different body part selection using a neural language model. [Figure 5D] This is one of the diagrams illustrating an exemplary scenario for human-object interaction generation and different body part selection using a neural language model. [Figure 5E]This is one of the diagrams showing an exemplary scenario for human-object interaction generation and different body part selection using a neural language model

[0015] [Figure 6] This is a diagram showing a flowchart of an exemplary method for 3D human-object interaction generation using a neural language model.

[0016] All are in accordance with at least one embodiment described in the present disclosure. [Detailed Description of the Invention]

[0017] Some embodiments described in the present disclosure may relate to methods and systems for human-object interaction generation and different body part selection using a neural language model. In the present disclosure, an input including an interaction position in a 3D environment and a text prompt describing the interaction between a human and an object may be received. A text-to-image generation model may be applied to the text prompt to generate an interaction image. Initial difference parameters may be determined for a 3D parametric model representing a person in the interaction image. A plurality of 2D views of the 3D environment may be extracted from a plurality of viewpoints around the interaction position. A sequence of prompts may be prepared based on the plurality of 2D views and the text prompt. A neural language model (NLM) may be applied to the sequence of prompts to identify a set of body parts of the 3D parametric model that must contact a set of positions in the 3D environment to achieve the interaction. A loss function may be initialized based on the set of body parts and the set of positions. The initial difference parameters may be updated sequentially and iteratively based on the loss function to obtain final difference parameters of the 3D parametric model. The 3D parametric model may be rendered in the 3D environment based on the final difference parameters.

[0018] The technical field of text-to-3D generation can be improved by configuring a system to generate common human-scene interactions as well as uncommon human-scene interactions described in text prompts, using a text-to-image generation model and a neural language model (NLM) for text prompts. The system can synthesize a 3D human model that interacts with a 3D environment based on specific instructions. The system can further synthesize a 3D parametric model by parameterizing pose and position. The system can generate interaction images using a text-to-image generation model for text prompts to achieve a desired interaction. The system can further estimate the 3D pose of the human in the interaction image. The interaction image may be a two-dimensional (2D) image generated from the text prompt. The system can identify human contact points by analyzing the text prompt and the 3D environment using an NLM. The system can further establish a loss function based on the identified contact points and synthesize 3D human-object interactions in the 3D environment by optimizing initial difference parameters based on the loss function. The system can utilize specific conditions for each human-scene interaction and a custom loss function that takes into account positional and orientation variability.

[0019] Human-scene interaction may involve contact with the 3D environment, particularly the contact of specific body parts with objects within the 3D environment. The contact between specific body parts and objects can vary with each interaction. The system can further utilize neural language models to effectively select relevant body parts and their corresponding contact locations for distance calculations between body parts and objects.

[0020] The disclosed approach can offer several advantages. The generation of realistic 3D human-object interactions (both frequent and infrequent interactions) based on the effective selection of one or more body parts that must come into contact with a set of positions in a 3D environment can be achieved by leveraging the image analysis capabilities of visual language models (VLMs) and the knowledge capabilities of large-scale language models (LLMs) without relying on ground truth data or human supervision. Due to the very little or no dependence on ground truth data, a loss function customized on specific conditions may be optimized, thereby leading to the generation of accurate static human-object interactions across a diverse range of different scenarios. Based on the use of neural language models (such as VLMs and LLMs), even common and infrequent human-object interactions can be generated in a zero-shot manner.

[0021] Furthermore, the proposed technique may include using a neural language model to leverage a customized loss function for each human-object interaction to generate specific conditions for each interaction and refine the generated human-object interactions with respect to posture, position, and orientation. To refine the generated human-object interactions with respect to posture, position, and orientation, a single dataset can be constructed based on a dataset of generative artificial intelligence (AI) models. This dataset may include both common interactions defined in the generative AI model and uncommon interactions not defined in the generative AI model. This makes it possible to effectively evaluate a wide range of static human-object interactions using a single dataset.

[0022] This disclosure provides a framework that leverages neural language models such as VLM and LLM to synthesize accurate and realistic human-object interactions with natural postures that precisely reflect prompts in challenging situations. This approach is optimized to efficiently generate a diverse range of human postures based on the effective selection of different human body parts, without requiring training data, and can generate unusual interactions. Examples include stepping on a garbage bag, doing a handstand on the saddle of a standing cow, kicking a chair, and doing a handstand on a chair. Furthermore, this approach can be used across a variety of applications, including augmented reality (AR), virtual reality (VR), digital twin technology, computer games, and data augmentation applications.

[0023] Embodiments of this disclosure will be described with reference to the accompanying drawings.

[0024] Figure 1 is a diagram representing an exemplary network environment related to the generation of 3D human-object interactions using a neural language model, configured according to at least one embodiment described herein. Referring to Figure 1, the network environment 100 is shown. The network environment 100 may include a system 102, a text-to-image generation model 116, a neural language model 122, and an optimizer 124. A database (not shown) may be provided to store information such as inputs 104, including interaction locations 106A in the 3D environment 106 and text prompts 112 describing the interaction between a person 110 and an object 108. Figure 1 further shows an interaction image 114, a 3D parametric model 118, a series of prompts 120, and a rendering 126 associated with the interaction between the person 110 and the object 108. The system 102 may communicate with one or more networks (such as a communication network 128).

[0025] System 102 may include appropriate logic, circuitry, interfaces, and / or code that can be configured to receive inputs 104 necessary for interaction between person 110 and object 108. In one embodiment, system 102 may control a display device (e.g., display device 206A in Figure 2). Display device 206A may be communicatively coupled to system 102 or may be a standalone device configured to render inputs 104 including interaction position 106A and text prompt 112. The text prompt 112 may describe the interaction between person 110 and object 108. Examples of system 102 may include, but are not limited to, computing devices, smartphones, mainframe machines, servers, consumer electronic (CE) devices, computer workstations, and / or devices with graphing capabilities (such as devices with a set of graphics processor units (GPUs)).

[0026] As used herein, the term “input 104” may be data received by the system 102, including an interaction location 106A in a 3D environment 106 and a text prompt 112 describing the interaction between person 110 and object 108. Input 104 may also include a 3D mesh, point cloud, or rendering of the 3D environment 106 specifying the 3D coordinates of the interaction location 106A.

[0027] As used herein, the term “text prompt 112” may be natural language text describing a task to be performed by the text-to-image generation model 116. The natural language text may also be in the form of structured instructions that can be interpreted and understood by the text-to-image generation model 116. The task may relate to the interaction between a person (e.g., person 110) and one or more objects (e.g., object 108). For example, a typical prompt may be a description of a desired output, such as “a person falling down the stairs,” “a person standing and stepping on a garbage bag,” or “a person sitting on the saddle of a standing cow.”

[0028] The text-to-image generation model 116 may be a machine learning model for analyzing a text prompt 112 and generating an interaction image 114 based on the analysis of the text prompt 112. The text-to-image generation model 116 may be trained on a large dataset of text-to-image pairs to interpret human language or other types of complex data. In certain cases, the dataset may be specific to human interactions with various types of objects.

[0029] In one embodiment, the text-to-image generation model 116 may be a type of artificial intelligence system (also known as an artificial deep neural network) configured to simultaneously process and understand multiple types of data modalities, such as text, images, audio, 3D data, and video. For example, the text-to-image generation model 116 may extend the capabilities of conventional large-scale language models (LLMs) by integrating various forms of data, enabling a more comprehensive understanding and generation of information across different media types.

[0030] In some embodiments, the text-to-image generation model 116 may be a large-scale language model, such as a transformer-based decoder-only model, an encoder-decoder model (using transformers), or a model using a neural network other than transformers. In one or more embodiments, the text-to-image generation model 116 may include multiple encoders specialized for handling different modalities of data (such as images and text). The text-to-image generation model 116 may include a fusion mechanism for integrating the outputs from the various encoders. Different data types (such as text and images in the text prompt 112) can be combined into a unified representation using techniques such as a cross-attention mechanism or a multimodal transformer. In some embodiments, the text-to-image generation model 116 may include decoders for generating outputs in various modalities. For example, a text decoder may generate a text response, and an image decoder may create visual content. During training, the text-to-image generation model 116 may use large datasets containing paired data from multiple modalities (e.g., interaction images, interaction-based image-caption pairs, videos with subtitles showing human-object interactions). Training may include supervised learning, reinforcement learning with human feedback (RLHF), and techniques such as fine-tuning to ensure the text-to-image generation model 116 performs well across different tasks. For applications such as image generation from text or text understanding, the text-to-image generation model 116 may include specialized heads that can be fine-tuned for specific tasks.

[0031] In some other embodiments, the text-to-image generation model 116 may use a diffusion process, which may start with a random noise image and iteratively process the image sequentially based on the text description to create a detail image. The architecture may include a text encoder (such as a transformer) for converting the text into a meaningful representation, a noise generator for creating an initial noise image, and a diffusion process that gradually reduces noise and adds detail using a denoising network (often a U-Net with an attention mechanism). Finally, an image decoder (typically a CNN) can convert the refined latent representation into a detail image. This guided, stepwise conversion ensures that the generated image closely matches the input text and can produce high-quality, contextually accurate results.

[0032] In these or other embodiments, the text-to-image generation model 116 may include electronic data that can be implemented, for example, as a software component of an application executable on system 102. The text-to-image generation model 116 may rely on libraries, external scripts, or other logic / instructions for execution by a processing device. The text-to-image generation model 116 may include code and routines configured to enable a computing device, such as an electronic device, to perform one or more actions, such as generating an interactive image 114, based on the application of the text-to-image generation model 116 to a text prompt 112. Additionally or alternatively, the text-to-image generation model 116 may be implemented using hardware including a processor, a microprocessor (e.g., one or more actions, or one or more actions, or controls their execution), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). Alternatively, in some embodiments, the text-to-image generation model 116 may be implemented using a combination of hardware and software.

[0033] As used herein, the term “3D Parametric Model 118” may refer to a computational model used to represent the human body with a high degree of realism. The model may use parameters to adjust the shape and posture of the body, and to include detailed anatomical structures of the face, hands, and torso, and to deform smoothly with movement. For example, the 3D Parametric Model 118 may be SMPL-X (Skinned Multi-Person Linear Model-extended) or a variation thereof.

[0034] In one exemplary embodiment, the 3D parametric model 118 may represent a person 110 in an interaction image 114. For the generation of 3D human-object interactions, the 3D parametric model 118 may include a set of body parts that must come into contact with a set of locations in the 3D environment 106 in order to achieve a desired interaction (specified in the text prompt 112).

[0035] As used herein, the term “prompt sequence 120” can be natural language text prepared based on multiple 2D views and text prompts 112. Each natural language text can be in the form of structured instructions that can be interpreted and understood by the neural language model 122. For example, a typical prompt sequence may be a description of a desired output. The prompt sequence 120 can be supplied to the neural language model 122 for the identification of a set of body parts of a 3D parametric model 118 that must be contacted by a set of positions in a 3D environment 106 in order to achieve interaction between an object 108 and a person 110.

[0036] The neural language model 122 may be a machine learning model configured to analyze each of the prompts from the sequence of prompts 120 and output a structured response, which may be used to identify a set of body parts of a 3D parametric model 118 that must contact a set of locations in a 3D environment to achieve interaction between an object 108 (specified in the text prompt 112) and a person 110. For example, each prompt in the set of prompts 120 may contain a sentence containing an instruction to identify at least one body part of the 3D parametric model 118. The neural language model 122 can generate a list having at least one body part of the 3D parametric model 118 that must contact at least one location in a 3D environment. The neural language model 122 may be trained or fine-tuned on a large dataset of question-answer pairs to interpret human language or other types of complex data. In certain cases, the dataset may be specific to human-object interactions in different 3D environments.

[0037] A neural language model 122 can be a type of artificial intelligence system (also called an artificial deep neural network) configured to process and understand different types of data modalities from prompts, including but not limited to text, images, audio, 3D data, 2D images, or video. By integrating various forms of data, a neural language model 122 can extend the capabilities of traditional large-scale language models (LLMs), enabling a more comprehensive understanding and generation of information across different media types. A neural language model 122 is also called a multimodal language model (MLM) or a Vision Language Model (VLM).

[0038] In some embodiments, the neural language model 122 may be a large-scale language model such as a transformer-based decoder-only model, an encoder-decoder model (using transformers), or a model that uses a neural network other than transformers. In one or more embodiments, the neural language model 122 may include multiple encoders specialized for processing different modalities of data (such as images and text). The neural language model 122 may include a fusion mechanism for integrating the outputs from the various encoders.

[0039] As an artificial deep neural network, the neural language model 122 may also be called a computational network, or a system of artificial neurons within a neural network arranged in multiple layers may be called nodes. The multiple layers of the neural network may include an input layer, one or more hidden layers, and an output layer. Each of the multiple layers may include one or more nodes (or artificial neurons). The outputs of all nodes in the input layer may be coupled to at least one node in one or more hidden layers. Similarly, the input of each hidden layer may be coupled to the output of at least one node in another layer of the neural network. The output of each hidden layer may be coupled to the input of at least one node in another layer of the neural network. The nodes in the final layer may receive input from at least one hidden layer and output a result. The number of layers and the number of nodes in each layer may be determined from the hyperparameters of the neural network. Such hyperparameters may be set before or after training the neural network on the training dataset.

[0040] Each node in the neural language model 122 may correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) with a set of parameters that can be tuned during network training. The set of parameters may include, for example, weight parameters, regularization parameters, etc. Each node may use these mathematical functions to compute an output based on one or more inputs from nodes in other layers of the neural network (e.g., one or more nodes in the previous layer). All or some nodes in the neural network may correspond to the same or different mathematical functions.

[0041] In training the neural language model 122, one or more parameters of each node in the neural network may be updated based on whether the output of the final layer for a given input (from the training dataset) matches the correct result, based on the loss function for the neural network. The above process may be repeated for the same or different inputs until the minimum value of the loss function is achieved and the training error is minimized. Several training methods are known in the art, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, and metaheuristics.

[0042] The optimizer 124 may be a computer program, a specialized circuit, or a combination thereof for optimizing the difference parameters of the 3D parametric model 118 to fit a set of body parts of the 3D parametric model 118 to a set of positions. The optimizer 124 may be configured to iteratively update the initial difference parameters of the 3D parametric model 118 based on the optimization of the loss function to obtain the final difference parameters of the 3D parametric model 118.

[0043] The communication network 128 may include a variety of communication media through which the system 102 can communicate with other electronic devices, such as a server (not shown). Examples of the communication network 128 may include, but are not limited to, the internet, cloud networks, wireless fidelity (Wi-Fi) networks, personal area networks (PANs), local area networks (LANs), cellular networks (e.g., long-term evolution (or 4G) cellular networks or 5G cellular networks), satellite networks (e.g., networks of low-Earth orbit satellites), and / or metropolitan area networks (MANs). Various devices within the exemplary network environment 100 may be configured to connect to the communication network 128 according to a variety of wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, Multihop Communication, Wireless Access Point (AP), Device-to-Device Communication, Cellular Communication Protocol, and / or Bluetooth® (BT) Communication Protocol, or any combination thereof.

[0044] In operation, system 102 may be configured to receive input 106 which includes an interaction location 104A in a 3D environment and a text prompt 112 which describes an interaction between a person (e.g., person 110) and one or more objects (e.g., object 108). For example, the text prompt 112 may be a command such as "kicking a car". In some examples, the text prompt 112 may be prefixed with "a person" and suffixed with ", full body", so the command is received as "a person kicking a car, full body". This may be done to ensure that the person's entire body is depicted in the interaction image 114. The receipt of input including the interaction location 106A and the text prompt 112 is further illustrated, for example, in Figures 3, 4, and 5A-5E.

[0045] System 102 may be further configured to generate interaction images 114 by applying a text-to-image generation model 116 to a text prompt 112. Interaction images 114 may be generated by interpreting the text prompt 112 and generating a corresponding visual representation. For example, an interaction image 114 generated from text describing a person's interaction with an object would visually depict a person 110 interacting with the specified object in the described manner. If the text prompt 112 says "a person sitting on a bench in a park, full body," the interaction image 114 would show a person 110 sitting on a park bench. It should be noted that objects in the generated interaction image 114 may differ in appearance or position from objects in the 3D environment 106. The accuracy of appearance or position may depend on the extent to which details of the 3D environment are incorporated into the text prompt 112. The generation of interaction images 114 is further illustrated, for example, in Figure 3.

[0046] System 102 may be further configured to determine initial difference parameters for a 3D parametric model 118 representing a person 110 in the interaction image 114. These initial difference parameters may include orientation, position, and posture parameters for the 3D parametric model 118. The determination of the initial difference parameters for the 3D parametric model 118 is further described, for example, in Figures 3 and 4.

[0047] System 102 may be further configured to extract multiple 2D views of the 3D environment 106 from multiple viewpoints around the interaction position 106A. The extraction of 2D views of the 3D environment 106 is further illustrated, for example, in Figure 5A.

[0048] System 102 may be configured to prepare a sequence of prompts 120 based on a plurality of 2D views and text prompts 112. The sequence of prompts 120 may be natural language instructions, each describing a task to be performed by a neural language model 122. For example, the sequence of prompts 120 may be processed by the neural language model 122 to determine a particular interaction in a 3D environment 106. First, a first prompt may be prepared, containing a basic instruction with a text prompt describing the interaction. The neural language model 122 may process this prompt and output a response indicating whether the interaction requires contact between object 108 and person 110. Subsequently, a second prompt may be prepared, containing a plurality of 2D views and a second basic instruction associated with these views. Based on the second prompt, the neural language model 122 may output a second response specifying the name of a major scene area shown in the 2D views. Then, a third prompt may be prepared, incorporating a third basic instruction and the second response specifying the major scene area. The neural language model 122 can process this third prompt and output a third response that identifies the main body parts that must be contacted at the main contact locations within the main scene area in order to achieve the interaction described in the text prompt 112. Subsequently, a fourth prompt may be prepared, which includes a fourth basic command having an interaction image 114, the text prompt 112, and a list of body parts. The neural language model 122 can process this fourth prompt and output a fourth response that specifies at least one body part from the list that, unlike the main body parts, must be contacted with object 108 in the interaction image 114 in order to achieve the interaction. Finally, a fifth prompt may be created, which includes an interaction image 114, the text prompt 112, a fifth basic command associated with the interaction image 114 and the text prompt 112, and a list of body parts.The neural language model 122 can process this fifth prompt and output a fifth response specifying the body part that must be in contact with the ground in the interaction image 114 in order to achieve the interaction. The preparation of the sequence of prompts 120 based on multiple 2D views and text prompts 112 is further described, for example, in Figures 5B, 5D, and 5E.

[0049] In one embodiment, the system 102 may be configured to apply a neural language model 122 to a sequence of prompts 120 to identify a set of body parts of a 3D parametric model 118 that must come into contact with a set of locations in a 3D environment 106 in order to achieve an interaction described in a text prompt 112. In one embodiment, the neural language model 122 may be applied to a prompt to output a response specifying whether the interaction described in the text prompt 112 requires contact between an object 108 and a person 110. For example, referring to Figure 1, the text prompt 112 may be prepared to include the command “sit in the chair”. The neural language model 122 can then be applied to the text prompt 112 to output a response specifying whether the interaction described in the text prompt requires contact between the chair and the person 110. For example, the response may specify the contact required between the chair and the person 110. The person 110 may be sitting in the chair, with the buttocks and upper thighs being the body parts that come into contact with the chair. The application of the neural language model 122 to the prompt sequence is further illustrated, for example, in Figures 5C to 5E.

[0050] System 102 may be further configured to initialize a loss function based on a set of body parts and a set of positions. The loss function may include, but is not limited to, a composite distance between the vertices of the set of body parts and the vertices of the set of positions, an intrusion loss specifying the signed distance between the body vertices of the 3D parametric model 118 and the scene vertices of the 3D environment, and a regularization loss measuring the deviation of the pose parameters of the 3D parametric model 118 from the initialized pose parameters of the initial difference parameters. The initialization of the loss function based on the set of body parts and the set of positions is further illustrated, for example, in Figures 3 and 4.

[0051] System 102 may be configured to use an optimizer 124 to sequentially and iteratively update initial difference parameters based on the loss function to obtain the final difference parameters of the 3D parametric model 118. The initial difference parameters may be sequentially and iteratively updated to minimize the value of the loss function until the value of the loss function satisfies the convergence condition. The sequential and iterative updating of the initial difference parameters is further illustrated, for example, in Figures 4 and 5C.

[0052] System 102 may be configured to render a 3D parametric model 118 in a 3D environment 106 based on final difference parameters. The 3D parametric model may be a model capable of representing a person 110 in an interaction image 114. Furthermore, the 3D parametric model 118 may be rendered to represent the interaction shown in the interaction image 114 and described in the text prompt 112 (a set of body parts coming into contact with a set of positions in the 3D environment 106 to achieve the interaction). Rendering of the 3D parametric model 118 in a 3D environment based on final difference parameters is further illustrated, for example, in Figures 4, 5A, 5D, and 5E.

[0053] Without departing from the scope of this disclosure, modifications, additions, or omissions may be made to Figure 1. For example, the network environment 100 may include more or fewer elements than those illustrated and described in this disclosure. For example, in some embodiments, the network environment 100 may include system 102 but not include optimizer 124. In addition, in some embodiments, the functionality of optimizer 124 may be incorporated into system 102 without departing from the scope of this disclosure.

[0054] Figure 2 is a block diagram of the exemplary system of Figure 1 for 3D human-object interaction generation using a neural language model, configured according to at least one embodiment described herein. Figure 2 is described in relation to the elements of Figure 1. Referring to Figure 2, a block diagram 200 of system 102 is shown. System 102 may include a processor 202, memory 204, a text-to-image generation model 116, a neural language model 122, an optimizer 124, an input / output (IO) device 206, and a network interface 208. The IO device 206 may include a display device 206A. The memory 204 may contain a large dataset (e.g., question-answer pairs or other types of complex data for interpreting human language).

[0055] The processor 202 may include appropriate logic, circuitry, interfaces, and / or code that can be configured to execute program instructions associated with different operations performed by the system 102. Operations may include, but are not limited to, input reception, interaction image generation, initial difference parameter determination, multiple 2D view extraction, preparation of a sequence of prompts, neural language model application, loss function initialization, sequential iterative updating of initial difference parameters, and 3D parametric model rendering. The processor 202 may also include any appropriate dedicated or general-purpose computer, computing entity, or processing device, including various computer hardware or software modules, which may be configured to execute instructions stored in any applicable computer-readable storage medium. For example, the processor 202 may include a microprocessor, microcontroller, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or any other digital or analog circuitry configured to interpret and / or execute program instructions and / or process data.

[0056] Although shown as a single processor in Figure 2, processor 202 may include any number of processors configured to individually or collectively perform or direct any number of operations of system 102, as described in this disclosure. Furthermore, one or more of the processors may reside on one or more systems, such as different servers.

[0057] In some embodiments, the processor 202 may be configured to interpret and / or execute program instructions and / or process data stored in memory 204. In some other embodiments, the processor 202 may fetch program instructions associated with the text image generation model 116 and the neural language model 122, and load the program instructions into memory 204. After the program instructions are loaded into memory 204, the processor 202 may execute the program instructions. Some examples of the processor 202 may be a graphical processing unit (GPU), a central processing unit (CPU), a reduced instruction set computer (RISC) processor, an application-specific integrated circuit (ASIC) processor, a composite instruction set computer (CISC) processor, a coprocessor, and / or a combination thereof.

[0058] Memory 204 may include appropriate logic, circuits, interfaces, and / or code that can be configured to store program instructions executable by the processor 202. In certain embodiments, memory 204 may be configured to store information such as, but is not limited to, interaction positions 106A in the 3D environment 106, text prompts 112, interaction images 114, a 3D parametric model 118, initial difference parameters of the 3D parametric model 118, final difference parameters of the 3D parametric model 118, multiple 2D views, multiple viewpoints, a sequence of prompts 120, and a set of positions. Memory 204 may further store a set of values ​​associated with an initialized loss function. The loss function may include, but is not limited to, a composite distance between vertices of a set of body parts and vertices of a set of positions, an intrusion loss specifying a signed distance between body vertices of the 3D parametric model 118 and scene vertices of the 3D environment, and a regularization loss measuring the deviation in the pose parameters of the 3D parametric model 118 from the initialized pose parameters of the initial difference parameters.

[0059] Memory 204 may include computer-readable storage media for carrying or storing computer-executable instructions or data structures. Such computer-readable storage media may include any available media that can be accessed by a general-purpose or dedicated computer, such as a processor 202. Such computer-readable storage media may include, but are not limited to, tangible or non-temporary computer-readable storage media, including CPU caches, hard disk drives (HDDs), solid-state drives (SSDs), random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM), secure digital (SD) cards, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or flash memory devices (e.g., solid-state memory devices). Computer-readable storage may also include any other storage media that can be used to carry or store specific program code in the form of computer-executable instructions or data structures, and that can be accessed by a general-purpose or dedicated computer. Combinations of the above may also be included within the scope of computer-readable storage media. Computer executable instructions may include, for example, instructions and data configured to cause a processor 202 to perform certain operations or sets of operations associated with system 102.

[0060] The I / O device 206 may include appropriate logic, circuitry, interfaces, and / or code that can be configured to receive a large dataset of inputs 104 containing question-answer pairs for interpreting human language or other types of complex data. For example, input 104 may include an interaction position 106A in a 3D environment and a text prompt 112 describing the interaction between a person 110 and an object 108. The I / O device 206 may be further configured to provide an output in response to input 104. For example, the output may correspond to a generated interaction image 114 representing a person 110 in contact with an object 108. The I / O device 206 may include a variety of input and output devices that can be configured to communicate with other components such as a processor 202 and a network interface 208. Examples of input devices may include, but are not limited to, a touchscreen, keyboard, mouse, joystick, and / or microphone. Examples of output devices may include, but are not limited to, a display device 206A and a speaker. The I / O device 206 may be located within system 102 or outside of system 102.

[0061] The display device 206A may include logic, circuitry, and interfaces configured to display interaction positions 106A in a 3D environment, text prompts 112, interaction images 114, a 3D parametric model 118, multiple 2D views, multiple viewpoints, a sequence of prompts 120, a set of identified body parts of the 3D parametric model 118, and a set of positions. The display device 206A may be a touchscreen that allows a user to provide user input through the display device 206A. The touchscreen may be at least one of a resistive touchscreen, a capacitive touchscreen, or a thermal touchscreen. The display device 206A may be implemented through several known technologies, including but not limited to liquid crystal display (LCD) displays, light-emitting diode (LED) displays, plasma displays, or organic LED (OLED) display technologies, or other display devices. According to one embodiment, the display device 206A may refer to a display screen for a head-mounted device (HMD), smart glasses device, see-through display, projection-based display, electrochromic display, or transparent display.

[0062] The network interface 208 may include appropriate logic, circuitry, and interfaces that can be configured to facilitate communication between the processor 202 (i.e., system 102) and a server (not shown) over the communication network 128. The network interface 208 may be implemented by using various known techniques to support wired or wireless communication of system 102 with the communication network 128. The network interface 208 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identification information module (SIM) card, or a local buffer circuit.

[0063] The network interface 208 may be configured to communicate wirelessly with networks such as the Internet, an intranet, or wireless networks such as cellular networks, wireless local area networks (LANs), and metropolitan area networks (MANs). Wireless communication may be configured to use one or more of several communication standards, protocols, and technologies, for example, the Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long-Term Evolution (LTE), Fifth Generation (5G) New Radio (NR), Global Mobile Communication System (GSM®), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Worldwide Interoperability for Microwave Access (Wi-MAX), Protocol for Email, Instant Messaging, and Short Message Service (SMS).

[0064] In certain embodiments, system 102 may be divided into a front-end subsystem and a back-end subsystem. The front-end subsystem may be configured independently to receive requests / commands from one or more of the following: user devices, third-party servers, web servers, client machines, and back-end subsystems. These requests may be communicated back to the back-end subsystems, which may be configured to act in response to these requests. For example, if system 102 communicates with multiple servers, some of the servers may be front-end servers configured to relay requests / commands to the remaining servers associated with the back-end subsystems.

[0065] Without departing from the scope of this disclosure, the exemplary system 102 may be modified, added to, or omitted. For example, in some embodiments, the exemplary system 102 may include any number of other components that are not expressly illustrated or described for the sake of brevity.

[0066] Figure 3 shows an exemplary execution pipeline for determining initial difference parameters of a 3D parametric model according to one embodiment of the present disclosure. Figure 3 is described in relation to the elements of Figures 1 and 2. Referring to Figure 3, an exemplary execution pipeline 300 is shown. The execution pipeline 300 may include a sequence of actions that may be performed by the processor 202 of the system 102 in Figure 1 for determining initial difference parameters of a 3D parametric model (for example, the 3D mesh 402B in Figure 4).

[0067] The execution pipeline 300 includes an action 302 for receiving a text prompt, an action 304 for applying the text imaging generation model 116 to the text prompt 310, an action 306 for generating an interaction image, and an action 308 for estimating initial 3D body pose parameters. Although only one text prompt 310 is shown in Figure 3, the scope of this disclosure is not limited thereto. Without departing from the scope of this disclosure, there may be two or more text prompts to which the text imaging generation model 116 is applied.

[0068] In 302, an action may be performed to receive a text prompt 310. Processor 202 may be configured to receive the text prompt 310. In one or more embodiments, processor 202 may be configured to receive an input 104 which includes an interaction location 402C in a 3D environment 502A (as shown in Figure 5A) and a text prompt 310 describing an interaction between person 314 and object 316. Interaction location 402C may refer to a specified location where an interaction occurs between object 316 and person 314 (rendered in 3D via a 3D parametric model (e.g., 3D mesh 402B)). Furthermore, interaction location 402C may also be a rough point location of a desired interaction in the 3D environment 502A and may be specified manually. For example, interaction location 402C may be a primary contact location 412A, an auxiliary contact location 412B, or a ground contact location (not shown) in the 3D environment 502A. The primary contact position 412A, the auxiliary contact position 412B, and the ground contact position are described in more detail in Figures 5D and 5E.

[0069] The action for receiving the text prompt 310 is crucial for generating the initial pose for the 3D parametric model (e.g., the 3D mesh 402B). For example, the text prompt 310 may contain a command such as "sitting on the saddle of a standing cow." This text prompt 310 can be supplied to the text-to-image generation model 116 to generate the interaction image 312. The receipt of the text prompt 310, which includes a description of the interaction between person 314 and object 316, is further illustrated, for example, in Figure 5A.

[0070] In 304, the text image generation model 116 may be applied to the text prompt 310. The processor 202 may be configured to apply the text image generation model 116 to the text prompt 310. The text image generation model 116 can correspond to at least one of the following: MLLM (multimodal large language model), VLM (vision language model), GAN (Generative Adversarial Network), or diffusion model. The application of the text image generation model 116 to the text prompt 310 is further illustrated, for example, in Figure 5A.

[0071] In 306, an operation for generating interaction images may be performed. In one embodiment, the processor 202 may be configured to generate an interaction image 312 based on the application of a text image generation model 116 to a text prompt 310. In at least one embodiment, the processor 202 may be further configured to utilize the text image generation model 116 to generate a set of interaction images based on the text prompt 310 or a set of text prompts (i.e., variations generated from the text prompt 310), thereby allowing the generation of a diverse range of interactions that match the description of the text prompt 310. The generated interaction image 312 may be used to estimate the 3D pose of a person 314 in the interaction image 312. The generation of the interaction image 312 is further illustrated, for example, in Figures 5B and 5E.

[0072] In 308, an operation for estimating initial difference parameters may be performed. The processor 202 may be configured to estimate the initial difference parameters of the person 314 in the interaction image 312. The initial difference parameters may represent the 3D pose of the person 314 in the interaction image 312. For example, the initial difference parameters may include orientation parameters, position parameters, and pose parameters, represented by "R", "t", and "θ", respectively. The vertices associated with the 3D body pose of the person 314 are,

number

[0073] Figure 4 shows an exemplary execution pipeline for generating 3D human-object interactions using a neural language model according to one embodiment of the present disclosure. Figure 4 is described in relation to elements from Figures 1, 2, and 3. Referring to Figure 4, an exemplary processing pipeline 400 for generating 3D human-object interactions using the knowledge capabilities of a neural language model 122 is shown. The processing pipeline 400 may include an input 402 having a text prompt 402A, a 3D mesh 402B, and an interaction position 402C in a 3D environment 502A (shown in Figure 5), and a set of actions including an action 404 for generating initial 3D pose parameters, an action 406 for body part selection, an action 408 for loss function customization, and an action 410 for 3D human-object interaction synthesis.

[0074] A text prompt 402A may describe the interaction between a person 314 and an object 316 in a 3D environment 502A. For example, a text prompt 402A may be an instruction prepared to include the description "person sitting on the saddle of a standing cow". A 3D mesh 402B may be a representation of a 3D parametric model that includes reference points on the X, Y, and Z axes to define a shape having height, width, and depth. Vertices associated with the 3D mesh 402B may be indicated by "V".

[0075] In one or more embodiments, the processor 202 may be configured to receive an input including an interaction location 402C, a text prompt 402A, and a 3D mesh 402B. In these or other embodiments, a database (not shown) may store a dataset having information such as an input 402 including the interaction location 402C and a plurality of text prompts (including text prompt 402A) describing the interaction between the person 314 and the object 316. For example, the database may store a plurality of text prompts, each describing an interaction requesting contact between the object 316 and the person 314 in a particular manner (e.g., sitting, standing on one's head, pushing, etc.).

[0076] The processor 202 may be configured to apply the text image generation model 116 to the text prompt 402A to generate the interaction image 312.

[0077] In 404, operations for generating initial differential parameters may be performed. In one or more embodiments, the processor 202 may be configured to determine initial differential parameters for a 3D mesh 402B (i.e., a 3D parametric model) representing a person 314 in the interaction image 312. Techniques for determining initial differential parameters from the interaction image 312 are well known to those skilled in the art, and therefore, additional details relating to the determination of initial differential parameters are omitted from this disclosure for the sake of brevity.

[0078] In some examples, the processor 202 may be configured to extract multiple 2D views 504A…504C (shown in Figure 5A) of the 3D environment 502A from multiple viewpoints around the interaction location 402C. The processor 202 may be further configured to prepare a sequence of prompts based on the multiple 2D views 504A…504C and the text prompt 402A. The sequence of prompts may be multiple natural language instructions, each describing a task to be performed by the neural language model 122. Furthermore, the neural language model 122 may be applied to the sequence of prompts to identify a set of body parts 404A~404B of the 3D mesh 402B (i.e., a 3D parametric model) for selection. The set of body parts 404A~404B must come into contact with a set of locations 412A~412B (as shown in Figure 5A) in the 3D environment 502A in order to achieve the interaction described in the text prompt 402A.

[0079] In 406, an action for body part selection may be performed. The processor 202 may be configured to select a set of body parts 404A-404B of a 3D mesh 402B (i.e., a 3D parametric model) that are in contact with a set of positions 412A-412B in the 3D environment 502A. The body part selection action may include at least one of primary contact body part selection 406A, auxiliary contact body part selection 406B, and ground contact body part selection 406C. The body part selection action may be divided into the three categories described above in order to precisely specify the required interaction conditions and reduce errors caused by models such as text image generation and neural language models.

[0080] In the first embodiment, an operation for selecting a primary contact body part 406A may be performed. A primary contact body part may be selected to calculate the distance to a specific location in the 3D environment 502A. In one embodiment, the processor 202 may be configured to identify a primary scene area from the 3D environment 502A by selecting k nearest neighbor vertices from the scene mesh of the 3D environment 502A. The k nearest neighbor vertices may be selected based on the interaction location 402C. Furthermore, multiple 2D views 504A~504C of the 3D environment 502A may be extracted to include the primary scene area from multiple viewpoints. For example, the primary scene area may be determined from a specified coarse point location "L" using equation (1) defined as follows:

number

[0081] In some cases, relying on the primary contact body part may result in unnatural positioning and orientation because the segmentation of the part is not sufficiently detailed. To address this, auxiliary contact body parts may be selected to calculate the distance to the 3D environment 502A, ensuring a more natural appearance with respect to human-object interaction. In a second embodiment, an operation for auxiliary contact body part selection 406B may be performed. The processor 202 may be configured to identify auxiliary contact position 412B from a set of positions 412A-412B and auxiliary body part 404B from a set of body parts 404A-404B, which must be in contact with the 3D asset representing object 316 in the 3D environment 502A. The auxiliary contact position 412B may be different from the primary contact position 412A. Similarly, the auxiliary body part 404B may be different from the primary body part 404A and must remain in contact with the 3D asset representing object 316 in the 3D environment 502A in order to achieve the interaction described in the text prompt 310.

[0082] In some examples, to prevent the human model from floating in certain interactions, ground-contacting body parts may be selected to calculate the distance to the ground. In a third embodiment, an action for ground-contacting body part selection 406C may be performed. The processor 202 may be configured to identify ground-contacting positions from a set of positions 412A-412B and to identify ground-contacting body parts (not shown) from a set of body parts 404A-404B that must contact the ground surface in the 3D environment 502A in order to achieve the interaction. The ground-contacting positions may be the positions where the ground-contacting body parts must contact the ground-contacting mesh representing the ground surface in the 3D environment 502A. For example, if the interaction requires a person 314 to push an object 316, the ground-contacting body parts may be identified as the feet of the person 314.

[0083] In 408, actions may be performed to customize the loss function. The loss function may be customized based on interaction requirements to ensure realistic and contextually accurate human-scene interactions. Processor 202 may be configured to initialize the loss function based on a set of body parts 404A-404B and a set of positions 412A-412B. The loss function may include, but is not limited to, a composite distance between the vertices of the set of body parts 404A-404B and the vertices of the set of positions 412A-412B, an intrusion loss specifying a signed distance between the body vertices of the 3D parametric model (e.g., 3D mesh 402B) and the scene vertices of the 3D environment 502A, and a regularization loss that measures the deviation of the pose parameters of the 3D parametric model (e.g., 3D mesh 402B) from the initialized pose parameters of the initial difference parameters.

[0084] The processor 202 may be configured to sequentially and iteratively update initial difference parameters based on a loss function to obtain the final difference parameters of a 3D parametric model (e.g., 3D mesh 402B). For example, the loss function may be minimized using sequentially iterative optimization techniques to adjust orientation, position, and attitude parameters represented by "R", "t", and "θ". Furthermore, the initial difference parameters may be sequentially and iteratively updated while minimizing the value of the loss function until the value of the loss function satisfies a convergence condition. For example, the initial difference parameters represented by "R", "t", and "θ" may be adjusted through appropriate sequentially iterative optimization techniques to minimize a customized loss function. This process may be repeated N times, and the final different parameters may be obtained as the final output to ensure the generation of realistic and diverse human-scene interactions. In one example, the values ​​of the initial difference parameters "R", "t", and "θ" may be sequentially and iteratively updated to minimize the value of the loss function until the value of the loss function falls below a loss threshold (e.g., 0.05).

[0085] At 410, an operation for 3D human-object interaction synthesis may be performed. Based on the final difference parameters, the processor 202 can render a 3D mesh 402B (i.e., a 3D parametric model) of person 314 in the 3D environment 502A. As shown in the figure, for example, the view 410A of the 3D environment 502A is rendered to include a 3D mesh 402B (i.e., a 3D parametric model) of person 314 sitting in the saddle area of ​​a cow (i.e., a 3D asset representing object 316).

[0086] Figures 5A, 5B, 5C, 5D, and 5E are diagrams illustrating a collective exemplary scenario for human-object interaction generation and selection of different body parts using a neural language model, according to one embodiment of the present disclosure. Figures 5A, 5B, 5C, 5D, and 5E are described in relation to the elements of Figures 1, 2, 3, and 4. Referring to Figures 5A-5E, an exemplary scenario 500 is shown. The exemplary scenario 500 shows a 3D environment 502A including an interaction location 402C. A text prompt 502B is further shown to describe the interaction between a person 314 and an object 316. For example, the text prompt 502B may include a natural language instruction such as "sit on the saddle of a standing cow." This text prompt 502B may be fed to a text-to-image generation model 116 to generate an interaction image 312. Alternatively, the text prompt 502B may be modified to "full body, seated on a standing cow saddle" to ensure that the generated interaction image 312 includes the full body of person 314. Based on the generated interaction image 312, initial difference parameters of a 3D parametric model (e.g., 3D mesh 402B) representing person 314 in the interaction image 312 may be determined.

[0087] Figure 5A further shows multiple 2D views 504A…504C of the 3D environment 502A around the interaction position 402C. The multiple 2D views 504A…504C may be taken from different viewpoints (i.e., viewing angles) within the 3D environment 502A, with an object 316, such as a standing cow, within the 3D environment 502A being the interaction position from different camera angles.

[0088] A sequence of prompts for body part selection 506 can be prepared using multiple 2D views 504A…504C, interaction images 312, and text prompts 502B. A neural language model 122 may be applied to the sequence of prompts to identify a set of body parts 404A–404B of a 3D mesh 402B (representing a 3D parametric model) that must come into contact with a set of positions 412A–412B in the 3D environment 502A to achieve interaction between object 316 and person 314. The set of body parts 404A–404B may include, but are not limited to, the primary body part 404A, the secondary body part 404B, and the ground-contacting body part (not shown).

[0089] For example, the neural language model 122 may be applied to a sequence of prompts to identify a primary body part 404A that must make contact with a primary contact location 412A of a set of locations 412A-412B within the primary scene area of ​​the 3D environment 502A. Furthermore, the neural language model 122 may be applied to a sequence of prompts to identify an auxiliary body part 404B that makes contact with a 3D asset at an auxiliary contact location 412B. The 3D asset may represent an object 316 in the 3D environment 502A. Furthermore, the neural language model 122 may be applied to a sequence of prompts to identify a ground-contacting body part that must make contact with a ground mesh. In the case of text prompt 502B, there may be no ground-contacting body part, as the interaction requires a person 314 to sit on the saddle / back of object 316. The ground mesh may represent the ground surface in the 3D environment 502A. The actions for selecting the main body part 404A are further described in Figures 5B to 5D, and the actions for selecting the auxiliary body part 404B and the ground-contacting body part are further described in Figure 5E.

[0090] For primary body part selection, the prompt sequence may include a first prompt prepared to include a first basic instruction having a text prompt 502B describing an interaction between object 316 and person 314. In one example, the processor 202 may be configured to prepare the first prompt based on a pre-configured prompt template. For example, the first prompt may be prepared to include a first basic instruction: "Does the interaction described in the text prompt request contact between the object and the person?"

[0091] In 508, the processor 202 may be configured to apply the neural language model 122 to a first prompt to output a first response (for example, yes or no). The first response may specify a response stating whether the interaction described in the text prompt 502B requests contact between object 316 and person 314. If the first response is "yes", control may be transferred to 510. Otherwise, control may be transferred to 516.

[0092] In 510, the processor 202 may further prepare a second prompt to include a plurality of 2D views 504A…504C and a second basic instruction associated with the plurality of 2D views 504A…504C. For example, the second prompt may be prepared to include a second basic instruction such as, "What is the name of the central part of the picture? Answer with as much detail as possible, in words or phrases. If you are not confident in specifying the name of the part, answer null." The second prompt 504B may be prepared to determine the name of the main scene area or main contact location (e.g., main contact location 512) where interaction with object 316 should take place. The processor 202 may further configure the second prompt to apply the neural language model 122 based on the first response to output a plurality of responses 510A corresponding to the plurality of 2D views 504A…504C. The plurality of responses 510A can reveal the body parts of object 316 at the center of each of the plurality of 2D views 504A…504C. For example, multiple responses 510A may include responses such as "Cow's Back," "Cow's Hip," and "Cow's Back."

[0093] In 512, the processor 202 may be further configured to apply majority voting behavior to multiple responses 510A to output a second response. The second response may specify the name of the main scene area 512A depicted in multiple 2D views 504A…504C. Majority voting may be a strategy used to reduce errors in identifying the name of the main scene area 512A. Using a multi-view rendering technique, multiple images centered on the interaction location 402C can be generated, and the neural language model 122 can be queried to determine the exact name of the main scene area 512A using the majority voting strategy. For example, using the majority voting strategy, the name of the main scene area 512A may be determined to be "Cow's Back".

[0094] In 514, the processor 202 may be further configured to prepare a third prompt, which includes a third basic instruction and a second response specifying the name of the main scene area 512A at a placeholder position within the third basic instruction. For example, the third prompt may be prepared to include a third basic instruction that says, "Generally speaking, when a person is sitting on the saddle of a standing cow, which part of the person's body should be in contact with the cow's back? Please select only one body part from the following options: <list of body parts>." The list is displayed as <'rightHand', 'rightUpLeg', 'leftArm', 'head', 'leftLeg', 'leftFoot', 'rightFoot', 'rightArm', 'rightLeg', 'leftForeArm', 'rightForeArm', 'neck', 'leftUpLeg', 'leftHand', 'hips'> [<"right hand", "right upper thigh", "left arm", "head", "left leg", "left foot", "right foot", "right arm", "right leg", "left forearm", "right forearm", "neck", "left upper thigh", "left hand", "hips">]. A second response specifying the name of the main scene area 512A may be included with a third basic instruction to output a third response 514A. A third response 514A may specify a primary body part 404A from a set of body parts 404A-404B that must contact the primary contact location 412A of a set of locations 412A-412B within the primary scene area 512A of the 3D environment 502A in order to achieve the interaction described in the text prompt 502B. For example, the third response 514A may specify the primary body part 404A of person 314 as the "buttocks". In some examples, the primary contact location 412A within the primary scene area 512A of the 3D environment 502A may correspond to a body part of a 3D asset representing object 316 within the 3D environment 502A. The primary contact locations 412A, 524B may be the same as the interaction location 402C, or may include the interaction location 402C.

[0095] The vertices of the selected main body part 404A are V k It can be written as: d keyThe distance loss function denoted by can be calculated based on V k . d key can be defined using the following Equations (2) and (3). [Equation] In Equations (2) and (3), A k specifies whether the interaction described in the text prompt 502B requires contact with a specific area on the object 316. If the answer to the specific area contact necessity evaluation A k is "No", the body does not need to be in contact with that specific area, and the distance between [Equation] and the entire scene (for example, the 3D environment 502A in FIG. 5A) can be calculated. For example, the text prompt 502B can be prepared to include the natural language instruction "sitting on the saddle of a cow". In this case, the person 314 may sit on a specific position of the cow, specifically on the back of the cow instead of the head or neck. Therefore, the answer to the specific position contact necessity evaluation A k is "Yes". Further, in the above equations [Equation]

[0096] For auxiliary body part selection, the prompt sequence may further include a fourth prompt prepared to include a fourth basic instruction, which contains a text prompt 502B in a placeholder position within the fourth basic instruction. In one example, the processor 202 may be configured to prepare a fourth prompt to contain a fourth basic instruction. For example, the fourth prompt may be prepared to include a fourth basic instruction 504C that reads: "In a given image, which body part of a person, other than the main body part 404A, i.e., the cow's back, is in contact with the object? Select only one body part from the following options: <list of body parts>." The list is displayed as <'rightHand', 'rightUpLeg', 'leftArm', 'head', 'leftLeg', 'leftFoot', 'rightFoot', 'rightArm', 'rightLeg', 'leftForeArm', 'rightForeArm', 'neck', 'leftUpLeg', 'leftHand', 'hips'> [<"Right Hand", "Right Upper Thigh", "Left Arm", "Head", "Left Leg", "Left Foot", "Right Foot", "Right Arm", "Right Leg", "Left Forearm", "Right Forearm", "Neck", "Right Upper Thigh", "Left Hand", "Buttocks">].

[0097] In 516, the processor 202 may be further configured to apply the neural language model 122 to a fourth prompt in order to output a fourth response 516A. The fourth response 516A may specify at least one body part from a list of body parts that is different from the primary body part 404A and must be in contact with object 316 in the interaction image 312 in order to achieve interaction between object 316 and person 314. The set of body parts 404A-404B may include auxiliary body parts 404B identified based on the fourth response 516A. The set of positions 412A-412B may include auxiliary contact positions 412B in which the auxiliary body part 404B is in contact with a 3D asset 502C representing object 316 in the 3D environment 502A. The auxiliary contact positions 412B may be different from the primary contact positions 412A. For example, the fourth response 516A may specify an auxiliary body part 404B of person 314 as a "leg". If no auxiliary body part is specified, control can be transferred to 518.

[0098] For example, specifying a single pair of body and scene parts is often insufficient to define natural interaction. For instance, relying solely on the distance between the buttocks (primary body part) and the cow's back when sitting on the cow's saddle can result in unrealistic interaction due to the large segmentation area of ​​the buttocks. To address this, one can select auxiliary body parts 404B or more auxiliary body parts that should come into contact with the 3D asset 502C of the scene (e.g., 3D environment 502A in Figure 5A), thereby ensuring a more natural position and orientation by adjusting the distance between the relevant auxiliary body parts and the 3D asset 502C of the 3D environment 502A.

[0099] The vertices of auxiliary body part 404B are,

number

number

[0100] For the selection of body parts in contact with the ground, the prompt sequence may include a fifth prompt prepared to include a fifth basic instruction associated with interaction image 312 and text prompt 502B. For example, the fifth prompt may be prepared to include a fifth basic instruction such as, "Given the person in the input image, do you think any of this person's body parts are touching the floor? Answer with yes / no only," and "When and which body part of the person in the given image should be touching the floor? Select only one body part from the following options: <list of body parts>." The list is displayed as <'rightHand', 'rightUpLeg', 'leftArm', 'head', 'leftLeg', 'leftFoot', 'rightFoot', 'rightArm', 'rightLeg', 'leftForeArm', 'rightForeArm', 'neck', 'leftUpLeg', 'leftHand', 'hips'> [<"Right Hand", "Right Upper Thigh", "Left Arm", "Head", "Left Leg", "Left Foot", "Right Foot", "Right Arm", "Right Leg", "Left Forearm", "Right Forearm", "Neck", "Right Upper Thigh", "Left Hand", "Buttocks">].

[0101] In 518, the processor 202 may be further configured to apply the neural language model 122 to a fifth prompt in order to output a fifth response. The fifth response may specify whether any body part from the list of body parts touches the ground surface in the interaction image 312 in order to achieve interaction between object 316 and person 314. In the case of interaction image 312, there are no body parts that touch the ground surface in interaction image 312. Therefore, the fifth response is no. However, in certain embodiments (not shown), the interaction may require contact with the ground surface. In such embodiments, the set of body parts 404A to 404B may include the ground-contacting body parts identified based on the fifth response. The set of positions 412A to 412B may also include ground-contacting positions where the ground-contacting body parts touch the ground mesh 414 representing the ground surface in the 3D environment 502A in order to achieve interaction. For example, if person 314 is standing on the ground surface in interaction image 312, the ground-contacting body parts in the fifth response may be "legs". In these or other embodiments, one or more ground-contacting parts may be identified. Such parts must be in contact with the ground to avoid unrealistic scenarios such as floating models. This can also ensure that the 3D parametric model (e.g., 3D mesh 402B) representing the person 314 in the interaction image 114 maintains physical validity within the scene.

[0102] The vertex of the body part in contact with the ground is,

number

number

[0103] After 516, control can move to 408 in Figure 4. key d auxiliary , and d ground Based on the calculation, the combined distance loss (L dis ) is L dis =α key ·d key +α auxiliary ·d auxiliary +α ground ·d ground It may also be defined as, where α key , α auxiliary , and α ground This represents the weight assigned to each distance. Furthermore, intrusion loss is used to prevent unrealistic mutual intrusion between the 3D mesh 402B of person 314 and the 3D environment 502A.

number

number

number

[0104] As shown in equation (6), the initial difference parameters represented by "R", "t", and "θ" can be adjusted through a sequential iterative optimization technique to minimize a customized loss function. This process may be repeated N times, with different final parameters obtained as the final output, ensuring the generation of realistic and diverse human-scene interactions. In one example, the values ​​of the initial difference parameters such as "R", "t", and "θ" can be updated sequentially and iteratively to obtain the final difference parameters by minimizing the value of the loss function until the value of the loss function matches a threshold.

[0105] The disclosed approach can offer several advantages. The generation of realistic 3D human-object interactions (both frequent and infrequent interactions) based on the effective selection of one or more body parts that must come into contact with a set of positions 412A-412B in a 3D environment can be achieved by leveraging the image analysis capabilities of a Visual Language Model (VLM) and the knowledge capabilities of a Large-Scale Language Model (LLM) without requiring reliance on ground truth data and supervision. Due to the very little or no reliance on ground truth data, customized loss functions based on specific conditions can be optimized, thereby leading to the generation of accurate static human-object interactions across a diverse range of different scenarios. Based on the use of VLM and LLM, even common and atypical human-object interactions can be generated in a zero-shot manner. Furthermore, the proposed technique may involve using VLM and LLM to leverage customized loss functions for each human-object interaction to generate specific conditions for each human-object interaction and to refine the generated human-object interactions with respect to posture, position, and orientation. To refine generated human-object interactions with respect to posture, position, and orientation, a single dataset can be constructed based on a dataset of generative artificial intelligence (AI) models. This dataset may include both common interactions defined in the generative AI model and non-general interactions not defined in the generative AI model. Consequently, this effective evaluation of a diverse range of static human-object interactions can be achieved by leveraging a single dataset. Therefore, this disclosure provides a framework that leverages the principles of VLM and LLM to synthesize accurate and realistic human-object interactions with natural postures that precisely reflect prompts in challenging situations.This approach can be optimized to efficiently generate a diverse range of human postures based on the effective selection of different human body parts, without requiring training data, to generate rare interactions such as stepping on a garbage bag, doing a handstand on the saddle of a standing cow, kicking a chair, or doing a handstand on a chair. Furthermore, this approach can be used across a variety of applications, including augmented reality (AR), virtual reality (VR), digital twin technology, computer games, and data augmentation applications.

[0106] It should be noted that the exemplary scenarios 500 in Figures 5A, 5B, 5C, 5D, and 5E are for illustrative purposes only and should not be construed as limiting the scope of this disclosure.

[0107] Figure 6 is a flowchart illustrating an exemplary method for generating 3D human-object interactions using a neural language model, according to one embodiment of the present disclosure. Figure 6 is described in relation to elements from Figures 1, 2, 3, 4, and 5A-5E. Referring to Figure 6, an exemplary flowchart 600 of a method for generating 3D human-object interactions using a neural language model is shown. Flowchart 600 may include operations 602 through 620 that may be performed by the processor 202 (in Figure 2) of system 102 (in Figure 1). Flowchart 600 may begin at 602 and proceed to 604.

[0108] At 604, input including interaction location in a 3D environment and text prompt may be received. Processor 202 may be configured to receive input 104 including interaction location 402C in a 3D environment 502A and text prompt 112. The text prompt 112 may describe an interaction between person 110 and object 108. The reception is further described, for example, in Figure 5A.

[0109] In 606, an interaction image may be generated by applying a text image generation model to a text prompt. The processor 202 may be configured to generate an interaction image 114 by applying a text image generation model 116 to a text prompt 112. The application of the text image generation model 116 to the text prompt 112 is further described, for example, in Figures 3 and 5A.

[0110] In 608, the initial difference parameters of the 3D parametric model can be determined. The processor 202 may be configured to determine the initial difference parameters for the 3D parametric model 118. The 3D parametric model may represent a person 110 in the interaction image 114. The determination of the initial difference parameters for the 3D parametric model is further described, for example, in Figures 3 and 4.

[0111] In 610, multiple 2D views of the 3D environment can be extracted from multiple viewpoints around the interaction location 402C. The processor 202 can be configured to extract multiple 2D views of the 3D environment 502A from multiple viewpoints around the interaction location 402C. The extraction of multiple 2D views of the 3D environment from multiple viewpoints is further illustrated, for example, in Figures 5A and 5B.

[0112] In 612, a sequence of prompts can be prepared based on multiple 2D views and text prompts. The processor 202 can be configured to prepare a sequence of prompts 120 based on multiple 2D views and text prompts 112. The preparation of a sequence of prompts based on multiple 2D views and text prompts is further illustrated, for example, in Figures 5B, 5D, and 5E.

[0113] In 614, a neural language model (NLM) may be applied to a sequence of prompts to identify a set of body parts. The processor 202 may be configured to apply the NLM to the sequence of prompts 120 to identify a set of body parts of a 3D parametric model 118 that must contact a set of positions in the 3D environment 502A in order to achieve interaction. The identified set of body parts may correspond to at least one of the primary body parts, auxiliary body parts, or ground-contacting body parts of the 3D parametric model 118 representing the person 110 in the interaction image 114. The application of the NLM to a sequence of prompts to identify a set of body parts of a 3D parametric model is further described, for example, in Figures 5A and 5C-5E.

[0114] In 616, the loss function can be initialized based on a set of body parts and a set of positions. The processor 202 may be configured to initialize the loss function based on a set of body parts and a set of positions. The loss function may include, but is not limited to, the composite distance between the vertices of the set of body parts and the vertices of the set of positions, an intrusion loss specifying the signed distance between the body vertices of the 3D parametric model 118 and the scene vertices of the 3D environment 502A, and a regularization loss that measures the deviation of the pose parameters of the 3D parametric model 118 from the initialized pose parameters of the initial difference parameters. The initialization of the loss function based on a set of body parts and a set of positions is further described, for example, in Figures 3 and 4.

[0115] In 618, the initial difference parameters may be iteratively updated based on the loss function to obtain the final difference parameters of the 3D parametric model. The processor 202 may be configured to iteratively update the initial difference parameters based on the loss function to obtain the final difference parameters of the 3D parametric model 118. The initial difference parameters may be iteratively updated to minimize the value of the loss function until the value of the loss function satisfies the convergence condition. Iterative updating of the initial difference parameters to obtain the final difference parameters is further illustrated, for example, in Figures 1, 5D, and 5E.

[0116] In 620, the 3D parametric model can be rendered in a 3D environment based on the final difference parameters. The processor 202 can be configured to render the 3D parametric model 118 in the 3D environment 502A based on the final difference parameters. The rendering of the 3D parametric model in the 3D environment based on the final difference parameters is further illustrated, for example, in Figures 1, 4, 5D, and 5E. Control can then be terminated.

[0117] While flowchart 600 is shown as discrete operations such as 604, 606, 608, 610, 612, 614, 616, 618, and 620, the disclosure is not limited in that way. However, in certain embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or deleted, depending on the specific implementation, without impairing the essence of the disclosed embodiments.

[0118] Various embodiments of the present disclosure may provide one or more non-temporary computer-readable storage media configured to store instructions causing a system (such as system 102) to perform a set of actions in response to being executed. The set of actions may include receiving an input including an interaction location in a 3D environment and text prompts describing the interaction between a person and an object. The set of actions may further include generating an interaction image by applying a text-to-image generation model to the text prompts. The set of actions may further include determining initial difference parameters for a 3D parametric model representing a person in the interaction image. The set of actions may further include extracting multiple 2D views of the 3D environment from multiple viewpoints around the interaction location. The set of actions may further include preparing a sequence of prompts based on the multiple 2D views and the text prompts. The set of actions may further include applying a neural language model to the sequence of prompts to identify a set of body parts of a 3D parametric model that must come into contact with a set of locations in the 3D environment to achieve the interaction. The set of actions may further include initializing a loss function based on the set of body parts and the set of locations. The set of actions may further include iteratively updating the initial difference parameters based on the loss function to obtain the final difference parameters of the 3D parametric model. The set of actions may further include rendering the 3D parametric model in a 3D environment based on the final difference parameters.

[0119] As used in this disclosure, the terms “module” or “component” may refer to a specific hardware implementation configured to perform the actions of a module or component, and / or a software object or software routine that can be stored and / or executed on general-purpose hardware of a computing system (e.g., computer-readable media, processing devices, etc.). In some embodiments, the different components, modules, engines, and services described in this disclosure may be implemented as objects or processes that run on a computing system (e.g., as separate threads). While some of the systems and methods described in this disclosure are generally described as being implemented in software (stored and / or executed on general-purpose hardware), specific hardware implementations or combinations of software and specific hardware implementations are also possible and conceivable. In this paper, a “computation entity” may be any computing system as previously defined in this disclosure, or any combination of modules or submodules that run on a computing system.

[0120] The terms used in this disclosure, and in particular in the attached claims (e.g., the body of the attached claims), are generally intended to be “open” terms (for example, “contains” should be interpreted as “contains, but is not limited to,” “has” should be interpreted as “has at least…,” and “contains” should be interpreted as “contains, but is not limited to,” etc.).

[0121] Furthermore, if a specific number is intended in the introduced claim, such intent is explicitly stated in the claim; if no such statement is made, such intent does not exist. For example, to aid understanding, the following appended claims may include the use of the introductory phrases “at least one” and “one or more” to introduce the claim. However, the use of such phrases is not permitted. Even if the same claim includes the introductory phrase "one or more" or "at least one" and an indefinite article such as "a" or "an," the introduction of a claim by the indefinite article "a" or "an" should not be interpreted as meaning that any particular claim containing such introduced claim is limited to embodiments that include only one such description (for example, "a" and / or "an" should be interpreted as meaning "at least one" or "one or more"), and the same applies to the use of definite articles used to introduce claim descriptions.

[0122] Furthermore, even if a specific number of claims introduced is explicitly stated, a person skilled in the art will recognize that such a statement should be interpreted as meaning at least that number (for example, the statement “two statements” without other modifiers means at least two statements, or two or more statements). Furthermore, when idiomatic expressions similar to “at least one of A, B, C, D, and E, etc.” or “one or more of A, B, C, D, and E, etc.” are used, such constructions are generally intended to include A only, B only, C only, D only, E only, a combination of A and B, a combination of A and C, a combination of B and C, a combination of A and D, a combination of B and D, a combination of C and D, a combination of A and E, a combination of B and E, a combination of C and E, a combination of D and E, or a combination of A, B, C, D, and E, etc.

[0123] Furthermore, any disjunctive phrase presenting two or more alternative terms should be understood in this paper, the claims, or the drawings as potentially including one of those terms, either of those terms, or both of those terms. For example, the phrase "A or B" should be understood as including the possibilities of "A" or "B" or "A and B".

[0124] All examples and conditional statements contained herein are intended for educational purposes to help readers understand the disclosure and the concepts to which the inventors have contributed to the advancement of the art, and should be construed as not being limited to such specifically described examples and conditions. While embodiments of this disclosure have been described in detail, various changes, substitutions, and modifications can be made thereto without departing from the spirit and scope of this disclosure.

[0125] With regard to embodiments including the above examples, the following additional information is disclosed. (Note 1) A method that can be executed by the processor: A stage of receiving input including the interaction location in a 3D environment and text prompts describing the interaction between a person and an object; The step of generating an interaction image by applying a text image generation model to the aforementioned text prompt; The steps include determining the initial difference parameters for the 3D parametric model representing the person in the aforementioned interaction image; The steps include: extracting multiple 2D views of the 3D environment from multiple viewpoints around the interaction location; A step of preparing a sequence of prompts based on the plurality of 2D views and the text prompts; The steps include: applying a neural language model to a sequence of prompts to identify a set of body parts of the 3D parametric model that must come into contact with a set of positions in the 3D environment in order to achieve the aforementioned interaction; A step of initializing a loss function based on the set of body parts and the set of positions; The steps include: sequentially updating the initial difference parameters based on the loss function to obtain the final difference parameters of the 3D parametric model; Based on the aforementioned final difference parameters, the step of rendering the 3D parametric model in the 3D environment is performed. Methods that include... (Note 2) The process further includes the step of identifying a major scene area from the 3D environment by selecting k nearest neighbor vertices from the scene mesh of the 3D environment, The k nearest neighbor vertex is selected based on the interaction position. The plurality of 2D views of the 3D environment are extracted from the plurality of viewpoints to include the main scene area. The method described in Appendix 1. (Note 3) The sequence of prompts includes a first prompt prepared to include a first basic instruction along with the text prompt describing the interaction, The neural language model is applied to the first prompt and outputs a first response that specifies whether the interaction described in the text prompt requests contact between the object and the person. The method described in Appendix 1. (Note 4) The sequence of prompts further includes a second prompt prepared to include the plurality of 2D views, and a second basic instruction associated with the plurality of 2D views. The neural language model is applied to the second prompt based on the first response and outputs a second response that specifies the names of the main scene areas depicted in the plurality of 2D views. The method described in Appendix 3. (Note 5) The sequence of prompts further includes a third prompt prepared to include a third basic instruction and a second response specifying the name of the main scene area at a placeholder position in the third basic instruction, The neural language model is applied to the third prompt to output a third response that identifies from the set of body parts the main body parts that must make contact with the main contact locations among the set of locations in the main scene area of ​​the 3D environment in order to achieve the interaction described in the text prompt. The method described in Appendix 4. (Note 6) The main contact location in the main scene area of ​​the 3D environment corresponds to the body part of the 3D asset representing the object in the 3D environment, as described in Appendix 5. (Note 7) The method according to Appendix 5, wherein the primary contact location is the same as the interaction location, or includes the interaction location. (Note 8) The sequence of prompts described above is further: The aforementioned interaction image; A fourth basic instruction, wherein the text prompt is included at the placeholder position in the fourth basic instruction; and List of body parts of the aforementioned person Includes a fourth prompt prepared to include, The neural language model is applied to the fourth prompt and outputs a fourth response specifying at least one body part from the list of body parts that, unlike the main body part, must make contact with the object in the interaction image in order to achieve the interaction. The method described in Appendix 5. (Note 9) The set of body parts includes an auxiliary body part identified based on the fourth response, The set of positions includes auxiliary contact positions where the auxiliary body portion contacts a 3D asset representing the object in the 3D environment. The auxiliary contact position is different from the main contact position. The method described in Appendix 8. (Note 10) The sequence of prompts further includes a fifth prompt, the fifth prompt being: The aforementioned interaction image, The aforementioned text prompt, A fifth basic instruction associated with the interaction image and the text prompt, List of body parts of the aforementioned person It is prepared to include, The neural language model is applied to the fifth prompt and outputs a fifth response specifying a body part from the list that must be in contact with the ground in the interaction image in order to achieve the interaction. The method described in Appendix 1. (Note 11) The set of body parts includes a ground-contacting body part identified based on the fifth response, The set of positions includes ground contact positions where the ground contact body portion must come into contact with the ground mesh representing the ground surface in the 3D environment in order to achieve the interaction. The method described in Appendix 10. (Note 12) The aforementioned loss function is: The composite distance loss between the vertices of the set of body parts and the vertices of the set of positions, An intrusion loss specifying the signed distance between the body vertices of the 3D parametric model and the scene vertices of the 3D environment, and Regularization loss that measures the deviation of the attitude parameters of the 3D parametric model from the initialized attitude parameters of the initial difference parameters. The method described in Appendix 1, including the method described in Appendix 1. (Note 13) The method according to Appendix 1, wherein the initial difference parameter is iteratively updated to minimize the value of the loss function until the value of the loss function satisfies the convergence condition. (Note 14) One or more non-temporary computer-readable storage media configured to store instructions that cause a system to perform an action in response to being executed, wherein the action is: A stage of receiving input including the interaction location in a 3D environment and text prompts describing the interaction between a person and an object; The step of generating an interaction image by applying a text image generation model to the aforementioned text prompt; The steps include determining the initial difference parameters for the 3D parametric model representing the person in the aforementioned interaction image; The steps include: extracting multiple 2D views of the 3D environment from multiple viewpoints around the interaction location; A step of preparing a sequence of prompts based on the plurality of 2D views and the text prompts; The steps include: applying a neural language model to a sequence of prompts to identify a set of body parts of the 3D parametric model that must come into contact with a set of positions in the 3D environment in order to achieve the aforementioned interaction; A step of initializing a loss function based on the set of body parts and the set of positions; The steps include: sequentially updating the initial difference parameters based on the loss function to obtain the final difference parameters of the 3D parametric model; Based on the aforementioned final difference parameters, the step of rendering the 3D parametric model in the 3D environment is performed. One or more non-temporary computer-readable storage media, including [the specified text]. (Note 15) The process further includes the step of identifying a major scene area from the 3D environment by selecting k nearest neighbor vertices from the scene mesh of the 3D environment, The k nearest neighbor vertex is selected based on the interaction position. The plurality of 2D views of the 3D environment are extracted from the plurality of viewpoints to include the main scene area. One or more non-temporary computer-readable storage media as described in Appendix 14. (Note 16) The sequence of prompts includes a first prompt prepared to include a first basic instruction along with the text prompt describing the interaction, The neural language model is applied to the first prompt and outputs a first response that specifies whether the interaction described in the text prompt requests contact between the object and the person. One or more non-temporary computer-readable storage media as described in Appendix 14. (Note 17) The sequence of prompts further includes a second prompt prepared to include the plurality of 2D views, and a second basic instruction associated with the plurality of 2D views. The neural language model is applied to the second prompt based on the first response and outputs a second response that specifies the names of the main scene areas depicted in the plurality of 2D views. One or more non-temporary computer-readable storage media as described in Appendix 16. (Note 18) The sequence of prompts further includes a third prompt prepared to include a third basic instruction and a second response specifying the name of the main scene area at a placeholder position in the third basic instruction, The neural language model is applied to the third prompt to output a third response that identifies from the set of body parts the main body parts that must make contact with the main contact locations among the set of locations in the main scene area of ​​the 3D environment in order to achieve the interaction described in the text prompt. One or more non-temporary computer-readable storage media as described in Appendix 17. (Note 19) The aforementioned loss function is: The composite distance loss between the vertices of the set of body parts and the vertices of the set of positions, An intrusion loss specifying the signed distance between the body vertices of the 3D parametric model and the scene vertices of the 3D environment, and Regularization loss that measures the deviation of the attitude parameters of the 3D parametric model from the initialized attitude parameters of the initial difference parameters. One or more non-temporary computer-readable storage media as described in Appendix 14, including the following: (Note 20) The memory that stores the instructions, and A processor coupled to the memory that executes the instructions and runs the process. has A system, wherein the process A stage of receiving input including the interaction location in a 3D environment and text prompts describing the interaction between a person and an object; The step of generating an interaction image by applying a text image generation model to the aforementioned text prompt; The steps include determining the initial difference parameters for the 3D parametric model representing the person in the aforementioned interaction image; The steps include: extracting multiple 2D views of the 3D environment from multiple viewpoints around the interaction location; A step of preparing a sequence of prompts based on the plurality of 2D views and the text prompts; The steps include: applying a neural language model to a sequence of prompts to identify a set of body parts of the 3D parametric model that must come into contact with a set of positions in the 3D environment in order to achieve the aforementioned interaction; A step of initializing a loss function based on the set of body parts and the set of positions; The steps include: sequentially updating the initial difference parameters based on the loss function to obtain the final difference parameters of the 3D parametric model; Based on the aforementioned final difference parameters, the step of rendering the 3D parametric model in the 3D environment is performed. A system that includes this. [Explanation of symbols]

[0126] 102 System 104 Input 106 3D environment 112 Text Prompt 116 Text-to-Image Generation Model 120 Prompt Sequence 122 Neural Language Models 124 Optimizer 128 Communication Networks 202 processors 206 I / O devices 206A Display device 204 memory 208 Network Interfaces 302 Text prompt received Applicable to 304 models 306 Text-to-Image Generation 308 Initial 3D body posture parameter estimation (t,R,θ) 310 Text prompt: "Sitting on the saddle of a standing cow" 312 Interaction Images 402 Input 402A Text Prompt 402B 3D Mesh 402C interaction position 404 Initial 3D Pose Parameter Generation 404A~404B Body parts set 406 Body part selection 406A Selection of primary contact body parts 406B Selection of auxiliary contact body parts 406C Ground Contact Body Part Selection 408 Loss Function Customization 410 3D Human-Object Interaction Synthesis 502B Text prompt: "Sitting on the saddle of a standing cow" 504A...504C Multiple 2D Views 508 First basic instruction: Does the interaction described in the text prompt require contact between an object and a person? 504A…504C Second Basic Instruction: What is the name of the central part of the picture? Answer with as much detail as possible, using a word or phrase. If you are unsure of the name of the part, answer "Null". 510A Multiple responses 512. Applying Majority Rule 512A Main Scenes / Areas 514 Third Basic Command: "Generally speaking, when a person is sitting on the saddle of a standing cow, which part of the person's body should be in contact with the cow's back? Please select only one body part from the following options: <"Right hand", "Right upper thigh", "Left arm", "Head", "Left leg", "Left foot", "Right foot", "Right arm", "Right leg", "Left forearm", "Right forearm", "Neck", "Right upper thigh", "Left hand", "Buttocks"> 514A Third Response 516 Fourth Basic Command: In a given image, which body part of a person, other than the primary body part, is in contact with the object? Select only one body part from the following options: <List of body parts> 516A Fourth Response 518 The fifth basic instruction: "Given a person in the input image, do you think any part of this person's body is touching the floor? Answer with yes / no only." and "When and which part of the person in the given image should be touching the floor? Select from the following options: <"Right hand", "Right upper thigh", "Left arm", "Head", "Left leg", "Left foot", "Right foot", "Right arm", "Right leg", "Left forearm", "Right forearm", "Neck", "Right upper thigh", "Left hand", "Buttocks"> 602 start 604 Receives input including interaction locations in a 3D environment and text prompts describing the interaction between a person and an object. 606 By applying a text image generation model to text prompts, interaction images are generated. 608 Determining initial difference parameters for a 3D parametric model representing a person in interaction images. 610 Extract multiple 2D views of a 3D environment from multiple viewpoints around the interaction location. 612 Prepare a sequence of prompts based on multiple 2D views and text prompts. 614 Applying a neural language model to a sequence of prompts to identify a set of body parts of a 3D parametric model that must come into contact with a set of positions in a 3D environment in order to achieve interaction. 616 Initialize the loss function based on a set of body parts and a set of positions. 618 The initial difference parameters are iteratively updated based on the loss function to obtain the final difference parameters of the 3D parametric model. Rendering a 3D parametric model in a 620 3D environment based on the final difference parameters.

Claims

1. A method that can be executed by the processor: A stage in which input is received, including the interaction location in a 3D environment and text prompts describing the interaction between a person and an object; The step of generating an interaction image by applying a text image generation model to the aforementioned text prompt; The steps include determining the initial difference parameters for the 3D parametric model representing the person in the interaction image; The steps include: extracting multiple 2D views of the 3D environment from multiple viewpoints around the interaction location; A step of preparing a sequence of prompts based on the plurality of 2D views and the text prompts; The steps include: applying a neural language model to a sequence of prompts to identify a set of body parts of the 3D parametric model that must come into contact with a set of positions in the 3D environment in order to achieve the aforementioned interaction; A step of initializing a loss function based on the set of body parts and the set of positions; The steps include: sequentially updating the initial difference parameters based on the loss function to obtain the final difference parameters of the 3D parametric model; Based on the aforementioned final difference parameters, the step of rendering the 3D parametric model in the 3D environment is performed. Methods that include...

2. The process further includes the step of identifying a major scene area from the 3D environment by selecting k nearest neighbor vertices from the scene mesh of the 3D environment, The k nearest neighbor vertex is selected based on the interaction position. The plurality of 2D views of the 3D environment are extracted from the plurality of viewpoints to include the main scene area. The method according to claim 1.

3. The sequence of prompts includes a first prompt prepared to include a first basic instruction along with the text prompt describing the interaction, The neural language model is applied to the first prompt and outputs a first response that specifies whether the interaction described in the text prompt requests contact between the object and the person. The method according to claim 1.

4. The sequence of prompts further includes a second prompt prepared to include the plurality of 2D views, and a second basic instruction associated with the plurality of 2D views. The neural language model is applied to the second prompt based on the first response and outputs a second response that specifies the names of the main scene areas depicted in the plurality of 2D views. The method according to claim 3.

5. The sequence of prompts further includes a third prompt prepared to include a third basic instruction and a second response specifying the name of the main scene area at a placeholder position in the third basic instruction, The neural language model is applied to the third prompt to output a third response that identifies from the set of body parts the main body parts that must make contact with the main contact locations among the set of locations in the main scene area of ​​the 3D environment in order to achieve the interaction described in the text prompt. The method according to claim 4.

6. The method according to claim 5, wherein the main contact location in the main scene area of ​​the 3D environment corresponds to a body part of a 3D asset representing the object in the 3D environment.

7. The method according to claim 5, wherein the primary contact position is the same as the interaction position or includes the interaction position.

8. The sequence of the aforementioned prompts is further: The aforementioned interaction image; A fourth basic instruction, wherein the text prompt is included at the placeholder position in the fourth basic instruction; and List of body parts of the aforementioned person Includes a fourth prompt prepared to include, The neural language model is applied to the fourth prompt and outputs a fourth response specifying at least one body part from the list of body parts that, unlike the main body part, must make contact with the object in the interaction image in order to achieve the interaction. The method according to claim 5.

9. The set of body parts includes an auxiliary body part identified based on the fourth response, The set of positions includes auxiliary contact positions where the auxiliary body portion contacts a 3D asset representing the object in the 3D environment. The auxiliary contact position is different from the main contact position. The method according to claim 8.

10. The sequence of prompts further includes a fifth prompt, the fifth prompt being: The aforementioned interaction image, The aforementioned text prompt, A fifth basic instruction associated with the interaction image and the text prompt, List of body parts of the aforementioned person It is prepared to include, The neural language model is applied to the fifth prompt and outputs a fifth response specifying a body part from the list that must be in contact with the ground in the interaction image in order to achieve the interaction. The method according to claim 1.

11. The set of body parts includes a ground-contacting body part identified based on the fifth response, The set of positions includes ground contact positions where the ground contact body portion must come into contact with the ground mesh representing the ground surface in the 3D environment in order to achieve the interaction. The method according to claim 10.

12. The aforementioned loss function is: The composite distance loss between the vertices of the set of body parts and the vertices of the set of positions, An intrusion loss specifying the signed distance between the body vertices of the 3D parametric model and the scene vertices of the 3D environment, and Regularization loss that measures the deviation of the attitude parameters of the 3D parametric model from the initialized attitude parameters of the initial difference parameters. The method according to claim 1, including the method described in claim 1.

13. The method according to claim 1, wherein the initial difference parameter is iteratively updated to minimize the value of the loss function until the value of the loss function satisfies the convergence condition.

14. One or more non-temporary computer-readable storage media configured to store instructions that cause a system to perform an action in response to being executed, wherein the action is: A stage in which input is received, including the interaction location in a 3D environment and text prompts describing the interaction between a person and an object; The step of generating an interaction image by applying a text image generation model to the aforementioned text prompt; The steps include determining the initial difference parameters for the 3D parametric model representing the person in the interaction image; The steps include: extracting multiple 2D views of the 3D environment from multiple viewpoints around the interaction location; A step of preparing a sequence of prompts based on the plurality of 2D views and the text prompts; The steps include: applying a neural language model to a sequence of prompts to identify a set of body parts of the 3D parametric model that must come into contact with a set of positions in the 3D environment in order to achieve the aforementioned interaction; A step of initializing a loss function based on the set of body parts and the set of positions; The steps include: sequentially updating the initial difference parameters based on the loss function to obtain the final difference parameters of the 3D parametric model; Based on the aforementioned final difference parameters, the step of rendering the 3D parametric model in the 3D environment is performed. One or more non-temporary computer-readable storage media, including [the specified text].

15. The process further includes the step of identifying a major scene area from the 3D environment by selecting k nearest neighbor vertices from the scene mesh of the 3D environment, The k nearest neighbor vertex is selected based on the interaction position. The plurality of 2D views of the 3D environment are extracted from the plurality of viewpoints to include the main scene area. One or more non-temporary computer-readable storage media according to claim 14.

16. The sequence of prompts includes a first prompt prepared to include a first basic instruction along with the text prompt describing the interaction, The neural language model is applied to the first prompt and outputs a first response that specifies whether the interaction described in the text prompt requests contact between the object and the person. One or more non-temporary computer-readable storage media according to claim 14.

17. The sequence of prompts further includes a second prompt prepared to include the plurality of 2D views, and a second basic instruction associated with the plurality of 2D views. The neural language model is applied to the second prompt based on the first response and outputs a second response that specifies the names of the main scene areas depicted in the plurality of 2D views. One or more non-temporary computer-readable storage media according to claim 16.

18. The sequence of prompts further includes a third prompt prepared to include a third basic instruction and a second response specifying the name of the main scene area at a placeholder position in the third basic instruction, The neural language model is applied to the third prompt to output a third response that identifies from the set of body parts the main body parts that must make contact with the main contact locations among the set of locations in the main scene area of ​​the 3D environment in order to achieve the interaction described in the text prompt. One or more non-temporary computer-readable storage media according to claim 17.

19. The aforementioned loss function is: The composite distance loss between the vertices of the set of body parts and the vertices of the set of positions, An intrusion loss specifying the signed distance between the body vertices of the 3D parametric model and the scene vertices of the 3D environment, and Regularization loss that measures the deviation of the attitude parameters of the 3D parametric model from the initialized attitude parameters of the initial difference parameters. One or more non-temporary computer-readable storage media according to claim 14, including the following:

20. The memory that stores the instructions, and A processor coupled to the memory that executes the instructions and runs the process. has A system, wherein the process A stage in which input is received, including the interaction location in a 3D environment and text prompts describing the interaction between a person and an object; The step of generating an interaction image by applying a text image generation model to the aforementioned text prompt; The steps include determining the initial difference parameters for the 3D parametric model representing the person in the interaction image; The steps include: extracting multiple 2D views of the 3D environment from multiple viewpoints around the interaction location; A step of preparing a sequence of prompts based on the plurality of 2D views and the text prompts; The steps include: applying a neural language model to a sequence of prompts to identify a set of body parts of the 3D parametric model that must come into contact with a set of positions in the 3D environment in order to achieve the aforementioned interaction; A step of initializing a loss function based on the set of body parts and the set of positions; The steps include: sequentially updating the initial difference parameters based on the loss function to obtain the final difference parameters of the 3D parametric model; Based on the aforementioned final difference parameters, the step of rendering the 3D parametric model in the 3D environment is performed. A system that includes this.

Images