Method for synthesizing human motion in three-dimensional scene based on unified scene semantic occupation

Abstract

The invention discloses a unified scene semantic occupancy-based human motion synthesis method in a three-dimensional scene, which is characterized in that unified scene semantic occupancy is adopted as scene representation, and local scene information is acquired through body centralization perception; performing scene feature extraction by adopting a bidirectional three-plane decomposition technology and a CLIP text encoder unified semantic coding; a motion controller is used for uniformly scheduling text instruction processing, modeling target-oriented character behavior and scene association, and motion sequence synthesis is completed by combining historical motion input and controlling a motion diffusion model. Compared with the prior art, the method has excellent performance on indexes such as execution time, target distance and non-penetrability score, solves the technical problem of scene semantic information and human behavior association modeling, can be widely applied to the fields of robot technology, game development, augmented reality and the like, and has a good application prospect.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and human motion generation technology, specifically a method for synthesizing human motion in a 3D scene based on unified scene semantic occupancy. Background Technology

[0002] Understanding and simulating human behavior in real-world 3D scenes has attracted significant attention due to its widespread applications in robotics, gaming, and augmented / virtual reality. Early research attempted to simulate human behavior and synthesize human motion to meet the requirements of a given scene. The PointNet family of sensors was used to perceive the entire scene and provide guidance for scene-aware human motion synthesis. More recent work focuses on utilizing fine-grained scene structures, with mesh sensors widely used due to their simplicity. However, the semantic information of the scene, which is highly correlated with human behavior, cannot be effectively represented and utilized in motion synthesis.

[0003] In scene-aware human motion synthesis, existing methods primarily focus on scene structure but neglect semantic understanding, and directly extracting features from scene semantics is extremely resource-intensive. Furthermore, the semantic category definitions in current scene understanding methods often depend on the dataset, making cross-dataset generalization challenging.

[0004] In summary, existing scene-aware human motion synthesis technologies mainly focus on scene structure while neglecting semantic understanding, resulting in a technical challenge in modeling the correlation between scene semantic information and human behavior. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for human motion synthesis in 3D scenes based on unified scene semantic occupancy (SSO). This method employs bidirectional triplane decomposition and CLIP text encoder for unified semantic encoding of scene features, using SSO as the scene representation and acquiring local scene information through body-centered perception. Specifically, a Gaussian kernel activation function is used on the depth map to highlight the influence of nearby objects, and the color map is normalized, enabling the acquisition of fine-grained scene semantic structure while significantly reducing redundant computation. In the motion generation stage, a motion controller uniformly schedules text instruction processing, models goal-oriented character behavior and its association with the scene, and combines historical motion input to control the motion diffusion model, completing motion sequence synthesis. The first stage trains a basic human motion generation model, and the second stage trains a scene instruction-controlled human motion generation model, achieving continuous scene adaptation and motion optimization.

[0006] This method performs well in terms of execution time, target distance, and non-penetration score, and effectively solves the technical challenge of modeling the relationship between scene semantic information and human behavior. It can be widely applied in fields such as robotics, game development, and augmented reality, and has good application prospects.

[0007] The specific technical solution to achieve the purpose of this invention is: a method for synthesizing human motion in a 3D scene based on unified scene semantic occupancy, characterized in that the method for synthesizing human motion in a 3D scene specifically includes the following steps: Step 1: Obtaining Scene Semantic Occupation: The 3D scene is represented as a compact scene semantic occupancy representation, where each element represents an occupancy voxel containing x, y, z coordinates, rgba color, and semantic label; Step 2: Two-way three-plane decomposition Obtain the semantic occupancy of the local scene centered on the human body, and render the local semantic occupancy along the positive and negative x, y, and z axes to obtain semantic occupancy maps Oyz, Ozy, Ozx, Oxz, Oxy, and Oyx. Each semantic occupancy map contains color, depth, and semantic label information. Step 3: Unified semantic feature mapping Semantic categories are embedded using the CLIP text encoder. A shared linear layer is used to reduce the dimensionality of all semantic category features. Then, the dimensionality-reduced CLIP features are distributed onto the semantic map to obtain a unified semantic feature map. Step 4: Scene Feature Extraction A Gaussian kernel is used as the activation function for the depth map to focus on nearby objects. The color map is normalized to the range [0,1]. Then, semantic features, geometric features and texture features are extracted respectively, and the three types of features are combined to obtain the perceived scene features. Step 5: Motion Control Calculate the target direction vector based on text instructions. For mobile tasks, calculate the direction from the current human pelvis to the target position. For interactive tasks, sample the target human posture and calculate the direction of all joints. Establish the target-oriented human-scene correlation through an attention mechanism. Step 6: Motion synthesis A motion diffusion model is used, which combines historical motion, scene features and interaction targets, to synthesize human motion sequences through an iterative denoising process.

[0008] The scene semantic occupancy is represented in a compact manner, where each element represents an occupancy voxel and consists of x, y, z coordinates, rgba color, and semantic label s.

[0009] The unified semantic feature mapping uses CLIP text encoder to embed semantic categories, realizing a unified semantic space representation of multiple datasets. After dimensionality reduction of CLIP features through a shared linear layer, they are distributed to the semantic graph, avoiding computational redundancy caused by direct extraction of high-dimensional features.

[0010] The depth map uses a Gaussian kernel activation function to make nearby objects easily perceived, and the color map is normalized to the range [0,1].

[0011] The motion control includes action intention cues, uses a learnable action codebook to represent action features, and clips the norm of the direction vector to a normalized range to provide directional cue features.

[0012] The goal-oriented human-scenario relevance is established in the following ways: 1) Map directional cue features to keys and values, convert motion features into frame-level queries, and calculate target-oriented human motion features using SoftMax attention; 2) Map the obtained target-oriented human motion features to frame-level queries, map scene features to keys and values, and obtain scene-aware motion features through an attention mechanism.

[0013] The motion control is constructed proportionally to the main branch, and multiple zero-initialized linear layers are used to inject control signals from the control branch into the main branch.

[0014] The human motion synthesis method in the 3D scene adopts a decoupled training strategy, using a base diffusion model and a control branch to be trained separately on different datasets. The base diffusion model is trained on a standard motion capture dataset; the control branch is trained using scene perception and the dataset, and the parameters of the base diffusion model are frozen.

[0015] The human motion synthesis method in the three-dimensional scene includes: multi-task motion synthesis of movement, action-to-motion generation, and human-scene interaction.

[0016] The basic diffusion model synthesizes S-frame motion trajectories through iterative denoising. For long-term motion synthesis, an autoregressive strategy is adopted, using the pose of the last H frames of the previous segment as the initial condition for the next generation, thus achieving a coherent target-oriented motion sequence.

[0017] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress: 1) By using bidirectional three-plane decomposition technology, the scene semantic occupancy is decomposed into a compact low-dimensional representation, which solves the problem of huge resource consumption when directly extracting features from the scene semantic occupancy. 2) A unified semantic representation is achieved through CLIP encoder and shared linear dimensionality reduction technique, which solves the generalization problem caused by different semantic category definitions in different datasets; 3) The problem of modeling the correlation between scene semantic information and human behavior was solved through goal-oriented human-scene correlation modeling; 4) High-quality long-term motion synthesis was achieved through decoupling training strategies and autoregressive generation methods. The quality and generalization ability of human motion synthesis in 3D scenes were improved by decomposing steps such as scene semantic occupancy acquisition, bidirectional three-plane decomposition, unified semantic feature mapping, scene feature extraction, target-guided scene character behavior control signal acquisition, and motion synthesis. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the process of the present invention; Figure 2 This is a diagram illustrating the acquisition of scene semantic occupancy. Figure 3 This is a schematic diagram of a two-way three-plane decomposition. Figure 4 A diagram illustrating unified semantic features; Figure 5 This is a schematic diagram of scene feature extraction; Figure 6 Create a diagram illustrating the human-scenario relevance for goal-oriented scenarios; Figure 7 This is a schematic diagram of the motion synthesis module. Detailed Implementation

[0019] See Figure 1 The present invention specifically includes the following steps: Step 100: Obtaining Scene Semantic Occupation Step 110: Convert the global scene into a sparse scene semantic occupancy, which contains RGB and semantic information; Step 120: Use a compact scene semantic occupancy storage, where each element represents an occupancy voxel containing x, y, and z coordinates, rgba color, and semantic label s.

[0020] Step 200: Two-way three-plane decomposition Step 210: Based on the current human body state, perceive the local scene centered on the human body, obtain the last few frames of the historical human body movement as guidance for the motion synthesis of the future scene perception, and use the last frame of the historical movement as the initial posture to perceive the surrounding environment. Step 220: Obtain the human body translation and horizontal orientation based on the human skeletal joints, and evenly distribute sensors around the human body. These sensors are transformed to world coordinates through a rotation matrix. Step 230: Render the local semantic occupancy along the positive and negative xyz axes to obtain semantic occupancy maps Oyz, Ozy, Ozx, Oxz, Oxy, and Oyx. Each semantic occupancy map contains color, depth, and semantic label information.

[0021] Step 300: Scene Feature Extraction Step 310: Embed semantic categories using CLIP text encoder to achieve unified semantic space representation. After dimensionality reduction of CLIP features through shared linear layer, they are distributed to semantic map to obtain unified semantic feature map. Step 320: For the depth map, we choose to focus more on nearby objects because the geometry of close-up scenes shows a stronger correlation with human behavior. We use a Gaussian kernel as the activation function to obtain an activated depth map, making it easier to perceive nearby objects. For the color map, we simply normalize it to the range [0,1]. Step 330: Perceived scene features combine semantic, geometric, and texture features for motion control in the 3D scene. Step 400: Target-guided scene character behavior control signals. Step 410: Based on the 3D scene, it is necessary to synthesize future motion according to instructions, using a learnable action codebook to represent motion features, and to navigate to the target location or reach the target object according to instructions. Step 420: For movement within the scene, calculate the direction from the current human pelvis to the target position; for other interactions, sample the target human pose, then calculate the direction of all joints, and clip the norm of the direction to the normalized range. Step 430: Map directional cue features to keys and values, generate frame-level queries through human motion features, obtain target-oriented human motion features through an attention mechanism, further map scene features to scene-related keys and values, map target-oriented human motion features to frame-level queries, obtain scene-aware motion features through an attention mechanism, and then establish scene-aware motion features.

[0022] Step 500: Motion Synthesis Step 510: The motion control branch is constructed according to the proportion of the main branch, and multiple zero-initialized linear layers are used to inject control signals from the control branch into the main branch; Step 520: A decoupled training strategy is adopted, and the basic diffusion model and the control branch are trained on different datasets respectively. The basic diffusion model is trained on a general motion dataset to obtain conditional generation capabilities, and the control branch is trained on a scene-aware dataset to integrate scene perception. The parameters of the basic diffusion model are frozen. Step 530: The model supports multi-task motion synthesis, including movement, action-to-motion generation, and human-scene interaction. The basic diffusion model synthesizes S-frame motion trajectories through iterative denoising. For long-term motion synthesis, an autoregressive strategy is adopted, with the pose of the last H frames of the previous segment serving as the initial condition for the next generation.

[0023] The technical solution of the present invention will be described in more detail below with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention. Example

[0024] The implementation method of human motion synthesis in a 3D scene based on unified scene semantic occupancy is as follows: 1) Data representation Human motion is represented as a sequence of body postures. The parametric human body model SMPL-X is used to represent human postures, mainly considering global translation, global orientation, and body joint rotation. The human body shape and hand posture remain unchanged during the motion.

[0025] 2) Scene representation See Figure 2 First, global scene information is obtained from 3D scene data and converted into a sparse scene semantic occupancy representation. The system discretizes the continuous 3D space through voxelization, with each voxel recording whether it is occupied and its corresponding attribute information. The converted scene semantic occupancy includes RGB color information and semantic label information, providing a data foundation for subsequent body-centered perception. A compact scene semantic occupancy storage method is adopted to improve storage and computation efficiency. For each occupied voxel, the system records its 3D coordinate position, RGBA color value, and corresponding semantic label. This compact representation avoids storing a large number of blank voxels, significantly reducing memory consumption, while maintaining the complete semantic and geometric information of the scene.

[0026] 3) Diffusion Model For any human motion, the diffusion process gradually adds noise to the original motion, and the reverse process gradually denoises the human motion. In the implementation, the supervisory network can directly predict the original motion given the noise motion at any time step.

[0027] 4) Two-way three-plane decomposition See Figure 3Based on the current human state perception and a human-centered local scene, the system acquires the last few frames of historical human motion as guidance for future scene perception motion synthesis. The last frame of historical motion is used as the initial pose to perceive the surrounding environment. By analyzing human skeletal joint information, the spatial position and orientation of the human body are determined, establishing a human-centered local coordinate system. Human translation and horizontal orientation information are obtained based on the human skeletal joints. A virtual sensor mesh is uniformly distributed around the human body, covering the three-dimensional spatial region around the body. The local coordinates of the sensors are transformed to the world coordinate system through a rotation matrix, achieving an accurate correspondence from body-centered coordinates to global coordinates. Local semantic occupancy is rendered and projected along the positive and negative x, y, and z axes, obtaining semantic occupancy maps in six different directions. Specifically, these include Oyz and Ozy projection maps along the positive and negative x-axis, Ozx and Oxz projection maps along the positive and negative y-axis, and Oxy and Oyx projection maps along the positive and negative z-axis. Each semantic occupancy map contains three types of information: color, depth, and semantic label, providing a multi-view scene representation for subsequent unified semantic feature extraction.

[0028] 5) Unified semantic occupancy representation See Figure 4 To achieve a unified semantic space representation, a CLIP text encoder is used to embed semantic categories. To address the issue of inconsistent semantic category definitions across different datasets, the system inputs each category name into the CLIP text encoder to obtain high-dimensional semantic features. Due to the repetitive nature of semantic information and the high dimensionality of CLIP features, all semantic category features are first dimensionality-reduced using a shared linear layer. Then, the dimensionality-reduced CLIP features are distributed across the semantic graph to obtain a unified semantic feature map representation.

[0029] 6) Scene Feature Extraction See Figure 5 For depth map processing, the system prioritizes nearby objects because near-field scene geometry exhibits a stronger correlation with human behavior. A Gaussian kernel is used as the activation function to obtain an activated depth map, making nearby objects easily perceived while the influence of distant objects gradually diminishes. For the color map, the system simply normalizes it to a specified range, preserving the relative relationships of color information. The perceived scene features combine semantic, geometric, and texture features. Semantic features are extracted from a unified semantic feature map, containing category information of objects in the scene; geometric features are obtained from the activated depth map, reflecting spatial structure and distance relationships; and texture features are extracted from the normalized color map, containing surface material information. These three types of features combine to form a comprehensive scene feature representation for motion control in 3D scenes.

[0030] 7) Motion controller See Figure 6Given a 3D scene, future motion needs to be synthesized according to instructions. A learnable action codebook is used to represent motion features. The direction vector from the current position to the target is calculated, and the direction norm is clipped to a normalized range. The target-oriented human-scene correlation is established, and the direction cue features are mapped to keys and values. Motion features are generated as frame-level queries, and target-oriented motion features are obtained through an attention mechanism. Then, the obtained target-oriented human motion features are further mapped to keys and values, and scene features are mapped to frame-level queries. The scene-aware motion features are obtained through an attention mechanism, establishing the correlation between human motion targets and the scene environment, and realizing the scene-aware motion feature representation.

[0031] 8) Training and Reasoning See Figure 7 A decoupled training strategy is adopted, dividing the entire system into two parts: a main branch and a control branch, which are trained separately. The main branch is responsible for basic human motion generation, while the control branch is responsible for scene perception and command control. The motion control branch is constructed proportionally to the main branch to ensure compatibility of the two branches in terms of network structure. Multiple zero-initialization linear layers are used as connecting bridges to gradually inject control signals from the control branch into the corresponding layers of the main branch, achieving effective transmission and fusion of control signals.

[0032] This invention employs a decoupled training strategy, training the base diffusion model and control branch separately on different datasets. The base diffusion model is trained on a standard motion capture dataset to acquire conditional generation capabilities and learn basic human motion patterns. The control branch is trained on a scene-aware dataset to integrate scene-aware capabilities. During training, the parameters of the base diffusion model are frozen to ensure the stability of the basic motion generation capability. The model supports multi-task motion synthesis, including movement, action-to-motion generation, and human-scene interaction. The base diffusion model synthesizes S-frame motion trajectories through iterative denoising. For long-term motion synthesis, an autoregressive strategy is used, employing the pose of the last H frames of the previous segment as the initial condition for the next generation, achieving coherent goal-oriented motion sequence generation.

[0033] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for synthesizing human motion in a 3D scene based on unified scene semantic occupancy, characterized in that, The method for synthesizing human motion in a 3D scene specifically includes the following steps: Step 1: Obtaining Scene Semantic Occupation: The 3D scene is represented as a compact scene semantic occupancy representation, where each element represents an occupancy voxel containing x, y, z coordinates, rgba color, and semantic label; Step 2: Two-way three-plane decomposition Obtain the semantic occupancy of the local scene centered on the human body, render the local semantic occupancy along the positive and negative x, y, and z axes to obtain semantic occupancy maps Oyz, Ozy, Ozx, Oxz, Oxy, and Oyx. Each semantic occupancy map contains color, depth, and semantic label information. Step 3: Unified semantic feature mapping Semantic categories are embedded using the CLIP text encoder. A shared linear layer is used to reduce the dimensionality of all semantic category features. Then, the dimensionality-reduced CLIP features are distributed onto the semantic map to obtain a unified semantic feature map. Step 4: Scene Feature Extraction A Gaussian kernel is used as the activation function for the depth map to focus on nearby objects. The color map is normalized to the range [0,1]. Then, semantic features, geometric features and texture features are extracted respectively, and the three types of features are combined to obtain the perceived scene features. Step 5: Motion Control Calculate the target direction vector based on text instructions. For mobile tasks, calculate the direction from the current human pelvis to the target position. For interactive tasks, sample the target human posture and calculate the direction of all joints. Establish the target-oriented human-scene correlation through an attention mechanism. Step 6: Motion synthesis A motion diffusion model is used, which combines historical motion, scene features and interaction targets, to synthesize human motion sequences through an iterative denoising process.

2. The method for synthesizing human motion in a 3D scene based on unified scene semantic occupancy as described in claim 1, characterized in that, The scene semantic occupancy is represented in a compact manner, where each element represents an occupancy voxel and consists of x, y, z coordinates, rgba color, and semantic label s.

3. The method for synthesizing human motion in a 3D scene based on unified scene semantic occupancy as described in claim 1, characterized in that, The unified semantic feature mapping uses CLIP text encoder to embed semantic categories, realizing a unified semantic space representation of multiple datasets. After dimensionality reduction of CLIP features through a shared linear layer, they are distributed to the semantic graph, avoiding computational redundancy caused by direct extraction of high-dimensional features.

4. The method for synthesizing human motion in a 3D scene based on unified scene semantic occupancy as described in claim 1, characterized in that, The depth map uses a Gaussian kernel activation function to make nearby objects easily perceived, and the color map is normalized to the range [0,1].

5. The method for synthesizing human motion in a 3D scene based on unified scene semantic occupancy according to claim 1, characterized in that, The motion control includes action intention cues, uses a learnable action codebook to represent action features, and clips the norm of the direction vector to a normalized range to provide directional cue features.

6. The method for synthesizing human motion in a 3D scene based on unified scene semantic occupancy according to claim 1, characterized in that, The goal-oriented human-scenario relevance is established in the following ways: 1) Map directional cue features to keys and values, convert motion features into frame-level queries, and calculate target-oriented human motion features using SoftMax attention; 2) Map the target human motion features to frame-level queries, map scene features to keys and values, and obtain scene-aware motion features through an attention mechanism.

7. The method for synthesizing human motion in a 3D scene based on unified scene semantic occupancy according to claim 1, characterized in that, The motion control is constructed proportionally to the main branch, and multiple zero-initialized linear layers are used to inject control signals from the control branch into the main branch.

8. The method for synthesizing human motion in a 3D scene based on unified scene semantic occupancy according to claim 1, characterized in that, The human motion synthesis method in the 3D scene adopts a decoupled training strategy, using a base diffusion model and a control branch to be trained separately on different datasets. The base diffusion model is trained on a standard motion capture dataset; the control branch is trained using scene perception and the dataset, and the parameters of the base diffusion model are frozen.

9. The method for synthesizing human motion in a 3D scene based on unified scene semantic occupancy according to claim 1 or claim 8, characterized in that, The human motion synthesis method in the three-dimensional scene includes: multi-task motion synthesis of movement, action-to-motion generation, and human-scene interaction.

10. The method for synthesizing human motion in a 3D scene based on unified scene semantic occupancy as described in claim 8, characterized in that, The basic diffusion model synthesizes S-frame motion trajectories through iterative denoising. For long-term motion synthesis, an autoregressive strategy is adopted, using the pose of the last H frames of the previous segment as the initial condition for the next generation, thus achieving a coherent target-oriented motion sequence.

Images