A three-dimensional human body and scene interactive reconstruction method and system

Abstract

The application belongs to the technical field of three-dimensional reconstruction, and specifically discloses a three-dimensional human body and scene interactive reconstruction method and system. The method is a three-dimensional human body and scene interactive reconstruction method for monocular video input, which can quickly complete three-dimensional Gaussian reconstruction of human bodies, static scenes and moving objects, and through introduction of posture correction, camera external parameter space regularization, joint reconstruction consistency loss and hierarchical fusion rendering technical means, ensures accurate alignment of the human body, the scene and the object in spatial position, occlusion relationship and illumination style, and realizes dynamic interactive reconstruction conforming to real physical logic. Finally, the application can generate high-fidelity, time-continuous and structure-consistent joint rendering images, provides stable and reliable three-dimensional expression capability for complex human body motion, object interaction and scene understanding, and greatly improves the three-dimensional interactive reconstruction quality and application value under monocular video conditions.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of three-dimensional reconstruction, and relates to a three-dimensional human body and scene interactive reconstruction method and system. BACKGROUND

[0002] Interactive reconstruction of three-dimensional human body and scene is a core supporting technology in the fields of virtual reality (VR), augmented reality (AR), digital twin and virtual content creation, and its goal is to construct high-fidelity three-dimensional human body model and scene model from visual input simultaneously, to provide a real three-dimensional environment basis for immersive experience and virtual interactive application. The current technology has irreplaceable application value in virtual fitting and other scenes, but it still faces multi-dimensional technical bottlenecks in actual landing, and the specific problems are as follows:

[0003] I. In the aspect of three-dimensional human body reconstruction, the existing technology is difficult to balance between "reconstruction efficiency", "detail fidelity" and "monocular input adaptability". Traditional multi-view reconstruction methods need to rely on multi-camera arrays to collect data from different angles, which can obtain high precision, but the device cost is high and the scene deployment is limited. In recent years, reconstruction methods based on monocular video have become a research hotspot. Among them, parameterized mesh models (such as SMPL-X) can quickly fit human posture and body shape by presetting human topological structure, but are limited by fixed mesh topology. Neural Radiance Field (NeRF) and its derivative technologies model human radiance field through neural network, which can generate high-fidelity appearance and geometry, but have the problems of long training cycle (usually several hours to several days) and weak real-time interaction capability. In addition, existing human body reconstruction methods focus on the human body itself and do not consider its interactive relevance with the scene, which makes it difficult for the reconstructed human model to naturally integrate into the scene and requires additional manual adjustment of spatial position and posture, which is inefficient.

[0004] II. In the aspect of three-dimensional scene reconstruction, static scene reconstruction technology is relatively mature, but scene reconstruction facing "human-scene dynamic interaction" still has obvious shortcomings. However, traditional static scene reconstruction can generate sparse point cloud or dense mesh of the scene, but cannot dynamically respond to the interaction between human and scene. Although static scene reconstruction based on 3D Gaussian Sputtering (3DGS) can realize real-time high-fidelity rendering, its original design is for static scene representation, and if it is directly used for scene reconstruction with human interaction, it is easy to cause geometric misalignment between scene and human interaction area.

[0005] III. In the aspect of interactive and collaborative reconstruction of human and scene, there is no effective solution. Most methods are prone to the problem of position deviation of scene objects that should be held by the human body. In addition, for dynamic interactive scenes of "human carrying scene object movement", the dynamic association between object and human body is easily ignored, resulting in that the reconstruction result cannot reflect the real interaction logic.

[0006] In summary, the current three-dimensional human and scene reconstruction technology has significant deficiencies in monocular input adaptability, detail fidelity, real-time performance, and interactive collaboration, and cannot efficiently and accurately realize dynamic interactive reconstruction of the two. Therefore, it is urgent to propose a technical solution that can simultaneously complete high-fidelity human, scene, and interactive relationship reconstruction from monocular input. SUMMARY

[0007] The purpose of the present application is to propose a three-dimensional human and scene interactive reconstruction method, which can quickly reconstruct three-dimensional human, three-dimensional scene and three-dimensional object from monocular video input, and can import the reconstructed three-dimensional models into the same three-dimensional space coordinate system, and optimize joint rendering according to the spatial relationship between human, scene and object, so as to realize three-dimensional reconstruction of high-fidelity human, scene and interactive relationship from monocular input.

[0008] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0009] A three-dimensional human and scene interactive reconstruction method, comprising the following steps:

[0010] Step 1. Perform three-dimensional pose estimation with monocular video as input, obtain human parameters, generate a standard mesh consistent with the human topology, and map the standard mesh vertices to a three-plane feature space to extract features of each vertex;

[0011] Input the extracted vertex features into a multi-layer perception network to predict the corresponding three-dimensional Gaussian attributes of each vertex, thereby converting the static standard mesh into a three-dimensional human Gaussian representation composed of Gaussian primitives;

[0012] Perform spatial transformation on the three-dimensional human Gaussian using linear blend skinning, and then generate a human image in the target pose through a three-dimensional Gaussian Splatting rendering pipeline to realize three-dimensional human model reconstruction;

[0013] Step 2. Input multi-view scene images and SfM calibrated camera parameters, estimate the scene point cloud through colmap, then initialize the scene point cloud as 3D Gaussian points, and project the 3D Gaussian points to generate 2D Gaussians;

[0014] Finally, the 2D Gaussians generate pixel colors through Alpha blending to complete three-dimensional scene model reconstruction;

[0015] Step 3. Input monocular video, construct a basic 3D Gaussian set as a stable prior, predict Gaussian frame-by-frame geometric transformation, render pixels through differentiable Gaussian rasterization, and optimize to realize three-dimensional object model reconstruction;

[0016] Step 4. With monocular video as input, camera intrinsic and extrinsic parameters are inferred, and a robust canonical world coordinate system is established by introducing quaternion orthogonality constraints, and the camera extrinsic parameters are optimized through cross-subject consistency loss;

[0017] After the camera extrinsic parameter optimization is completed, the human body, scene and object three types of Gaussian mappings are mapped to the camera coordinate system and projected to the image plane, and a hierarchical hybrid rendering strategy is adopted to realize the natural fusion of the human body, object and scene;

[0018] Through joint reconstruction consistency loss optimization of details, color and occlusion, a high-fidelity dynamic rendering image is finally generated.

[0019] In addition, on the basis of the above-mentioned three-dimensional human body and scene interactive reconstruction method, the application also proposes a corresponding three-dimensional human body and scene interactive reconstruction system, both of which are based on the same inventive concept, which adopts the following technical scheme:

[0020] A three-dimensional human body and scene interactive reconstruction system comprises the following modules:

[0021] The three-dimensional human body reconstruction module is used for three-dimensional pose estimation with monocular video as input, obtaining human body parameters, generating a canonical mesh consistent with the human body topology, and mapping the canonical mesh vertices to a three-plane feature space to extract the features of each vertex.

[0022] The extracted vertex features are input into a multi-layer perception network to predict the corresponding three-dimensional Gaussian attributes of each vertex, thereby converting the static canonical mesh into a three-dimensional human body Gaussian representation composed of Gaussian primitives.

[0023] The three-dimensional human body Gaussian is spatially transformed using linear blend skinning, and then a three-dimensional Gaussian Splatting rendering pipeline is used to generate a human body image at the target pose, realizing three-dimensional human body model reconstruction.

[0024] The three-dimensional scene model reconstruction module is used for inputting multi-view scene images and SfM calibrated camera parameters, estimating the scene point cloud through colmap, initializing the scene point cloud as 3D Gaussian points, and then projecting the 3D Gaussian points to generate 2D Gaussian; finally, the 2D Gaussian generates pixel color through Alpha blending, completing the three-dimensional scene model reconstruction.

[0025] The three-dimensional object model reconstruction module is used for inputting monocular video, constructing a basic 3D Gaussian set as a stable prior, predicting Gaussian frame-by-frame geometric transformation, and generating pixels through differentiable Gaussian rasterization rendering and optimization, realizing three-dimensional object model reconstruction.

[0026] and an interactive reconstruction module configured to infer camera intrinsic and camera extrinsic with monocular video as input, and introduce quaternion orthogonality constraint to establish a robust canonical world coordinate system, and optimize camera extrinsic through cross-subject consistency loss;

[0027] After the camera extrinsic optimization, the human body, scene and object are mapped to the camera coordinate system and projected to the image plane, and a hierarchical hybrid rendering strategy is adopted to realize the natural fusion of the human body, object and scene.

[0028] Through the joint reconstruction consistency loss optimization of details, color and occlusion, a high-fidelity dynamic rendering image is finally generated.

[0029] The present application has the following advantages:

[0030] As described above, the present application proposes an interactive reconstruction method for three-dimensional human body, scene and object with monocular video input, which realizes the unified reconstruction of the three in the geometric, spatial and visual aspects through the cooperation of posture correction, camera extrinsic space regularization, joint reconstruction consistency loss and hierarchical fusion rendering. The posture correction first dynamically compensates the human body mesh, so that the human body geometry remains stable under complex motion. On this basis, the camera extrinsic space regularization constrains the cross-frame extrinsic parameter change with the corrected geometric structure as the reference, and realizes the accurate alignment of the human body, scene and object in the same world coordinate system. Then, the joint reconstruction consistency loss feedback corrects the rendering result through pixel, occlusion and style, etc. visual error, further unifies the appearance and relationship of the human body, scene and object. Finally, hierarchical fusion rendering completes correct depth sorting and Gaussian fusion based on the unified geometry and appearance parameters, and the rendering result is re-input into the joint reconstruction consistency loss to form a closed loop optimization. Through the sequential connection and mutual dependence of the above modules, the present application realizes the high-fidelity, time-continuous and interactive logic correct three-dimensional joint reconstruction of human body motion, object movement and static scene under monocular condition. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A flowchart of the three-dimensional human body and scene interactive reconstruction method in the embodiment of the present application;

[0032] Figure 2 A frame diagram of the three-dimensional human body model reconstruction in the embodiment of the present application;

[0033] Figure 3 A result diagram of the drivable human body in the embodiment of the present application;

[0034] Figure 4 A result diagram of the driven human body in the embodiment of the present application;

[0035] Figure 5A schematic diagram of a framework for three-dimensional scene (static scene) model reconstruction in an embodiment of the present application;

[0036] Figure 6 A schematic diagram of a rendering result of a static scene in an embodiment of the present application;

[0037] Figure 7 A schematic diagram of a framework for three-dimensional object (moving object) model reconstruction in an embodiment of the present application;

[0038] Figure 8 A schematic diagram of a rendering result of a moving object in an embodiment of the present application;

[0039] Figure 9 A schematic diagram of a joint rendering result of a human body, a scene and an object in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] Embodiment 1

[0042] This embodiment 1 describes a three-dimensional human body and scene interactive reconstruction method. Through a unified three-dimensional Gaussian expression framework and an optimized calculation process, the method can efficiently complete the three core tasks of human body reconstruction, scene reconstruction and object reconstruction under the premise of ensuring reconstruction accuracy, and significantly improve the overall processing efficiency from monocular video input to three-dimensional reconstruction output.

[0043] In order to ensure the consistency of the human body, the scene and the object in the spatial structure and the time dimension, the method of the present application introduces posture correction, camera external parameter space regularization mechanism, joint reconstruction consistency loss and hierarchical Gaussian fusion rendering based on depth sorting, so that the human body, the scene and the object are accurately aligned in the same world coordinate system.

[0044] Through the above optimization, the method of the present application can stably fit the interactive relationship of the human body, the object and the scene in the real video, dynamically correct the misplacement, drift and occlusion errors caused by monocular input, so that the joint rendering result is highly consistent with the real video frame in terms of geometric structure, motion continuity and lighting style.

[0045] Finally, the method of the present application can generate a multi-agent joint rendering diagram that is complete in detail, visually realistic and correct in spatial logic, thereby realizing high-fidelity reconstruction and presentation of the interactive process of the human body, the object and the scene in a complex dynamic scene.

[0046] As shown in Figure 1 The three-dimensional human body and scene interactive processing method in this embodiment includes the following steps:

[0047] Step 1. Three-dimensional human body model reconstruction.

[0048] The monocular video is input to perform three-dimensional pose estimation, obtain human parameters, generate a canonical mesh consistent with the human topology, and map the vertices of the canonical mesh to a three-plane feature space to extract features of each vertex.

[0049] The extracted vertex features are input into a multi-layer perception network to predict the three-dimensional Gaussian attributes corresponding to each vertex, thereby converting the static canonical mesh into a three-dimensional human Gaussian representation composed of Gaussian primitives.

[0050] The three-dimensional human Gaussian including a face Gaussian and a body Gaussian is spatially transformed using linear blend skinning, and then a human image in a target pose is generated through a three-dimensional Gaussian Splatting rendering pipeline to realize three-dimensional human model reconstruction.

[0051] As Figure 2 The three-dimensional human model reconstruction process is shown, which specifically includes the following steps:

[0052] First, monocular video is used as a data source, and three-dimensional pose estimation is performed on the person in the video through SMPL-X parameter regression and pose estimation to obtain human parameters, and a canonical mesh consistent with the human topology is generated.

[0053] The human parameters include joint pose parameters , human body shape parameters , and facial expression parameters .

[0054] Among them the rotation angle of 55 joints is described to represent the whole body action; as a 100-dimensional body shape vector shared across frames, used to generate an individualized initial body shape; a 50-dimensional expression code used to drive the geometric changes of the facial region.

[0055] The above generated joint pose parameters , human body shape parameters , and facial expression parameters can be used to generate a canonical mesh consistent with the human structure, providing a stable three-dimensional reference space for subsequent appearance and geometric expression.

[0056] After the canonical mesh is constructed, the vertices of the canonical mesh are encoded using a three-plane feature representation, and the position encoding of the canonical mesh is mapped to a three-plane feature space to obtain the features of each vertex.

[0057] Each vertex feature includes the local appearance and fine-grained geometric features of the vertex.

[0058] The three-dimensional pose is obtained by combining the vertex normal and the SMPL-X parameter regression and pose estimation, and the pose is corrected by a multi-layer perception network to compensate for the geometric deviation of the canonical mesh when the human body pose changes greatly.

[0059] Specifically, the specific process of correcting the pose by the multi-layer perception network in the embodiment is as follows:

[0060] First, the vertices, vertex normals and three-dimensional pose parameters of the canonical mesh are collectively used as inputs of the multi-layer perception network, so that the multi-layer perception network can simultaneously perceive the local geometric shape and the global human joint motion state;

[0061] Subsequently, the multi-layer perception network predicts a vertex-level pose correction variable according to the inputs, which is used to represent the real offset of the vertex relative to the canonical mesh under the current pose; the offset is directly added to the vertex position of the canonical mesh, thereby compensating for the deformation error caused by large pose changes, joint bending or local geometric distortion, so that the corrected mesh still maintains a coherent, stable and real human body structure geometric shape in the scenarios of limb movement, torso twisting and the like.

[0062] Since the pose correction can explicitly model the geometric error caused by the pose change, the stability and consistency of the canonical mesh when the pose changes are improved, and the problem that the traditional canonical space based on the mesh is difficult to maintain geometric continuity for complex motions is solved.

[0063] The vertex features of the canonical mesh are input into the multi-layer perception network to predict the corresponding three-dimensional Gaussian attributes of each vertex, thereby converting the static canonical mesh into a three-dimensional human body Gaussian representation composed of Gaussian primitives, thereby avoiding the problems of topology uncertainty or texture drift in the traditional point cloud or radiance field method. The multi-layer perception network predicts the corresponding three-dimensional Gaussian attributes for each vertex through a learnable regression structure, including position fine-tuning, scale parameter and color parameter in the canonical space.

[0064] Meanwhile, thanks to the results of the pose correction, the three-dimensional Gaussian representation has good drivability and can stably reuse the pose and expression space of SMPL-X for deformation, so that the human body still has continuous and consistent three-dimensional detail performance under different poses.

[0065] The reconstructed three-dimensional drivable human body is shown in FIG. 1. Figure 3

[0066] The three-dimensional human Gaussian including the face Gaussian and the body Gaussian is spatially transformed by using linear blend skinning, and combined with a three-dimensional Gaussian Splatting (3DGS) rendering pipeline to accurately map the human pose corresponding to the driving signal to the three-dimensional Gaussian set, realize dynamic deformation modeling of the whole body region, and obtain a three-dimensional human reconstruction model.​

[0067] The driven human body is as shown in Figure 4 The application relies on the differentiable rendering characteristics of Gaussian representation, can obtain high-quality and artifact-free dynamic rendering results while maintaining real-time, makes the mapping relationship of three-dimensional human body Gaussian representation converted from the standard mesh to the target posture human body more stable and reliable, and significantly improves the realism and driving consistency of the dynamic virtual image.

[0068] The facial Gaussian changes with the expression formula as follows:

[0069] ;

[0070] Wherein is the facial Gaussian position after expression superposition, is the standard mesh facial vertex position, , are the identity-related position offset and posture-related position offset of the facial Gaussian respectively; is the facial expression offset.

[0071] The body Gaussian related offset formula is as follows:

[0072] .

[0073] Wherein is the body and hand Gaussian position after posture offset fusion, is the standard mesh body and hand vertex position, , are the identity-related position offset and posture-related position offset of the corresponding parts respectively.

[0074] Step 2. Three-dimensional scene model reconstruction.

[0075] With multi-view scene images and SfM calibrated camera parameters as input, the point cloud of the scene is obtained through colmap estimation. The point cloud is initialized as a 3D Gaussian point, and then the 3D Gaussian point is projected to generate a 2D Gaussian.

[0076] Finally, the 2D Gaussian generates pixel color through Alpha blending, and the three-dimensional scene model reconstruction is completed.

[0077] As Figure 5 shows the three-dimensional scene model reconstruction process, which specifically includes:

[0078] First, with multi-view images of a static scene and SfM calibrated camera parameters as input, the point cloud of the scene is obtained through colmap estimation. The point cloud is initialized as a 3D Gaussian point, which is used as a scene representation unit.

[0079] Then the 3D Gaussian is projected to the image plane to get the projected 2D Gaussian to support rendering.

[0080] The present application aims at the problem of sparsity imbalance in scene reconstruction, and performs a density control operation on the 3D Gaussian points according to a fixed number of iterations in the 3D Gaussian point optimization process, which dynamically determines the retention, deletion or expansion of each Gaussian point according to its contribution degree, color and local distribution.

[0081] Specifically, for 3D Gaussian points with low contribution degree, long time not participating in rendering or high error in multi-view projection, they are deleted from the 3D Gaussian point set to avoid redundancy and cause computational overhead and noise.

[0082] And for areas with strong texture edges, deep dramatic change areas or high error but stable gradient, several 3D Gaussian points are added in the neighborhood of the 3D Gaussian point, so that the area can express the real geometry with higher density.

[0083] Finally, the pixel color is generated by Alpha blending combined with the GPU parallel computing capability, and the three-dimensional scene model reconstruction is completed.

[0084] The 3D Gaussian is used as the scene representation unit, and its mathematical expression is:

[0085] .

[0086] Wherein is the position mean of each Gaussian, is the covariance matrix, which determines the shape and orientation of the Gaussian, is any point in space, represents the density value of the three-dimensional Gaussian point at point .

[0087] The covariance matrix of the projected 2D Gaussian is calculated as follows:

[0088] .

[0089] Wherein represents the covariance matrix of the projected 2D Gaussian, is the view transformation matrix, which converts the 3D Gaussian from the world space to the camera space, is the Jacobian matrix of the projection transformation.

[0090] Finally, the 2D Gaussian is generated by the Alpha blending formula combined with the GPU parallel computing capability to generate the pixel color:

[0091] .

[0092] Wherein, to accumulate transparency, for Gaussian color, to represent opacity, to represent the final pixel color.

[0093] Through the above three-dimensional scene model reconstruction step, the scene rendering diagram obtained is as shown in Figure 6 .

[0094] Step 3. Three-dimensional object model reconstruction.

[0095] Taking monocular video as input, a basic 3D Gaussian set is constructed as a stable prior, a Gaussian frame-by-frame geometric transformation is predicted, pixels are generated through differentiable Gaussian rasterization rendering and optimization, and three-dimensional object model reconstruction is realized.

[0096] As Figure 7 shows a three-dimensional object model reconstruction process, which includes the following steps:

[0097] Taking a monocular video sequence as input data, first, a basic 3D Gaussian point set composed of position, size and color is constructed as a canonical Gaussian, which is used as a stable prior for subsequent dynamic deformation and multi-frame rendering.

[0098] The canonical Gaussian is input into the time-conditioned deformation network, and the time sequence information corresponding to the canonical Gaussian is input at the same time. A multi-layer perception network is used to predict the spatial offset of each Gaussian in the current frame, so as to obtain the frame-by-frame geometric transformation from the canonical space to the observed space. The canonical Gaussian and the predicted spatial offset are added to obtain the deformed Gaussian point parameters.

[0099] The time-conditioned deformation network models the local motion in the dynamic scene in the form of a deep network, so that the scene geometry and appearance can change continuously over time, solving the problem of dynamic detail loss or time discontinuity in previous methods.

[0100] To ensure that the deformation process remains smooth and structurally consistent in the time dimension, an annealing smoothing constraint mechanism is introduced. By gradually reducing the weight of the time smoothing term, the time-conditioned deformation network obtains stable time alignment capability in the early training stage.

[0101] Specifically, a strong time smoothing constraint is added in the early training stage to make the deformation results of adjacent times as close as possible, so that the time-conditioned deformation network learns the global consistent time alignment capability, avoiding shaking, distortion or local time drift.

[0102] However, if a strong time smoothing constraint is maintained throughout the training process, the time-conditioned deformation network's ability to express real dynamic details will be limited. Therefore, the present application uses an annealing strategy to gradually decrease the weight of the time smoothing term during training, allowing the time-conditioned deformation network to freely learn more detailed and faster motion changes in the later stage.

[0103] After obtaining the deformed Gaussian point parameters of the current frame, the deformed Gaussian point set is input into a rendering module, and a three-dimensional Gaussian point is projected onto an image plane through a differentiable rasterization algorithm based on a Gaussian kernel.

[0104] The reconstruction error between the rendered image obtained by projection and the real image is used to minimize the color error between the rendered image and the real image, and the canonical Gaussian, the time-conditioned deformation network and the rendering parameters are optimized in reverse according to this goal to realize the reconstruction of the three-dimensional object model. The obtained object rendering image is as shown in Figure 8 .

[0105] At any time , the deformed Gaussian point parameters are obtained according to the offset output by the time-conditioned deformation network:

[0106] .

[0107] wherein is the output of the time-conditioned deformation network at time t, which dynamically adjusts with the object motion, , , represent the center point coordinates, the rotation direction and the size of the canonical Gaussian respectively, , , represent the center point coordinates, the rotation direction and the size of the deformed Gaussian point respectively.

[0108] Step 4. Optimize the camera parameters and the joint reconstruction consistency loss, and simultaneously render the human body, the scene and the object.

[0109] Taking monocular video as input, the camera intrinsic and extrinsic parameters are inferred, and a robust canonical world coordinate system is established by introducing quaternion orthogonal constraint. The camera extrinsic parameters are optimized through cross-subject consistency loss.

[0110] After the optimization of the camera extrinsic parameters, the human body, the scene and the object are mapped to the camera coordinate system, and are projected onto the image plane. A hierarchical hybrid rendering strategy is adopted to realize the natural fusion of the human body, the object and the scene.

[0111] Through the joint reconstruction consistency loss, the details, the color and the occlusion are optimized, and finally a high-fidelity dynamic rendering image is generated.

[0112] Specifically, the specific process of this step 4 is as follows:

[0113] First, taking the input monocular video sequence as the calibration basis, the initial intrinsic and extrinsic parameters of the camera are inferred through the structure from motion (SfM) algorithm, wherein the intrinsic parameters include the focal length and the principal point position, and the extrinsic parameters are composed of a rotation matrix and a translation vector.

[0114] To ensure that the three types of 3D Gaussians of human body, scene and object can be uniformly rendered in the same space system, the application introduces a space regularization constraint mechanism in the camera extrinsic parameter calibration stage.

[0115] The space regularization constraint mechanism ensures that the rotation matrix always maintains orthogonality in the optimization process through quaternion reparameterization, so that it always falls within the real scene volume, thereby establishing a robust "canonical world coordinate system".

[0116] Compared with the traditional method of directly using the SfM result, the structural constraint (space regularization constraint mechanism) introduced in the calibration stage of the application effectively avoids the mis-calibration caused by dynamic human body or fast-moving object.

[0117] The space regularization parameters include a quaternion for parameterizing the camera rotation matrix; and a camera translation parameter .

[0118] Based on the unified space, the three-dimensional human body model, the three-dimensional object model and the three-dimensional scene model are respectively input into the extrinsic parameter optimization module, and the pixel consistency of the joint rendering image and the real video frame is used as a supervision signal to reversely optimize the camera extrinsic parameter.

[0119] In addition, during the camera extrinsic parameter optimization process, the application designs a cross-subject consistency loss for simultaneously constraining the geometric projection relationship of the three types of Gaussian models of human body, scene and object to remain consistent in space.

[0120] The cross-subject consistency loss consists of two parts, which are: a projection consistency term for ensuring that the human body boundary, object boundary and scene geometry remain aligned on the projection plane; and a time sequence stability term for suppressing the rendering drift caused by jumping and jittering.

[0121] The cross-subject consistency loss optimization strategy significantly improves the stability and cross-frame continuity of the extrinsic parameter tracking.

[0122] The formula expression of the projection consistency term is as follows:

[0123] .

[0124] Wherein represents the projection consistency term, , , represent the two-dimensional centers of the three-dimensional human body, the three-dimensional object and the three-dimensional scene Gaussian projected onto the image plane, respectively; represents the L1 norm, which is the sum of the absolute values of the vector.

[0125] The formula expression of the time sequence stability term is as follows:

[0126] .

[0127] where denotes the temporal stabilization term, 、 denotes the neighboring frame extrinsic change; denotes the Frobenius norm, which computes the square root of the sum of squares of matrix elements; denotes the L2 norm, which computes the square root of the sum of squares of a vector.

[0128] After the camera extrinsic parameters converge, the three-dimensional human Gaussian and the three-dimensional object Gaussian are mapped to the canonical world coordinate system through their respective dynamic deformation, and are naturally aligned with the static Gaussian of the scene.

[0129] Subsequently, the human, scene and object Gaussians are all converted to the camera coordinate system using the optimized camera extrinsic matrix, and are projected onto the image plane based on the camera intrinsic parameters.

[0130] After projection, a hierarchical hybrid rendering strategy is proposed to globally sort the human, scene and object Gaussians according to the camera space depth. This sorting makes the scene have the lowest priority, and the human and object have higher priority, which visually conforms to the natural occlusion relationship in the real world "background → human → object", and also avoids sorting errors caused by the mixing of different Gaussians during rendering. Finally, color synthesis is completed through differentiable Gaussian Alpha blending.

[0131] The hierarchical hybrid rendering parameters include , which denotes the depth of the i-th Gaussian point in the camera space.

[0132] In the interfacial areas of the human, scene and object Gaussians, a continuous boundary transition mechanism based on Gaussian covariance is used to automatically realize the natural fusion of the human, object and scene, so that the rendering result maintains a unified visual style.

[0133] Define denotes the boundary transition function, denotes a globally adjustable hyperparameter, which is used to control the decay rate of the boundary transition; denotes the scale of the i-th Gaussian covariance matrix, which is used to reflect the expansion size of the Gaussian in space.

[0134] In addition, in order to further improve the rendering quality, the present application also designs a joint reconstruction consistency loss, which unifies the differences of the three types of Gaussians into the same optimization target, so that the finally generated joint rendering image achieves high fidelity effect in detail continuity, color consistency and occlusion relationship.

[0135] The joint reconstruction consistency loss includes an occlusion consistency constraint and a multi-scale pixel reconstruction loss, and is expressed as follows:

[0136] .

[0137] wherein denotes the joint reconstruction consistency loss, denotes the multi-scale pixel reconstruction loss, denotes the occlusion consistency loss, , are respectively , corresponding weights of each loss.

[0138] .

[0139] wherein denotes a result of the rendered image at the kth scale, denotes an image obtained by scaling the real video frame to the same scale, denotes an L1 norm, and the absolute values of a vector are summed.

[0140] .

[0141] wherein denotes a foreground mask obtained by rendering, denotes a real foreground mask.

[0142] The final output joint rendered image is as shown in Figure 9 .

[0143] The method can quickly complete three core tasks of human body reconstruction, scene reconstruction and object reconstruction from monocular video under the premise of ensuring reconstruction accuracy, generate a joint rendered image which is rich in details, visually realistic and consistent with spatial logic, and provide high-quality basic data support for subsequent virtual simulation, film production, virtual reality and other fields of application.

[0144] Embodiment 2

[0145] This embodiment 2 describes a three-dimensional human body and scene interactive reconstruction system, which is based on the same inventive concept as the three-dimensional human body and scene interactive reconstruction method in the above embodiment 1.

[0146] The three-dimensional human body and scene interactive reconstruction system in this embodiment includes the following modules:

[0147] The three-dimensional human body reconstruction module is used for three-dimensional pose estimation with monocular video as input, obtaining human body parameters, generating a standard mesh consistent with the topology of the human body, and mapping the standard mesh vertices to a three-plane feature space to extract features of each vertex.

[0148] The extracted vertex features are input into a multi-layer perception network to predict the three-dimensional Gaussian attributes corresponding to each vertex, thereby converting the static canonical mesh into a three-dimensional human body Gaussian representation composed of Gaussian primitives;

[0149] Finally, the Gaussian points of the three-dimensional human body Gaussian representation are spatially transformed using linear blend skinning, and a three-dimensional Gaussian Splatting rendering pipeline is used to generate a human body image in a target pose, thereby realizing three-dimensional human body model reconstruction;

[0150] The three-dimensional scene model reconstruction module is configured to input a multi-view scene image and SfM calibrated camera parameters, estimate a scene point cloud through colmap, initialize the scene point cloud as a 3D Gaussian point, project the 3D Gaussian point to generate a 2D Gaussian, and finally generate pixel colors through Alpha blending to complete three-dimensional scene model reconstruction;

[0151] The three-dimensional object model reconstruction module is configured to input a monocular video, construct a basic 3D Gaussian set as a stable prior, predict Gaussian frame-by-frame geometric transformation, render pixels through differentiable Gaussian rasterization, and optimize to realize three-dimensional object model reconstruction;

[0152] The interactive reconstruction module is configured to input a monocular video, infer camera intrinsic and extrinsic parameters, introduce a quaternion orthogonal constraint to establish a robust canonical world coordinate system, and optimize the camera extrinsic parameters through cross-subject consistency loss;

[0153] After the camera extrinsic parameter optimization is completed, the human body, scene, and object Gaussian mappings are projected to the image plane, and a hierarchical blending rendering strategy is used to realize the natural fusion of the human body, object, and scene;

[0154] Through joint reconstruction consistency loss optimization of details, colors, and occlusions, a high-fidelity dynamic rendering image is finally generated.

[0155] It should be noted that the contents not mentioned in the above-mentioned various functional modules of the system described in Embodiment 2 can be referred to the description of the steps of the corresponding method in Embodiment 1 above, and will not be described in detail here.

[0156] Of course, the above description is only for the preferred embodiments of the present application, and the present application is not limited to the above-mentioned embodiments. It should be noted that any skilled person in the art can make all equivalent substitutions and obvious modifications under the guidance of the present application, which are within the scope of the present application, and should be protected by the present application.

Claims

1. A method for interactive reconstruction of a three-dimensional human body and scene, characterized in that, Includes the following steps: Step 1. Perform 3D pose estimation using monocular video as input, obtain human body parameters, generate a standardized mesh consistent with human body topology, and map the vertices of the standardized mesh to a three-plane feature space to extract the features of each vertex; The extracted vertex features are input into a multilayer perceptron network to predict the three-dimensional Gaussian properties corresponding to each vertex, thereby converting the static canonical mesh into a three-dimensional human Gaussian representation composed of Gaussian elements. The three-dimensional human body is reconstructed by using linear blending skin to perform spatial transformation on the Gaussian model of the three-dimensional human body and generating human body images of the target pose through the three-dimensional Gaussian Splatting rendering pipeline. Step 2. Using the multi-view scene image and SfM-calibrated camera parameters as input, the point cloud of the scene is obtained through colmap estimation. The point cloud is initialized as 3D Gaussian points, and then the 3D Gaussian points are projected to generate 2D Gaussian points. Finally, the 2D Gaussian color is generated by Alpha blending to complete the reconstruction of the 3D scene model; Step 3. Using monocular video as input, construct a basic 3D Gaussian set as a stable prior, predict Gaussian frame-by-frame geometric transformation, generate pixels through differentiable Gaussian rasterization rendering and optimize, and realize the reconstruction of the 3D object model; Step 4. Using monocular video as input, infer camera intrinsic and extrinsic parameters, and introduce quaternion orthogonal constraints to establish a robust and standardized world coordinate system. Optimize camera extrinsic parameters through cross-subject consistency loss. After the camera extrinsic parameters are optimized, the three types of Gaussian mappings of human body, scene and object are mapped to the camera coordinate system and projected onto the image plane. A layered blending rendering strategy is used to achieve natural fusion of human body, object and scene. By jointly reconstructing consistency loss to optimize details, colors, and occlusions, a high-fidelity dynamic rendering image is ultimately generated.

2. The three-dimensional human body and scene interactive reconstruction method according to claim 1, characterized in that, Step 1 specifically involves: First, using monocular video as the data source, the three-dimensional pose of the person in the video is estimated by SMPL-X parameter regression and pose estimation to obtain human body parameters and generate a standardized mesh consistent with the human body topology. The vertices of the normalized mesh are encoded using a three-plane feature representation. By mapping the position encoding of the normalized mesh to the three-plane feature space, the features of each vertex are obtained, including the local appearance and geometric features of the vertex. Simultaneously, by combining the vertex normals and human body parameters obtained from SMPL-X parameter regression and pose estimation, pose correction is performed through a multilayer perceptron network to compensate for the geometric deviations of the normalized mesh when the human body pose changes significantly. The vertex features of the normalized mesh are input into a multilayer perceptron network to predict the three-dimensional Gaussian properties corresponding to each vertex, thereby converting the static normalized mesh into a three-dimensional human Gaussian representation composed of Gaussian elements. The pose and expression space of SMPL-X is reused to deform the 3D human Gaussian representation to obtain the driving signal; A spatial transformation of the 3D human body Gaussian is performed using linear blending skin. Then, combined with the 3D Gaussian Splatting rendering pipeline, the human body pose corresponding to the driving signal is accurately mapped to the 3D Gaussian set to obtain a 3D human body reconstruction model.

3. The three-dimensional human body and scene interactive reconstruction method according to claim 2, characterized in that, In step 1, the human body parameters include joint posture parameters. Human body shape parameters and facial expression parameters ; in Describe the rotation angles of 55 joints to characterize whole-body movements; As a 100-dimensional body shape vector shared across frames, it is used to generate individualized initial body shapes; Encodes 50-dimensional facial expressions to drive geometric changes in facial regions; The 3D human body Gaussian includes two main parts: the facial Gaussian and the body Gaussian. The formula for the Gaussian variation of facial expressions is expressed as follows: ； in This represents the Gaussian position of the face after the expression is overlaid. To standardize the position of the face vertices in the mesh, , These are the identity-related positional offset and pose-related positional offset of the facial Gaussian face, respectively. Facial expression shift; The Gaussian-related offset formula for the body is expressed as follows: ； in The Gaussian positions of the body and hands after pose shift fusion. To standardize the positions of the body and hand vertices in the mesh, , These are the identity-related position offset and attitude-related position offset for the corresponding parts.

4. The three-dimensional human body and scene interactive reconstruction method according to claim 1, characterized in that, Step 2 specifically involves: First, the multi-view image of the static scene and the camera parameters after SfM calibration are used as input. The point cloud of the scene is obtained by colmap estimation. Each point cloud is initialized with a 3D Gaussian point as the scene representation unit. The 3D Gaussian is then projected onto the image plane to obtain the projected 2D Gaussian to support the rendering process; To address the density imbalance issue in scene reconstruction, density control operations are performed on 3D Gaussian points during the 3D Gaussian point optimization process at a fixed number of iterations, including the deletion and addition of Gaussian points. Finally, 2D Gaussian is used to generate pixel colors through Alpha blending, thus completing the reconstruction of the 3D scene model.

5. The three-dimensional human body and scene interactive reconstruction method according to claim 4, characterized in that, In step 2, a 3D Gaussian is used as the scene representation unit, and its mathematical expression is: ； in The mean value for each Gaussian position. The covariance matrix determines the shape and orientation of the Gaussian. For any point in space, This indicates that the three-dimensional Gaussian point is at point... Density value at; The covariance matrix of the projected 2D Gaussian is calculated as follows: ； in This represents the covariance matrix of the projected 2D Gaussian. The view transformation matrix transforms the 3D Gaussian from world space to camera space. Let be the Jacobian matrix of the projection transformation; Finally, leveraging the parallel computing capabilities of the GPU, the 2D Gaussian color is generated using the Alpha mixing formula: ； in To accumulate transparency, Gaussian color, Indicates opacity. This indicates the final pixel color.

6. The three-dimensional human body and scene interactive reconstruction method according to claim 1, characterized in that, Specifically, step 3 is as follows: Using monocular video sequences as input data, a basic 3D Gaussian point set consisting of position, scale, and color is first constructed as a normalized Gaussian, which serves as a stable prior for subsequent dynamic deformation and multi-frame rendering. The normalized Gaussian is input into the temporal conditional deformation network. By utilizing temporal information, the spatial offset of each 3D Gaussian point in the current frame is predicted using a multilayer perceptron, resulting in a frame-by-frame geometric transformation from the normalized space to the observation space. The deformed Gaussian point parameters are obtained by adding the normalized Gaussian and the predicted spatial offset. To ensure that the deformation process remains smooth and consistent with the structure in the time dimension, an annealing smoothing constraint mechanism is introduced. By gradually reducing the weight of the time smoothing term, the time-conditional deformation network can obtain stable time alignment capability in the early stages of training. After obtaining the Gaussian point parameters after deformation in the current frame, the set of deformed Gaussian points is input into the rendering module, and the three-dimensional Gaussian points are projected onto the image plane through a differentiable rasterization algorithm based on Gaussian kernel. The final pixel values ​​are synthesized using a depth sorting and color blending strategy. By analyzing the reconstruction error between the observed image and the rendered image, the normalized Gaussian network, the temporal conditional deformation network, and the rendering parameters are optimized in reverse to achieve the reconstruction of the 3D object model.

7. The three-dimensional human body and scene interactive reconstruction method according to claim 1, characterized in that, Step 4 specifically involves: First, the initial intrinsic and extrinsic parameters of the camera are inferred using the input monocular video sequence as the calibration basis and the structure for motion recovery (SfM) algorithm. The intrinsic parameters include the focal length and principal point position, while the extrinsic parameters consist of the rotation matrix and the translation vector. In the calibration phase, a spatial regularization constraint mechanism is introduced. Quaternion reparameterization ensures that the rotation matrix remains orthogonal throughout the optimization process, ensuring that it always falls within the volume of the real scene and establishing a robust standard world coordinate system. By using the pixel consistency between the jointly rendered image and the real video frame as a supervision signal, the camera extrinsic parameters are optimized in reverse. During the camera extrinsic optimization process, a cross-subject consistency loss is designed to simultaneously constrain the geometric projection relationship of the three types of Gaussian models—human body, scene, and object—in space. After the camera extrinsic converges, the 3D human body Gaussian and the 3D object Gaussian are mapped to the normalized world coordinate system through their respective dynamic deformations, naturally aligning with the static Gaussian of the scene. Then, the optimized camera extrinsic matrix was used to transform all three types of Gaussians—human body, scene, and object—to the camera coordinate system, and the three types of Gaussians were projected onto the image plane based on the camera intrinsic parameters. After projection, a layered hybrid rendering strategy is adopted. The three types of Gaussian elements, namely human body, scene and object, are globally sorted according to camera space depth. Color synthesis is completed by differentiable Gaussian Alpha fusion. In the three Gaussian border areas of human body, scene and object, a continuous boundary transition mechanism based on Gaussian covariance is adopted to automatically realize the natural integration of human body, scene and object, so that the rendering results maintain a unified visual style. The design incorporates a joint reconstruction consistency loss, unifying the differences among the three types of Gaussian rendering into a single optimization objective, so that the final jointly rendered image achieves high fidelity in terms of detail continuity, color consistency, and occlusion relationships.

8. The three-dimensional human body and scene interactive reconstruction method according to claim 7, characterized in that, In step 4, the cross-entity consistency loss consists of two parts: The projection consistency term is used to ensure that human body boundaries, object boundaries, and scene geometry remain aligned on the projection plane; Timing stability term, used to suppress rendering drift caused by jumps and jitter; The formula for the projection consistency term is expressed as follows: ； in Indicates the projection consistency term. , , These are respectively represented as the two-dimensional centers of the Gaussian projection of the 3D human body, 3D object, and 3D scene onto the image plane; Denotes the L1 norm, which is the sum of the absolute values ​​of vectors; The formula for the time-stability term is expressed as follows: ； in Represents time-stable terms. , Indicates the change in extrinsic parameters between adjacent frames; This represents the Frobenius norm, which is used to calculate the square root of the sum of squares of the matrix elements. Let L2 norm be used to calculate the square root of the sum of squares of a vector.

9. The three-dimensional human body and scene interactive reconstruction method according to claim 7, characterized in that, In step 4, the formula for the joint reconstruction consistency loss is expressed as follows: ； in This indicates a loss of consistency in joint reconstruction. This represents the multi-scale pixel reconstruction loss. This indicates the loss of consistency due to occlusion. , They are respectively , The weights corresponding to each type of loss; ； in Indicates the rendered image at the 1st Results at each scale This represents an image of a real video frame scaled to the same size. Denotes the L1 norm, which is the sum of the absolute values ​​of vectors; ； in This represents the foreground mask obtained from rendering. Represents the true foreground mask.

10. A three-dimensional human body and scene interactive reconstruction system, characterized in that, Includes the following modules: The 3D human body reconstruction module is used to perform 3D pose estimation with monocular video as input, obtain human body parameters, generate a standardized mesh consistent with human body topology, and map the vertices of the standardized mesh to a three-plane feature space to extract the features of each vertex. The extracted vertex features are input into a multilayer perceptron network to predict the three-dimensional Gaussian properties corresponding to each vertex, thereby converting the static canonical mesh into a three-dimensional human Gaussian representation composed of Gaussian elements. The three-dimensional human body Gaussian model is reconstructed by using linear blending skin to perform spatial transformation on the Gaussian skin of the face and body. Then, the human body image of the target pose is generated through the three-dimensional Gaussian Splatting rendering pipeline. The 3D scene model reconstruction module takes multi-view scene images and SfM-calibrated camera parameters as input, estimates the point cloud of the scene through colmap, initializes the scene point cloud as 3D Gaussian points, projects the 3D Gaussian points to generate 2D Gaussian points, and finally generates pixel colors through alpha mixing of the 2D Gaussian points to complete the 3D scene model reconstruction. The 3D object model reconstruction module is used to construct a basic 3D Gaussian set as a stable prior using monocular video as input, predict Gaussian frame-by-frame geometric transformation, generate pixels through differentiable Gaussian rasterization rendering and optimization, and realize the reconstruction of the 3D object model. And an interactive reconstruction module, which uses monocular video as input to infer camera intrinsic and extrinsic parameters, and introduces quaternion orthogonal constraints to establish a robust and standardized world coordinate system, and optimizes camera extrinsic parameters through cross-subject consistency loss; After the camera extrinsic parameters are optimized, the three types of Gaussian mappings of human body, scene and object are mapped to the camera coordinate system and projected onto the image plane. A layered blending rendering strategy is used to achieve natural fusion of human body, object and scene. By jointly reconstructing consistency loss to optimize details, colors, and occlusions, a high-fidelity dynamic rendering image is ultimately generated.

Images