Human-centered video scene reconstruction and separation method, system, medium and device

Abstract

This application provides a method, system, medium, and device for human-centered video scene reconstruction and separation. The method includes: for a first-person perspective video sequence, initializing a 3D Gaussian set covering the background, hands, and objects based on a priori knowledge of structure recovery from motion; assigning a learnable dynamic category probability vector to each Gaussian point in the 3D Gaussian set; constructing a dedicated deformation branch; according to the learnable dynamic category probability vector of each Gaussian point, assigning each Gaussian point to the dedicated deformation branch for processing through a preset soft-hard two-stage routing mechanism, determining the Gaussian points processed by the dedicated deformation branch; rendering the Gaussian points processed by the dedicated deformation branch to determine the 4D scene reconstruction image and the decomposed reconstruction images of the background, hands, and objects. This application achieves 4D scene reconstruction of human-centered video and explicit, fine-grained separation of the background, hands, and objects.

Description

Technical Field

[0001] This application relates to the fields of computer vision and computer graphics technology, and more specifically, to a method, system, medium, and device for human-centered video scene reconstruction and separation. Background Technology

[0002] With the widespread adoption of head-mounted cameras and AR / VR devices, human-centered video has become an important window into understanding human-environment interaction. Reconstructing dynamic 4D (3D spatial + temporal) scenes from such videos is crucial for achieving immersive experiences and intelligent agent-environment interaction.

[0003] In recent years, Neural Radiation Field (NeRF) and 3D Gaussian Splatting techniques have made significant progress in dynamic scene reconstruction. 3D Gaussian Splatting, in particular, has attracted attention due to its high-quality rendering and efficient rasterization. Its dynamic scaling method models the deformation of Gaussian points over time by introducing a deformation network or HexPlane encoder.

[0004] However, existing dynamic reconstruction schemes have significant shortcomings when dealing with complex human-centered videos.

[0005] First, most methods use a single network to process the entire scene, which not only increases computational overhead but also fails to learn the independent motion patterns of different components in the scene (such as static backgrounds, dynamic hands, and objects). Second, existing scene decomposition methods often assume a fixed camera viewpoint or can only perform coarse foreground and background separation. For example, some methods rely on manually segmenting video clips, which is time-consuming and laborious; while other methods can automatically separate moving and static components, their pixel-intensity-based masks fail in violent self-motion modes, and they fail to fully utilize scene priors when initializing dynamic regions, resulting in coarse separation and an inability to further distinguish hands from objects.

[0006] For example, in the patent "CN109272513B; Hand and Object Interaction Segmentation Method and Apparatus Based on Depth Camera", it achieves static segmentation of background, hand and object through image alignment, background extraction, HSV color segmentation, morphological filtering and contour extraction. It adopts traditional image processing, ignores motion law learning, and cannot adapt to the independent movement of different parts in dynamic scenes. In addition, it relies on depth information for background removal, the camera view is fixed, and there is no compensation mechanism to adapt to changes in camera view. Its separation accuracy depends on the large color contrast between the object and the hand. If the colors are similar, the segmentation boundary is blurred and there is roughness. Moreover, it does not introduce any scene prior information, and selects the effective area by "contour area and perimeter ratio". There is no prior support when initializing dynamic areas, and the segmentation accuracy is greatly affected by the scene complexity.

[0007] Meanwhile, reconstruction methods specifically designed for hand-object interaction typically rely on predefined 3D models, complex optimization processes, or multi-view calibration settings, making it difficult to effectively scale up in unconstrained, human-centered scenes with intense movement and frequent occlusion.

[0008] Therefore, there is an urgent need in this field for a technical solution that can overcome the above-mentioned defects and achieve high-quality 4D reconstruction of human-centered scenes without the need for prior models of hands or objects, while simultaneously completing fine-grained, explicit decomposition of background, hands, and objects. Summary of the Invention

[0009] In view of the deficiencies in the prior art, the purpose of this application is to provide a human-centered video scene reconstruction and separation method, system, medium and device.

[0010] The first aspect of this application provides a human-centered video scene reconstruction and separation method, including: For first-person perspective video sequences, based on the prior of reconstructing structure from motion, a 3D Gaussian set covering the background, hands, and objects is initialized; Assign a learnable dynamic class probability vector to each Gaussian point in the 3D Gaussian set, and determine the learnable dynamic class probability vector for each Gaussian point. Construct dedicated deformation branches, including a background-specific deformation branch, a hand-specific deformation branch, and an object-specific deformation branch; Based on the learnable dynamic category probability vector of each Gaussian point, each Gaussian point is assigned to the dedicated deformation branch for processing through a preset soft-hard two-stage routing mechanism, and the Gaussian points processed by the dedicated deformation branch are determined. The Gaussian points processed by the dedicated deformation branch are rendered to determine the 4D scene reconstruction image and the decomposed reconstruction images of the background, the hand, and the object.

[0011] Optionally, for the first-person perspective video sequence, initializing a 3D Gaussian set covering the background, hands, and objects based on a priori reconstruction of structure from motion includes: The Colmap method is used to perform motion-to-structure reconstruction on the first-person perspective video sequence to determine the camera parameters and initialize the point cloud for each frame. Based on the initialized point cloud, perform 3D Gaussian initialization to determine the initialized 3D Gaussian; The camera parameters of each frame image are used to optimize the initial 3D Gaussian, and the 3D Gaussian set covering the background, hand, and object is determined.

[0012] Optionally, the construction of a dedicated deformable branch includes: A static deformation branch is used as the background-specific deformation branch, and the static deformation branch is an identity mapping branch; The hand-specific deformation branch is constructed based on the shared HexPlane spatiotemporal encoder and the first MLP decoder. The shared HexPlane spatiotemporal encoder is used to extract the spatiotemporal features of the hand, and the first MLP decoder is used to predict the Gaussian deformation of the hand. Based on the shared HexPlane spatiotemporal encoder and the second MLP decoder, a deformation branch for the object is constructed. The shared HexPlane spatiotemporal encoder is also used to extract the spatiotemporal features of the object, and the second MLP decoder is used to predict the Gaussian deformation of the object.

[0013] Optionally, the preset soft-hard two-stage routing mechanism includes a soft-gating stage and a hard-gating stage, with the soft-gating stage executed first, followed by the hard-gating stage. The step of assigning each Gaussian point to a dedicated deformation branch for processing based on the learnable dynamic category probability vector of each Gaussian point through a preset soft-hard two-stage routing mechanism, and determining the Gaussian points processed by the dedicated deformation branch, includes: In the soft gating stage, each Gaussian point is simultaneously input into the background-specific deformation branch, the hand-specific deformation branch, and the object-specific deformation branch to determine the deformation amount of each specific deformation branch. The deformation amount of each dedicated deformation branch is weighted and fused using the learnable dynamic category probability vector of each Gaussian point to determine the deformation amount of each Gaussian point. The learnable dynamic category probability vector of each Gaussian point includes a learnable dynamic category probability vector belonging to the background category, a learnable dynamic category probability vector belonging to the object category, and a learnable dynamic category probability vector belonging to the hand category. Each Gaussian point is self-updated based on the deformation amount of each Gaussian point to determine each updated Gaussian point. In the hard gating stage, a category label is assigned to each updated Gaussian point based on the largest category probability vector among the learnable dynamic category probability vectors corresponding to each updated Gaussian point, thereby determining the category of each updated Gaussian point. The index time of each updated Gaussian point and the first-person perspective video sequence are input into the dedicated deformation branch corresponding to the category of each updated Gaussian point to determine the time-aware Gaussian deformation amount corresponding to the category of each updated Gaussian point. Each updated Gaussian point is updated based on the time-aware Gaussian deformation amount corresponding to the category of each updated Gaussian point, and the Gaussian points processed by the dedicated deformation branch are determined. The Gaussian points processed by the dedicated deformation branch include Gaussian points processed by the background dedicated deformation branch, Gaussian points processed by the hand dedicated deformation branch, and Gaussian points processed by the object dedicated deformation branch.

[0014] Optionally, rendering the Gaussian points processed by the dedicated deformation branch to determine the 4D scene reconstruction image and the decomposed reconstruction images of the background, the hand, and the object includes: Render all Gaussian points processed by the dedicated deformation branch to determine the 4D scene reconstruction image; Render the Gaussian points processed by the background-specific deformation branch to determine the decomposed and reconstructed image of the background; The Gaussian points processed by the hand-specific deformation branch are rendered to determine the decomposed and reconstructed image of the hand. The Gaussian points processed by the object-specific deformation branch are rendered to determine the decomposed and reconstructed image of the object.

[0015] Optionally, the method further includes: During the rendering process, a brightness map is generated based on the learnable brightness properties of the Gaussian points processed by the background-specific deformation branch. A preset piecewise function is used to perform brightness compensation on the brightness map to determine the brightness map after brightness compensation.

[0016] Optionally, the method further includes: During the rendering process, dynamic optical flow is determined based on the difference between real optical flow and camera motion optical flow; The predicted optical flow of the hand and the object is optimized based on the dynamic optical flow.

[0017] Optionally, the method further includes: Based on the segmentation masks of the hand and the object, the reconstruction loss of the hand and the object, as well as the transparency loss of the hand and the object, are determined, and the decomposed and reconstructed images of the hand and the object are optimized.

[0018] Optionally, the method further includes: When calculating gradients during backpropagation, morphological dilation is performed on the non-currently learned category mask to determine the dilated mask region; Gradient truncation is performed on the currently learned category mask within the inflated mask region.

[0019] A second aspect of this application provides a human-centered video scene reconstruction and separation system, comprising: The 3D Gaussian set initialization module is used to initialize a 3D Gaussian set covering the background, hands, and objects for a first-person view video sequence, based on the prior of recovering structure from motion. The dynamic category probability vector allocation module is used to assign a learnable dynamic category probability vector to each Gaussian point in the 3D Gaussian set, and to determine the learnable dynamic category probability vector for each Gaussian point. A dedicated deformation branch construction module is used to construct dedicated deformation branches, which include a background-specific deformation branch, a hand-specific deformation branch, and an object-specific deformation branch. The decomposition deformation module is used to assign each Gaussian point to the dedicated deformation branch for processing based on the learnable dynamic category probability vector of each Gaussian point through a preset soft-hard two-stage routing mechanism, and to determine the Gaussian points processed by the dedicated deformation branch. The scene reconstruction and decomposition reconstruction module is used to render the Gaussian points processed by the dedicated deformation branch to determine the 4D scene reconstruction image and the decomposition reconstruction images of the background, the hand, and the object.

[0020] A third aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the methods provided in the first aspect of this application.

[0021] A fourth aspect of this application provides an electronic device comprising: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of any of the methods provided in the first aspect of this application.

[0022] The human-centered video scene reconstruction and separation method of this application introduces a learnable dynamic category probability vector for each Gaussian point in a 3D Gaussian set and constructs dedicated deformation branches for background, hands, and objects. Through a preset soft-hard two-stage routing mechanism, each Gaussian point is assigned to a dedicated deformation branch for processing, and Gaussian points processed by the dedicated deformation branches are obtained. Finally, each type of Gaussian point processed by the dedicated deformation branch is rendered, realizing 4D scene reconstruction of human-centered video and explicit and fine separation of background, hands, and objects. It overcomes the common problems of blurred boundaries and component adhesion in hand-object interaction areas, and provides an effective tool for human-centered interactive video scene understanding and interactive editing.

[0023] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0024] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a human-centered video scene reconstruction and separation method according to an exemplary embodiment.

[0025] Figure 2 This is a schematic diagram illustrating a human-centered video scene reconstruction and separation process according to an exemplary embodiment.

[0026] Figure 3 This is a schematic diagram showing a comparison between the reconstruction effect of a human-centered video scene reconstruction and separation method and a baseline model, according to an exemplary embodiment.

[0027] Figure 4 This is a schematic diagram showing a comparison of the separation effect of a human-centered video scene reconstruction and separation method with a baseline model, according to an exemplary embodiment.

[0028] Figure 5 This is a schematic diagram illustrating the structure of a human-centered video scene reconstruction and classification system according to an exemplary embodiment. Detailed Implementation

[0029] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0030] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0031] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0032] Existing dynamic reconstruction methods use a single network to process the entire scene, increasing computational overhead and failing to learn the independent motion patterns of different components within the scene. Some scene decomposition methods assume a fixed camera viewpoint, achieving only coarse foreground-background separation. Other methods rely on manual video segmentation, which is time-consuming and labor-intensive. Some automatic scene decomposition methods fail in pixel-intensity-based masks during intense self-motion, and their initialization of dynamic regions fails to fully utilize scene priors, resulting in coarse separation and an inability to further distinguish hands from objects. To address these issues, this application provides a human-centered video scene reconstruction method to resolve these problems.

[0033] Figure 1 This is a flowchart illustrating a human-centered video scene reconstruction and separation method according to an exemplary embodiment. Figure 2 This is a schematic diagram illustrating a human-centered video scene reconstruction and separation process according to an exemplary embodiment.

[0034] Reference Figure 1 , Figure 2 As shown in the figure, one embodiment of this application provides a human-centered video scene reconstruction and separation method, including S11 to S15.

[0035] S11, for first-person perspective video sequences, initializes a 3D Gaussian set covering the background, hands, and objects based on the prior of recovering structure from motion.

[0036] S12 assigns a learnable dynamic class probability vector to each Gaussian point in the 3D Gaussian set, thus determining the learnable dynamic class probability vector for each Gaussian point.

[0037] S13, construct a dedicated deformation branch.

[0038] Specifically, the special deformation branches include background-specific deformation branches, hand-specific deformation branches, and object-specific deformation branches.

[0039] Among them, the background-specific deformation branch is used to predict the amount of background deformation, the hand-specific deformation branch is used to predict the amount of hand deformation, and the object-specific deformation branch is used to predict the amount of object deformation.

[0040] S14. Based on the learnable dynamic category probability vector of each Gaussian point, each Gaussian point is assigned to a dedicated deformation branch for processing through a preset soft-hard two-stage routing mechanism, and the Gaussian points processed by the dedicated deformation branch are determined.

[0041] S15 renders the Gaussian points processed by a dedicated deformation branch to determine the 4D scene reconstruction image and the decomposed reconstruction images of the background, hands, and objects.

[0042] The embodiments described above in this application introduce a learnable dynamic category probability vector for each Gaussian point in a 3D Gaussian set and construct dedicated deformation branches for the background, hand, and object. Through a preset soft-hard two-stage routing mechanism, each Gaussian point is assigned to a dedicated deformation branch for processing, and Gaussian points processed by the dedicated deformation branches are obtained. Finally, each type of Gaussian point processed by the dedicated deformation branch is rendered, realizing 4D scene reconstruction of human-centered video and explicit and fine separation of background, hand, and object. This overcomes the common problems of blurred boundaries and component adhesion in the hand-object interaction area, providing an effective tool for human-centered interactive video scene understanding and interactive editing.

[0043] In order to initialize the 3D Gaussian set of the first-person perspective video sequence, in some specific embodiments of this application, for S11, for the first-person perspective video sequence, based on the prior of recovering the structure from motion, the initialization of the 3D Gaussian set covering the background, hand and object can be carried out by: S111 to S113.

[0044] S111 uses the Colmap method to perform motion recovery processing on the first-person view video sequence to determine the camera parameters and initialize the point cloud for each frame.

[0045] Specifically, the camera parameters for each frame include camera intrinsics and camera extrinsics. The camera intrinsics include focal lengths fx and fy (in pixels), principal points cx and cy (in pixels), and distortion coefficients. The camera extrinsics include rotation matrices (or quaternions) and translation vectors representing the camera pose.

[0046] S112, Perform 3D Gaussian initialization based on the initial point cloud, and determine the initial 3D Gaussian.

[0047] S113, use the camera parameters of each frame image to optimize the initialization of 3D Gaussian, and determine the 3D Gaussian set covering the background, hand and object.

[0048] Specifically, steps S112 to S113 above correspond to the standard initialization process in the 3D Gaussian splashing method, which will not be elaborated here. The sparse point cloud from motion recovery structure is converted into a 3D Gaussian set with position, color, scaling, rotation and opacity parameters, providing initial geometry and appearance priors for subsequent 4D scene reconstruction and separation of the three levels of hand, object and background.

[0049] The embodiments described above in this application, based on the prior of reconstructing the structure from motion, can obtain the camera view of each frame of the image for 3D Gaussian learning and optimization, and can obtain the point cloud prior of the entire scene, that is, the 3D Gaussian set covering the back, hands and objects.

[0050] To introduce learnable dynamic category probability vectors, in some specific embodiments of this application, for S12, a learnable dynamic category probability vector is assigned to each Gaussian point in the 3D Gaussian set. The learnable dynamic category probability vector for each Gaussian point can be determined by: The learnable dynamic class probability of each Gaussian point is used to characterize the likelihood that the Gaussian point belongs to the background, hand, or object, and also represents the probability distribution of the Gaussian point belonging to the corresponding class.

[0051] Specifically, the sum of the learnable dynamic class probability vectors for each Gaussian point is 1:

[0052] in, This represents a learnable, dynamic class probability vector for a Gaussian point. This indicates that the Gaussian point belongs to the categories of background, hand, and object.

[0053] For example, ,in, This represents a learnable, dynamic class probability vector indicating whether a Gaussian point belongs to the background class. This represents a learnable, dynamic class probability vector indicating whether a Gaussian point belongs to an object category. This represents a learnable dynamic class probability vector that a Gaussian point belongs to the hand category.

[0054] The sum of the learnable dynamic category probability vectors for each Gaussian point belonging to the background category, the object category, and the hand category is 1.

[0055] Each Gaussian point has a learnable dynamic class probability vector. It does not have a fixed initial value; it only needs to satisfy the sum of three values ​​being 1. It can be assigned based on empirical values ​​(the probability of the static background branch is higher) or randomly assigned. It can be optimized based on the network's learning.

[0056] For example, the learnable dynamic class probability vector of Gaussian points is set to [0.8, 0.1, 0.1] to ensure that most Gaussian points in the entire scene have the highest probability of belonging to the background.

[0057] In the embodiments described above, by assigning a learnable dynamic category probability vector to each Gaussian point in the 3D Gaussian set, it is possible to dynamically learn which category each Gaussian point belongs to, and to learn and correct unreasonable 3D Gaussian initializations.

[0058] To achieve dynamic scene modeling, in some specific embodiments of this application, for S13, constructing a dedicated deformation branch can be done using S131 to S133.

[0059] S131, uses a static deformation branch as a dedicated deformation branch for the background. .

[0060] Specifically, the background 3D Gaussian is a static Gaussian. Therefore, the static deformation branch is an identity mapping branch. The static deformation branch can output directly without parametric deformation.

[0061] S132, based on the shared HexPlane spatiotemporal encoder and the first MLP decoder, constructs a hand-specific deformation branch.

[0062] Specifically, the shared HexPlane spatiotemporal encoder is used to extract the spatiotemporal features of the hand and store the object attributes, while the first MLP decoder is used to predict the Gaussian deformation of the hand.

[0063] The 3D Gaussian hand model belongs to the dynamic Gaussian model, a special deformation branch for hands. For the dynamic deformation branch, the deformation amount of the Gaussian properties of the hand is predicted separately by multiple heads through the first MLP decoder.

[0064] S133 constructs object-specific deformation branches based on the shared HexPlane spatiotemporal encoder and the second MLP decoder.

[0065] Specifically, the shared HexPlane spatiotemporal encoder is also used to extract the spatiotemporal features of the object and store the object attributes, while the second MLP decoder is used to predict the Gaussian deformation of the object.

[0066] 3D Gaussian deformation of objects belongs to dynamic Gaussian deformation, a branch specific to objects. For the dynamic deformation branch, the deformation amount of the object's Gaussian properties is predicted separately by multiple heads through the second MLP decoder.

[0067] The embodiments described above in this application construct dedicated deformation branches for background, hand, and object, and predict the deformation amount of Gaussian points of corresponding categories to achieve dynamic scene modeling.

[0068] In the embodiments of this application, both the hand-specific deformation branch and the object-specific deformation branch use a shared HexPlane spatiotemporal encoder.

[0069] In some specific embodiments of this application, a globally shared probability branch is set up to update the learnable dynamic class probability vector. This globally shared probability branch also receives latent space encoded features output by the spatiotemporal encoder of the shared Hexplane, which are input into the MLP prediction network to obtain the updated learnable dynamic class probability vector, which is used to assist Gaussian points in being input into different dedicated deformation branches.

[0070] Specifically, globally shared probability branches Increased output probability based on spatiotemporal coding To optimize the learnable dynamic class probability vector, i.e., probability augmentation. The summation is performed with the original probability vector value, and the category classification is optimized.

[0071] The embodiments described above in this application improve the accuracy of Gaussian point classification by setting a globally shared probability branch and updating the learnable dynamic category probability vector, thereby improving the accuracy of 4D scene reconstruction and the separation of background, hand, and object.

[0072] In some specific embodiments of this application, the preset soft-hard two-stage routing mechanism includes a soft-gating stage and a hard-gating stage, with the soft-gating stage executed first and the hard-gating stage executed later.

[0073] To achieve coarse-to-fine decomposition, in some specific embodiments of this application, for S14, based on the learnable dynamic category probability vector of each Gaussian point, each Gaussian point is assigned to a dedicated deformation branch for processing through a preset soft-hard two-stage routing mechanism. The Gaussian points processed by the dedicated deformation branch can be determined by: S141 to S146.

[0074] S141, In the soft gating stage, each Gaussian point is simultaneously input into the background-specific deformation branch, the hand-specific deformation branch, and the object-specific deformation branch to determine the deformation amount of each specific deformation branch.

[0075] Specifically, the deformation amount of each dedicated deformation branch includes the background deformation amount, the hand deformation amount, and the object deformation amount.

[0076] S142, the deformation amount of each dedicated deformation branch is weighted and fused using the learnable dynamic category probability vector of each Gaussian point to determine the deformation amount of each Gaussian point.

[0077] Specifically, the learnable dynamic category probability vector for each Gaussian point includes a learnable dynamic category probability vector belonging to the background category, a learnable dynamic category probability vector belonging to the object category, and a learnable dynamic category probability vector belonging to the hand category.

[0078] The learnable dynamic category probability vector belonging to the background category is multiplied by the background deformation, the learnable dynamic category probability vector belonging to the hand category is multiplied by the hand deformation, and the learnable dynamic category probability vector belonging to the object category is multiplied by the object deformation. The results are then summed to achieve weighted fusion and determine the Gaussian point deformation.

[0079] S143, self-update each Gaussian point based on the deformation of each Gaussian point, and determine each updated Gaussian point.

[0080] Specifically, based on the above steps S141 to S143: Gaussian point deformation :

[0081] in, This represents the deformation at the Gaussian point. This represents a learnable dynamic class probability vector for Gaussian points belonging to the background class. This indicates the background deformation amount output by the background-specific deformation branch. This represents a learnable, dynamic class probability vector representing Gaussian points belonging to an object category. This indicates the amount of object deformation output by the object-specific deformation branch. This represents a learnable dynamic class probability vector for Gaussian points belonging to the hand category. This indicates the amount of hand deformation output by the dedicated hand deformation branch.

[0082] Through the aforementioned weighted soft gating mechanism, each dedicated deformation branch can obtain an effective gradient in the early stages of training, learning the Gaussian probability attribute from random uncertainty to an almost certain probability.

[0083] S144, In the hard-gating stage, class labels are assigned to each updated Gaussian point based on the largest class probability vector among the learnable dynamic class probability vectors corresponding to each updated Gaussian point, thus determining the class of each updated Gaussian point.

[0084] Specifically, the category labels are for:

[0085] S145, input the index time of each updated Gaussian point and the first-person view video sequence into the dedicated deformation branch corresponding to the category of each updated Gaussian point, and determine the time-aware Gaussian deformation amount corresponding to the category of each updated Gaussian point.

[0086] Based on the above steps S144 to S145, each updated Gaussian point is classified into the category corresponding to the largest category probability vector in its learnable dynamic category probability vector, and the Gaussian point is input into the dedicated deformation branch corresponding to the category corresponding to the largest category probability vector in its learnable dynamic category probability vector, thereby obtaining the time-aware Gaussian deformation amount corresponding to the category of each updated Gaussian point.

[0087] S146, update each updated Gaussian point according to the time-aware Gaussian deformation amount corresponding to the category of each updated Gaussian point, and determine the Gaussian point processed by the dedicated deformation branch.

[0088] Specifically, the update process is as follows: sum the time-aware Gaussian deformation amount with the original Gaussian point.

[0089] Specifically, the Gaussian points processed by the dedicated deformation branch include Gaussian points processed by the background dedicated deformation branch, Gaussian points processed by the hand dedicated deformation branch, and Gaussian points processed by the object dedicated deformation branch.

[0090] The embodiments described above in this application achieve explicit and detailed separation of the background, hand, and object by adopting a preset two-stage soft-hard routing mechanism, and obtain time-aware Gaussian deformation, thereby obtaining the dynamic changes of the modeled Gaussian in the spatiotemporal dimension, improving the image quality, efficiency, and motion coherence of subsequent 4D scene reconstruction, restoring fine dynamic textures, ensuring dynamic coherence, strengthening motion modeling capabilities, accurately capturing complex motion, reducing storage costs, and supporting real-time rendering.

[0091] To achieve human-centered video scene reconstruction and decomposition reconstruction of components, in some specific embodiments of this application, for S15, rendering the Gaussian points processed by a dedicated deformation branch to determine the 4D scene reconstruction image and the decomposition reconstruction images of the background, hands, and objects may include S151 to S154.

[0092] S151 renders all Gaussian points processed by a dedicated deformation branch to determine the 4D scene reconstruction image.

[0093] Based on the above step S151, the Gaussian points processed by each dedicated deformation branch are rendered, namely the Gaussian points processed by the background dedicated deformation branch, the Gaussian points processed by the hand dedicated deformation branch, and the Gaussian points processed by the object dedicated deformation branch, to obtain a 4D scene reconstruction image of the video centered on the person, i.e., a panoramic image.

[0094] S152 renders the Gaussian points processed by the background-specific deformation branch to determine the decomposed and reconstructed image of the background.

[0095] S153 renders the Gaussian points processed by the hand-specific deformation branch to determine the decomposed and reconstructed image of the hand.

[0096] S154 renders the Gaussian points processed by the object-specific deformation branch to determine the decomposed and reconstructed image of the object.

[0097] Based on the above steps S152 to S153, the Gaussian points of each dedicated deformation branch are rendered to obtain the decomposed and reconstructed images of the corresponding components.

[0098] The embodiments described above in this application, by introducing learnable dynamic category probability vectors and dedicated deformation branches for different categories, achieve explicit and fine separation of background, hand, and object in human-centered video scene reconstruction. Fine-grained decoupling overcomes the problems of blurred boundaries and component adhesion that are common in the hand-object interaction area in existing technologies, laying the foundation for subsequent scene understanding and interactive editing.

[0099] To improve the reconstruction and separation quality of each part, this application introduces a category-level control strategy.

[0100] In some specific embodiments of this application, a human-centered video scene reconstruction and separation method further includes steps S17 to S18.

[0101] S17: During the rendering process, a luminance map is generated based on the learnable luminance properties of Gaussian points processed by the background-specific deformation branch.

[0102] S18, use a preset piecewise function to perform brightness compensation on the brightness map, and determine the brightness map after brightness compensation.

[0103] Specifically, based on the above steps S17 to S18, a luminance map is generated for the background using the learnable luminance attribute b. And use a preset piecewise function to process the brightness map Perform brightness compensation:

[0104] in, Represents a brightness diagram. Represents the brightness modeling coefficient. This represents a brightness map after brightness compensation.

[0105] The embodiments described above in this application generate a brightness map using learnable brightness attributes and compensate the brightness map using a preset piecewise function, which effectively suppresses artifacts caused by changes in illumination and improves the quality of 4D scene reconstruction images and background decomposition reconstruction images.

[0106] In some specific embodiments of this application, a human-centered video scene reconstruction and separation method further includes steps S19 to S20.

[0107] S19 determines the dynamic optical flow during the rendering process based on the difference between the real optical flow and the camera motion optical flow.

[0108] S20 optimizes the predicted optical flow for the hand and object based on dynamic optical flow.

[0109] Specifically, for the dynamic parts such as hands and objects, the dynamic optical flow is the removal of the camera motion component from the real optical flow:

[0110] in, Indicates dynamic optical flow. This represents the optical flow of camera motion from time t to time t+1. This represents the actual optical flow from time t to time t+1.

[0111] Optimize motion modeling of hands and objects using motion flow constraints:

[0112] in, Indicates the predicted optical flow. This represents the difference in loss between the predicted optical flow and the actual dynamic optical flow.

[0113] The embodiments described above in this application improve the temporal consistency of motion trajectories, enhance the tracking accuracy for fast motion, and improve the spatial detail and geometric fidelity of the reconstruction results by employing motion flow constraints for dynamic hand and object motion modeling.

[0114] In some specific embodiments of this application, a human-centered video scene reconstruction and separation method further includes S21.

[0115] S21, based on the segmentation masks of the hand and the object, determine the reconstruction loss of the hand and the object as well as the transparency loss of the hand and the object, and optimize the decomposed and reconstructed images of the hand and the object.

[0116] Specifically, the reconstruction loss item is:

[0117] in, Indicates the losses incurred during reconstruction. Indicates the rendered image. This represents a segmentation mask for a certain region. Represents the truth image.

[0118] The transparency loss term is:

[0119] in, This indicates a loss of transparency. This indicates the transparency obtained during the rendering process.

[0120] In the above embodiments of this application, the boundaries of each instance are strengthened through backpropagation of the reconstruction loss term and transparency loss term based on the supervision mechanism of the segmentation mask M.

[0121] In some specific embodiments of this application, a human-centered video scene reconstruction and separation method may further include S23 to S24.

[0122] S23, when calculating the gradient during backpropagation, morphological dilation is performed on the mask of categories not currently being learned to determine the dilated mask region.

[0123] S24, perform gradient truncation on the currently learned class mask in the inflated mask region.

[0124] Specifically,

[0125] in, Indicates the total loss. Indicates the current image. The mask represents the occluded portion, i.e., the union of the masked areas of the hand and the object. Dialate indicates that appropriate morphological dilation is performed.

[0126] The above formula indicates that gradient update backpropagation is only performed in the non-inflated mask region.

[0127] For example, when updating the background mask, the background mask is used as the currently learned category mask, the background is occluded by the hand and the object, and the hand and the object are used as non-currently learned category masks. Morphological dilation is performed on the mask of the hand and the object to determine the dilated mask region, and the background update is performed by gradient truncation in the dilated mask region.

[0128] The embodiments described above in this application prevent cross-contamination caused by the occlusion relationship between different parts by performing morphological dilation on the non-currently learned category mask and gradient truncation on the currently learned category mask in the dilated mask region.

[0129] In some specific embodiments of this application, during the training process, soft-gating training is performed first, followed by hard-gating training to solidify component separation. The total loss function is:

[0130] in, Represents the total loss function. Indicates L1 loss, Indicates optical flow loss, Indicates the losses incurred during reconstruction. This indicates a loss of transparency. This represents the SSIM similarity loss. This represents the entropy regularization term.

[0131] Specifically, transparency loss is used as mask supervision loss.

[0132] The aforementioned total loss function works together throughout both the soft-gated and hard-gated training phases.

[0133] Through the above technical solution, this application achieves high-quality full-scene reconstruction and, for the first time, completes three-level fine-grained decomposition of background-hand-object in the dynamic reconstruction of human-centered video.

[0134] This application presents a human-centered video scene reconstruction and separation method. By introducing dynamic probability vectors and category-specific deformation branches, it pioneers explicit and fine-grained separation of background, hands, and objects in human-centered visual 4D reconstruction. Unlike existing dynamic scene reconstruction methods that can only perform coarse foreground-background dichotomy, this fine-grained decoupling effectively overcomes the problems of blurred boundaries and component adhesion commonly found in existing methods in the hand-object interaction area, laying a solid foundation for subsequent scene understanding and interactive editing.

[0135] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0136] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0137] HO4D, EPIC-FIELD, and Hot3D were used as datasets. Four models—4DGaussians, MotionGS, NeuralDiff, and DeGauss—were used as baseline models. 4DGaussians and MotionGS were used only for the full-scene reconstruction task, while NeuralDiff and DeGauss performed separation tasks in addition to reconstruction.

[0138] Figure 3 This is a schematic diagram showing a comparison between the reconstruction effect of a human-centered video scene reconstruction and separation method and a baseline model, according to an exemplary embodiment.

[0139] The reconstruction results in the same scenario were qualitatively compared and presented with images for reference. Figure 3 As shown, the peak signal-to-noise ratio (PSNR) is used to quantitatively measure the pixel-level error between the reconstructed image and the original image; the higher the value, the smaller the distortion. The structural similarity index (SSIM) is also used to evaluate image similarity from three dimensions: brightness, contrast, and structure, which is more in line with the characteristics of the human visual system. The closer the SSIM value is to 1, the better the structure is preserved. The learned perceptual image patch similarity index (LPIPS) is also used to calculate perceptual differences based on deep learning features. The lower the value, the higher the perceptual quality and the closer it is to human subjective judgment.

[0140]

[0141] Table 1 Reference Figure 3 As shown in Table 1, the method provided in this application achieves peak signal-to-noise ratios (PSNR) of 33.69 dB, 34.60 dB, and 26.12 dB on the HO4D, EPIC-FIELD, and Hot3D datasets, respectively, all outperforming all baseline models and demonstrating a significant improvement in pixel-level fidelity. Regarding perceptual quality, the method provided in this application achieves a learned perceptual patch similarity (LPIPS) value of 0.043 on the HO4D dataset, comparable to the best result. On the more challenging Hot3D dataset, the method provided in this application achieves the best performance with a learned perceptual patch similarity (LPIPS) value of 0.262, demonstrating its superior perceptual quality in complex hand-object interaction scenarios. Furthermore, the method provided in this application maintains a leading position in the structural similarity index (SSIM) across all three datasets, further validating the advantage of its reconstruction results in terms of structural similarity.

[0142] Figure 4 This is a schematic diagram showing a comparison of the separation effect of a human-centered video scene reconstruction and separation method with a baseline model, according to an exemplary embodiment.

[0143] Reference Figure 4 As shown, in the separation task, the method provided in this application is compared with the baseline models NeuralDiff and DeGauss, which support the separation task. These two baseline models only support the separation of foreground and background and do not support the separation of object-hand-background at three levels. The separation accuracy and performance of the method provided in this application are better than the baseline models.

[0144] Figure 5 This is a schematic diagram illustrating the structure of a human-centered video scene reconstruction and classification system according to an exemplary embodiment.

[0145] Reference Figure 5 As shown, this application provides a human-centered video scene reconstruction and classification system 100, including: a 3D Gaussian set initialization module 110, a dynamic category probability vector allocation module 120, a dedicated deformation branch construction module 130, a decomposition deformation module 140, and a scene reconstruction and decomposition reconstruction module 150.

[0146] The 3D Gaussian set initialization module 110 is used to initialize a 3D Gaussian set covering the background, hands, and objects for a first-person perspective video sequence based on the prior of recovering structure from motion. The dynamic category probability vector assignment module 120 is used to assign a learnable dynamic category probability vector to each Gaussian point in the 3D Gaussian set, and to determine the learnable dynamic category probability vector of each Gaussian point. The dedicated deformation branch construction module 130 is used to construct dedicated deformation branches, including a background dedicated deformation branch, a hand dedicated deformation branch, and an object dedicated deformation branch; The decomposition deformation module 140 is used to assign each Gaussian point to a dedicated deformation branch for processing based on the learnable dynamic category probability vector of each Gaussian point through a preset soft-hard two-stage routing mechanism, and to determine the Gaussian points processed by the dedicated deformation branch. The scene reconstruction and decomposition reconstruction module 150 is used to render Gaussian points processed by a dedicated deformation branch, and to determine the 4D scene reconstruction image as well as the decomposition reconstruction images of the background, hands, and objects.

[0147] The embodiments described above in this application introduce a learnable dynamic category probability vector for each Gaussian point in a 3D Gaussian set and construct dedicated deformation branches for the background, hand, and object. Through a preset soft-hard two-stage routing mechanism, each Gaussian point is assigned to a dedicated deformation branch for processing, and Gaussian points processed by the dedicated deformation branches are obtained. Finally, each type of Gaussian point processed by the dedicated deformation branch is rendered, realizing 4D scene reconstruction of human-centered video and explicit and fine separation of background, hand, and object. This overcomes the common problems of blurred boundaries and component adhesion in the hand-object interaction area, providing an effective tool for human-centered interactive video scene understanding and interactive editing.

[0148] Regarding the embodiments of the above system, the specific ways in which each module performs operations have been described in detail in the embodiments of the method, and will not be elaborated here.

[0149] Based on the same technical concept, in some specific embodiments of this application, a terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and a method that the processor can use to execute when executing the program.

[0150] Based on the same technical concept, in some specific embodiments of this application, a computer-readable storage medium is provided on which a computer program is stored, which can be used to execute a method when the program is executed by a processor.

[0151] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0152] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0153] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0154] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0155] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0157] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0159] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A human-centered video scene reconstruction and separation method, characterized in that, include: For first-person perspective video sequences, based on the prior of reconstructing structure from motion, a 3D Gaussian set covering the background, hands, and objects is initialized; Assign a learnable dynamic class probability vector to each Gaussian point in the 3D Gaussian set, and determine the learnable dynamic class probability vector for each Gaussian point. Construct dedicated deformation branches, including a background-specific deformation branch, a hand-specific deformation branch, and an object-specific deformation branch; Based on the learnable dynamic category probability vector of each Gaussian point, each Gaussian point is assigned to the dedicated deformation branch for processing through a preset soft-hard two-stage routing mechanism, and the Gaussian points processed by the dedicated deformation branch are determined. The Gaussian points processed by the dedicated deformation branch are rendered to determine the 4D scene reconstruction image and the decomposed reconstruction images of the background, the hand, and the object.

2. The human-centered video scene reconstruction and separation method according to claim 1, characterized in that, For the first-person perspective video sequence, based on the prior knowledge of structure recovery from motion, a 3D Gaussian set covering the background, hands, and objects is initialized, including: The Colmap method is used to perform motion-to-structure reconstruction on the first-person perspective video sequence to determine the camera parameters and initialize the point cloud for each frame. Based on the initialized point cloud, perform 3D Gaussian initialization to determine the initialized 3D Gaussian; The camera parameters of each frame image are used to optimize the initial 3D Gaussian, and the 3D Gaussian set covering the background, hand, and object is determined.

3. The human-centered video scene reconstruction and separation method according to claim 1, characterized in that, The construction of the dedicated deformation branch includes: A static deformation branch is used as the background-specific deformation branch, and the static deformation branch is an identity mapping branch; The hand-specific deformation branch is constructed based on the shared HexPlane spatiotemporal encoder and the first MLP decoder. The shared HexPlane spatiotemporal encoder is used to extract the spatiotemporal features of the hand, and the first MLP decoder is used to predict the Gaussian deformation of the hand. Based on the shared HexPlane spatiotemporal encoder and the second MLP decoder, a deformation branch for the object is constructed. The shared HexPlane spatiotemporal encoder is also used to extract the spatiotemporal features of the object, and the second MLP decoder is used to predict the Gaussian deformation of the object.

4. The human-centered video scene reconstruction and separation method according to claim 1, characterized in that, The preset soft-hard two-stage routing mechanism includes a soft-gating stage and a hard-gating stage. The soft-gating stage is executed first, followed by the hard-gating stage. The step of assigning each Gaussian point to a dedicated deformation branch for processing based on the learnable dynamic category probability vector of each Gaussian point through a preset soft-hard two-stage routing mechanism, and determining the Gaussian points processed by the dedicated deformation branch, includes: In the soft gating stage, each Gaussian point is simultaneously input into the background-specific deformation branch, the hand-specific deformation branch, and the object-specific deformation branch to determine the deformation amount of each specific deformation branch. The deformation amount of each dedicated deformation branch is weighted and fused using the learnable dynamic category probability vector of each Gaussian point to determine the deformation amount of each Gaussian point. The learnable dynamic category probability vector of each Gaussian point includes a learnable dynamic category probability vector belonging to the background category, a learnable dynamic category probability vector belonging to the object category, and a learnable dynamic category probability vector belonging to the hand category. Each Gaussian point is self-updated based on the deformation amount of each Gaussian point to determine each updated Gaussian point. In the hard gating stage, a category label is assigned to each updated Gaussian point based on the largest category probability vector among the learnable dynamic category probability vectors corresponding to each updated Gaussian point, thereby determining the category of each updated Gaussian point. The index time of each updated Gaussian point and the first-person perspective video sequence are input into the dedicated deformation branch corresponding to the category of each updated Gaussian point to determine the time-aware Gaussian deformation amount corresponding to the category of each updated Gaussian point. Each updated Gaussian point is updated based on the time-aware Gaussian deformation amount corresponding to the category of each updated Gaussian point, and the Gaussian points processed by the dedicated deformation branch are determined. The Gaussian points processed by the dedicated deformation branch include Gaussian points processed by the background dedicated deformation branch, Gaussian points processed by the hand dedicated deformation branch, and Gaussian points processed by the object dedicated deformation branch.

5. The human-centered video scene reconstruction and separation method according to claim 4, characterized in that, The process of rendering the Gaussian points processed by the dedicated deformation branch to determine the 4D scene reconstruction image and the decomposed reconstruction images of the background, the hand, and the object includes: Render all Gaussian points processed by the dedicated deformation branch to determine the 4D scene reconstruction image; Render the Gaussian points processed by the background-specific deformation branch to determine the decomposed and reconstructed image of the background; The Gaussian points processed by the hand-specific deformation branch are rendered to determine the decomposed and reconstructed image of the hand. The Gaussian points processed by the object-specific deformation branch are rendered to determine the decomposed and reconstructed image of the object.

6. The human-centered video scene reconstruction and separation method according to claim 5, characterized in that, The method further includes: During the rendering process, a brightness map is generated based on the learnable brightness properties of the Gaussian points processed by the background-specific deformation branch. A preset piecewise function is used to perform brightness compensation on the brightness map to determine the brightness map after brightness compensation; The method further includes: During the rendering process, dynamic optical flow is determined based on the difference between real optical flow and camera motion optical flow; The predicted optical flow of the hand and the object is optimized based on the dynamic optical flow; The method further includes: Based on the segmentation masks of the hand and the object, the reconstruction loss of the hand and the object, as well as the transparency loss of the hand and the object, are determined, and the decomposed and reconstructed images of the hand and the object are optimized.

7. The human-centered video scene reconstruction and separation method according to claim 6, characterized in that, The method further includes: When calculating gradients during backpropagation, morphological dilation is performed on the non-currently learned category mask to determine the dilated mask region; Gradient truncation is performed on the currently learned category mask within the inflated mask region.

8. A human-centered video scene reconstruction and separation system, characterized in that, include: The 3D Gaussian set initialization module is used to initialize a 3D Gaussian set covering the background, hands, and objects for a first-person view video sequence, based on the prior of recovering structure from motion. The dynamic category probability vector allocation module is used to assign a learnable dynamic category probability vector to each Gaussian point in the 3D Gaussian set, and to determine the learnable dynamic category probability vector for each Gaussian point. A dedicated deformation branch construction module is used to construct dedicated deformation branches, which include a background-specific deformation branch, a hand-specific deformation branch, and an object-specific deformation branch. The decomposition deformation module is used to assign each Gaussian point to the dedicated deformation branch for processing based on the learnable dynamic category probability vector of each Gaussian point through a preset soft-hard two-stage routing mechanism, and to determine the Gaussian points processed by the dedicated deformation branch. The scene reconstruction and decomposition reconstruction module is used to render the Gaussian points processed by the dedicated deformation branch to determine the 4D scene reconstruction image and the decomposition reconstruction images of the background, the hand, and the object.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.

10. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-7.

Images