A dynamic three-dimensional gaussian update method driven by two-dimensional point tracking

Abstract

The application relates to a dynamic three-dimensional Gaussian updating method driven by two-dimensional point tracking. The method comprises the following steps: acquiring a continuous image frame sequence, camera parameters and an initial three-dimensional Gaussian unit set synchronously collected under multiple perspectives; performing two-dimensional point tracking on a foreground area of a dynamic target under each perspective to obtain two-dimensional tracking points between adjacent frames; performing cross-perspective association matching and confidence weighting on the two-dimensional tracking points falling into the same three-dimensional Gaussian unit corresponding image area under each perspective based on the camera parameters; solving motion variables of the corresponding three-dimensional Gaussian unit; pre-updating geometric parameters of the three-dimensional Gaussian unit according to the solving result; constructing a joint loss function for iterative training and correction to obtain a final three-dimensional Gaussian unit set of the next frame; and iteratively updating the three-dimensional Gaussian unit set frame by frame to obtain a time-sequentially continuous dynamic three-dimensional Gaussian sequence. The method can effectively improve the convergence efficiency, geometric consistency and time sequence stability of three-dimensional reconstruction under a few-perspective dynamic scene.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a dynamic 3D Gaussian update method driven by 2D point tracking. Background Technology

[0002] With the rapid development of free-viewpoint video, digital human content production, virtual shooting, and immersive interactive technologies, dynamic human 3D reconstruction and rendering technology has become a research hotspot in the field of computer vision. Its goal is to recover the continuously changing geometric shape and texture of the human body from multi-viewpoint videos and support high-quality rendering output from any viewpoint. In recent years, 3D Gaussian sputtering technology, with its advantages of high differentiable rendering efficiency, strong detail restoration capability, and controllable memory usage, has gradually become one of the mainstream technical solutions for dynamic 3D reconstruction. Dynamic 3D Gaussian reconstruction continuously expresses the deformation of non-rigidly moving human bodies and clothing by updating the position, rotation, scale, color, and opacity of 3D Gaussian units frame by frame. It can be widely used in scenarios such as digital human asset generation, sports event playback, and medical motion assessment.

[0003] Most mainstream dynamic 3D Gaussian update schemes currently focus on optimizing frame-by-frame photometric reconstruction errors, relying on the backpropagation gradient of rendering loss to drive Gaussian parameter iteration. Some schemes improve temporal continuity by introducing deformation field networks to predict overall deformation or adding local spatiotemporal regularization constraints. In practical applications, a deployment scheme using a small number of fixed cameras for surround shooting has high application value in low-to-medium cost content production and home fitness scenarios due to its low hardware cost, small deployment space, and simple operation process.

[0004] However, existing dynamic 3D Gaussian update methods still face numerous technical bottlenecks in scenarios with few fixed-view cameras. Firstly, in scenarios involving large displacement motion, the gradient dependency is high, resulting in insufficient convergence efficiency and stability. When the human body undergoes rapid limb movements or overall displacement, the local luminosity gradient of a single frame is insufficient to guide Gaussian units to complete long-distance position migration, easily leading to gradient mismatch. This causes Gaussian units to deviate from the real human body surface, resulting in problems such as local drift, ghosting, color mismatch, and inter-frame flickering. Furthermore, the fewer the number of viewpoints, the more pronounced the limitations of gradient information become.

[0005] Secondly, the temporal motion information in multi-view videos has not been fully explored and utilized. Multi-view synchronous videos contain a large amount of two-dimensional point trajectory information that can reflect the local motion trend of the human body. However, existing solutions usually only use point trajectories as auxiliary weak constraints in the training phase, and do not transform them into direct motion priors for Gaussian parameter updates. At the same time, existing trajectory constraints mostly only act on the center position of the Gaussian, and cannot synchronously drive the coherent update of rotation and scale parameters. It is difficult to accurately express non-rigid deformations such as limb extension and contraction, clothing swaying, and changes in surface orientation, resulting in insufficient temporal geometric consistency in dynamic reconstruction.

[0006] Finally, the parameter optimization process is fragmented, making it easy for errors to accumulate and propagate. In the existing workflow, camera parameter calibration, trajectory solving, and Gaussian parameter optimization are usually performed independently in stages, and the initial camera calibration error will be propagated to the subsequent reconstruction process step by step. In scenarios with few views, the view coverage is limited and there are many occluded areas, making it impossible to offset errors with redundant information from multiple views. This can easily lead to problems such as cross-view geometric misalignment and trajectory constraint offset. Furthermore, there is a lack of systematic fault-tolerant processing mechanisms for occlusion, low-confidence trajectories, and stationary areas, which further reduces the stability of the reconstruction results. Summary of the Invention

[0007] Therefore, it is necessary to provide a two-dimensional point tracking-driven dynamic 3D Gaussian update method that can improve the convergence efficiency, temporal geometric consistency, and operational robustness of dynamic 3D reconstruction in scenarios with few viewpoints, in order to address the aforementioned technical problems.

[0008] A dynamic 3D Gaussian update method driven by 2D point tracking, the method comprising: Acquire a continuous sequence of image frames acquired synchronously from multiple viewpoints, camera parameters corresponding to each viewpoint, and an initial set of three-dimensional Gaussian units corresponding to the initial frame. The three-dimensional Gaussian units include center position, rotation parameters, scale parameters, color parameters, and opacity parameters. In each viewpoint, two-dimensional point tracking is performed on the foreground region of the dynamic target in each frame of the continuous image frame sequence to obtain the two-dimensional displacement data of each two-dimensional tracking point between adjacent frames and the corresponding tracking confidence. Based on the set of three-dimensional Gaussian units corresponding to the current frame, and based on the camera parameters, combined with the corresponding two-dimensional displacement data and tracking confidence, the two-dimensional tracking points that fall into the image region corresponding to the same three-dimensional Gaussian unit after projection under each viewpoint are cross-viewpoint association matching and confidence weighting. Based on the multi-view two-dimensional trajectory distribution after association, the three-dimensional center displacement, rotation increment and scale increment of the corresponding three-dimensional Gaussian unit are solved. Based on the obtained 3D center displacement, rotation increment and scale increment, the geometric parameters of the corresponding 3D Gaussian element are pre-updated to generate the pre-updated 3D Gaussian element set for the next frame. A joint loss function is constructed, which includes color reconstruction error, structural similarity error, two-dimensional trajectory reprojection constraint and neighborhood motion smoothing constraint. The optimization objective is to minimize the value of the joint loss function. The pre-updated three-dimensional Gaussian unit set is iteratively trained and corrected to obtain the final three-dimensional Gaussian unit set of the next frame. By iteratively updating the set of three-dimensional Gaussian units frame by frame, a temporally continuous dynamic three-dimensional Gaussian sequence is obtained.

[0009] In one embodiment, obtaining the two-dimensional displacement data of each two-dimensional tracking point between adjacent frames and the corresponding tracking confidence score includes: Perform bidirectional tracking consistency detection, local neighborhood motion consistency detection, and cross-view visibility detection on the two-dimensional tracking points from various perspectives; Adjust the tracking confidence of each two-dimensional tracking point based on the detection results, and remove outlier tracking points whose tracking confidence is lower than the preset threshold.

[0010] In one embodiment, the three-dimensional Gaussian unit is divided into trajectory Gaussian unit and ordinary Gaussian unit, wherein the trajectory Gaussian unit is used to undertake the geometric anchoring function, and the ordinary Gaussian unit is used to undertake the expression of appearance details; When solving for the three-dimensional center displacement, rotation increment, and scale increment of the three-dimensional Gaussian element: For the trajectory Gaussian unit, the three-dimensional motion increment is directly solved based on the high-confidence two-dimensional trajectory within the projection region of the trajectory Gaussian unit itself. For the ordinary Gaussian unit, motion information is propagated based on the spatial distance, surface orientation similarity, and foreground attribution relationship between each ordinary Gaussian unit and its neighboring trajectory Gaussian units to obtain the corresponding three-dimensional motion increment.

[0011] In one embodiment, when solving for the rotation increment and scale increment in the three-dimensional Gaussian element: Based on the changes in the covariance distribution of the two-dimensional trajectory set in adjacent frames under multiple perspectives, the three-dimensional covariance increment of the corresponding three-dimensional Gaussian unit is estimated. The three-dimensional covariance increment is decomposed into rotation increment and scale increment, which correspond to the orientation change and local deformation degree of the local surface of the dynamic target, respectively.

[0012] In one embodiment, after generating the pre-updated 3D Gaussian unit set for the next frame and before iterative training and correction, the method further includes: Motion reliability is determined for each 3D Gaussian unit after pre-update. When a 3D Gaussian unit meets at least one of the following conditions, it is determined to be an unreliable Gaussian unit: insufficient number of visible viewpoints, tracking confidence is lower than a preset threshold, 3D ray intersection residual is greater than a preset threshold, scale change exceeds a preset range, or motion direction deviates from the neighborhood trend. For unreliable Gaussian cells, reduce the corresponding motion estimation weights or correct them by propagating motion information from nearby reliable Gaussian cells.

[0013] In one embodiment, when iteratively training and correcting the pre-updated 3D Gaussian unit set, the correction amount of the camera parameters of each viewpoint is also used as an optimization variable. The camera parameters are synchronously corrected during the iterative training process to achieve joint closed-loop optimization of camera parameters and 3D Gaussian unit parameters.

[0014] In one embodiment, the iterative training and correction of the pre-updated 3D Gaussian unit set employs a staged training strategy, including: First warm-up training phase: fix the parameters of the ordinary Gaussian unit, and only correct the geometric parameters of the trajectory Gaussian unit and the camera parameters to stabilize the geometric constraint relationship; The second complete training phase: release the parameter constraints of all three-dimensional Gaussian units and synchronously update all parameters in each three-dimensional Gaussian unit.

[0015] In one embodiment, the joint loss function further includes a three-dimensional trajectory consistency constraint, a trajectory Gaussian scaling constraint, and a geometric regularization constraint; The three-dimensional trajectory consistency constraint is used to constrain the spatial continuity of the three-dimensional trajectory points associated with the two-dimensional trajectory, the trajectory Gaussian scaling constraint is used to maintain the geometric anchoring properties of the trajectory Gaussian unit, and the geometric regularization constraint is used to suppress unreasonable parameter mutations of the Gaussian unit.

[0016] This application also proposes a dynamic three-dimensional Gaussian update device driven by two-dimensional point tracking, the device comprising: The data acquisition module is used to acquire the continuous image frame sequence synchronously acquired from multiple perspectives, the camera parameters corresponding to each perspective, and the initial three-dimensional Gaussian unit set corresponding to the initial frame. The three-dimensional Gaussian unit includes the center position, rotation parameters, scale parameters, color parameters, and opacity parameters. The inter-frame tracking module is used to perform two-dimensional point tracking on the foreground region of the dynamic target in each frame of the continuous image frame sequence from each viewpoint, so as to obtain the two-dimensional displacement data of each two-dimensional tracking point between adjacent frames and the corresponding tracking confidence. The incremental solution module is used to take the set of three-dimensional Gaussian units corresponding to the current frame as a reference, and based on the camera parameters, combined with the corresponding two-dimensional displacement data and tracking confidence, to perform cross-view association matching and confidence weighting on the two-dimensional tracking points that fall into the image region corresponding to the same three-dimensional Gaussian unit after projection from each viewpoint, and to solve the three-dimensional center displacement, rotation increment and scale increment of the corresponding three-dimensional Gaussian unit according to the multi-view two-dimensional trajectory distribution after association. The parameter pre-update module is used to pre-update the geometric parameters of the corresponding three-dimensional Gaussian element based on the solved three-dimensional center displacement, rotation increment and scale increment, and generate the pre-updated three-dimensional Gaussian element set for the next frame. The training and correction module is used to construct a joint loss function that includes color reconstruction error, structural similarity error, two-dimensional trajectory reprojection constraint and neighborhood motion smoothing constraint. The optimization objective is to minimize the value of the joint loss function. The module iteratively trains and corrects the pre-updated three-dimensional Gaussian unit set to obtain the final three-dimensional Gaussian unit set for the next frame. The temporal iteration module is used to obtain a temporally continuous dynamic three-dimensional Gaussian sequence by iteratively updating the three-dimensional Gaussian unit set frame by frame.

[0017] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above-described two-dimensional point tracking driven dynamic three-dimensional Gaussian update method.

[0018] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a dynamic three-dimensional Gaussian update method driven by two-dimensional point tracking.

[0019] The aforementioned dynamic 3D Gaussian update method driven by 2D point tracking acquires inter-frame motion information by performing 2D point tracking on the foreground region of a dynamic target from multiple perspectives. Based on the tracked 2D displacement and confidence level, it solves for the center displacement, rotation increment, and scale increment of the 3D Gaussian unit across perspectives, performing a pre-update of geometric parameters. This provides higher-quality initial parameter values ​​for subsequent training and correction, effectively alleviating the gradient mismatch problem caused by relying solely on photometric gradients in large displacement scenes, improving parameter convergence efficiency, and reducing local drift, ghosting, and inter-frame flicker. Simultaneously, by jointly solving multi-view trajectories, it synchronously drives the coherent update of position, rotation, and scale, fully exploring the value of multi-view temporal motion information. This allows for a more accurate representation of the non-rigid deformation of the dynamic target, improving the temporal geometric consistency of dynamic reconstruction. Finally, iterative fine correction is performed using a multi-dimensional joint loss function, optimizing the appearance restoration effect while ensuring the accuracy of geometric parameters. Combined with a frame-by-frame iterative temporal update mechanism, it can stably output high-quality continuous dynamic 3D Gaussian sequences, comprehensively improving the operational efficiency and result stability of dynamic 3D reconstruction in scenes with few perspectives. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a dynamic 3D Gaussian update method driven by 2D point tracking in one embodiment. Figure 2 This is a schematic diagram illustrating four fixed cameras as multi-view inputs for point tracking and Gaussian updating in one embodiment. Figure 3 This is a schematic diagram of point tracking-driven 3D Gaussian pre-update in one embodiment; Figure 4 This is a schematic diagram illustrating the relationship between loss and constraint in one embodiment; Figure 5 This is a schematic diagram of the overall process of the method in one embodiment; Figure 6 This is a structural block diagram of a dynamic three-dimensional Gaussian update device driven by two-dimensional point tracking in one embodiment. Figure 7This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] To address the issues in existing technologies regarding dynamic 3D Gaussian updates in multi-view scenarios, such as gradient mismatch during large displacement movements, low convergence efficiency, and susceptibility to drift and flickering, insufficient conversion of 2D temporal motion information into geometric update priors, poor temporal consistency in non-rigid deformation, and fragmented parameter optimization across multiple modules leading to easy accumulation and propagation of errors, [further solutions are needed]. Figure 1 As shown, a dynamic 3D Gaussian update method driven by 2D point tracking is provided, including the following steps: Step S100: Obtain the continuous image frame sequence acquired synchronously from multiple viewpoints, the camera parameters corresponding to each viewpoint, and the initial three-dimensional Gaussian unit set corresponding to the initial frame. The three-dimensional Gaussian unit includes the center position, rotation parameters, scale parameters, color parameters, and opacity parameters.

[0023] Step S110: Under each viewpoint, perform two-dimensional point tracking on the foreground region of the dynamic target in each frame of the continuous image frame sequence to obtain the two-dimensional displacement data of each two-dimensional tracking point between adjacent frames and the corresponding tracking confidence.

[0024] Step S120: Based on the set of three-dimensional Gaussian units corresponding to the current frame, and based on camera parameters, combined with the corresponding two-dimensional displacement data and tracking confidence, cross-view association matching and confidence weighting are performed on the two-dimensional tracking points that fall into the image region corresponding to the same three-dimensional Gaussian unit after projection from each viewpoint. Based on the multi-view two-dimensional trajectory distribution after association, the three-dimensional center displacement, rotation increment and scale increment of the corresponding three-dimensional Gaussian unit are solved.

[0025] Step S130: Based on the solved three-dimensional center displacement, rotation increment and scale increment, the geometric parameters of the corresponding three-dimensional Gaussian element are pre-updated to generate the pre-updated three-dimensional Gaussian element set for the next frame.

[0026] Step S140: Construct a joint loss function that includes color reconstruction error, structural similarity error, two-dimensional trajectory reprojection constraint, and neighborhood motion smoothing constraint. With minimizing the value of the joint loss function as the optimization objective, iteratively train and correct the pre-updated three-dimensional Gaussian unit set to obtain the final three-dimensional Gaussian unit set for the next frame.

[0027] Step S150: By iteratively updating the three-dimensional Gaussian unit set frame by frame, a temporally continuous dynamic three-dimensional Gaussian sequence is obtained.

[0028] In this embodiment, the dynamic target refers to any three-dimensional object whose spatial position and / or surface morphology change over time. In one or more embodiments of this application, the dynamic target is specifically all or part of the human body, including but not limited to the torso, limbs, joints, hands, face, and combinations thereof. It is understood that during human movement, there are accompanying limb swings, torso twists, clothing undulations, and non-rigid deformations of the skin surface. Its movement pattern includes various components such as rigid body displacement, joint rotation, and local flexible deformation, which places high demands on the temporal continuity and geometric accuracy of three-dimensional reconstruction.

[0029] Furthermore, the dynamic target is not limited to the human body; in other embodiments, it can also be an animal, a robot, a deformable object, or other object with dynamic deformation capabilities. Those skilled in the art will understand that any object that can be simultaneously captured by multi-view cameras, has traceable texture or contour features, and exhibits positional or morphological changes at different times can be used as the processing object of the method proposed in this application. This application uses the human body as a dynamic target for description, but this should not be construed as limiting the scope of application of this application.

[0030] In step S100, multi-view acquisition can be performed using four fixed cameras surrounding the central subject shooting area. The first to fourth cameras are fixed around the central shooting area, with the optical axes of each camera pointing towards the central area. Adjacent cameras have overlapping fields of view. Figure 2 As shown. Acquired images are organized according to camera number, frame number, and timestamp; camera synchronization can be hardware synchronization, software timestamp synchronization, or post-processing frame alignment. After acquisition, frame alignment, distortion correction, human foreground segmentation, and effective region sampling are required, preferably forming a set of tracking points near the human contour, limb edges, clothing texture, and joints. Four fixed cameras acquire data around the central figure area, which is low-cost and requires little installation space, but is affected by the back of the human body, overlapping limbs, and clothing folds. Therefore, it is necessary to fully utilize cross-frame point tracking and cross-viewpoint trajectory consistency to transform the motion information acquired by the limited number of cameras into a dynamic Gaussian updated prior.

[0031] It should be noted that this method does not limit the specific number of cameras. The four cameras and their arrangement described above are merely preferred embodiments, achieving a good balance between hardware cost, deployment space, and view coverage, making them particularly suitable for low-to-medium cost content production and home fitness scenarios. However, in other embodiments, the number of cameras can be flexibly adjusted according to the actual application scenario, accuracy requirements, and hardware cost. For example, in scenarios with a large acquisition space or high reconstruction accuracy requirements, six, eight, or more cameras can be arranged in a ring to increase multi-view redundancy information, improve the coverage of occluded areas, and enhance reconstruction robustness. In scenarios with extremely limited hardware costs or small deployment space, two or three cameras can be arranged at a certain angle. Although the view coverage is limited, this application can still fully utilize the temporal motion information under limited view through a point tracking-driven pre-update mechanism, reducing the dependence on multi-view spatial redundancy. The increase or decrease in the number of cameras does not affect the core concept of this application, namely, using the two-dimensional point tracking results under multiple views to drive the pre-update of three-dimensional Gaussian units. Therefore, changes in the number of cameras should not be considered as a limitation on the scope of protection of this application.

[0032] In this embodiment, camera parameters include intrinsic and extrinsic parameters. Pre-calibrated parameters can be read directly, or camera parameters can be corrected based on cross-view feature trajectories. When using trajectory correction, two-dimensional tracking points of the same human body in multiple viewpoints are associated as a cross-view global trajectory. The three-dimensional trajectory points are projected onto the camera image plane, and the reprojection error is calculated. Simultaneously, the three-dimensional consistency error is calculated using depth or rendering depth, thereby constraining the camera's intrinsic and extrinsic parameters and the initial three-dimensional trajectory points.

[0033] In this embodiment, an initial set of three-dimensional Gaussian units is also generated based on the initial frame, key frame, or multi-view reconstruction results in a continuous image frame sequence from different viewpoints. The set of three-dimensional Gaussian units includes multiple three-dimensional Gaussian units, and each Gaussian unit includes at least a center position, rotation parameter, scale parameter, color parameter, and opacity parameter.

[0034] Specifically, the 3D Gaussian units in the 3D Gaussian unit set are divided into trajectory Gaussian units and ordinary Gaussian units. Gaussian units corresponding to stable feature trajectories are set as trajectory Gaussian units, while Gaussian units that supplement appearance details are set as ordinary Gaussian units, respectively serving as geometric anchors and rendering representations. Trajectory Gaussian units are used for geometric anchoring, while ordinary Gaussian units are used for appearance detail representation, ensuring both the spatial accuracy of trajectory constraints and retaining the ability of 3D Gaussian units to express high-quality appearance and details.

[0035] Furthermore, the process of dividing the trajectory Gaussian unit into ordinary Gaussian units includes: projecting each three-dimensional Gaussian unit onto the image of each viewpoint to obtain the corresponding two-dimensional projection center and projection radius; searching for two-dimensional tracking points within a neighborhood window expanded based on the two-dimensional projection center and the projection radius; when the same three-dimensional Gaussian unit is associated with two-dimensional tracking points in at least a preset number of viewpoints where the tracking confidence is higher than a first threshold, the displacement between consecutive frames is continuous, and the epipolar residual across viewpoints is less than a second threshold, the three-dimensional Gaussian unit is marked as a trajectory Gaussian unit; when the three-dimensional Gaussian unit does not meet the above stable trajectory association conditions, but its projection is still located in the foreground region of the dynamic target and participates in color or transparency reconstruction, it is marked as an ordinary Gaussian unit.

[0036] Optionally, during the partitioning process, adjacency relationships are established based on the three-dimensional distance between Gaussian units and labeled trajectory Gaussian units, the similarity of the main directions of normal or covariance, and the consistency of foreground instance affiliation. If an ordinary Gaussian unit is adjacent to multiple trajectory Gaussian units, its motion propagation source is determined according to the distance decay weight and direction similarity weight, so that the trajectory Gaussian unit plays the role of sparse and reliable geometric anchors, and the ordinary Gaussian unit plays the role of dense appearance compensation, thereby avoiding the forced binding of all Gaussian units to unstable two-dimensional trajectories.

[0037] In step S110, pixels, feature points, or sampling points in the human foreground region are tracked within each viewpoint. For the first... Frame and the In each frame, the system acquires the displacement of each tracking point on the image plane and records the tracking confidence, visibility, foreground attribution, and occlusion status.

[0038] In this embodiment, the point tracking method can be implemented using optical flow, sparse feature matching, depth point tracking model or a combination thereof. This application does not limit the specific algorithm, but only requires that its output can represent the point trajectory between adjacent frames or multiple frames.

[0039] Furthermore, the two-dimensional displacement data and corresponding tracking confidence of each two-dimensional tracking point between adjacent frames are obtained. This includes performing bidirectional tracking consistency detection, local neighborhood motion consistency detection, and cross-view visibility detection on the two-dimensional tracking points under each viewpoint. Based on the detection results, the tracking confidence of each two-dimensional tracking point is adjusted, and outlier tracking points with tracking confidence below a preset threshold are removed. If the reverse tracking deviation exceeds the threshold, or the trajectory direction within the same region is significantly outlier, the trajectory confidence is reduced. Through the above detection mechanism, occlusion, drift, or cross-view inconsistent trajectory points can be effectively removed, improving the reliability of trajectory data and providing high-quality input for subsequent three-dimensional motion solving.

[0040] In step S120, the local two-dimensional affine motion is first estimated based on the two-dimensional point tracking results within the Gaussian element projection area. Then, combined with camera parameters, the three-dimensional center displacement of the Gaussian element is solved by triangulation, weighted least squares, or multi-view ray intersection.

[0041] The extent of the Gaussian element projection region is determined based on the three-dimensional center position, three-dimensional covariance, and camera projection function of the Gaussian element. Specifically, the first... The center position of the three-dimensional Gaussian element is projected onto the first... The two-dimensional projection center is obtained from each viewpoint. Its three-dimensional covariance is mapped to the two-dimensional covariance through the projection Jacobian matrix, and the ellipse projection radius is determined by the eigenvalues ​​of the two-dimensional covariance. The pixel region or the set of two-dimensional tracking points that satisfy the Mahalanobis distance not greater than a preset threshold is determined as the projection region of the Gaussian unit under that viewpoint. For Gaussian units with too small a projection area, the minimum window radius is used for expansion. For Gaussian units where the overlap ratio between the projection region and the foreground mask is less than a preset threshold, the observation weight of that viewpoint is reduced or that viewpoint is not used in the motion solution.

[0042] Specifically, first obtain the first The image of each frame from different viewpoints and the corresponding foreground region of the human body are obtained, and two-dimensional point tracking is performed within the foreground region of each viewpoint to obtain the first frame. Frame to the Two-dimensional trajectories between frames are then generated. Subsequently, the system correlates, filters, and evaluates the confidence of multi-view two-dimensional trajectories projected onto the same three-dimensional Gaussian cell, eliminating occluded, drifting, or inconsistent trajectory points across different viewpoints. For the retained trajectory points, the three-dimensional center displacement of the corresponding three-dimensional Gaussian cell is solved by combining camera intrinsic and extrinsic parameters with multi-view geometric relationships.

[0043] In this embodiment, the three-dimensional Gaussian element is divided into trajectory Gaussian elements and ordinary Gaussian elements. When solving for the three-dimensional center displacement, rotation increment, and scale increment of the three-dimensional Gaussian element: for trajectory Gaussian elements, the three-dimensional motion increment is directly solved based on the high-confidence two-dimensional trajectory within the projection region of the trajectory Gaussian element itself. For ordinary Gaussian elements, motion information is propagated based on the spatial distance between each ordinary Gaussian element and its neighboring trajectory Gaussian elements, surface orientation similarity, and foreground attribution relationship to obtain the corresponding three-dimensional motion increment.

[0044] In this embodiment, by distinguishing between trajectory Gaussian units and ordinary Gaussian units, some Gaussian units serve as geometric anchors, while others are responsible for representing appearance details. This ensures both the spatial accuracy of trajectory constraints and the ability of 3D Gaussian units to express high-quality appearance and details. Thus, a small number of stable trajectory Gaussian units can provide motion priors for the surrounding ordinary Gaussian units, maintaining temporal continuity in the overall Gaussian set.

[0045] Furthermore, when solving for the rotation increment and scale increment in the three-dimensional Gaussian cell: based on the change in the covariance distribution of the two-dimensional trajectory set in adjacent frames under multiple views, the three-dimensional covariance increment of the corresponding three-dimensional Gaussian cell is estimated, and the three-dimensional covariance increment is decomposed into rotation increment and scale increment, which correspond to the orientation change and local deformation degree of the local surface of the dynamic target, respectively.

[0046] Specifically, in addition to the center displacement, the three-dimensional covariance variation can be estimated based on the two-dimensional covariance variation in multiple perspectives, and the three-dimensional covariance increment can be decomposed into rotational variation and scale variation to obtain the rotational increment and scale increment of the Gaussian unit, so that the Gaussian unit can express the local surface orientation change of the human body, limb extension and contraction, and clothing movement.

[0047] Furthermore, the estimation process of transforming 2D covariance into 3D rotation and scale increments includes: statistically analyzing the local 2D covariance matrix of the Gaussian unit for the associated 2D trajectory points in adjacent frames, calculating the 2D covariance increment for each viewpoint, back-projecting the 2D covariance increment into a 3D covariance increment candidate based on the Jacobian matrix of the camera projection function at the current 3D center position; fusing the 3D covariance increment candidates from multiple viewpoints according to observation weights to obtain the 3D covariance increment; and performing eigenvalue decomposition or polar decomposition on the fused 3D covariance matrix to convert principal axis changes into rotation increments and eigenvalue changes into scale increments. Thus, the 3D Gaussian unit not only updates its center position based on the 2D trajectory but also updates its rotation and scale parameters based on local 2D deformation.

[0048] In this embodiment, the specific solution process for calculating the three-dimensional center displacement, rotation increment, and scale increment of the corresponding three-dimensional Gaussian unit based on the associated multi-view two-dimensional trajectory distribution in step S120, i.e., the conversion relationship between the two-dimensional trajectory and the three-dimensional Gaussian pre-update amount, can be expressed by the following formula: (1) (2) (3) In formulas (1) to (3), Indicates the first From the perspective of the first The two-dimensional tracking point at the th The two-dimensional position of the frame. This indicates that the two-dimensional tracking point starts from the first... Frame to the Two-dimensional displacement of the frame, Indicates the first The projection function of each camera. Indicates the relationship with the first A set of two-dimensional trajectories associated with a three-dimensional Gaussian unit projection region. The weights are determined by tracking confidence, visibility, and cross-view consistency. , , and These represent the center position, rotation, scale, and covariance of the Gaussian element, respectively.

[0049] Specifically, the weights can be expressed by the following formula: (4) In formula (4), This represents the tracking confidence score output by the two-dimensional point tracking algorithm. Indicates the first The two-dimensional tracking point falls into the first The correlation degree of a three-dimensional Gaussian unit projection region This indicates that the two-dimensional tracking point is at the th Visibility from a different perspective This represents the cross-perspective consistency factor. Among them, the degree of correlation... The correlation can be determined based on the Mahalanobis distance from the 2D tracking point to the center of the Gaussian projection; the smaller the distance, the higher the correlation. Visibility It can be determined jointly by the foreground mask, depth occlusion relationship, and projection boundary. Cross-viewpoint consistency factor. The consistency factor can be determined by the ray intersection residuals, epipolar constraint residuals, or reprojection residuals from different viewpoints; the smaller the residual, the higher the consistency factor. After normalizing the weights of each two-dimensional trajectory point associated with the same Gaussian cell, it is used for the weighted solution of center displacement, rotation increment, and scale increment.

[0050] Furthermore, formula (2) is used to inversely calculate the Gaussian center displacement from the multi-view two-dimensional trajectory, while formula (3) is used to synchronously transform the trajectory distribution change into Gaussian rotation and scale update.

[0051] Through formulas (1) to (3), the two-dimensional point trajectory information is systematically transformed into the geometric update quantity of the three-dimensional Gaussian unit, so that the dynamic Gaussian update can break through the expression range of only center translation and further cover the changes in the orientation of the local surface of the human body, the stretching and contraction of limbs, the swinging of clothing and non-rigid deformation.

[0052] In step S130, for the first The three-dimensional Gaussian elements of the frame are used to obtain the center position increment, rotation increment, and scale increment based on the multi-view trajectory solution results (i.e., the results obtained through step S120), and the first... Pre-updated Gaussian unit of the frame.

[0053] In this embodiment, the center displacement, rotation increment, and scale increment are applied together to the first... Frame Gaussian unit, generating the first The pre-updated Gaussian units of the frame. These pre-updated Gaussian units are moved to positions close to those of a real human body after motion. Therefore, subsequent rendering and training do not need to rely entirely on local image gradients to find distant targets, reducing gradient mismatch under rapid motion. The pre-update relationship is as follows: Figure 3 As shown, Figure 3 A schematic diagram of point tracking-driven 3D Gaussian pre-update.

[0054] Specifically, the three-dimensional Gaussian element pre-update process can be described as follows:

[0055] (5)

[0056] In formula (5), , , These represent the numbers after the pre-update. The Gaussian unit in the first The frame's center position, rotation, and scale. Motion information obtained through point tracking is applied to the Gaussian units before photometric training, so that the Gaussian units are moved to positions close to those of real human motion before training begins or at the beginning of each time step, reducing the dependence on local photometric gradients under large displacement conditions.

[0057] In this embodiment, after generating the pre-updated 3D Gaussian unit set for the next frame and before iterative training and correction, motion regularization processing is performed, including: determining the motion reliability of each pre-updated 3D Gaussian unit. A 3D Gaussian unit is determined to be unreliable if it meets at least one of the following conditions: insufficient visible viewpoints, tracking confidence below a preset threshold, 3D ray intersection residual greater than a preset threshold, scale change exceeding a preset range, or motion direction deviating from the neighborhood trend. The motion estimation weights of unreliable Gaussian units are then reduced, or they are corrected by propagating motion information from neighboring reliable Gaussian units. Through motion reliability judgment, abnormal trajectory removal, neighborhood motion propagation, stationary Gaussian identification, and motion regularization, the dynamic update stability under occlusion, low visibility, low texture, and tracking error conditions is improved.

[0058] The aforementioned motion regularization process improves the stability of dynamic Gaussian pre-updates. For reliable Gaussian cells, displacement, rotation, and scale changes can be smoothed based on spatial proximity, foreground semantic region, and surface orientation similarity. For occluded Gaussian cells that belong to the human motion region, the system can use motion estimation from adjacent visible regions for completion. For Gaussian cells with low tracking displacement and stable foreground affiliation, the system can identify them as stationary Gaussian cells, avoiding the propagation of erroneous motion.

[0059] In step S140, after the pre-update is completed, the pre-updated 3D Gaussian unit set is trained and corrected by constructing a joint loss function. The parameters involved in the training and correction in this step include: the center position, rotation parameters, scale parameters, color parameters, and opacity parameters of each 3D Gaussian unit; the position increment, rotation increment, and scale increment at each time step; and the intrinsic and extrinsic parameter corrections for each viewpoint camera when joint optimization is required. The constructed joint loss function includes at least color reconstruction error, structural similarity error, 2D trajectory reprojection constraints, and neighborhood motion smoothing constraints. It may further include 3D trajectory consistency constraints, trajectory Gaussian scale constraints, and geometric regularization constraints. These loss terms work synergistically in the dynamic Gaussian training and parameter correction process, and their relationship is as follows: Figure 4 As shown.

[0060] In this embodiment, the joint loss function is expressed as: (6) (7) (8) In formula (6), This indicates color reconstruction error. Indicates structural similarity error. This represents the two-dimensional trajectory reprojection constraint. This represents the consistency constraint of the three-dimensional trajectory. This indicates a Gaussian scaling constraint on the trajectory. This indicates a neighborhood motion smoothing constraint. This represents a geometric regularity constraint. , , , , , and These are the corresponding weighting coefficients.

[0062] In formula (7), This represents a three-dimensional trajectory point associated with a two-dimensional trajectory point. N Indicates the first The spatial neighborhood of Gaussian units. Through the above loss term, it is possible to constrain the trajectory, Gaussian parameters, and local motion continuity while maintaining the consistency of image appearance.

[0063] Specifically, color reconstruction error and structural similarity error To ensure that the rendered image is consistent with the appearance of the real acquired image, its calculation method is the same as the photometric loss calculation method in the existing 3D Gaussian sputtering technology. That is, the current 3D Gaussian unit set is rendered into the predicted image under each view through a differentiable renderer, and then the L1 loss and SSIM loss are calculated pixel by pixel with the real acquired image.

[0064] Specifically, two-dimensional trajectory reprojection constraints Used to ensure consistency between tracking points and 3D trajectory points, it aligns the positions of 3D spatial points on the dynamic target surface, projected by the camera, with the 2D tracking results, thereby transferring observation information from the multi-view 2D trajectory to 3D space. 3D trajectory consistency constraint This constraint is used to maintain the spatial continuity of 3D trajectory points in a correlated 2D trajectory, ensuring smooth 3D position changes of the same trajectory point between adjacent frames. (Trajectory Gaussian scaling constraint) Used to maintain the geometric anchoring properties of trajectory Gaussian units, preventing them from deviating from their geometric anchoring function due to excessive scaling or shrinking during training. Motion smoothing constraint Local motion consistency is achieved by penalizing the difference in motion increments between adjacent Gaussian units. Geometric regularization constraint. Used to suppress unreasonable parameter abrupt changes in Gaussian elements, such as drastic changes in transparency or discontinuous rotation parameters.

[0065] Furthermore, the above loss function can be refined into the following formula by incorporating the point tracking-driven 3D Gaussian update process: (9) (10) (11) (12) (13) In formulas (9) to (13), Indicates the first From the perspective of the first The two-dimensional tracking point at the th Position in the frame, This represents the three-dimensional trajectory point or the Gaussian center of the trajectory associated with the two-dimensional tracking point. Indicates the first Camera projection function for each viewpoint, The observation weights are determined by tracking confidence, visibility, and cross-view consistency; T represents the set of Gaussian cells for the trajectory. Indicates the first The initial scale of a trajectory Gaussian unit, This represents a penalty term for exceeding the scale limit, used when the scale is less than the lower limit. or greater than the upper limit A penalty is incurred at that time; N represents the set of spatially adjacent Gaussian units. This represents the neighborhood weight determined by spatial distance, surface orientation similarity, and foreground attribution consistency. Indicates the first The Gaussian unit in the first The center displacement increment of the frame, Indicates rotation parameters, The opacity parameter is represented. The design reference point tracking constraint and the reprojection constraint, trajectory smoothing constraint and parameter regularization ideas in the literature on 3D Gaussian optimization are used for the loss term, and are adapted to the characteristics of the division of labor between trajectory Gaussian units and ordinary Gaussian units in this application.

[0066] Furthermore, to clarify the parameter relationships and summation range of formulas (9) to (13), the summation range of formula (9) includes the viewpoints that satisfy the visibility and trajectory association conditions. Two-dimensional tracking points and time frame Formula (10) applies to the same trajectory point The three-dimensional position sequence on adjacent time frames is summed; Formula (11) sums the scale parameters in the trajectory Gaussian cell set T, where the scale over-limit penalty term can adopt a piecewise quadratic penalty, that is, the penalty is only generated when the scale of the trajectory Gaussian cell is less than the preset lower scale limit or greater than the preset upper scale limit; Formula (12) sums the spatially adjacent Gaussian cell pairs (i, j) ∈ N and time frames. Summation; Formula (13) is for Gaussian elements. and time frame The summation is performed, where the difference in rotation parameters can be calculated using the Frobenius norm of the rotation matrix, the normalized quaternion distance, or the equivalent rotation parameter distance, to ensure that the rotation constraints are consistent with the form of the rotation parameters used.

[0067] In this embodiment, during iterative training and correction of the pre-updated 3D Gaussian unit set, the correction amount of camera parameters from each viewpoint is also used as an optimization variable. The camera parameters are simultaneously corrected during iterative training, achieving joint closed-loop optimization of camera parameters and 3D Gaussian unit parameters. When the camera parameters have been obtained through fixed calibration and the accuracy requirements are met, the training and correction process can fix the camera parameters and only correct the center position, rotation parameters, scale parameters, color parameters, and opacity parameters of each 3D Gaussian unit, as well as the position increment, rotation increment, and scale increment corresponding to each time step. However, when there are errors in the camera parameters, insufficient texture in the acquisition area, or when it is desirable to reduce complex calibration procedures, the correction amounts of the camera intrinsic and extrinsic parameters are added as constraint variables to the joint loss function for optimization, forming a closed loop between camera parameter estimation and dynamic Gaussian training. Incorporating camera parameter correction, trajectory Gaussian constraints, ordinary Gaussian appearance representation, and rendering loss constraints into the same training and correction process can maintain multi-view geometric consistency in low-camera scenarios such as four fixed cameras, reducing camera pose drift and geometric misalignment caused by relying solely on pre-calibrated parameters or local photometric errors.

[0068] In this embodiment, a phased training strategy is adopted for iterative training and correction of the pre-updated 3D Gaussian unit set.

[0069] Specifically, the first stage is the warm-up training stage. In this stage, the parameters of ordinary Gaussian units are fixed, and only the geometric parameters and camera parameters of trajectory Gaussian units are calibrated. This allows the geometric relationships and trajectory constraints to stabilize first, providing reliable geometric anchors for subsequent full training. The second stage is the full training stage. After convergence in the first stage, the parameter constraints of all 3D Gaussian units are released, and all parameters such as position, rotation, scale, color, and opacity in each 3D Gaussian unit are updated synchronously. At the same time, the geometric anchoring effect of the trajectory Gaussian units is maintained through scale constraints and trajectory constraints. Through the above-mentioned phased training strategy, it is possible to avoid ordinary Gaussian units deviating from the real surface due to gradient instability in the early stages of training, while ensuring that the geometric anchoring effect of the trajectory Gaussian units is fully utilized.

[0070] Furthermore, the convergence criteria for the first stage include at least one of the following: the decrease in the joint loss function or trajectory reprojection constraint is less than a preset proportion in several consecutive iterations; the average 2D reprojection error of the trajectory Gaussian unit is less than a preset pixel threshold; the 3D trajectory consistency constraint of adjacent frames tends to stabilize; or the number of iterations reaches a preset upper limit. In the first stage, the weight of trajectory-related constraints is higher than the weight of color reconstruction error and structural similarity error to prioritize obtaining stable geometric anchor points; after entering the second stage, the weights of color reconstruction error and structural similarity error are increased, while the non-zero weights of 2D trajectory reprojection constraints, 3D trajectory consistency constraints, and trajectory Gaussian scale constraints are retained, so that the optimization of appearance details will not destroy the established trajectory geometric relationships.

[0071] Optionally, the weights between the two stages can be varied using a piecewise constant or smooth scheduling method. For example, during the warm-up training stage, the trajectory reprojection weight and the 3D trajectory consistency weight can be set to larger values, while the color reconstruction weight and the structural similarity weight can be set to smaller values. During the full training stage, the color reconstruction weight and the structural similarity weight can be gradually increased, while the trajectory reprojection weight, the 3D trajectory consistency weight, and the scale constraint weight can be kept above a preset lower limit. Through the above convergence determination and weight scheduling, the 3D motion can be stabilized first under conditions of few viewpoints, occlusion, or low texture, and then the Gaussian appearance expression can be refined.

[0072] like Figure 5 The diagram shown illustrates the complete workflow of this method. Combining the steps described above, after inputting a multi-view synchronized video sequence, this method executes sequentially along the following main lines: First, during the multi-view video input stage, the synchronous image sequences acquired by each camera undergo foreground extraction processing to separate the dynamic target region, providing an effective sampling range for subsequent point tracking. At the same time, the initial frame is initialized with 3D Gaussian to establish an initial 3D Gaussian point cloud set, and the camera parameters are determined or corrected.

[0073] Secondly, in the initialization and point tracking stage, two-dimensional point tracking is performed on the foreground region of each frame image from each viewpoint to obtain the two-dimensional displacement data, tracking confidence, and visibility information of each tracking point between adjacent frames. Combining the camera parameters from each viewpoint, the two-dimensional trajectories projected onto the same spatial region from multiple viewpoints are subjected to cross-viewpoint correlation matching and confidence weighting to solve for the center displacement, rotation increment, and scale increment of the corresponding three-dimensional Gaussian points.

[0074] Next, in the motion compensation and Gaussian pre-update stage, the solved 3D center displacement, rotation increment, and scale increment are applied to the 3D Gaussian points of the current frame to generate a pre-updated 3D Gaussian point cloud set for the next frame. This allows the Gaussian points to be moved to a spatial position close to the actual motion of the dynamic target before rendering training begins, reducing the dependence on large displacement photometric gradients. For Gaussian points with low confidence, occlusion, or insufficient visible view, compensation is performed through reliable neighborhood motion propagation or motion regularization.

[0075] Finally, in the loss constraint and rendering output stage, based on the pre-updated 3D Gaussian point cloud set, a joint loss function is constructed, which includes photometric loss, 2D trajectory reprojection constraint, 3D trajectory consistency constraint, geometric constraint, scale constraint, and motion smoothness constraint. This function is used to iteratively train and correct the Gaussian point parameters and, if necessary, camera parameters. After correction, a dynamic 3D Gaussian sequence is output through differentiable rendering, supporting dynamic rendering output from any viewpoint. Thus, through frame-by-frame iterative updates, a temporally continuous dynamic 3D Gaussian representation is obtained.

[0076] Thus, through the frame-by-frame iterative processing of steps S100 to S150 described above, this embodiment can obtain a continuously changing three-dimensional Gaussian sequence of the target dynamic scene on the time axis. Each frame in this sequence contains a set of three-dimensional Gaussian points that have undergone motion pre-update and joint loss optimization correction, fully recording the spatial geometry, surface texture color, and motion state of local areas of the dynamic target at each moment, thereby achieving a complete three-dimensional representation of the dynamic scene.

[0077] Building upon this foundation, the dynamic 3D Gaussian sequence output by this method can be directly applied to various practical 3D vision tasks. Specifically, based on this sequence, 3D Gaussian points at each moment can be projected onto any specified virtual viewpoint through a differentiable rendering pipeline, synthesizing a 2D image of the corresponding moment from that viewpoint, thereby generating a free-viewpoint video. Users can freely switch viewing angles during playback, watching the movement of dynamic targets from any direction. This free-viewpoint video can be widely used in scenarios such as multi-angle playback of sports events, dance movement analysis, and film and television special effects preview. In addition, this dynamic 3D Gaussian sequence can also be used to construct dynamic digital human models, digitizing the movement and appearance of real people, thereby driving real-time animation rendering of virtual images, serving application needs such as virtual shooting, immersive interaction, and game character generation. In the field of autonomous driving, this sequence can also serve as a spatiotemporal representation of dynamic obstacles or pedestrians, providing 3D geometric input for surrounding environment perception and behavior prediction.

[0078] It should be understood that the above application scenarios are merely illustrative examples of the uses of the output results of this method and do not constitute any limitation on the scope of protection of the method itself. The core of this invention lies in providing a method for generating dynamic 3D Gaussian sequences from multi-view videos. The output 3D Gaussian sequence serves as a general 3D representation of dynamic scenes and can be flexibly applied by those skilled in the art to various downstream 3D vision tasks according to actual needs.

[0079] In the aforementioned dynamic 3D Gaussian update method driven by 2D point tracking, the steps are not executed in isolation, but rather form an organic whole around the core idea of ​​using 2D point tracking motion priors to guide 3D Gaussian pre-updates, and then using joint optimization to achieve fine parameter correction. Specifically, the multi-view 2D point trajectories extracted in step S110 are transformed into center displacement, rotation increment, and scale increment in 3D space in step S120 through cross-view association and confidence weighting; step S130 uses these increments to complete the pre-update of Gaussian parameters before rendering training, so that the Gaussian point cloud spatially approximates the distribution after real motion in advance; step S140 further refines the pre-update results through a joint loss function, unifying image appearance information, trajectory observation information, and physical priors of motion continuity into the optimization framework, and finally forming a complete dynamic 3D Gaussian sequence frame by frame in step S150.

[0080] The aforementioned technical effects are not achieved by a single step, but rather by the coordinated and interconnected actions of these steps. First, the point-tracking-driven pre-update mechanism ensures that the Gaussian points obtain reasonable initial motion values ​​before the backpropagation of the photometric gradient, effectively alleviating the problems of gradient mismatch and convergence difficulties caused by large displacement motion in scenarios with few viewpoints. Second, the 3D motion increment solved across viewpoints simultaneously covers center displacement, rotational changes, and scale changes, enabling the Gaussian points to more comprehensively represent the local non-rigid deformation of dynamic targets. This process relies on the constraints of multi-view geometric relationships in step S120 and the guarantee of motion smoothness constraints in step S140; executing either step alone cannot achieve this effect. Third, camera parameters participate in the constraints of motion solving in step S120 and joint optimization in step S140 simultaneously, allowing camera calibration errors to be continuously corrected during iteration. This cross-step parameter coupling is difficult to achieve with independent piecewise processing methods. Finally, the confidence weighting in step S120, the motion reliability determination before step S130, and the regularization of multiple loss terms in step S140 together constitute a complete robustness guarantee mechanism, which effectively suppresses the negative impacts of occlusion, low texture, and tracking errors.

[0081] In summary, this method organically combines 2D point tracking, multi-view geometric solving, 3D Gaussian pre-update, and joint optimization training, creating a positive loop between each step: point tracking provides motion priors, pre-update narrows the optimization range, the joint loss function achieves fine correction, and the corrected Gaussian parameters provide a more accurate projection area for point tracking in the next frame. Under this closed-loop architecture, the convergence speed, geometric consistency, temporal stability, and rendering quality of dynamic 3D Gaussian reconstruction are systematically improved, making it particularly suitable for applications such as dynamic human reconstruction, free-view video generation, digital human asset generation, and virtual shooting in scenarios with few viewpoints and low cost, such as four fixed cameras.

[0082] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0083] In one embodiment, such as Figure 6 As shown, a dynamic three-dimensional Gaussian update device driven by two-dimensional point tracking is provided, including: a data acquisition module 200, an inter-frame tracking module 210, an incremental solution module 220, a parameter pre-update module 230, a training correction module 240, and a temporal iteration module 250, wherein: The data acquisition module 200 is used to acquire a continuous image frame sequence synchronously acquired from multiple perspectives, camera parameters corresponding to each perspective, and an initial three-dimensional Gaussian unit set corresponding to the initial frame. The three-dimensional Gaussian unit includes center position, rotation parameters, scale parameters, color parameters, and opacity parameters. The inter-frame tracking module 210 is used to perform two-dimensional point tracking on the foreground region of the dynamic target in each frame of the continuous image frame sequence from each viewpoint, so as to obtain the two-dimensional displacement data of each two-dimensional tracking point between adjacent frames and the corresponding tracking confidence. The incremental solution module 220 is used to take the three-dimensional Gaussian unit set corresponding to the current frame as a reference, based on the camera parameters, combined with the corresponding two-dimensional displacement data and tracking confidence, to perform cross-view association matching and confidence weighting on the two-dimensional tracking points that fall into the image area corresponding to the same three-dimensional Gaussian unit after projection from each viewpoint, and to solve the three-dimensional center displacement, rotation increment and scale increment of the corresponding three-dimensional Gaussian unit according to the multi-view two-dimensional trajectory distribution after association. The parameter pre-update module 230 is used to pre-update the geometric parameters of the corresponding three-dimensional Gaussian unit based on the solved three-dimensional center displacement, rotation increment and scale increment, and generate the pre-updated three-dimensional Gaussian unit set for the next frame. The training and correction module 240 is used to construct a joint loss function that includes color reconstruction error, structural similarity error, two-dimensional trajectory reprojection constraint and neighborhood motion smoothing constraint. The optimization objective is to minimize the value of the joint loss function. The module then iteratively trains and corrects the pre-updated three-dimensional Gaussian unit set to obtain the final three-dimensional Gaussian unit set for the next frame. The temporal iteration module 250 is used to obtain a temporally continuous dynamic three-dimensional Gaussian sequence by iteratively updating the three-dimensional Gaussian unit set frame by frame.

[0084] Specific limitations regarding the 2D point tracking-driven dynamic 3D Gaussian update device can be found in the limitations of the 2D point tracking-driven dynamic 3D Gaussian update method described above, and will not be repeated here. Each module in the aforementioned 2D point tracking-driven dynamic 3D Gaussian update device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0085] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a dynamic three-dimensional Gaussian update method driven by two-dimensional point tracking. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0086] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0087] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps: Acquire a continuous sequence of image frames acquired synchronously from multiple viewpoints, camera parameters corresponding to each viewpoint, and an initial set of three-dimensional Gaussian units corresponding to the initial frame. The three-dimensional Gaussian units include center position, rotation parameters, scale parameters, color parameters, and opacity parameters. In each viewpoint, two-dimensional point tracking is performed on the foreground region of the dynamic target in each frame of the continuous image frame sequence to obtain the two-dimensional displacement data of each two-dimensional tracking point between adjacent frames and the corresponding tracking confidence. Based on the set of three-dimensional Gaussian units corresponding to the current frame, and based on the camera parameters, combined with the corresponding two-dimensional displacement data and tracking confidence, the two-dimensional tracking points that fall into the image region corresponding to the same three-dimensional Gaussian unit after projection under each viewpoint are cross-viewpoint association matching and confidence weighting. Based on the multi-view two-dimensional trajectory distribution after association, the three-dimensional center displacement, rotation increment and scale increment of the corresponding three-dimensional Gaussian unit are solved. Based on the obtained 3D center displacement, rotation increment and scale increment, the geometric parameters of the corresponding 3D Gaussian element are pre-updated to generate the pre-updated 3D Gaussian element set for the next frame. A joint loss function is constructed, which includes color reconstruction error, structural similarity error, two-dimensional trajectory reprojection constraint and neighborhood motion smoothing constraint. The optimization objective is to minimize the value of the joint loss function. The pre-updated three-dimensional Gaussian unit set is iteratively trained and corrected to obtain the final three-dimensional Gaussian unit set of the next frame. By iteratively updating the set of three-dimensional Gaussian units frame by frame, a temporally continuous dynamic three-dimensional Gaussian sequence is obtained.

[0088] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire a continuous sequence of image frames acquired synchronously from multiple viewpoints, camera parameters corresponding to each viewpoint, and an initial set of three-dimensional Gaussian units corresponding to the initial frame. The three-dimensional Gaussian units include center position, rotation parameters, scale parameters, color parameters, and opacity parameters. In each viewpoint, two-dimensional point tracking is performed on the foreground region of the dynamic target in each frame of the continuous image frame sequence to obtain the two-dimensional displacement data of each two-dimensional tracking point between adjacent frames and the corresponding tracking confidence. Based on the set of three-dimensional Gaussian units corresponding to the current frame, and based on the camera parameters, combined with the corresponding two-dimensional displacement data and tracking confidence, the two-dimensional tracking points that fall into the image region corresponding to the same three-dimensional Gaussian unit after projection under each viewpoint are cross-viewpoint association matching and confidence weighting. Based on the multi-view two-dimensional trajectory distribution after association, the three-dimensional center displacement, rotation increment and scale increment of the corresponding three-dimensional Gaussian unit are solved. Based on the obtained 3D center displacement, rotation increment and scale increment, the geometric parameters of the corresponding 3D Gaussian element are pre-updated to generate the pre-updated 3D Gaussian element set for the next frame. A joint loss function is constructed, which includes color reconstruction error, structural similarity error, two-dimensional trajectory reprojection constraint and neighborhood motion smoothing constraint. The optimization objective is to minimize the value of the joint loss function. The pre-updated three-dimensional Gaussian unit set is iteratively trained and corrected to obtain the final three-dimensional Gaussian unit set of the next frame. By iteratively updating the set of three-dimensional Gaussian units frame by frame, a temporally continuous dynamic three-dimensional Gaussian sequence is obtained.

[0089] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0090] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A dynamic three-dimensional Gaussian update method driven by two-dimensional point tracking, characterized in that, The method includes: Acquire a continuous sequence of image frames acquired synchronously from multiple viewpoints, camera parameters corresponding to each viewpoint, and an initial set of three-dimensional Gaussian units corresponding to the initial frame. The three-dimensional Gaussian units include center position, rotation parameters, scale parameters, color parameters, and opacity parameters. In each viewpoint, two-dimensional point tracking is performed on the foreground region of the dynamic target in each frame of the continuous image frame sequence to obtain the two-dimensional displacement data of each two-dimensional tracking point between adjacent frames and the corresponding tracking confidence. Based on the set of three-dimensional Gaussian units corresponding to the current frame, and based on the camera parameters, combined with the corresponding two-dimensional displacement data and tracking confidence, the two-dimensional tracking points that fall into the image region corresponding to the same three-dimensional Gaussian unit after projection under each viewpoint are cross-viewpoint association matching and confidence weighting. Based on the multi-view two-dimensional trajectory distribution after association, the three-dimensional center displacement, rotation increment and scale increment of the corresponding three-dimensional Gaussian unit are solved. Based on the obtained 3D center displacement, rotation increment and scale increment, the geometric parameters of the corresponding 3D Gaussian element are pre-updated to generate the pre-updated 3D Gaussian element set for the next frame. A joint loss function is constructed, which includes color reconstruction error, structural similarity error, two-dimensional trajectory reprojection constraint and neighborhood motion smoothing constraint. The optimization objective is to minimize the value of the joint loss function. The pre-updated three-dimensional Gaussian unit set is iteratively trained and corrected to obtain the final three-dimensional Gaussian unit set of the next frame. By iteratively updating the set of three-dimensional Gaussian units frame by frame, a temporally continuous dynamic three-dimensional Gaussian sequence is obtained.

2. The dynamic three-dimensional Gaussian update method driven by two-dimensional point tracking according to claim 1, characterized in that, The process of obtaining the two-dimensional displacement data of each two-dimensional tracking point between adjacent frames and the corresponding tracking confidence includes: Perform bidirectional tracking consistency detection, local neighborhood motion consistency detection, and cross-view visibility detection on the two-dimensional tracking points from various perspectives; Adjust the tracking confidence of each two-dimensional tracking point based on the detection results, and remove outlier tracking points whose tracking confidence is lower than the preset threshold.

3. The dynamic three-dimensional Gaussian update method driven by two-dimensional point tracking according to claim 1, characterized in that, The three-dimensional Gaussian unit is divided into trajectory Gaussian unit and ordinary Gaussian unit. The trajectory Gaussian unit is used to undertake the geometric anchoring function, and the ordinary Gaussian unit is used to undertake the expression of appearance details. When solving for the three-dimensional center displacement, rotation increment, and scale increment of the three-dimensional Gaussian element: For the trajectory Gaussian unit, the three-dimensional motion increment is directly solved based on the high-confidence two-dimensional trajectory within the projection region of the trajectory Gaussian unit itself. For the ordinary Gaussian unit, motion information is propagated based on the spatial distance, surface orientation similarity, and foreground attribution relationship between each ordinary Gaussian unit and its neighboring trajectory Gaussian units to obtain the corresponding three-dimensional motion increment.

4. The dynamic three-dimensional Gaussian update method driven by two-dimensional point tracking according to claim 1, characterized in that, When solving for the rotation increment and scale increment in the three-dimensional Gaussian element: Based on the changes in the covariance distribution of the two-dimensional trajectory set in adjacent frames under multiple perspectives, the three-dimensional covariance increment of the corresponding three-dimensional Gaussian unit is estimated. The three-dimensional covariance increment is decomposed into rotation increment and scale increment, which correspond to the orientation change and local deformation degree of the local surface of the dynamic target, respectively.

5. The dynamic three-dimensional Gaussian update method driven by two-dimensional point tracking according to claim 1, characterized in that, After generating the pre-updated 3D Gaussian unit set for the next frame and before iterative training and correction, the process also includes: Motion reliability is determined for each 3D Gaussian unit after pre-update. When a 3D Gaussian unit meets at least one of the following conditions, it is determined to be an unreliable Gaussian unit: insufficient number of visible viewpoints, tracking confidence is lower than a preset threshold, 3D ray intersection residual is greater than a preset threshold, scale change exceeds a preset range, or motion direction deviates from the neighborhood trend. For unreliable Gaussian cells, reduce the corresponding motion estimation weights, or correct them by propagating motion information from nearby reliable Gaussian cells.

6. The dynamic three-dimensional Gaussian update method driven by two-dimensional point tracking according to claim 1, characterized in that, When iteratively training and correcting the pre-updated 3D Gaussian unit set, the correction amount of the camera parameters from each viewpoint is also used as an optimization variable. The camera parameters are simultaneously corrected during the iterative training process to achieve joint closed-loop optimization of camera parameters and 3D Gaussian unit parameters.

7. The dynamic three-dimensional Gaussian update method driven by two-dimensional point tracking according to claim 3, characterized in that, The iterative training and correction of the pre-updated 3D Gaussian unit set adopts a staged training strategy, including: First warm-up training phase: fix the parameters of the ordinary Gaussian unit, and only correct the geometric parameters of the trajectory Gaussian unit and the camera parameters to stabilize the geometric constraint relationship; The second complete training phase: release the parameter constraints of all three-dimensional Gaussian units and synchronously update all parameters in each three-dimensional Gaussian unit.

8. The dynamic three-dimensional Gaussian update method driven by two-dimensional point tracking according to claim 1, characterized in that, The joint loss function also includes three-dimensional trajectory consistency constraints, trajectory Gaussian scaling constraints, and geometric regularization constraints; The three-dimensional trajectory consistency constraint is used to constrain the spatial continuity of the three-dimensional trajectory points associated with the two-dimensional trajectory, the trajectory Gaussian scaling constraint is used to maintain the geometric anchoring properties of the trajectory Gaussian unit, and the geometric regularization constraint is used to suppress unreasonable parameter mutations of the Gaussian unit.

Images