Dynamic scene reconstruction method, computer equipment and program product

Abstract

The invention discloses a dynamic scene reconstruction method, computer equipment and a program product. According to the method, picture information and camera poses of the same scene are obtained, a standard field composed of three-dimensional Gaussian primitives is constructed in a three-dimensional space, a deformation field is constructed, the standard field and the deformation field at different times are subjected to joint rendering in a three-dimensional Gaussian splashing mode, and various parameters are trained and optimized according to the difference between a synthetic image and a real image. Performing motion perception according to the motion condition of the three-dimensional Gaussian primitives, taking the primitives with smaller motion as static primitives, and performing partition modeling on the primitives with larger motion in time to obtain a layered dynamic scene model; and calling the model for rendering according to the given time and the camera pose in the reasoning stage, and outputting a target image. According to the method, the reconstruction precision of the high-dynamic target can be improved while the rendering efficiency is ensured, and the continuity and visual quality of the dynamic scene in the time dimension are improved by introducing the time consistency constraint at the boundary of the time subinterval.

Description

Technical Field

[0001] This application relates to the fields of computer graphics technology and deep learning technology, and in particular to a dynamic scene reconstruction method, computer device and program product. Background Technology

[0002] Currently, 3D reconstruction techniques for static scenes can generate high-quality image results. NeRF (Neural Radiance Field) uses an implicit MLP (Multi-Layer Perceptron) to learn and encode a continuous representation of a five-dimensional light field, and renders images from new perspectives using ray stepping. In contrast, 3DGS (3D Gaussian Splatting) uses a three-dimensional Gaussian representation method to explicitly represent 3D space and generates 2D images through projection and high-speed rasterization.

[0003] Currently, dynamic scene reconstruction solutions are mainly based on NeRF and 3DGS, capturing dynamic changes by learning deformation fields. NeRF-based methods are limited by their ray sampling and point-by-point query MLP characteristics, resulting in high training computation costs and difficulties in real-time rendering. To achieve real-time rendering, 3DGS-based models such as E-D3DGS (Embedding-Based Deformable 3DGaussian Splatting) employ globally shared canonical fields and deformation models. However, because a single model cannot perfectly capture long sequences of complex motion, its "temporal averaging effect" is significant, easily causing image distortion in highly dynamic scenes and blurring and ghosting of rapidly moving targets. SWinGS (Sliding Windows for Dynamic 3D Gaussian Splatting) attempts to divide video sequences into multiple windows and train independent models to alleviate the temporal averaging effect, but the connection between models may produce image roughness, and the complex pre- and post-processing workflows also have obvious limitations. All of the above methods struggle to balance reconstruction accuracy and inference efficiency.

[0004] Therefore, there is an urgent need to propose a new dynamic scene reconstruction method that can save computational costs, retain the high-efficiency rendering advantages of 3D Gaussian splashing, and at the same time ensure the temporal coherence of the global model and the details of the local model, so as to achieve high-fidelity dynamic scene reconstruction. Summary of the Invention

[0005] Embodiments of this application provide a dynamic scene reconstruction method, computer device, and program product to solve the above-mentioned technical problems.

[0006] This application proposes a dynamic scene reconstruction method in its first aspect, comprising: acquiring multi-view images and camera poses of the same dynamic scene at multiple times; constructing a canonical field composed of multiple three-dimensional Gaussian primitives in three-dimensional space based on the camera poses; constructing a deformation field that associates temporal information with the three-dimensional Gaussian primitives and outputs the changes in primitive parameters; jointly rendering the canonical field and the deformation field at different times using a three-dimensional Gaussian splashing method; and training and optimizing the canonical field and the deformation field based on the difference between the rendering result and the real image; performing time-separated modeling of the dynamic part according to the motion of the three-dimensional Gaussian primitives during or after training to obtain a hierarchical dynamic scene model including the canonical field and local deformation parameters of multiple time sub-intervals; and, in the inference stage, calling the hierarchical dynamic scene model to perform three-dimensional Gaussian splashing rendering for any given time and camera pose, and outputting the target image at the corresponding time and viewpoint.

[0007] Furthermore, the construction of a canonical field composed of multiple three-dimensional Gaussian primitives includes: performing structural self-motion recovery based on the multi-view image and camera pose to obtain a sparse three-dimensional point cloud; using the point coordinates in the sparse three-dimensional point cloud as the three-dimensional position mean of the corresponding three-dimensional Gaussian primitive; and using isotropic covariance, unit quaternion pose, initial opacity value, and color parameters obtained by statistical analysis of point cloud color and / or the multi-view image color as the initial parameters of each three-dimensional Gaussian primitive in the canonical field.

[0008] Furthermore, the deformation field includes: a temporal embedding module for encoding continuous time or temporal index as a temporal embedding vector; a primitive embedding vector for assigning a learnable latent vector to each three-dimensional Gaussian primitive; and a multilayer perceptron decoding network that takes the concatenated vector of the temporal embedding vector and the primitive embedding vector as input, the decoding network being used to output the amount of change in the position, shape, and / or appearance parameters of the three-dimensional Gaussian primitive.

[0009] Furthermore, a three-dimensional covariance matrix is ​​constructed based on the rotation matrix and scale vector of each three-dimensional Gaussian element. This three-dimensional covariance matrix is ​​then transformed into a two-dimensional covariance matrix using the projection mapping involved in camera pose determination and its Jacobian matrix, to determine the center position and shape of the two-dimensional Gaussian distribution in the image plane. Viewpoint-related colors are calculated based on the incident line of sight and the spherical harmonic coefficients learned for each three-dimensional Gaussian element. The two-dimensional Gaussian distribution is then cumulatively synthesized based on opacity according to the depth order of the projection centers of each three-dimensional Gaussian element to obtain a synthesized image.

[0010] Furthermore, the training and optimization of the canonical field and the deformation field based on the difference between the rendered result and the real image includes: using the weighted sum of the L1 loss between the synthetic image and the corresponding real image and the structural dissimilarity metric as the image reconstruction constraint, and using an adaptive moment estimation algorithm to iteratively update the parameters of the canonical field, the parameters of the deformation field, and the embedding vectors related to time and primitives.

[0011] Furthermore, the step of performing time-based partitioning modeling of the dynamic part based on the motion of the three-dimensional Gaussian primitives includes: calculating the maximum displacement and position variance of each three-dimensional Gaussian primitive at multiple training time sampling points, and performing quantile normalization on the maximum displacement and the position variance to obtain displacement components and variance components; linearly combining the displacement components and the variance components according to preset weights to obtain motion scores; and marking three-dimensional Gaussian primitives with motion scores not exceeding a preset threshold as static primitives and three-dimensional Gaussian primitives with motion scores exceeding the preset threshold as dynamic primitives.

[0012] Furthermore, the time-based partitioning modeling process is a recursive process, including: dividing the current time range carrying the dynamic primitive into two time sub-intervals, and copying the canonical representation and local deformation parameters associated with the dynamic primitive for each time sub-interval; recalculating the motion score based on the updated 3D position in each time sub-interval; when the motion score in a certain time sub-interval exceeds the preset threshold, continuing to divide the time sub-interval into two and copying the local deformation parameters, until the motion scores in all time sub-intervals do not exceed the preset threshold, at which point the recursive time partitioning terminates.

[0013] Furthermore, the training optimization also includes: at the boundary time or boundary neighborhood of each time sub-interval, for the same camera pose and / or adjacent camera poses, rendering the canonical field using the local deformation parameters on both sides of the boundary respectively, calculating the difference measure of color and / or density of the rendering results on both sides of the boundary, and using the difference measure as a temporal consistency constraint; the temporal consistency constraint and the image reconstruction constraint together constitute a joint loss function, and the canonical field and the local deformation parameters corresponding to each time sub-interval are updated according to the joint loss function.

[0014] In a second aspect, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to, when executing the computer program, implement any of the dynamic scene reconstruction methods described in the first aspect above.

[0015] In a third aspect, this application provides a program product stored in a non-transitory computer-readable storage medium. When the program product is run on a computer device having the non-transitory computer-readable storage medium, the computer device executes the program product to implement any of the dynamic scene reconstruction methods described in the first aspect above.

[0016] The above-mentioned technical solutions have the following advantages or beneficial effects: By introducing gauge fields and deformation fields into the 3D Gaussian splashing framework, the static reference shape of the scene is decoupled from the temporal changes, making the expression of dynamic scenes clearer and improving parameter interpretability and training stability; based on the motion statistics of 3D Gaussian primitives in the time series, motion scores are constructed, and 3D Gaussian primitives with smaller motion are used as static primitives, without applying deformation modeling to them, thereby reducing unnecessary parameters and computational load; for 3D Gaussian primitives with larger motion, their time range is divided into multiple time sub-intervals through recursive time partitioning, and local deformation parameters are configured for each sub-interval, achieving more accurate processing of highly dynamic regions. Detailed time-series modeling improves the reconstruction accuracy of rapidly moving targets; temporal consistency constraints are introduced at the boundaries of time sub-intervals, incorporating the color and / or density differences of the rendering results on both sides of the boundary into the joint loss function, which, together with image reconstruction constraints, optimizes the normal field and local deformation parameters, effectively alleviating the visual seam problem at the boundaries of time sub-intervals and improving the continuity of dynamic scenes in the temporal dimension; an explicit scene representation and rasterization rendering method using 3D Gaussian splashing are adopted, combined with the above-mentioned layered dynamic scene modeling strategy, which ensures high rendering efficiency while maintaining rendering quality, making it suitable for application scenarios with high requirements for real-time performance and image quality, such as virtual reality, free-viewpoint video, and digital asset creation. Attached Figure Description

[0017] Figure 1 A flowchart of a dynamic scene reconstruction method provided in this application; Figure 2 A schematic diagram of the dynamic reconstruction method provided in this application; Figure 3 The training optimization logic block diagram provided for this application; Figure 4 This is a schematic diagram of the structure of a computer device provided in this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and beneficial effects of this application clearer, the following detailed description, in conjunction with the accompanying drawings and specific embodiments, further illustrates this application. It should be understood that the specific embodiments described in this specification are merely for explaining this application and are not intended to limit it.

[0019] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified. The terms "comprising," "including," "having," and any variations thereof should be understood as non-exclusive inclusion, meaning that other elements or steps not explicitly listed may also exist. The term "and / or" indicates that the listed items may exist individually or simultaneously in any combination.

[0020] In the description of this application, terms such as "module / unit / device / processor / storage medium / computer program" can be implemented by software, hardware, or a combination of both; "computer-readable storage medium" can be any non-transitory storage medium. Terms such as "acquire / collect / read / receive / calculate / determine / output / generate" are used interchangeably without ambiguity, representing routine data processing operations performed by the system / device / program. Those skilled in the art can reasonably understand the above terms based on the specific context and make equivalent substitutions for their specific implementations without departing from the spirit and scope of this application.

[0021] In the description of this application, the terms related to dynamic scene reconstruction and 3D Gaussian representation are explained as follows: The term "3D Gaussian primitive" refers to a class of 3D primitives used to explicitly represent a scene, each primitive having at least 3D position, covariance parameter, opacity, and color parameters, used to approximately represent a local volume or surface region in 3D space; the term "gaussian field" refers to a 3D Gaussian set composed of multiple 3D Gaussian primitives used to characterize the static reference shape of a scene, which typically does not change explicitly over time; the term "deformation field" refers to the mapping relationship between temporal information and 3D Gaussian primitives, outputting the changes in the position, shape, and / or appearance parameters of the primitive, which can be implemented through neural networks or other functional forms; the term "local deformation parameter" typically refers to a parameter related to a specific... Deformation field parameters defined within a sub-interval or local time range; the term "gaussian representation" can be understood as the set of reference parameters corresponding to a certain three-dimensional Gaussian element in the gauge field; the term "three-dimensional Gaussian splash" refers to an explicit scene representation based on three-dimensional Gaussian elements, which projects a three-dimensional Gaussian onto the image plane to obtain a two-dimensional Gaussian distribution, and then accumulates and synthesizes it according to depth order and opacity to generate an image; the term "sparse three-dimensional point cloud" refers to a set of points that are relatively discretely distributed in three-dimensional space, obtained through multi-view observation or other three-dimensional reconstruction methods, used to roughly represent the geometric structure of the scene; the term "structure self-motion recovery" refers to the reconstruction process based on the joint estimation of the three-dimensional structure of the scene and camera motion parameters of multi-view images, which can be understood as a common method in this field. One of the 3D reconstruction algorithms; the term "camera pose" refers to the position and orientation information of the camera in 3D space, usually including translation and rotation, used to determine the projection relationship from 3D space to the camera's imaging plane; the term "temporal embedding vector" refers to the vector representation obtained by encoding continuous time or time index, used to express the differences at different time points in the model; the term "primitive embedding vector" refers to the latent vector assigned to each 3D Gaussian primitive, used to characterize the motion or appearance features of that primitive; the term "decoding network" usually refers to a neural network structure such as a multilayer perceptron that takes temporal embedding vector and primitive embedding vector as inputs and outputs the changes in primitive parameters; the term "motion score" refers to motion statistics based on the displacement and / or position variance of 3D Gaussian primitives over time. The scalar obtained from the quantity calculation is used to characterize the magnitude of the motion of the primitive; the term "temporal sub-interval" refers to a sub-time period divided for a certain three-dimensional Gaussian primitive within the global time range, which can be used to configure different local deformation parameters respectively; the term "hierarchical dynamic scene model" refers to a model that represents a dynamic scene in a hierarchical manner, including the gauge field and the local deformation parameters corresponding to multiple temporal sub-intervals; the term "temporal consistency constraint" refers to a class of constraints used to constrain the rendering results of different temporal sub-intervals to remain continuous or similar in terms of color and / or density at the boundary; the term "image reconstruction constraint" refers to the loss or constraint constructed based on the difference between the synthetic image and the real image, which can include a combination of pixel-level error and structural dissimilarity measure.

[0022] Figure 1 A flowchart illustrating a dynamic scene reconstruction method provided in this application. Figure 1 As shown, an exemplary dynamic scene reconstruction method includes: S1, acquiring scene information and constructing a normalized field; S2, constructing a deformation field, jointly rendering the normalized field and the deformation field, and training and optimizing them; S3, performing time-partition modeling and dynamic scene layering; S4, calling the layered dynamic scene model to perform three-dimensional Gaussian splash rendering. Specifically, step S1 includes acquiring multi-view images and camera poses of the same dynamic scene at multiple times, and constructing a canonical field composed of multiple three-dimensional Gaussian primitives in three-dimensional space based on the camera poses; step S2 includes constructing a deformation field that associates time information with the three-dimensional Gaussian primitives and outputs the changes in primitive parameters, jointly rendering the canonical field and deformation field at different times using a three-dimensional Gaussian splashing method, and training and optimizing the canonical field and deformation field based on the difference between the rendering result and the real image; step S3 includes performing time-segregated modeling of the dynamic part according to the motion of the three-dimensional Gaussian primitives during or after training, obtaining a hierarchical dynamic scene model including the canonical field and local deformation parameters of multiple time sub-intervals; step S4 includes, during the inference stage, calling the hierarchical dynamic scene model to perform three-dimensional Gaussian splashing rendering for any given time and camera pose, and outputting the target image at the corresponding time and viewpoint.

[0023] In step S1 above, the construction of the canonical field includes: performing structural self-motion recovery based on multi-view images and camera poses to obtain a sparse 3D point cloud; using the point coordinates in the sparse 3D point cloud as the mean 3D position of the corresponding 3D Gaussian elements; and using isotropic covariance, unit quaternion pose, initial opacity value, and color parameters obtained from point cloud color and / or multi-view image color statistics as the initial parameters of each 3D Gaussian element in the canonical field. In some specific embodiments, the COLMAP tool is used to process the video sequence of the scene, estimate the camera position for each frame, and simultaneously reconstruct a sparse 3D point cloud. For example, given one or more video sequences, COLMAP first decomposes them into a series of image frames. Subsequently, through an incremental reconstruction process using SfM (Structure from Motion), the sparse 3D point cloud structure of the scene is calculated. and the camera pose corresponding to each frame. The deformation field includes: a temporal embedding module for encoding continuous time or temporal indexes as temporal embedding vectors; a primitive embedding vector for assigning a learnable latent vector to each 3D Gaussian primitive; and a multilayer perceptron decoding network that takes the concatenated vector of the temporal embedding vector and the primitive embedding vector as input, the decoding network being used to output the amount of change in the position, shape, and / or appearance parameters of the 3D Gaussian primitive.

[0024] In step S2 above, the 3D Gaussian splashing method is implemented through the following process: A 3D covariance matrix is ​​constructed based on the rotation matrix and scale vector of each 3D Gaussian primitive. The 3D covariance matrix is ​​then transformed into a 2D covariance matrix using the projection mapping involved in camera pose determination and its Jacobian matrix, to determine the center position and shape of the 2D Gaussian distribution in the image plane. Viewpoint-related colors are calculated based on the incident line of sight and the spherical harmonic coefficients learned for each 3D Gaussian primitive. The 2D Gaussian distribution is then cumulatively synthesized based on opacity according to the depth order of the projection centers of each 3D Gaussian primitive to obtain a synthesized image. The process of training and optimizing the gauge field and deformation field based on the difference between the rendered result and the real image is as follows: The weighted sum of the L1 loss between the synthesized image and the corresponding real image and the structural dissimilarity metric is used as an image reconstruction constraint. An adaptive moment estimation algorithm is then used to iteratively update the parameters of the gauge field, the parameters of the deformation field, and the embedding vectors related to time and primitives. The training optimization process can further include: at the boundary time or boundary neighborhood of each time sub-interval, for the same camera pose and / or adjacent camera poses, rendering the canonical field using the local deformation parameters on both sides of the boundary respectively, calculating the difference measure of color and / or density of the rendering results on both sides of the boundary, and using the difference measure as a temporal consistency constraint; combining the temporal consistency constraint and the image reconstruction constraint to form a joint loss function, and updating the canonical field and the local deformation parameters corresponding to each time sub-interval according to the joint loss function.

[0025] In step S3 above, the dynamic part is modeled by time partitioning based on the motion of the three-dimensional Gaussian primitives. The implementation method is as follows: calculate the maximum displacement and position variance of each three-dimensional Gaussian primitive at multiple training time sampling points, and perform quantile normalization on the maximum displacement and position variance to obtain displacement components and variance components; perform linear combination of displacement components and variance components according to preset weights to obtain motion scores; use three-dimensional Gaussian primitives whose motion scores do not exceed a preset threshold as static primitives, and mark three-dimensional Gaussian primitives whose motion scores exceed the threshold as dynamic primitives.

[0026] This application provides a recursive time-based partitioning modeling process, including: dividing the current time range carrying the dynamic primitive into two time sub-intervals, and copying the canonical representation and local deformation parameters associated with the dynamic primitive for each time sub-interval; recalculating the motion score in each time sub-interval based on the updated 3D position; when the motion score in a certain time sub-interval exceeds the preset threshold, continuing to divide the time sub-interval into two and copying the local deformation parameters, until the motion scores in all time sub-intervals do not exceed the preset threshold, at which point the recursive time partitioning is terminated.

[0027] Figure 2 A schematic diagram illustrating the dynamic reconstruction method provided in this application. (See attached image.) Figure 2 As shown, the main graphics processing in the exemplary dynamic reconstruction method provided in this application can be summarized into the following four parts: (a) deformation process; (b) dynamic score calculation; (c) segmentation based on dynamic score; and (d) rendering and loss.

[0028] Figure 3 The training optimization logic block diagram provided for this application is shown below. Figure 3 As shown, the training optimization process can be summarized as follows: During the training phase, time and camera viewpoint are randomly sampled from the training data. The sampled time and viewpoint are input into the global deformation field and interact with the 3D Gaussian primitives in the normalized field to obtain the reconstructed 3D Gaussian primitives at that time. Subsequently, the reconstructed 3D Gaussian primitives are rasterized to generate a rendered image at that time and viewpoint, which is then compared with the corresponding real image to calculate the image loss. Based on the image loss, the global deformation field and related model parameters are optimized and updated, thereby continuously improving the reconstruction accuracy and rendering quality of the 3D Gaussian primitives at any user-specified time and camera viewpoint.

[0029] The following is combined Figure 1 , Figure 2 and Figure 3 The above-described exemplary dynamic reconstruction method will be described in detail.

[0030] In embodiments of this application, a global gauge field is represented as a parameter list. The scenario is... It consists of three-dimensional Gaussian elements, each Gaussian element All are explicitly defined by the following parameters: from a three-dimensional vector Mean of location ; by a scaling vector and a rotation quaternion The covariance of the composition Opacity Regarding the direction of view function That is, the spherical harmonic function. A global deformation field is formed by the temporal embedding vector of each frame of the video. and each Gaussian element Exclusive latent vector Composition. The embedding vector is processed by an MLP coarse-form decoder. Or another independent fine deformation decoder The change can be obtained. or Based on all deformations Moment Gaussian body After rasterization processing involving depth sorting and alpha blending, the final rendered image is synthesized. Compare rendered images with real-world images. Compare and calculate reconstruction loss After backpropagation and parameter update optimization of the gauge field Deformation field and The network weights, recording the embedding vectors. , Embedded module Perform this update using the Adam optimizer and repeat the iteration until the model converges.

[0031] In the aforementioned global gauge field, each three-dimensional Gaussian element calculates a dynamic fraction based on its deformation distance and variance over time. Adaptive motion-aware partitioning is performed. Generally, if 3D Gaussian modeling is performed separately for each frame and dynamic images are synthesized, the image fidelity should be optimal. However, this strategy relies on huge computational costs and involves repeated sampling of features for repetitive slow motion parts. Therefore, this embodiment applies the idea of ​​partitioning modeling to Gaussian primitives and time, classifying primitives with dynamic scores below a specified threshold as static backgrounds and removing their deformation calculations; recursively partitioning primitives with dynamic scores above a specified threshold, the partitioning including: the original normalized primitives... Assigned to the first sub-period And create a copy of it with the same specification properties. To be responsible for the second sub-period Simultaneously, dedicated local deformation fields are replicated and trained for each of these two sub-time periods. and To perform refined motion modeling, the high-speed motion time intervals of highly dynamic Gaussian primitives are segmented, allowing for better representation of their initial shape and deformation. Then, the canonical primitives (originals or their copies) corresponding to their respective time periods are selected, and a deformation field specific to that time period is applied for transformation. The resulting image is then obtained through rasterization. The rendered composite image is compared with the real image to calculate the reconstruction loss. And calculate the cross-frame consistency loss at the partition boundary. Calculate the overall cross-frame consistency loss The global and all local gauge field and deformation field parameters are jointly trained and optimized until the entire hierarchical dynamic scene model converges to meet the expectations, at which point the hierarchical dynamic scene model is frozen.

[0032] In step S2 above, the specific process of training and optimizing the gauge field and deformation field includes: first, performing a random sampling process, from... Randomly select a time point from the frame of the video. and their corresponding camera pose and real images Usage time point As an index, its deformation is obtained from the deformation field. Where c is a low-dynamic Gaussian element and f is a high-dynamic Gaussian element. Dynamic positions are generated using deformation. Dynamic rotation Dynamic scaling Dynamic opacity Dynamic SH coefficient: Thus, a completely new and complete set of three-dimensional Gaussian meta-parameters has been generated at time t. Based on all deformations Moment Gaussian body After rasterization processing involving depth sorting and alpha blending, the final rendered image is a composite image. First, the dynamic rotation... Convert to rotation matrix Dynamic scaling Convert to diagonal scaling matrix The 3D covariance matrix is ​​calculated for subsequent transformations; the specific formula is as follows. ,in, This represents the calculated 3D covariance matrix; Represents the rotation matrix; This represents a diagonal scaling matrix.

[0033] In the above rendering process, it is necessary to project the 3D Gaussian onto the 2D image plane and calculate the 2D mean. and 2D covariance The specific formula is as follows: , ,in, This represents the calculated 2D mean coordinates; This represents the view matrix, also known as the camera extrinsic parameters. Represents the camera intrinsic parameter matrix; Indicates the first Gauss in time 3D mean coordinates; This represents the calculated 2D covariance; Indicates perspective projection The Jacobian matrix at that location.

[0034] This application provides specific formulas for calculating the dynamic color of each 3D Gaussian primitive projection: , ,in, Indicates the first Gauss in time The perspective and direction; Indicates the first Gauss in time The 3D center position; This indicates the position of the camera origin in the world coordinate system; Indicates the first Gauss in time From a specific direction The final color observed; Indicates the order of SH; Indicates the order of SH; Indicates the first Gauss in time SH coefficient; This represents the spherical harmonic basis functions.

[0035] The 2D Gaussian image obtained after projection is used to form the final image through alpha blending. For any pixel on the screen... The specific formula is as follows: , ,in, Indicates the first A 2D Gaussian point at a pixel The lack of transparency in contributions; This indicates the base opacity of the Gaussian point; Represents pixels Compared to the first Gaussian 2D center 2D offset; This represents the aforementioned 2D covariance; Represents pixels The final color; Indicates the covered pixels The total number of 2D Gaussians is ; Indicates the Gaussian index sorted by depth from front to back. ; The aforementioned obtained first A Gaussian color; This indicates that the light reaches the first... Cumulative transmittance before Gauss.

[0036] For each pixel After rendering and calculation, the result is Rendered image at any moment , and real images Compare and calculate reconstruction loss The specific formula is as follows: ,in, This represents the reconstruction loss between the rendered image and the real image; Indicates the weighting factor of the hyperparameter; This represents L1 loss, also known as mean absolute error; This represents the loss due to structural dissimilarity.

[0037] According to reconstruction losses After backpropagation and parameter update optimization of the standard field Deformation field and The network weights, recording the embedding vectors. , Embedded module Perform this update using the Adam optimizer and repeat the iteration until the model converges.

[0038] In step S3 above, the motion-aware partitioning process includes: calculating the dynamic score of each Gaussian cell to assess its motion amplitude within a given time period, and performing subsequent time segmentation based on this motion amplitude, specifically using the following formula: ; ; ; ; .

[0039] In the above formula, The index representing each three-dimensional Gaussian unit; This represents the historical position index of the three-dimensional Gaussian element; Indicates that the index is The maximum displacement of the three-dimensional Gaussian element; Indicates that the index is The three-dimensional Gaussian element in the index is The displacement below; Indicates that the index is The motion variance of the three-dimensional Gaussian element; Indicates that the index is The average position of the three-dimensional Gaussian elements; This represents the maximum displacement after percentile standardization; This represents the variance of motion after percentile standardization; For indicator functions; for The kth percentile ; for The kth percentile ; For Gaussian dynamic fractions.

[0040] Based on the obtained Gaussian dynamic score, adaptive motion-aware partitioning is performed for the global time range. Each three-dimensional Gaussian primitive records its segmentation hierarchy initialized to 0. , and its initialization Time segmentation range For all Gaussian elements with dynamic scores above a given threshold, the corresponding deformation field is expressed as: The Gaussian element replicates itself and its corresponding deformation field, and its application is over the time range. Its hierarchical division is denoted as Its deformation field is applied to The replicated Gaussian unit is applied to the time range. Its hierarchical division is denoted as The replicated deformation field is applied to ;in Repeat the steps of dividing the Gaussian elements and their deformation fields by time until no Gaussian elements with dynamic fractions higher than a given threshold remain.

[0041] For 3D Gaussian units with dynamic scores below a given threshold, their corresponding embedded modules are removed. Embedded vectors in , Treating it as a static 3D Gaussian primitive reduces its computational cost. During image rendering, for a given time... It is necessary to first calculate the three-dimensional Gaussian elements with high dynamic fractions after deformation. After combining it with the static Gaussian primitives with low dynamic scores, the overall rendered image is calculated. At this point, all the 3D Gaussian primitive parameters can be regarded as the result of deformation, and the rendering formula is consistent with the rendering function described in step S2 above.

[0042] For 3D Gaussian primitives with high dynamic scores, an overall cross-frame consistency is introduced to ensure image coherence at temporal partition boundaries after reconstruction. The rendered image is compared with the real image to calculate the reconstruction loss, and the cross-frame consistency loss at partition boundaries is calculated. The overall cross-frame consistency loss is then calculated using the following formula: , , ,in, This represents the reconstruction loss; Indicates time All three-dimensional Gaussian elements used in the reconstruction; Indicates a given observation perspective; Indicates time Based on the three-dimensional Gaussian element and perspective The rendered image; Indicates time The corresponding real image below; This represents the cross-frame consistency loss; Indicates time All three-dimensional Gaussian elements used in the reconstruction; Indicates time Based on the three-dimensional Gaussian element and perspective The rendered image; This represents the overall cross-frame consistency loss.

[0043] According to the obtained After backpropagation and parameter update optimization of the standard field Specific three-dimensional Gaussian elements Its corresponding deformation field and The network weights and their corresponding embedding vectors , Embedded module Perform this update using the Adam optimizer and repeat the iteration until the model converges, at which point the parameters are frozen.

[0044] Figure 4 This is a schematic diagram of the internal structure of a computer device provided in this application. Figure 4As shown, the computer device can be a terminal, including a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements any of the dynamic scene reconstruction methods provided in the embodiments of this application. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad provided on the casing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0045] The aforementioned memory stores a computer program, and when the processor executes the computer program, it performs the following steps: S1. Acquire multi-view images and camera poses of the same dynamic scene at multiple times, and construct a canonical field composed of multiple three-dimensional Gaussian primitives in three-dimensional space based on the camera poses; S2. Construct a deformation field that associates time information with three-dimensional Gaussian primitives and outputs the changes in primitive parameters, and jointly render the canonical field and deformation field at different times using a three-dimensional Gaussian splashing method, and train and optimize the canonical field and deformation field based on the difference between the rendering results and the real image; S3. During or after training, perform time-segregated modeling of the dynamic part according to the motion of the three-dimensional Gaussian primitives to obtain a hierarchical dynamic scene model including the canonical field and local deformation parameters of multiple time sub-intervals; S4. In the inference stage, for any given time and camera pose, call the hierarchical dynamic scene model to perform three-dimensional Gaussian splashing rendering, and output the target image at the corresponding time and viewpoint.

[0046] This application also provides a computer program product, which is stored in a non-transitory computer-readable storage medium. When the program product is run on a computer device having the non-transitory computer-readable storage medium, the computer device executes the program product and implements the following steps: S1. Acquire multi-view images and camera poses of the same dynamic scene at multiple times, and construct a canonical field composed of multiple three-dimensional Gaussian primitives in three-dimensional space based on the camera poses; S2. Construct a deformation field that associates time information with three-dimensional Gaussian primitives and outputs the changes in primitive parameters, and jointly render the canonical field and deformation field at different times using a three-dimensional Gaussian splashing method, and train and optimize the canonical field and deformation field based on the difference between the rendering result and the real image; S3. During or after training, perform time-separated modeling of the dynamic part according to the motion of the three-dimensional Gaussian primitives to obtain a layered dynamic scene model including the canonical field and local deformation parameters of multiple time sub-intervals; S4. In the inference stage, for any given time and camera pose, call the layered dynamic scene model to perform three-dimensional Gaussian splashing rendering and output the target image at the corresponding time and viewpoint. In this embodiment, the program product includes a program code portion for performing the steps of the dynamic scene reconstruction method in the various embodiments of this application when the program product is executed by one or more computing devices. The program product may be stored on a computer-readable recording medium and may also be provided via a data network (e.g., via RAN, via the Internet, and / or via RBS) for download. Alternatively or additionally, the method may be encoded in a field-programmable gate array (FPGA) and / or application-specific integrated circuit (ASIC), or the functionality may be provided by means of a hardware description language for download.

[0047] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0048] In summary, this application proposes a dynamic scene reconstruction technology based on 3D Gaussian splashing. By introducing a canonical field composed of multiple 3D Gaussian primitives, the static reference shape of the scene is decoupled from the time-varying deformation field. Combined with 3D Gaussian splashing rendering, efficient and high-fidelity reconstruction of dynamic scenes is achieved. Furthermore, the motion of the 3D Gaussian primitives is measured using motion fractions based on maximum displacement and position variance. Primitives with less motion are treated as static primitives, while primitives with greater motion are recursively partitioned over time and local deformation parameters are configured for each time sub-interval. This constructs a hierarchical dynamic scene model, which not only improves the accuracy of representing violently moving targets and locally highly dynamic regions but also avoids the redundant overhead of uniformly complex deformation modeling across the entire time range. Meanwhile, a temporal consistency constraint is introduced at the boundary of the temporal sub-interval and forms a joint loss with the image reconstruction constraint. The normal field and local deformation parameters are jointly optimized, which effectively reduces the visual seams at the boundary of the temporal partition and ensures the continuity and visual quality of the dynamic scene in the temporal dimension. Thus, while taking into account the rendering efficiency, the accuracy and stability of the new perspective rendering of the dynamic scene are significantly improved.

[0049] Those skilled in the art will understand that, without departing from the core concept of this invention, all equivalent substitutions or modifications made to the initialization method of the gauge field, the specific network structure of the deformation field, the calculation form of the motion fraction, the division strategy of the time sub-interval, the specific implementation of the time consistency constraint and the image reconstruction constraint, as well as the division and logical deployment of each functional unit in the system, are within the protection scope of this invention.

[0050] The specific embodiments described above are only used to illustrate the technical solutions of this application and are not intended to limit this application. For those skilled in the art, various modifications and changes can be made without departing from the spirit and scope of the technical solutions of this application, and all such modifications and changes should fall within the protection scope of this application.

Claims

1. A dynamic scene reconstruction method, characterized in that, include: Acquire multi-view images and camera poses of the same dynamic scene at multiple times, and construct a canonical field composed of multiple three-dimensional Gaussian primitives in three-dimensional space based on the camera poses; A deformation field is constructed that associates time information with the three-dimensional Gaussian primitives and outputs the changes in primitive parameters. The normal field and the deformation field at different times are jointly rendered using a three-dimensional Gaussian splashing method. The normal field and the deformation field are trained and optimized based on the difference between the rendering results and the real image. During or after training, the dynamic part is modeled in time-separated areas based on the motion of the three-dimensional Gaussian primitives, resulting in a hierarchical dynamic scene model that includes the gauge field and local deformation parameters of multiple time sub-intervals. During the inference phase, for any given time and camera pose, the layered dynamic scene model is invoked to perform 3D Gaussian splash rendering, outputting the target image at the corresponding time and viewpoint.

2. The dynamic scene reconstruction method as described in claim 1, characterized in that, The construction of the gauge field, which consists of multiple three-dimensional Gaussian elements, includes: Based on the multi-view images and camera pose, a structure-based self-motion recovery is performed to obtain a sparse 3D point cloud. The point coordinates in the sparse 3D point cloud are used as the mean 3D position of the corresponding 3D Gaussian elements. The isotropic covariance, unit quaternion pose, initial opacity value, and color parameters obtained by statistical analysis of point cloud color and / or multi-view image color are used as the initial parameters of each 3D Gaussian element in the normalized field.

3. The dynamic scene reconstruction method as described in claim 1, characterized in that, The deformation field includes: A time embedding module used to encode continuous time or time index as a time embedding vector; Assign a primitive embedding vector that can learn a latent vector to each 3D Gaussian primitive; And a multilayer perceptron decoding network that takes the concatenated vector of the temporal embedding vector and the primitive embedding vector as input, the decoding network being used to output the changes in the position, shape and / or appearance parameters of the three-dimensional Gaussian primitives.

4. The dynamic scene reconstruction method as described in claim 1, characterized in that, The three-dimensional Gaussian splashing method includes: A three-dimensional covariance matrix is ​​constructed based on the rotation matrix and scale vector of each three-dimensional Gaussian element. The three-dimensional covariance matrix is ​​then transformed into a two-dimensional covariance matrix through the projection mapping involved in camera pose determination and its Jacobian matrix, so as to determine the center position and shape of the two-dimensional Gaussian distribution in the image plane. The view-dependent color is calculated based on the incident line of sight and the spherical harmonic coefficients learned for each 3D Gaussian unit. Based on the depth order of the projection centers of each 3D Gaussian primitive, the 2D Gaussian distribution is cumulatively synthesized based on opacity to obtain a composite image.

5. The dynamic scene reconstruction method as described in claim 1, characterized in that, The training and optimization of the normalized field and the deformation field based on the difference between the rendered result and the real image includes: The weighted sum of the L1 loss between the synthetic image and the corresponding real image and the structural dissimilarity metric is used as the image reconstruction constraint, and the parameters of the gauge field, the parameters of the deformation field, and the time- and primitive-related embedding vectors are iteratively updated using an adaptive moment estimation algorithm.

6. The dynamic scene reconstruction method as described in claim 1, characterized in that, The step of performing time-based partitioning modeling of the dynamic part based on the motion of the three-dimensional Gaussian primitives includes: The maximum displacement and position variance of each 3D Gaussian element at multiple training time sampling points are calculated, and quantile normalization is performed on the maximum displacement and the position variance to obtain the displacement component and variance component. The displacement component and the variance component are linearly combined according to preset weights to obtain the motion score; Three-dimensional Gaussian primitives with motion scores not exceeding a preset threshold are designated as static primitives, while three-dimensional Gaussian primitives with motion scores exceeding the preset threshold are designated as dynamic primitives.

7. The dynamic scene reconstruction method as described in claim 6, characterized in that, The process of time-based partitioning modeling is a recursive process, including: Divide the current time range carrying the dynamic primitive into two time sub-intervals, and copy the canonical representation and local deformation parameters associated with the dynamic primitive for each time sub-interval; Within each time sub-interval, the motion score is recalculated based on the updated 3D position. When the motion score in a certain time sub-interval exceeds the preset threshold, the time sub-interval is further divided into two parts and the local deformation parameters are copied until the motion scores in all time sub-intervals do not exceed the preset threshold, at which point the recursive time partitioning is terminated.

8. The dynamic scene reconstruction method as described in claim 5, characterized in that, The training optimization also includes: At the boundary time or boundary neighborhood of each time sub-interval, for the same camera pose and / or adjacent camera poses, the normal field is rendered using the local deformation parameters on both sides of the boundary, and the difference measure of the rendering results on both sides of the boundary in color and / or density is calculated, and the difference measure is used as a time consistency constraint. The temporal consistency constraint and the image reconstruction constraint are combined to form a joint loss function, and the local deformation parameters corresponding to the canonical field and each time sub-interval are updated according to the joint loss function.

9. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to implement the dynamic scene reconstruction method as described in any one of claims 1 to 8 when executing the computer program.

10. A program product, characterized in that, The program product is stored in a non-transitory computer-readable storage medium. When the program product is run on a computer device having the non-transitory computer-readable storage medium, the computer device executes the program product to implement the dynamic scene reconstruction method as described in any one of claims 1 to 8.

Images