A training method of a sparse-view dynamic scene reconstruction model, a sparse-view dynamic scene reconstruction method, an electronic device, and a medium

Abstract

The application discloses a dynamic scene reconstruction model training method, an electronic device and a medium, and relates to the technical field of computer graphics, which comprises the following steps: acquiring a reference view collected by an image collection device from a perspective; generating a plurality of perspectives of the reference view by using a generative model to obtain a generated view of a virtual image collection device from a perspective; taking the reference view and the generated view as training supervision basis to train an initial dynamic scene reconstruction model, so that the trained dynamic scene reconstruction model can represent scene characteristics corresponding to the reference view and the generated view, and learn scene compensation information corresponding to the generated view; and in the case that a reconstruction index of the trained dynamic scene reconstruction model is greater than or equal to a preset threshold, taking the trained dynamic scene reconstruction model as an optimized dynamic scene reconstruction model. The application has the characteristics and advantages of guaranteeing efficient and high-quality dynamic scene reconstruction under sparse perspectives.

Description

Technical Field

[0001] This invention relates to the field of computer graphics technology, and in particular to a training method for a dynamic scene reconstruction model with sparse perspective, a dynamic scene reconstruction method with sparse perspective, an electronic device, and a computer-readable medium. Background Technology

[0002] 4D Gaussian Splatting (4DGS) is a representation model for dynamic scenes and is widely used in the field of dynamic scene reconstruction.

[0003] Existing 4DGS-based dynamic scene reconstruction techniques extend the representation of each 3D Gaussian parameter to a time-varying Gaussian parameter by temporally modeling it. This allows information such as the scene's geometry, appearance, and dynamic changes to be encoded into the representation model. In practical applications, this technique typically relies on a dense array system of dozens or even hundreds of cameras. Based on multiple viewpoints and loss optimization mechanisms, the Gaussian parameters are continuously adjusted to make the rendered image closer to the real scene. Furthermore, only by relying on the dense viewpoint constraints provided by the dense camera array can the accuracy of the 4DGS model training be guaranteed, avoiding issues such as flickering and drift in the rendered image, thereby achieving high-quality 4D reconstruction of dynamic scenes.

[0004] Therefore, how to achieve high-precision, spatiotemporally consistent 4D dynamic scene reconstruction with only a small number of cameras and low cost is a key problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] This invention aims to address one of the technical problems in related technologies to a certain extent. To this end, this invention provides a training method for a dynamic scene reconstruction model, a dynamic scene reconstruction method, an electronic device, and a computer-readable medium. These features and advantages include effectively reducing dependence on dense camera arrays, ensuring high-efficiency and high-quality dynamic scene reconstruction even under sparse viewpoints, while simultaneously reducing computational costs, improving model generalization ability, and enhancing practical deployment applicability.

[0006] To achieve the above objectives, as a first aspect of the present invention, a training method for a dynamic scene reconstruction model is provided, wherein the training method includes: Obtain a reference view captured from the perspective of the image acquisition device; Generative models are used to generate multi-view views of the reference view to obtain a generated view of the virtual image acquisition device. Using the reference view and the generated view as training supervision, the initial dynamic scene reconstruction model is trained so that the trained dynamic scene reconstruction model can represent the scene features corresponding to the reference view and the generated view, and learn the scene compensation information corresponding to the generated view. If the reconstruction index of the trained dynamic scene reconstruction model is greater than or equal to the preset threshold, the trained dynamic scene reconstruction model will be used as the optimized dynamic scene reconstruction model.

[0007] Optionally, the initial dynamic scene reconstruction model is trained using the reference view and the generated view as training supervision, so that the trained dynamic scene reconstruction model can represent the scene features corresponding to the reference view and the generated view, and learn the scene compensation information corresponding to the generated view, including: In each round of training, the initial dynamic scene reconstruction model is trained using the reference view as the training supervision basis, so that the trained dynamic scene reconstruction model can fit the scene features corresponding to the reference view. Using the generated view as the training supervision basis, the scene compensation information corresponding to the generated view is learned, and the initial dynamic scene reconstruction model and the scene compensation information are jointly trained so that the trained dynamic scene reconstruction model can fit the overall scene features corresponding to the reference view and the generated view.

[0008] Optionally, the step of learning scene compensation information corresponding to the generated view as training supervision includes: Based on the pose and time information of the virtual image acquisition device corresponding to the generated view, distortion features are extracted from the initial dynamic scene reconstruction model; The distortion features are decoded using a decoder to obtain scene compensation information.

[0009] Optionally, the step of extracting distortion features from the initial dynamic scene reconstruction model based on the pose and time information of the virtual image acquisition device corresponding to the generated view includes: Using the pose and time information of the virtual image acquisition device corresponding to the generated view as a reference, the position information of the Gaussian sphere is extracted from the initial dynamic scene reconstruction model, and the position information, pose information and time information are combined and projected onto multiple two-dimensional feature planes respectively. Interpolate the features on each two-dimensional feature plane to obtain multiple distorted sub-features corresponding to the generated view; The distortion feature is obtained by cascading and fusing multiple distortion sub-features.

[0010] Optionally, the distortion features include distortion position features, distortion scaling features, and distortion rotation features; The process of decoding the distortion features using a decoder to obtain scene compensation information includes: The distortion position features are decoded to obtain the scene position compensation correction amount; Decode the distortion scaling features to obtain the scene scaling compensation correction amount; The distortion rotation features are decoded to obtain the scene rotation compensation correction amount.

[0011] Optionally, the training method further includes: The scene compensation information after each round of training is fused with the dynamic scene reconstruction model after training to obtain the fused dynamic scene reconstruction model. The trained dynamic scene reconstruction model is used for rendering to obtain the first rendered image; The second rendered image is obtained by rendering using a fused dynamic scene reconstruction model; The loss values ​​between the first rendered image and the reference view, and between the second rendered image and the generated view, are determined based on the loss function. Update the parameters of the current dynamic scene reconstruction model and scene compensation information based on the loss value; The step of using the trained dynamic scene reconstruction model as the optimized dynamic scene reconstruction model when the reconstruction index of the trained dynamic scene reconstruction model is greater than or equal to a preset threshold includes: If the similarity between the first rendered image and the reference view, and the similarity between the second rendered image and the generated view are both greater than or equal to a preset threshold, the dynamic scene reconstruction model obtained in this round of training is used as the optimized dynamic scene reconstruction model.

[0012] Optionally, the loss function includes at least one of photometric loss, perceptual loss, variation loss, and smoothness regularization.

[0013] As a second aspect of the present invention, a dynamic scene reconstruction method is provided, wherein the dynamic scene reconstruction method includes: A dynamic scene reconstruction model is obtained; wherein the dynamic scene reconstruction model is a model trained by a training method for a dynamic scene reconstruction model provided according to the first aspect of the present invention. The target time information and target pose information are input into the dynamic scene reconstruction model to obtain the reconstructed image corresponding to the target time information and target pose information.

[0014] As a third aspect of the present invention, an electronic device is provided, comprising: One or more processors; A memory storing one or more computer programs, which, when executed by one or more processors, cause the one or more processors to implement a training method for a dynamic scene reconstruction model provided in the first aspect of the invention, and a dynamic scene reconstruction method provided in the second aspect.

[0015] Furthermore, as a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a training method for a dynamic scene reconstruction model provided in the first aspect of the present invention and a dynamic scene reconstruction method provided in the second aspect.

[0016] This invention addresses the problem that existing 4DGS-based dynamic scene reconstruction technologies require dense camera array systems and have high hardware costs, proposing a training method for dynamic scene reconstruction models. This method can achieve high-precision, spatiotemporally consistent 4D dynamic scene reconstruction using only a small number of cameras.

[0017] This invention first acquires a small number of reference views captured by image acquisition devices at corresponding viewpoints. Then, a generative model is used to generate multi-view views from these reference views, resulting in generated views corresponding to the viewpoints of the virtual image acquisition devices, thus supplementing the viewpoint information not observed by the actual camera. Considering that the generated views output by the generative model suffer from spatiotemporal inconsistencies, artifacts, and distortions, which directly lead to low accuracy and poor performance in dynamic scene reconstruction, this invention uses both the reference views and generated views as training supervision to jointly train the initial dynamic scene reconstruction model. This allows the model to simultaneously represent the scene features corresponding to both the reference and generated views, and adaptively learn scene compensation information matching the generated views to correct for biases and distortions introduced by the generated views. In each training round, the model's performance is evaluated based on preset reconstruction metrics. When the model's reconstruction metrics reach a preset threshold, the trained model is determined as the optimized dynamic scene reconstruction model.

[0018] Therefore, based on the above technical solution, this invention not only enables high-quality dynamic scene reconstruction with a small number of cameras, but also effectively improves the spatiotemporal inconsistency problem introduced by the generated view, enhancing the spatiotemporal consistency and visual accuracy of the reconstruction results. This invention significantly reduces reliance on dense camera hardware, greatly lowers the deployment cost of dynamic scene reconstruction, and effectively improves the flexibility and applicability of dynamic reconstruction technology in real-world scenarios.

[0019] These features and advantages of the present invention will be disclosed in detail in the following specific embodiments and accompanying drawings. The preferred embodiments or means of the present invention will be shown in detail in conjunction with the accompanying drawings, but are not intended to limit the technical solutions of the present invention. In addition, each of these features, elements and components appearing in the following text and drawings is a plurality of, and different symbols or numbers are used for convenience of representation, but all represent parts with the same or similar construction or function. Attached Figure Description

[0020] The present invention will be further described below with reference to the accompanying drawings: Figure 1 A flowchart illustrating a training method for a dynamic scene reconstruction model provided by this invention; Figure 2 A flowchart illustrating one embodiment of step S130 of the training method provided by the present invention; Figure 3 A flowchart illustrating another embodiment of step S132 of the training method provided by the present invention; Figure 4 A flowchart illustrating one embodiment of step S132a of the training method provided by the present invention; Figure 5 A flowchart of one embodiment of step S132b of the training method provided by the present invention; Figure 6 A flowchart illustrating another implementation of step S130 of the training method provided by the present invention; Figure 7 A flowchart illustrating one embodiment of step S140 of the training method provided by the present invention; Figure 8 A flowchart of the dynamic scene reconstruction method provided by the present invention; Figure 9 is a comparison diagram of the reconstruction effects of the present invention and the prior art; Figure 10 A schematic diagram illustrating the entire process of the training method and application reasoning provided by this invention; Figure 11 A module diagram of an electronic device provided by the present invention; Figure 12 This invention provides a schematic diagram of a computer-readable medium.

[0021] Explanation of reference numerals in the attached figures Among them, 101 is the processor; 102 is the memory; 103 is the I / O interface; and 104 is the bus. Detailed Implementation

[0022] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described are intended to explain the present invention and should not be construed as limiting the invention.

[0023] The terms "an embodiment," "example," or "trademark" used in this specification refer to a particular feature, structure, or characteristic described in connection with the embodiment itself that may be included in at least one embodiment disclosed in this invention. The phrase "in an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0024] The reason why existing 4DGS-based dynamic scene reconstruction technologies suffer from problems such as reliance on dense camera arrays, high application costs, and poor scene adaptability lies in the strong dependence of their technical principles on viewpoint information. Specifically, this can be summarized as follows: First, 4DGS encodes the scene's geometric structure, appearance information, and dynamic changes into the model by temporally modeling 3D Gaussian parameters. This encoding process is highly dependent on constraints provided by multiple viewpoints. Only by collecting sufficient viewpoint data through a dense camera array can sufficient basis be provided for adjusting Gaussian parameters, avoiding model training bias. Second, existing 4DGS models have insufficient adaptability to sparse viewpoints. When the number of cameras decreases and the viewpoint distribution is sparse, the model cannot obtain complete spatial information of the scene, making it difficult to accurately recover the scene's geometric structure. This leads to problems such as texture blurring and inconsistent artifacts across multiple viewpoints, preventing high-quality reconstruction. Furthermore, existing 4DGS... The temporal modeling of dynamic scenes is not perfect. Even with a dense camera array, it is difficult to completely solve spatiotemporal consistency problems such as flickering, structural jumps, and discontinuous motion in the time dimension, which affects the reconstruction effect. Finally, the massive amount of data collected by the dense camera array will significantly increase the computational burden of the model, resulting in a long optimization process and high computational complexity for the 4DGS model. At the same time, the model has weak generalization ability and is difficult to adapt to different scene types, viewpoint distributions, and dynamic modes, which further limits its application in consumer and mobile scenarios and cannot meet the requirements of low cost and high efficiency for actual deployment.

[0025] In view of this, in order to solve the above problems, as a first aspect of the present invention, a training method for a dynamic scene reconstruction model is provided, such as... Figure 1 As shown, the training method includes: In step S110, a reference view acquired from the perspective of the image acquisition device is obtained; In step S120, the reference view is generated from multiple perspectives using a generative model to obtain a generated view of the virtual image acquisition device from that perspective. In step S130, the initial dynamic scene reconstruction model is trained using the reference view and the generated view as training supervision, so that the trained dynamic scene reconstruction model can characterize the scene features corresponding to the reference view and the generated view, and learn the scene compensation information corresponding to the generated view. In step S140, if the reconstruction index of the trained dynamic scene reconstruction model is greater than or equal to a preset threshold, the trained dynamic scene reconstruction model is used as the optimized dynamic scene reconstruction model.

[0026] This invention first acquires a small number of reference views captured by image acquisition devices at corresponding viewpoints. Then, a generative model is used to generate multi-view views from these reference views, resulting in generated views corresponding to the viewpoints of the virtual image acquisition devices, thus supplementing the viewpoint information not observed by the actual camera. Considering that the generated views output by the generative model suffer from spatiotemporal inconsistencies, artifacts, and distortions, which directly lead to low accuracy and poor performance in dynamic scene reconstruction, this invention uses both the reference views and generated views as training supervision to jointly train the initial dynamic scene reconstruction model. This allows the model to simultaneously represent the scene features corresponding to both the reference and generated views, and adaptively learn scene compensation information matching the generated views to correct for biases and distortions introduced by the generated views. In each training round, the model's performance is evaluated based on preset reconstruction metrics. When the model's reconstruction metrics reach a preset threshold, the trained model is determined as the optimized dynamic scene reconstruction model.

[0027] Therefore, based on the above technical solution, this invention not only enables high-quality dynamic scene reconstruction with a small number of cameras, but also effectively improves the spatiotemporal inconsistency problem introduced by the generated view, enhancing the spatiotemporal consistency and visual accuracy of the reconstruction results. This invention significantly reduces reliance on dense camera hardware, greatly lowers the deployment cost of dynamic scene reconstruction, and effectively improves the flexibility and applicability of dynamic reconstruction technology in real-world scenarios.

[0028] It should be noted that step S120 includes, but is not limited to, generative models mentioned in this application: generation methods based on diffusion models, conditional prediction methods based on autoregressive models or Transformers, and learning-based image interpolation and extrapolation models. Using generative models, scene information that physical cameras cannot observe can be supplemented even when the number of actual acquisition devices is relatively sparse. For example, viewpoint supplementation of images or features from unobserved perspectives, and temporal supplementation of scene states at unsampled time points. This significantly alleviates the information loss problem caused by sparse input and provides sufficient and reliable viewpoint and temporal supervision information for subsequent joint training and spatiotemporal consistency optimization of the model.

[0029] After obtaining the multi-view input data according to step S120, the training method of the present invention is then used to complete the training. As an optional implementation of step S130, such as... Figure 2 As shown, the initial dynamic scene reconstruction model is trained using a reference view and a generated view as training supervision, so that the trained dynamic scene reconstruction model can represent the scene features corresponding to the reference view and the generated view, and learn the scene compensation information corresponding to the generated view, including: In step S131, during each training round, the initial dynamic scene reconstruction model is trained using the reference view as the training supervision basis, so that the trained dynamic scene reconstruction model can fit the scene features corresponding to the reference view. In step S132, the generated view is used as the training supervision basis to learn the scene compensation information corresponding to the generated view, and the initial dynamic scene reconstruction model and the scene compensation information are jointly trained so that the trained dynamic scene reconstruction model can fit the overall scene features corresponding to the reference view and the generated view.

[0030] The above steps constitute the training strategy of this invention. Its core idea is: in each round of training, the real-world acquired reference view is used as the sample and supervision label to train the initial dynamic scene reconstruction model, so that the dynamic scene reconstruction model can learn accurate dynamic scene information under the reference view (i.e., the viewpoint of the image acquisition device), such as spatial location information, temporal change information, geometric structure information, appearance texture information, and dynamic motion information, ensuring that the model first masters the basic features of a real and clean scene.

[0031] Similarly, during each training round, considering the inherent defects of views generated by generative models, such as spatiotemporal inconsistencies, artifacts, and distortions, directly using them for training would pollute the training process of the initial dynamic scene reconstruction model. Specifically, directly using distorted generated views to train the initial dynamic scene reconstruction model will cause the model to learn incorrect scene features, leading to problems such as rendering flickering and model parameter collapse, ultimately resulting in a significant decrease in the quality of dynamic scene reconstruction. Therefore, this invention simultaneously uses generated views as samples and supervision labels. To solve the above problems and improve model training accuracy, scene compensation information corresponding to the generated views is learned and trained synchronously during training. This scene compensation information compensates for and counteracts the distortions, artifacts, and spatiotemporal inconsistencies present in the generated views.

[0032] In this way, the model can learn the multi-view dynamic scene information corresponding to the generated view, and can also avoid the defects brought by the generated view by the scene compensation information, thus realizing the joint training of the initial dynamic scene reconstruction model and the scene compensation information.

[0033] The essence of the training strategy of this invention is as follows: A dynamic scene reconstruction model is trained using a clean reference view to fit the real scene under the reference view; scene compensation information is trained synchronously using the generated view as supervision to compensate for the inherent defects of the generated view before joint supervised training is performed. It should be noted that steps S131 and S132 are not two independent training phases, nor are they separate training steps; they are trained simultaneously and completed synchronously within the same training round. The core advantage of this strategy is that it can fully utilize the generated view to supplement the viewpoint information not observed by the physical camera, expanding the model's scene representation range, while avoiding the contamination of the model training process by the defects of the generated view, ensuring that the final trained model is pure and has high accuracy, and can stably represent the overall multi-view dynamic scene corresponding to the reference view and the generated view.

[0034] As explained above, step S132 will be further described. As an optional implementation of step S132, such as... Figure 3 As shown, the step of learning scene compensation information corresponding to the generated view, using the generated view as training supervision, includes: In step S132a, distortion features are extracted from the initial dynamic scene reconstruction model based on the pose and time information of the virtual image acquisition device corresponding to the generated view; In step S132b, the distortion features are decoded using a decoder to obtain scene compensation information.

[0035] It should be noted that, in the above embodiments, the initial dynamic scene reconstruction model used in this invention is a 4DGS-based dynamic scene representation model. The model internally encodes the position, scaling, rotation, and temporal-related basic features corresponding to each Gaussian element. Combining the camera pose and temporal information corresponding to the generated view, a set of features prone to deviations and offsets under the corresponding viewpoint can be extracted from the basic features within the model. This invention uniformly defines these features as distortion features. It should be clarified that the distortion features do not only refer to image distortion-related features, but rather are a comprehensive term for features that characterize geometric shifts, texture deviations, and spatiotemporal inconsistencies that are prone to occur in a scene during viewpoint changes and temporal evolution. Based on the multi-dimensional distortion features extracted above, the decoder performs mapping transformation and parameter parsing on each dimension of the features, enabling targeted decoding of scene compensation information for correcting various scene deviations, thereby achieving compensation and correction of defects in the generated view.

[0036] The core of obtaining scene compensation information lies in accurately extracting distortion features corresponding to defects in the generated view. Accurate extraction of distortion features is a crucial prerequisite for achieving generated view defect compensation and ensuring model training accuracy. Only by accurately extracting distortion features that characterize spatiotemporal inconsistencies, geometric offsets, and texture deviations in the generated view can a reliable basis be provided for the subsequent decoder to obtain effective scene compensation information, thereby preventing generated view defects from contaminating the model training process. Therefore, the following provides a detailed explanation of the distortion feature extraction process in step S132a. As an optional implementation of step S132a, such as... Figure 4 As shown. The step of extracting distortion features from the initial dynamic scene reconstruction model based on the pose and time information of the virtual image acquisition device corresponding to the generated view includes: In step S132a1, the position information of the Gaussian sphere is extracted from the initial dynamic scene reconstruction model with reference to the posture information and time information of the virtual image acquisition device corresponding to the generated view. The position information, posture information and time information are combined and projected onto multiple two-dimensional feature planes respectively. In step S132a2, the features on each two-dimensional feature plane are interpolated to obtain multiple distorted sub-features corresponding to the generated view; In step S132a3, multiple distortion sub-features are cascaded and fused to obtain the distortion feature.

[0037] Specifically, since the initial dynamic scene reconstruction model is a full-temporal and spatiotemporal dynamic scene representation model built on 4DGS, it can completely encode the spatial distribution, geometric / shape parameters, and temporal changes of all Gaussian elements within the scene. Therefore, it can accurately sample and extract the Gaussian distribution features at the corresponding three-dimensional spatial position from the overall Gaussian field based on specified time information and camera pose information. For a single-frame generated view in the video corresponding to a virtual image acquisition device with a certain pose, the three-dimensional spatial coordinates corresponding to the center of each Gaussian sphere within the scene can be located, and all information corresponding to each coordinate point can be obtained. Subsequently, the obtained three-dimensional spatial coordinates, camera pose information, and time information are combined in pairs to obtain multiple feature planes. Taking the three-dimensional spatial coordinates as x, y, z, the pose as p, and the time as t as an example, a total of nine two-dimensional feature planes can be obtained: xp, xt, xy, xz, yp, yt, zp, zt, and yz.

[0038] Each feature plane corresponds to a set of original features. Considering that the original features in each two-dimensional feature plane are discrete and sparsely distributed, and are prone to feature mutations and noise interference, interpolation is used to smooth and make the features continuous, ensuring the integrity and stability of the deviation features, thus obtaining the distorted sub-features of each feature plane. The distorted sub-features are then cascaded and fused to obtain the distorted features.

[0039] The distortion features specifically include distortion location features, distortion scaling features, and distortion rotation features. As an optional implementation of step S132b, such as... Figure 5 As shown. The step of decoding the distortion features using a decoder to obtain scene compensation information includes: In step S132b1, the distortion position features are decoded to obtain the scene position compensation correction amount; In step S132b2, the distortion scaling feature is decoded to obtain the scene scaling compensation correction amount; In step S132b3, the distortion rotation feature is decoded to obtain the scene rotation compensation correction amount.

[0040] It should be noted again that this invention does not specifically limit the network structure of the decoder. The decoder in this invention includes, but is not limited to, multilayer perceptrons (MLPs), residual network modules, and Transformer decoder layers.

[0041] The scene compensation information specifically includes scene position compensation correction, scene scaling compensation correction, and scene rotation compensation correction. The initial dynamic scene reconstruction model internally encodes the position, scaling, rotation, and time-related basic features corresponding to each Gaussian element. During training, the scene compensation corrections of the corresponding dimensions can be superimposed on the original Gaussian parameters for correction. That is, the scene position compensation correction is superimposed on the original position parameters, the scene scaling compensation correction is superimposed on the original scaling parameters, and the scene rotation compensation correction is superimposed on the original rotation parameters, thereby obtaining a corrected and adapted fused dynamic scene reconstruction model that matches the generated view features.

[0042] It should be noted that the fusion dynamic scene reconstruction model corresponding to the scene compensation information is only used as an auxiliary tool in the training process. Its core function is to correct the deviation of the generated view, obtain defect-free multi-view scene information, reduce dependence on hardware devices, and help the initial dynamic scene reconstruction model complete the full-scene learning. Once a dynamic scene reconstruction model that meets the preset standards has been learned, it is not necessary to obtain or use the scene compensation information again during the inference phase.

[0043] During each training round, the initial dynamic scene reconstruction model and scene compensation information are updated and optimized simultaneously. This invention utilizes a dual-supervised simultaneous training mechanism: the dynamic scene reconstruction model after each training round supervises the rendering of the real reference view, while the fused dynamic scene reconstruction model with superimposed scene compensation information supervises the rendering of the generated view. Since the update results of the initial dynamic scene reconstruction model parameters directly affect the extraction results of scene compensation information, further influencing the correction effect of the fused dynamic scene reconstruction model, the two are mutually constrained, interconnected, and collaboratively optimized. Therefore, through this joint training method, it is possible to ensure that the trained dynamic scene reconstruction model always learns the real and reliable scene features corresponding to the reference view, ensuring the purity of the model, while continuously correcting various deviations and defects in the generated view with the help of scene compensation information, and extending the learning to the multi-view scene information corresponding to the generated view after defect correction, thereby achieving the goal of constructing a multi-view, unbiased real scene. Finally, the constructed loss functions and model evaluation metrics are combined to complete iterative optimization, converging to obtain the globally optimized dynamic scene representation model.

[0044] It is important to note that the fusion dynamic scene reconstruction model is only applied to the supervised constraint process during the training phase. Once all training iterations are complete and model inference is performed, there is no need to call scene compensation information or perform parameter overlay and fusion operations; the final optimized and clean dynamic scene reconstruction model can be used directly. This is because during joint training, the model has already completed the integration and learning of accurate scene information across all perspectives and time periods, possessing complete multi-view geometric structure, appearance texture, and temporal dynamic representation capabilities, enabling stable and high-quality reconstruction of dynamic scenes across the entire domain.

[0045] Therefore, as described above. As another optional implementation of this application, such as... Figure 6 As shown, the training method of this application further includes: In step S131, the scene compensation information after each round of training is fused with the trained dynamic scene reconstruction model to obtain a fused dynamic scene reconstruction model. In step S132, the trained dynamic scene reconstruction model is used for rendering to obtain the first rendered image; In step S133, the fused dynamic scene reconstruction model is used for rendering to obtain the second rendered image; In step S134, the loss values ​​between the first rendered image and the reference view, and between the second rendered image and the generated view, are determined according to the loss function; In step S135, the parameters of the current dynamic scene reconstruction model and scene compensation information are updated according to the loss value.

[0046] During training, both the dynamic scene reconstruction model and the fused dynamic scene reconstruction model carry scene representation information after each round of training. The trained dynamic scene reconstruction model learns unbiased real-world scene features from various perspectives, such as the scene's spatial geometry, appearance, texture details, and temporal dynamic changes. The fused dynamic scene reconstruction model learns scene features from the compensated and corrected generated view perspective.

[0047] Rendering refers to the aggregation and imaging of attributes such as color, transparency, spatial distribution, shape, and rotation of the Gaussian elements encoded within the model, based on specified camera pose and temporal information. This transforms the implicitly stored scene features into a visualized 2D image from the corresponding viewpoint. After obtaining the image from the corresponding viewpoint through the rendering process, pixel-level and feature-level comparisons can be performed with the reference view and the generated view. This provides a direct and accurate basis for subsequent loss value calculations and model parameter tuning, ensuring that model training is always optimized in a direction that closely matches the real scene from each viewpoint and compensates for the defects of the generated view.

[0048] Based on the above dual-rendering image generation and loss constraint optimization process, the image similarity index can be used to determine whether the model training has reached the convergence optimization criterion. This serves as the basis for stopping model training, thereby obtaining the optimal dynamic scene reconstruction model. As an optional implementation of step S140, such as... Figure 7 As shown, the step of using the trained dynamic scene reconstruction model as the optimized dynamic scene reconstruction model when the reconstruction index of the trained dynamic scene reconstruction model is greater than or equal to a preset threshold includes: In step S141, if the similarity between the first rendered image and the reference view, and the similarity between the second rendered image and the generated view are greater than or equal to a preset threshold, the dynamic scene reconstruction model obtained in this round of training is used as the optimized dynamic scene reconstruction model.

[0049] The similarity evaluation index is used to quantify the degree of fit between the rendered image and the corresponding view. It includes, but is not limited to, structural similarity (SSIM), peak signal-to-noise ratio (PSNR), mean square error (MSE), normalized cross-correlation coefficient (NCC), and pixel-level matching accuracy. This application does not limit the specific type of similarity evaluation index. Any evaluation index that can accurately measure image similarity and characterize the model reconstruction accuracy can be applied to this embodiment.

[0050] To ensure the stability and physical rationality of the training process and reconstruction results, this application constructs loss functions including, but not limited to, photometric loss, perceptual loss, constraint loss, variability loss, and smoothness regularization. Specifically, photometric loss is a basic pixel-level loss used to constrain pixel photometric and color differences between the first rendered image and the reference view, and between the second rendered image and the generated view, ensuring basic image reconstruction accuracy. Perceptual loss is used to extract high-level visual features of the image and calculate feature differences, further constraining the overall structure, texture details, and visual perception consistency based on pixel alignment. Constraint loss is used to constrain the optimization range of camera pose parameters, avoiding excessive deviation of pose parameters from their initial values ​​during iteration, ensuring a unified viewpoint reconstruction benchmark. Variation loss is used to constrain the smooth changes of Gaussian parameters and feature plane data, suppressing parameter mutations and scene structure distortion drift, ensuring spatial geometric stability. Smoothness regularization encompasses temporal smoothness constraints and feature space smoothness constraints, enabling the smooth and continuous dynamic changes between consecutive frames, suppressing temporal image flicker and multi-view feature distribution disorder, achieving overall spatiotemporal consistency constraints. The above multiple loss functions are combined according to preset weights to form the total loss function. Combined with the model training process, bidirectional constraint optimization is completed, taking into account the basic reconstruction accuracy, structural stability and temporal dynamic continuity.

[0051] Thus, after the above training, the optimized dynamic scene reconstruction model can reconstruct complete scenes across multiple perspectives, effectively suppressing scene structure drift, ensuring consistent and coordinated reconstruction results from all perspectives, and improving temporal image flickering. It achieves a continuous and stable dynamic process, with high overall reconstruction accuracy, complete texture details, and adherence to the physical laws of real scenes. This model effectively reduces dependence on dense camera arrays, achieving high-efficiency and high-quality dynamic scene reconstruction even under sparse perspective conditions. It also boasts advantages such as low computational cost, strong model generalization ability, and wide applicability in practical deployment. Due to these characteristics and advantages, this optimized dynamic scene reconstruction model can be applied to fields such as high-fidelity dynamic rendering, virtual reality / augmented reality, film production, short video content generation, and live streaming.

[0052] As a second aspect of the present invention, a dynamic scene reconstruction method is provided, such as... Figure 8 As shown, the dynamic scene reconstruction method includes: In step S210, a dynamic scene reconstruction model is obtained; wherein the dynamic scene reconstruction model is a model trained by a training method for a dynamic scene reconstruction model provided in the first aspect of the present invention. In step S220, the target time information and target pose information are input into the dynamic scene reconstruction model to obtain a reconstructed image corresponding to the target time information and target pose information.

[0053] Specifically, steps S210-S220 above are the specific inference process of the optimized dynamic scene reconstruction model obtained using the training method of the first aspect of this invention. The entire process does not rely on any scene compensation information or the fusion of the dynamic scene reconstruction model, because the optimized dynamic scene reconstruction model internally stores complete global scene information composed of multiple perspectives, including the scene's spatial geometric structure, appearance texture details, and full-time dynamic change patterns, possessing complete scene representation capabilities. Given the pose (i.e., camera position) and target time information (i.e., which frame in the video), the model can retrieve the scene features of the corresponding perspective and time, and render a visualized scene image conforming to the pose and time based on the internally encoded Gaussian elements, obtaining the reconstructed image under the specified input. It should be emphasized again that scene compensation information only participates in joint optimization during the model training phase to ensure the global convergence calibration of model parameters in the dynamic scene reconstruction model; it does not participate in computation during the inference phase, such as steps S210-S220. Therefore, the dynamic scene reconstruction method of this application does not increase the computational complexity of the inference stage, nor does it require the introduction of additional network parameters, effectively ensuring the lightweight and real-time advantages of the inference process.

[0054] Tests have shown that the dynamic scene reconstruction model trained according to this application can quickly and accurately generate real-time, multi-angle, and high-fidelity dynamic scene reconstruction images during the inference process, effectively reducing the overall computational overhead. It has excellent real-time rendering performance and flexible multi-view generation capabilities, while also effectively reducing computational costs, improving model generalization ability and practical deployment applicability.

[0055] In the embodiments of this application, it is repeatedly mentioned that a small number of cameras and sparse viewpoints are used to complete dynamic scene reconstruction. It should be noted here that sparse viewpoints refer to the acquisition viewpoints of the actual image acquisition devices (such as cameras), and the number of image acquisition devices directly corresponds to the number of sparse viewpoints in this application.

[0056] Through repeated verification by a large number of experiments, the method of this application only requires a small number of image acquisition devices to complete training and reconstruction. The number of image acquisition devices required is usually less than 10. In typical applications, only 2-6 image acquisition devices are needed to achieve high-efficiency and high-quality full-domain dynamic scene reconstruction.

[0057] Compared to existing solutions that often require a dense array of dozens or even hundreds of cameras to complete reconstruction, this application significantly reduces the dependence on the number of acquisition devices, which can greatly reduce hardware construction costs. At the same time, due to the reduction in the amount of input data and the lower pressure on the front-end data acquisition, the overall computational burden is reduced, thereby effectively reducing computational overhead, lowering the threshold for model deployment, and making it more applicable to practical applications.

[0058] In summary, the method of this application has the combined advantages of low dependence on the number of devices, excellent reconstruction effect, and high operating efficiency, which can effectively broaden the application scenarios of dynamic scene reconstruction technology.

[0059] To visually demonstrate the reconstruction advantages of this application, a comparative analysis of the reconstruction effects of this application and the prior art is conducted below with reference to the reconstruction effect diagrams shown in Figure 9. Figures 9(a) and (b) are reconstruction effect diagrams under the same sparse perspective. Figure 9(a) is the reconstruction effect diagram of this application; Figure 9(b) is the reconstruction effect diagram of the prior art.

[0060] From the perspective of the overall reconstruction effect, the reconstruction effect of this application, as shown in Figure 9(a), shows that: the overall image is complete and coherent, the color and brightness are accurately reproduced, the image base is clear, and there is no blurring of color blocks; the overall structure of the scene is complete and unified, and the texture details of various decorative elements in the background are fully preserved; the perspective reconstruction benchmark is stable and unified, and there is no spatial structure offset problem; the transition of the entire image is smooth and natural, without distortion problems such as structural drift, geometric discontinuity, and spatial artifacts; there are also no distortion phenomena such as image ghosting, trailing, and edge tearing, and the overall scene is spatiotemporally coherent and stable. In contrast, the reconstruction effect of the prior art, as shown in Figure 9(b), shows that: the overall image is severely blurred and fuzzy, with obvious ghosting, trailing, structural artifacts, and image discontinuity problems, a large amount of background details are lost, and the overall scene coherence is poor.

[0061] From the perspective of local reconstruction results, the focus is on the reconstruction effect of the main body area of ​​the image. The reconstruction effect of this application, as shown in Figure 9(a), shows that the outline of the person is complete and clear, the clothing texture and facial details are fully restored, the edges of the main body are regular, and there is no distortion. In contrast, the reconstruction effect of the prior art, as shown in Figure 9(b), shows that the person area is severely blurred, details are significantly lost, the outline edges are disordered, and there is obvious image degradation.

[0062] In summary, the above comparison shows that under sparse view reconstruction conditions, the proposed solution can achieve high-precision, high-quality and stable scene reconstruction across the entire domain. The overall image is clean and complete, without any distortion defects, and the core target details are restored with excellent effect. The overall reconstruction effect is significantly better than the existing technology.

[0063] Furthermore, to more intuitively and completely demonstrate the overall technical process and end-to-end system architecture of the dynamic scene reconstruction scheme based on sparse perspective of this application, a schematic diagram of the entire process principle of training and application of this application is shown below. Figure 10As shown in the diagram, the process is divided into two main phases: model training and model inference. In the training phase, sparse reference views acquired using a limited number of real image acquisition devices are combined with a generative model to complete the virtual generated viewpoint. Scene compensation information is constructed through spatiotemporal distortion feature extraction and a decoder compensation module, jointly optimizing the parameters of the dynamic scene reconstruction model and the scene compensation information. Simultaneously, iterative optimization begins using dual-rendering supervision constraints and a multi-dimensional loss function until the model converges, outputting the optimized dynamic scene reconstruction model. In the inference phase, only the target pose and time information need to be input into the optimized dynamic scene reconstruction model to directly infer and render the scene reconstruction image at the corresponding viewpoint and time.

[0064] It is important to note the input data used in this application. In this application, the input data consists of a reference view and a generated view. This input data includes, but is not limited to, multi-timeframe image sequences or video streams, multi-view observation data from a small number of cameras, and optional camera parameter information (including intrinsic parameters, extrinsic parameters, and time synchronization information). In some embodiments, it may also include: depth information (e.g., from an RGB-D device), optical flow or motion estimation information, and external sensor data (e.g., from an IMU).

[0065] Similarly, the output formats of the dynamic scene reconstruction model trained in this application include, but are not limited to, continuous-time dynamic scene representation (4D representation), arbitrary-viewpoint rendered image sequences, video output, 3D structures, or explicit models. In some implementations, in addition to supporting real-time or near-real-time rendering, interactive viewpoint control may also be supported.

[0066] Compared with existing technologies, this application has the following advantages: 1) Significantly reduced data acquisition costs: No complex multi-camera arrays are required; dynamic scene reconstruction can be completed with only a small number of devices. 2) Improved reconstruction quality under sparse perspectives: Generative completion mechanism effectively compensates for insufficient observations, improving structural integrity and texture quality. 3) Enhanced spatiotemporal consistency: Significantly reduces flickering, discontinuities, and structural drift problems in dynamic scenes. 4) Improved computational efficiency and flexibility: Supports multiple representation forms and model structures, possessing good engineering adaptability. 5) Good scalability and application prospects.

[0067] As a third aspect of the present invention, an electronic device is provided, such as... Figure 11 As shown, it includes: One or more processors 101; The memory 102 stores one or more computer programs that, when executed by the one or more processors 101, cause the one or more processors 101 to implement a training method for a dynamic scene reconstruction model provided in the first aspect of the present invention and a dynamic scene reconstruction method provided in the second aspect.

[0068] The tool may also include one or more I / O interfaces 103 connected between the processor 101 and the memory 102, configured to enable information interaction between the processor 101 and the memory 102.

[0069] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit 101 (CPU); the first memory 102 is a device with data storage capabilities, including but not limited to random access memory 102 (RAM, more specifically SDRAM, DDR, etc.), read-only memory 102 (ROM), electrically erasable programmable read-only memory 102 (EEPROM), and flash memory (FLASH); the I / O interface 103 (read-write interface) is connected between the processor 101 and the memory 102, enabling information interaction between the processor 101 and the memory 102, including but not limited to a data bus 104 (Bus).

[0070] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0071] Furthermore, as a fourth aspect of the present invention, a computer-readable medium is also provided having a computer program stored thereon, such as... Figure 12 As shown, when the computer program is executed by the processor, it implements the training method for a dynamic scene reconstruction model provided in the first aspect of the present invention and the dynamic scene reconstruction method provided in the second aspect.

[0072] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can implement the methods of any of the above embodiments. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0073] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes, but is not limited to, the contents described in the accompanying drawings and the specific embodiments above. Any modifications that do not depart from the functional and structural principles of the present invention will be included within the scope of the claims.

Claims

1. A training method for a dynamic scene reconstruction model, characterized in that, The training method includes: Obtain a reference view captured from the perspective of the image acquisition device; Generative models are used to generate multi-view views of the reference view to obtain a generated view of the virtual image acquisition device. Using the reference view and the generated view as training supervision, the initial dynamic scene reconstruction model is trained so that the trained dynamic scene reconstruction model can represent the scene features corresponding to the reference view and the generated view, and learn the scene compensation information corresponding to the generated view. If the reconstruction index of the trained dynamic scene reconstruction model is greater than or equal to the preset threshold, the trained dynamic scene reconstruction model will be used as the optimized dynamic scene reconstruction model.

2. The training method according to claim 1, characterized in that, The process of training an initial dynamic scene reconstruction model using a reference view and a generated view as training supervision, so that the trained dynamic scene reconstruction model can represent the scene features corresponding to the reference view and the generated view, and learn scene compensation information corresponding to the generated view, includes: In each round of training, the initial dynamic scene reconstruction model is trained using the reference view as the training supervision basis, so that the trained dynamic scene reconstruction model can fit the scene features corresponding to the reference view. Using the generated view as the training supervision basis, the scene compensation information corresponding to the generated view is learned, and the initial dynamic scene reconstruction model and the scene compensation information are jointly trained so that the trained dynamic scene reconstruction model can fit the overall scene features corresponding to the reference view and the generated view.

3. The training method according to claim 2, characterized in that, The step of learning scene compensation information corresponding to the generated view, using the generated view as training supervision, includes: Based on the pose and time information of the virtual image acquisition device corresponding to the generated view, distortion features are extracted from the initial dynamic scene reconstruction model; The distortion features are decoded using a decoder to obtain scene compensation information.

4. The training method according to claim 3, characterized in that, The step of extracting distortion features from the initial dynamic scene reconstruction model based on the pose and time information of the virtual image acquisition device corresponding to the generated view includes: Using the pose and time information of the virtual image acquisition device corresponding to the generated view as a reference, the position information of the Gaussian sphere is extracted from the initial dynamic scene reconstruction model, and the position information, pose information and time information are combined and projected onto multiple two-dimensional feature planes respectively. Interpolate the features on each two-dimensional feature plane to obtain multiple distorted sub-features corresponding to the generated view; The distortion feature is obtained by cascading and fusing multiple distortion sub-features.

5. The training method according to claim 3, characterized in that, The distortion features include distortion position features, distortion scaling features, and distortion rotation features; The step of decoding the distortion features using a decoder to obtain scene compensation information includes: The distortion position features are decoded to obtain the scene position compensation correction amount; Decode the distortion scaling features to obtain the scene scaling compensation correction amount; The distortion rotation features are decoded to obtain the scene rotation compensation correction amount.

6. The training method according to any one of claims 1 to 5, characterized in that, The training method also includes: The scene compensation information after each round of training is fused with the dynamic scene reconstruction model after training to obtain the fused dynamic scene reconstruction model. The trained dynamic scene reconstruction model is used for rendering to obtain the first rendered image; The second rendered image is obtained by rendering using a fused dynamic scene reconstruction model; The loss values ​​between the first rendered image and the reference view, and between the second rendered image and the generated view, are determined based on the loss function. Update the parameters of the current dynamic scene reconstruction model and scene compensation information based on the loss value; The step of using the trained dynamic scene reconstruction model as the optimized dynamic scene reconstruction model when the reconstruction index of the trained dynamic scene reconstruction model is greater than or equal to a preset threshold includes: If the similarity between the first rendered image and the reference view, and the similarity between the second rendered image and the generated view are both greater than or equal to a preset threshold, the dynamic scene reconstruction model obtained in this round of training is used as the optimized dynamic scene reconstruction model.

7. The training method according to claim 6, characterized in that, The loss function includes at least one of photometric loss, perceptual loss, constraint loss, variation loss, and smoothness regularization.

8. A dynamic scene reconstruction method, characterized in that, The dynamic scene reconstruction method includes: A dynamic scene reconstruction model is obtained; wherein the dynamic scene reconstruction model is a model trained by a training method for a dynamic scene reconstruction model provided according to any one of claims 1 to 7. The target time information and target pose information are input into the dynamic scene reconstruction model to obtain the reconstructed image corresponding to the target time information and target pose information.

9. An electronic device, characterized in that, include: One or more processors; A memory having stored thereon one or more computer programs that, when executed by the one or more processors, cause the one or more processors to implement the method according to any one of claims 1 to 8.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 8.

Images