Scene reconstruction method, electronic device, storage medium, and program product

Abstract

The application discloses a scene reconstruction method, an electronic device, a storage medium and a program product, relates to the technical field of computer graphics, and comprises the following steps: scene reconstruction is performed on video frames in a preset dynamic sliding window in a video frame sequence through a first four-dimensional Gaussian splash model, a plurality of restored scene frames are obtained, and the first four-dimensional Gaussian splash model is reversely optimized according to the restored scene frames and corresponding video frames; the updated parameters are sent to a client, so that the client updates a second four-dimensional Gaussian splash model according to the received parameters, and scene reconstruction is performed through the updated second four-dimensional Gaussian splash model; the dynamic sliding window is moved according to a preset sliding step, and the reconstruction and optimization process is repeatedly executed until the entire video frame sequence is covered. Through the setting of the dynamic sliding window, the high-delay problem that a traditional offline method needs to wait for complete data is effectively avoided, and the response efficiency of scene reconstruction is improved.

Description

Technical Field

[0001] This application relates to the field of computer graphics technology, and in particular to scene reconstruction methods, electronic devices, storage media and program products. Background Technology

[0002] In recent years, Gaussian splashing technology based on explicit representation of three-dimensional Gaussian primitives has been widely used in fields such as digital twins, virtual reality, and film and television production due to its extremely fast training speed and rendering efficiency, giving rise to many dynamic scene reconstruction methods.

[0003] Common scene reconstruction methods include offline reconstruction using 4DGS (4D Gaussian Splatting) technology, which uses a set of static Gaussian primitives combined with a temporally conditional deformable network to represent the dynamic scene. However, this type of method requires training on the target dynamic scene in the complete time domain and undergoing tens of thousands of iterations of training to gradually optimize the parameters of the Gaussian primitives and deformable network, resulting in a high response delay between data acquisition and the generation of the final reconstruction result.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a scene reconstruction method, electronic device, storage medium and program product, which aims to solve the technical problem of how to improve the response efficiency of scene reconstruction based on four-dimensional Gaussian splashing.

[0006] To achieve the above objectives, this application proposes a scene reconstruction method, which includes:

[0007] Generate a video frame sequence based on the obtained video frame sequence;

[0008] Using the first four-dimensional Gaussian splash model, the video frames within the preset dynamic sliding window in the video frame sequence are reconstructed to obtain multiple restored scene frames. Based on each restored scene frame and its corresponding video frame, the first four-dimensional Gaussian splash model is optimized in reverse.

[0009] The updated parameters in the optimized first four-dimensional Gaussian splash model are sent to the client so that the client can update the second four-dimensional Gaussian splash model according to the received parameters and reconstruct the scene using the updated second four-dimensional Gaussian splash model.

[0010] According to the preset sliding step size, the dynamic sliding window is moved, and the steps of reconstructing the scene of the video frames within the preset dynamic sliding window through the first four-dimensional Gaussian splash model to obtain multiple restored scene frames are repeated until the dynamic sliding window moves to the end of the video frame sequence.

[0011] In one embodiment, the first four-dimensional Gaussian splash model includes three-dimensional Gaussian primitives and a deformable network. The step of reconstructing the scene of video frames within a preset dynamic sliding window in the video frame sequence using the first four-dimensional Gaussian splash model to obtain multiple restored scene frames, and then back-optimizing the first four-dimensional Gaussian splash model based on each restored scene frame and its corresponding video frame, includes:

[0012] When the dynamic sliding window is the starting window, the video frames within the dynamic sliding window are reconstructed using the three-dimensional Gaussian primitives to obtain multiple static restored frames. Based on each static restored frame and the corresponding video frame, the three-dimensional Gaussian primitives are optimized in reverse until the number of iterations reaches a preset first iteration threshold.

[0013] The video frames within the dynamic sliding window are reconstructed using the optimized 3D Gaussian primitives and the deformable network to obtain multiple dynamically restored frames. Based on each dynamically restored frame and the corresponding video frame, the optimized 3D Gaussian primitives and the deformable network are then optimized in reverse.

[0014] In one embodiment, the four-dimensional Gaussian splash model includes three-dimensional Gaussian primitives and a deformable network. The step of back-optimizing the parameters of the first four-dimensional Gaussian splash model based on each of the restored scene frames and their corresponding video frames includes:

[0015] When the dynamic sliding window is not the starting window and the number of iterations does not exceed the preset second iteration threshold, the deformation network is optimized in reverse according to each restored scene frame and the corresponding video frame;

[0016] When the dynamic sliding window is not the starting window and the number of iterations exceeds the second iteration threshold, the spatial gradient of the three-dimensional Gaussian unit is determined according to each restored scene frame and the corresponding video frame, and the deformable network and the three-dimensional Gaussian unit with spatial gradients higher than the preset gradient threshold are optimized in reverse.

[0017] In one embodiment, the step of determining the spatial gradient of the three-dimensional Gaussian unit based on each of the restored scene frames and the corresponding video frames includes:

[0018] In each round of iterative training, the gradient values ​​of the three-dimensional Gaussian primitives are determined based on the backpropagation algorithm, each restored scene frame, and the corresponding video frame.

[0019] The spatial gradient of the three-dimensional Gaussian primitive is obtained by averaging the gradient values ​​corresponding to each round of iterative training.

[0020] In one embodiment, the first four-dimensional Gaussian splash model includes HexPlane. Before the step of reconstructing the scene from video frames within a preset dynamic sliding window using the first four-dimensional Gaussian splash model to obtain multiple restored scene frames, the method further includes:

[0021] When the dynamic sliding window is not the starting window and the iteration number is the first time, the overlapping frame between the current window and the previous window is determined;

[0022] The HexPlane parameter corresponding to the overlapping frame of the previous window is determined as the HexPlane parameter corresponding to the overlapping frame of the current window, and the HexPlane parameter corresponding to the last frame of the previous window is determined as the HexPlane parameter of the non-overlapping frame of the current window.

[0023] The parameters of other components trained in the previous window are used as the parameters of other components in the current window, wherein the other components are the components other than HexPlane in the first four-dimensional Gaussian splash model.

[0024] In one embodiment, the HexPlane includes a spatial plane and a spatiotemporal plane. Before the step of reconstructing the scene from video frames within a preset dynamic sliding window using a first four-dimensional Gaussian splash model to obtain multiple restored scene frames, the method further includes:

[0025] Delete the high-resolution plane in the aforementioned spatial plane;

[0026] The temporal resolution of the spatiotemporal plane is determined based on the window length of the dynamic sliding window, wherein the temporal resolution is directly proportional to the window length.

[0027] In one embodiment, the step of generating a video frame sequence based on the acquired video frame sequence includes:

[0028] A video frame sequence containing at least two perspectives is acquired, and the video frame sequence is sparsely reconstructed using motion structure recovery technology to obtain the video frame sequence.

[0029] Before the step of reconstructing the scene from video frames within a preset dynamic sliding window using the first four-dimensional Gaussian splash model, the method further includes:

[0030] The first four-dimensional Gaussian splash model is initialized based on the sparse point cloud frames of the video frame sequence.

[0031] Furthermore, to achieve the above objectives, this application also proposes a scene reconstruction apparatus, the scene reconstruction apparatus comprising:

[0032] The point cloud generation module is used to generate a video frame sequence based on the acquired video frame sequence.

[0033] The model optimization module is used to reconstruct the scene of the video frames in the preset dynamic sliding window in the video frame sequence using the first four-dimensional Gaussian splash model, to obtain multiple restored scene frames, and to optimize the first four-dimensional Gaussian splash model in reverse according to each restored scene frame and its corresponding video frame.

[0034] The parameter sending module is used to send the updated parameters in the optimized first four-dimensional Gaussian splash model to the client, so that the client updates the second four-dimensional Gaussian splash model according to the received parameters and performs scene reconstruction through the updated second four-dimensional Gaussian splash model;

[0035] The window moving module is used to move the dynamic sliding window according to a preset sliding step size, repeating the steps of reconstructing the scene of the video frames within the preset dynamic sliding window using the first four-dimensional Gaussian splash model to obtain multiple restored scene frames, until the dynamic sliding window moves to the end of the video frame sequence.

[0036] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the scene reconstruction method as described above.

[0037] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the scene reconstruction method described above.

[0038] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the scene reconstruction method described above.

[0039] The one or more technical solutions proposed in this application have at least the following technical effects: First, based on the acquired video frame sequence, a video frame sequence is generated, converting the original image data into an explicit geometric representation more suitable for Gaussian splash processing, thereby improving subsequent processing efficiency; then, through the first four-dimensional Gaussian splash model, scene reconstruction is performed on the video frames within a preset dynamic sliding window to obtain multiple restored scene frames. By limiting the processing range of the model, the computational load of a single optimization of the model is reduced, so as to achieve a fast response; furthermore, based on each restored scene frame and its corresponding video frame, the parameters of the first four-dimensional Gaussian splash model are optimized in reverse, and the model parameters are quickly fitted to the current scene content using data within a finite sliding window, ensuring the rapid convergence of the model parameters under finite data conditions, so as to maintain the... The system maintains the stability of client-side reconstruction quality. Furthermore, it sends the updated parameters from the optimized first four-dimensional Gaussian splash model to the client, enabling the client to update the second four-dimensional Gaussian splash model based on the received parameters. Scene reconstruction is then performed using the updated second four-dimensional Gaussian splash model. By transmitting the updated parameters corresponding to each sliding window in real time, the client achieves real-time reconstruction capabilities, improving the efficiency of scene reconstruction based on the four-dimensional Gaussian splash model. Then, according to a preset sliding step size, the dynamic sliding window moves, repeatedly reconstructing the scene from the video frames within the preset dynamic sliding window using the first four-dimensional Gaussian splash model, obtaining multiple restored scene frames, until the dynamic sliding window moves to the end of the video frame sequence, completing the full processing of the video frame sequence. This application effectively avoids the high latency problem caused by traditional offline methods that require waiting for complete data training, thus achieving low-latency, high-real-time scene reconstruction based on four-dimensional Gaussian splash while ensuring high-quality dynamic scene reconstruction. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating an embodiment of the scenario reconstruction method of this application.

[0043] Figure 2 This is a diagram of the overall training framework based on a dynamic sliding window provided in Embodiment 2 of this application;

[0044] Figure 3 This is a comparison diagram of the effects of different scene reconstruction methods provided in Embodiment 2 of this application;

[0045] Figure 4 This is a schematic diagram of the module structure of the scene reconstruction device according to an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the scene reconstruction method in this application embodiment.

[0047] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0048] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0049] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0050] Common scene reconstruction methods include offline reconstruction using 4DGS (4D Gaussian Splatting) technology, which uses a set of static Gaussian primitives combined with a temporally conditional deformable network to represent the dynamic scene. However, these methods rely on training the target dynamic scene across the entire temporal domain and undergo tens of thousands of iterations to gradually optimize the parameters of the Gaussian primitives and deformable network, resulting in a high response latency between data acquisition and the generation of the final reconstruction result. In streaming media scenarios, users often have to wait tens of minutes to obtain the reconstructed results of the dynamic scene, severely impacting the interactive experience.

[0051] Online reconstruction methods (such as 3DGStream) employ a frame-by-frame training strategy, starting from the static Gaussian units in the first frame and training the deformable network for each frame based on the results of the previous frame. This adapts to streaming application scenarios. However, the independent reconstruction of each frame results in a large number of parameters in the deformable network per frame, which increases the network transmission bandwidth requirements and incurs additional time overhead due to parameter training.

[0052] This application provides a solution. First, based on the acquired video frame sequence, a video frame sequence is generated, converting the original image data into an explicit geometric representation more suitable for Gaussian splashing processing, thereby improving subsequent processing efficiency. Then, using a first four-dimensional Gaussian splashing model, scene reconstruction is performed on the video frames within a preset dynamic sliding window, resulting in multiple reconstructed scene frames. By limiting the model's processing range, the computational load of a single optimization is reduced, enabling rapid response. Next, based on each reconstructed scene frame and its corresponding video frame, the parameters of the first four-dimensional Gaussian splashing model are optimized in reverse. Utilizing data within a finite sliding window, the model parameters quickly fit the current scene content, ensuring rapid convergence of the model parameters under limited data conditions, thus maintaining the client-side reconstruction quality. The stability of the data is ensured. Furthermore, the updated parameters in the optimized first four-dimensional Gaussian splash model are sent to the client, allowing the client to update the second four-dimensional Gaussian splash model based on the received parameters. Scene reconstruction is then performed using the updated second four-dimensional Gaussian splash model. By transmitting the updated parameters corresponding to each sliding window in real time, the client's real-time reconstruction capability is achieved, improving the efficiency of scene reconstruction based on the four-dimensional Gaussian splash model. Then, according to a preset sliding step size, the dynamic sliding window is moved, and the process of reconstructing the scene from the video frames within the preset dynamic sliding window using the first four-dimensional Gaussian splash model is repeated, resulting in multiple restored scene frames, until the dynamic sliding window moves to the end of the video frame sequence, completing the full processing of the video frame sequence. This application effectively avoids the high latency problem caused by the need for complete data training in traditional offline methods through dynamic sliding windows, and also overcomes the shortcomings of large parameter quantities and high bandwidth consumption in online frame-by-frame methods. Thus, while ensuring high-quality dynamic scene reconstruction, it achieves low latency, low bandwidth, and high real-time performance.

[0053] It should be noted that the executing entity in this embodiment can be an electronic device with data processing, network communication and program execution functions, such as a tablet computer, personal computer, mobile phone, etc.

[0054] Based on this, the embodiments of this application provide a scene reconstruction method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the scenario reconstruction method of this application.

[0055] In this embodiment, the scene reconstruction method includes steps S10 to S40:

[0056] Step S10: Obtain the video frame sequence;

[0057] A video frame sequence is a set of two-dimensional pixel matrices indexed by timestamps and containing visual information from different angles. Each pixel matrix contains RGB color information and / or brightness information, representing the camera's projection observation of the scene at a specific moment. In addition, each frame (per pixel matrix) also includes data such as camera intrinsic and extrinsic parameters to facilitate sparse reconstruction and construct sparse video frames.

[0058] In one possible implementation, prior to step S10, the method further includes:

[0059] Step S01: Using motion structure recovery technology, sparse reconstruction is performed on the initial video frames of the video frame sequence to obtain sparse point cloud frames.

[0060] Structure from Motion (SfM) technology refers to a processing method that recovers sparse point cloud information of a scene and camera motion trajectory from a sequence of continuously captured two-dimensional images. It includes steps such as feature point extraction, matching, camera pose solving, and three-dimensional scene point cloud generation.

[0061] A sparse point cloud frame is an image frame represented in the form of a point cloud after sparse reconstruction. It consists of a large number of three-dimensional spatial points and can be supplemented with information such as color and brightness of each point to achieve discretization and explicit expression of the three-dimensional geometric structure of the scene.

[0062] For example, key feature points in the initial video frame are detected using feature extraction algorithms. Common feature extraction algorithms include SIFT (Scale-invariant feature transform), ORB (OrientedFAST and Rotated BRIEF), and SuperPoint. Then, feature point matching is performed between images in different frames to establish a two-dimensional correspondence across viewpoints and time. Based on the feature point matching results, geometric constraints and mathematical models are used to estimate the camera's motion parameters, such as rotation matrices and translation vectors, as well as the relative motion information of objects in the scene. Finally, based on the estimated camera motion parameters and the two-dimensional correspondence of feature points, the coordinates of the feature points in three-dimensional space are calculated using algorithms such as triangulation, thereby constructing a sparse point cloud model and obtaining a sparse point cloud frame.

[0063] Understandably, by restoring the motion structure and utilizing the geometric constraints between multiple viewpoints, the video frame sequence can be transformed into accurate point cloud data, providing a visible three-dimensional geometric representation for the Gaussian splash model. This reduces the uncertainty of model optimization and minimizes training failures or instability caused by geometric ambiguity.

[0064] Step S02: Initialize the first four-dimensional Gaussian splash model based on the sparse point cloud frame.

[0065] The four-dimensional Gaussian splash model is a model used to represent and reconstruct dynamic scenes in four-dimensional space (three-dimensional space plus time). It is a collection of three-dimensional Gaussian primitives. Each Gaussian not only has spatial attributes, but also associates with time variables through a deformable network, together representing a four-dimensional spatiotemporal scene.

[0066] Optionally, the initial parameters of the first four-dimensional Gaussian splash model can be set according to the spatial position and color of the sparse point cloud frame. The initial parameters include the initial position, color, size, and orientation of each three-dimensional Gaussian element.

[0067] For example, each 3D point in the sparse point cloud frame can be converted into a 3D Gaussian primitive, and the spatial coordinates of each 3D point can be directly set as the center position of the corresponding Gaussian primitive. Furthermore, the shape of each Gaussian ellipsoid can be set to isotropic (i.e., the degree of blurring is the same in all directions), or the shape of each Gaussian ellipsoid can be estimated based on the initial shape of the sparse point cloud frame. Then, the color, transparency, and other attributes of each Gaussian ellipsoid are determined from the sparse point cloud frame data, thereby completing the initialization of the first 4D Gaussian splash model. Further, this initialized first 4D Gaussian splash model can start from this initial state, perform its first rendering based on the camera pose of the first frame, and prepare to accept joint optimization of subsequent frames within a sliding window.

[0068] In this embodiment, since the sparse point cloud frames are derived from accurate multi-view SFM, which has a metric scale and high precision, the first four-dimensional Gaussian splash model is initialized based on this, ensuring that the model is in a geometrically and semantically correct initial state from the first moment, thus shortening the model startup time. At the same time, rendering based on the accurate initial state effectively reduces the number of subsequent optimization adjustments, so that the model can converge quickly and improve the model training efficiency.

[0069] Step S20: Using the first four-dimensional Gaussian splash model, the video frames within the preset dynamic sliding window in the video frame sequence are reconstructed to obtain multiple restored scene frames. Based on each restored scene frame and its corresponding video frame, the parameters of the first four-dimensional Gaussian splash model are optimized in reverse.

[0070] A dynamic sliding window is a fixed-length window that moves along the timeline, containing several consecutive frames of point cloud data; the length of the window and the sliding step can be adjusted according to specific needs.

[0071] A reconstructed scene frame refers to a 3D scene frame generated by rendering a 4D Gaussian splash model. It is a reconstruction of the scene appearance captured by the camera pose at that moment.

[0072] For example, the first four-dimensional Gaussian splash model includes three-dimensional Gaussian primitives and a deformation network. The steps of scene reconstruction using the first four-dimensional Gaussian splash model include: inputting each video frame in the dynamic sliding window and its corresponding timestamp t, and the initialized three-dimensional Gaussian primitives into the deformation network to obtain the instantaneous position, rotation, scale and other deformation parameters of each three-dimensional Gaussian primitive at time t; then, projecting the deformed instantaneous three-dimensional Gaussian primitives corresponding to time t onto the two-dimensional image plane according to the camera pose to synthesize the restored scene frames corresponding to each time t.

[0073] For example, after reconstruction is complete, the restored scene frames and corresponding real video frames corresponding to each video frame within the dynamic sliding window can be compared frame by frame to calculate the comprehensive loss function of the dynamic sliding window, such as pixel difference loss and structural similarity loss. Then, through automatic differentiation, the gradient of the comprehensive loss with respect to all trainable parameters in the first four-dimensional Gaussian splash model is calculated, and the gradients of the trainable parameters are updated according to a preset optimization algorithm (such as stochastic gradient descent algorithm, Adam algorithm, etc.) and the gradients of each trainable parameter. Among them, the trainable parameters include the attribute values ​​of the three-dimensional Gaussian primitives (such as position, color, transparency, scale, etc.) and the weight parameters in the deformable network. This embodiment does not impose specific limitations on these.

[0074] Understandably, by limiting offline scene reconstruction and model optimization to a finite sliding window, the search space for each round of training iterations is reduced, enabling a faster response. At the same time, by using all frames within this sliding window for optimization, the model is provided with joint constraints at multiple time points, effectively reducing problems such as large parameter changes and local optima in single-frame optimization, which can effectively improve model training efficiency and scene reconstruction quality.

[0075] In one feasible implementation, the first four-dimensional Gaussian splash model includes three-dimensional Gaussian primitives and a deformable network, and step S20 includes:

[0076] Step S21: When the dynamic sliding window is the starting window, the video frames in the dynamic sliding window are reconstructed using three-dimensional Gaussian elements to obtain multiple static restored frames. Based on each static restored frame and the corresponding video frame, the three-dimensional Gaussian elements are optimized in reverse until the number of iterations reaches the preset first iteration threshold.

[0077] In a 3D Gaussian splash model, a set of attribute parameters used to represent the geometry and appearance of a scene in 3D space includes parameters such as position, color (spherical harmonic coefficients), transparency, scale, and rotation.

[0078] Static frame reconstruction refers to a two-dimensional image generated by differentiable rendering using only three-dimensional Gaussian primitives and the camera pose corresponding to each video frame, ignoring time conditions.

[0079] The first iteration threshold refers to the upper limit of the number of iterations used to terminate the static optimization of this stage during the training process in the initial window.

[0080] For example, for each video frame within the initial window, projection and lighting calculations can be performed based on the three-dimensional Gaussian primitives in the first four-dimensional Gaussian splash model and the camera pose corresponding to each video frame, and then static restoration frames corresponding to each video frame can be generated. Then, the rendered static restoration frames are compared frame by frame with the video frames corresponding to each video frame, the loss function is calculated, and the attribute values ​​of the three-dimensional Gaussian primitives are updated through backpropagation until the number of iterations reaches the preset first iteration threshold or the loss function converges, at which point the optimization of this stage stops.

[0081] Step S22: The video frames within the dynamic sliding window are reconstructed using the optimized 3D Gaussian primitives and deformable network to obtain multiple dynamically restored frames. Based on each dynamically restored frame and the corresponding video frame, the optimized 3D Gaussian primitives and deformable network are then reverse-engineered.

[0082] The optimized 3D Gaussian primitives refer to the set of Gaussian attribute parameters that can express the common geometric state of the scene to be reconstructed with high quality after the above static optimization stage.

[0083] Deformation networks are neural networks that use timestamps as conditional parameters to learn the mapping from the canonical space (static space) of three-dimensional Gaussian elements to the instantaneous observation space. They can receive optimized three-dimensional Gaussian elements and video frames and their corresponding current time t as inputs, and output the deformation parameters such as displacement, rotation, and scaling of each three-dimensional Gaussian element at the current time t.

[0084] Dynamically reconstructed frames refer to instantaneous 3D Gaussian primitives with adjusted deformation parameters output by a deformation network, which generate a 2D image through differentiable rendering based on the camera pose corresponding to each video frame within the initial window.

[0085] For example, for each video frame and its corresponding timestamp t within the initial window, the deformation parameters of the 3D Gaussian elements corresponding to each video frame (or each timestamp t) are output through the deformable network. Then, the deformation parameters at time t are applied to the optimized 3D Gaussian elements to obtain the instantaneous 3D Gaussian elements at time t. After projection and lighting calculations are performed using the instantaneous 3D Gaussian elements and the camera pose of the video frame at time t, the dynamic reconstruction frame corresponding to the video frame is generated. Then, the rendered dynamic reconstruction frame is compared frame by frame with the video frames corresponding to each video frame, the loss function is calculated, and the attribute values ​​of the 3D Gaussian elements and the weight parameters of the deformable network are updated through backpropagation until the number of iterations reaches the preset number of iterations or the loss function converges.

[0086] In this embodiment, the training difficulty of four-dimensional reconstruction in the first window is effectively reduced by the two-stage initialization training of the initial window, namely, first optimizing the three-dimensional Gaussian primitives and then jointly optimizing them with the deformable network; at the same time, through the subsequent joint training, the static geometric representation and dynamic motion law of the reconstructed scene are mutually refined and optimized, thereby improving the quality of the model scene reconstruction.

[0087] In one feasible implementation, the four-dimensional Gaussian splash model includes three-dimensional Gaussian primitives and a deformable network. Step S20, which involves inversely optimizing the parameters of the first four-dimensional Gaussian splash model based on each reconstructed scene frame and its corresponding video frame, includes:

[0088] Step S23: When the dynamic sliding window is not the starting window and the number of iterations does not exceed the preset second iteration threshold, the deformation network is optimized in reverse according to each restored scene frame and the corresponding video frame.

[0089] Step S24: When the dynamic sliding window is not the starting window and the number of iterations exceeds the second iteration threshold, determine the spatial gradient of the three-dimensional Gaussian primitives based on each restored scene frame and the corresponding video frame, and inversely optimize the deformation network and the three-dimensional Gaussian primitives with spatial gradients higher than the preset gradient threshold.

[0090] In one feasible embodiment, when the dynamic sliding window is not the initial window, the training process of the window is divided into two optimization stages by setting a second iteration threshold. In the first optimization stage, the parameters of the three-dimensional Gaussian primitives are kept frozen, and only the parameters of the deformable network are updated in reverse to adapt to the motion changes of the new window. In the second optimization stage, while continuing to optimize the deformable network, the three-dimensional Gaussian primitives with spatial gradients higher than the preset gradient threshold are selectively updated in reverse. That is, the parts of the three-dimensional Gaussian primitives that are significantly different from the new video frames are selectively corrected, thereby ensuring high-quality reconstruction while effectively reducing the storage overhead of model training and improving training efficiency.

[0091] The second iteration threshold is a preset boundary point for dividing the number of iterations into two stages of the optimization process that is not the first window. It can be set according to training needs.

[0092] Spatial gradient refers to the rate of change of each attribute parameter in a 3D Gaussian primitive in space. The magnitude of the gradient value of each 3D Gaussian primitive reflects the difference between the current 3D Gaussian primitive and the 3D points of the video frame in the current sliding window. The larger the spatial gradient value of a 3D Gaussian primitive, the greater the rendering difference between it and the corresponding 3D point cloud, which may lead to inaccuracies in the rendered image and requires optimization.

[0093] It is understood that the reverse update process is similar to the reverse update process in the specific implementation of steps S21 and S22 in this embodiment, so it will not be described again. During the reverse update process, the three-dimensional Gaussian primitives with spatial gradients less than or equal to the preset gradient threshold are not completely in a static state. They can still be driven to change by the deformation parameters output by the deformable network. That is, the three-dimensional Gaussian primitives with spatial gradients less than or equal to the preset gradient threshold may still be in a low-motion state, which can better adapt to different scene changes and effectively improve the adaptability and generalization of the four-dimensional Gaussian splash model.

[0094] In this implementation, by locking Gaussian units and optimizing only the deformable network in the early stages of optimization, the model can quickly learn the changes in motion patterns of video frames within the current sliding window, while ensuring that the static geometric structure of the scene learned in the previous window is not destroyed or forgotten. Furthermore, a spatial gradient threshold filtering mechanism is introduced in the later stages of optimization to selectively optimize Gaussian units that conflict significantly with the video frames within the current sliding window, avoiding unnecessary global parameter perturbations and effectively improving the training efficiency of the model. This allows the updated model parameters to be quickly applied to actual scene reconstruction, thereby improving response efficiency.

[0095] In one feasible implementation, step S24, which involves determining the spatial gradient of the three-dimensional Gaussian primitives based on each reconstructed scene frame and its corresponding video frame, includes:

[0096] Step S241: In each round of iterative training, based on the backpropagation algorithm, each restored scene frame and the corresponding video frame, determine the gradient value of the three-dimensional Gaussian unit in each round of iterative training.

[0097] Backpropagation is an efficient algorithm that starts from the final output of the model, calculates the loss function, and then calculates the partial derivative (i.e., gradient) of each training parameter in the model with respect to the loss function layer by layer.

[0098] For example, during any round of training in the first and second optimization phases (excluding the first dynamic sliding window), the differences between each restored scene frame and the corresponding video frame are compared using a preset loss function. The loss value of each 3D Gaussian unit is calculated. Commonly used loss functions include structural similarity loss and L1 loss, which are used to measure the difference between the rendered image (restored scene frame) and the real image (video frame) from different perspectives. Then, the partial derivatives of each attribute parameter (such as position, color, etc.) of the 3D Gaussian unit with respect to the loss function are calculated to obtain the gradient value of each parameter in the 3D Gaussian unit.

[0099] Step S242: Based on the gradient values ​​corresponding to each round of iterative training, the spatial gradient of the three-dimensional Gaussian primitive is obtained by averaging them.

[0100] For example, after each round of iterative training is completed, the gradient values ​​corresponding to each three-dimensional Gaussian element in that round of training are accurately recorded. Then, after the number of iterations exceeds the second iteration threshold, i.e., the training enters the second optimization stage, in each round of iteration, for any three-dimensional Gaussian element, the gradient values ​​recorded in all rounds of iterative training are mathematically averaged, that is, these gradient values ​​are added together and divided by the number of iterations to obtain an average gradient value, and this average gradient value is determined as the spatial gradient corresponding to the three-dimensional Gaussian element.

[0101] In one feasible implementation, in each round of iterative training, the gradient values ​​of the position parameters of the 3D Gaussian primitives are determined based on the backpropagation algorithm, each restored scene frame and the corresponding video frame; and the spatial gradient of the 3D Gaussian primitives is obtained by averaging the gradient values ​​of the position parameters of the 3D Gaussian primitives in each round of iterative training.

[0102] Position parameters are parameters used to determine the position of a 3D Gaussian primitive in 3D space. In a 3D coordinate system, the position of a point is usually represented by three coordinate values ​​(such as x, y, z). For a 3D Gaussian primitive, its position parameters are the parameters that determine the coordinates of the primitive's center in 3D space.

[0103] For example, in each round of training iteration, the gradient value of the position parameter of the 3D Gaussian primitive is calculated using the backpropagation algorithm, and the gradient values ​​obtained in each round of iteration are added together and averaged to obtain the spatial gradient of the 3D Gaussian primitive.

[0104] Understandably, in 3D scene reconstruction tasks, the goal is usually to restore the shape of objects. By focusing only on the gradient values ​​of the position parameters of 3D Gaussian primitives for inverse optimization, on the one hand, the calculation of other space-independent parameters such as color can be reduced, accelerating model convergence and improving model training speed; on the other hand, by optimizing the position parameters, the distribution of primitives can be directly adjusted to more efficiently approximate the target shape or surface, thereby achieving efficient scene reconstruction while ensuring high-quality dynamic scene reconstruction.

[0105] In this embodiment, the gradient of each three-dimensional Gaussian primitive is continuously monitored during each iteration of training. After the number of iterations reaches the second iteration threshold, the average gradient value of multiple iterations is determined as the spatial gradient of the three-dimensional Gaussian primitive, which is used as the basis for judging model training optimization. This reduces misjudgments caused by outliers in a single iteration, ensures the training effectiveness of the Gaussian splash model, and thus improves training efficiency and parameter update efficiency.

[0106] Step S30: Send the updated parameters in the optimized first four-dimensional Gaussian splash model to the client so that the client can update the second four-dimensional Gaussian splash model according to the received parameters and reconstruct the scene using the updated second four-dimensional Gaussian splash model.

[0107] A client is an application terminal that receives scene reconstruction requests and corresponding input data from users.

[0108] The second four-dimensional Gaussian splash model refers to a model example with the same structure as the first four-dimensional Gaussian splash model, deployed on the client, and used for real-time scene rendering on the client.

[0109] For example, after the server completes training for each dynamic sliding window, it can send the sliding step number of the current sliding window and the updated parameters in the optimized first four-dimensional Gaussian splash model of that window to the client. Then, the client can move its sliding window on the video frame sequence to be reconstructed on its own end according to the received sliding step number of the sliding window, and directly replace the corresponding parameters in the second four-dimensional Gaussian splash model of that window according to the received updated parameters to obtain the optimized second four-dimensional Gaussian splash model. The optimized second four-dimensional Gaussian splash model is then used to reconstruct the scene of the video frame to be reconstructed in the sliding window.

[0110] Understandably, by timely updating and synchronizing parameters to the client, the second-dimensional Gaussian splash model of each sliding window on the client is kept in sync with the server-side model, thereby improving the timeliness and response speed of scene reconstruction.

[0111] Step S40: According to the preset sliding step size, move the dynamic sliding window and repeat the process of reconstructing the scene of the video frames within the preset dynamic sliding window through the first four-dimensional Gaussian splash model to obtain multiple restored scene frames until the dynamic sliding window moves to the end of the video frame sequence.

[0112] The sliding step size is a parameter used to control the distance the sliding window moves forward on the time axis. It is usually expressed in frames and determines how many frames are added and how many old frames are discarded when the window slides forward.

[0113] For example, let the length of the dynamic sliding window be L and the sliding step size be S; if the time corresponding to the first frame of the current sliding window is T, and the four-dimensional representation of the previous window is known as... The multi-view observations from time T to T+L are I T:T+L The training process can then be represented as:

[0114]

[0115] This embodiment provides a scene reconstruction method. By setting a dynamic sliding window, the parameters are updated to the client synchronously after training is completed on the video frames within the current sliding window. This allows the client's four-dimensional Gaussian splash model to perform scene reconstruction without completing training in the full time domain. Naturally, the efficiency advantage of offline reconstruction representation is introduced into streaming input, effectively improving the response efficiency of scene reconstruction based on the four-dimensional Gaussian splash model.

[0116] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. In addition, the deformed network includes HexPlane, and before step S20, it further includes:

[0117] Step S201: When the dynamic sliding window is not the starting window and the iteration number is the first time, determine the overlapping frame between the current window and the previous window;

[0118] Overlapping frames refer to video frames that are jointly covered by the current dynamic sliding window and the previous dynamic sliding window on the timeline. Their range is determined by the window length and the sliding step size.

[0119] Step S202: Determine the HexPlane parameter corresponding to the overlapping frame of the previous window as the HexPlane parameter corresponding to the overlapping frame of the current window, and determine the HexPlane parameter corresponding to the last frame of the previous window as the HexPlane parameter of the non-overlapping frame in the current window.

[0120] HexPlane is a neural feature structure used to represent dynamic 3D scenes. It decomposes features into six feature planes, each spanning a different pair of coordinate axes. Specifically, it includes spatial planes (XY, XZ, YZ) and spatiotemporal planes (XT, YT, ZT). The spatial planes are used to capture the static structure of the scene, while the spatiotemporal planes are used to capture the changes in the scene over time.

[0121] For example, for each overlapping frame t, the feature parameters corresponding to the overlapping frame t are extracted from the Hexplane trained in the previous window, and these feature parameters are directly copied to the position of the corresponding frame t in the Hexplane of the current window; for non-overlapping frames in the current window, the feature parameters corresponding to the last frame are extracted from the Hexplane of the previous window, and the feature parameters corresponding to the last frame are directly determined as the position of the non-overlapping frame in the Hexplane of the current window.

[0122] Understandably, for overlapping frames, directly copying the optimized and precise parameters makes these frames nearly optimal at the starting point of the new sliding window, avoiding repeated learning of known information, thereby reducing the number of iterations. At the same time, it eliminates the dynamic jump or flickering problem at the window boundary that may be caused by parameter reinitialization, thus improving the training efficiency and training effect of the model.

[0123] Step S203: The parameters of other components trained in the previous window are determined as the parameters of other components in the current window, wherein the other components are the components other than HexPlane in the first four-dimensional Gaussian splash model.

[0124] Other components refer to components other than HexPlane in the first four-dimensional Gaussian splash model, such as three-dimensional Gaussian primitives and multilayer perceptrons (MLP).

[0125] For example, after the reverse optimization of the previous window is completed, all parameters of the other components in the first four-dimensional Gaussian splash model, except for HexPlane, can be saved as a snapshot; then, before the first iteration training begins in the current window, the parameters in the above snapshot are directly overwritten with the parameters of the other components in the current window.

[0126] For example, please refer to Figure 2 , Figure 2 A holistic training framework based on dynamic sliding windows is proposed, wherein the first four-dimensional Gaussian splashing module includes three-dimensional Gaussian primitives and a deformable network, the deformable network including compressed HexPlane and at least one multilayer perceptron. First, the video frame sequence can be divided into at least N sliding windows according to the length L and the sliding step size. Figure 2Part a illustrates the training process for the initial window (window 0). First, the 3D Gaussian primitives are trained. Since no deformation network is used to adjust the 3D Gaussian primitives, they are referred to as static Gaussian primitives. The specific training process involves reconstructing video frames within window 0 using the static Gaussian primitives and then optimizing these primitives based on the reconstruction results, continuing until the first preset iteration threshold is reached. Next, the static Gaussian primitives and the deformation network are trained together. The training process specifically includes: inputting the trained static Gaussian primitives and video frames with time t∈{1,2,...,L} into the deformation network, which outputs the deformation parameters corresponding to each attribute (such as displacement and rotation) of the Gaussian primitives. The deformation network includes a compressed HexPlane and at least one multilayer perceptron. The compressed HexPlane refers to retaining only the single-resolution spatial plane and the HexPlane with a smaller temporal resolution in the spatiotemporal plane. Finally, the static Gaussian primitives are adjusted according to the deformation parameters to obtain dynamic Gaussian primitives. Figure 2 Part b illustrates the training process for non-first sliding windows. Before iterative training, the current window is initialized, specifically including: for the overlapping part of the current window with the previous window in the spatiotemporal plane of HexPlane, the feature parameters of the corresponding frame in the previous window can be directly copied; for the non-overlapping part, the feature parameters of the last frame of the previous window are directly copied. For other components (including Gaussian units, the spatial plane of HexPlane, and the multilayer perceptron), the parameters trained by the previous window can be directly copied. In addition, the training process for non-first sliding windows can be divided into two stages. In the first stage, all Gaussian units are in a frozen state, and their parameters remain unchanged. Other components are trained normally. In the second stage, the parameters of high-gradient Gaussian units (3D Gaussian units with spatial gradients greater than a preset gradient threshold) and other components are trained and updated, while 3D Gaussian units with spatial gradients less than or equal to the preset gradient threshold remain in a frozen state.

[0127] In this embodiment, before the iterative training of the current window begins, by inheriting the parameters of the previous window (including HexPlane and other components in the first four-dimensional Gaussian splash model), the jitter, flickering or degradation of static scene geometry and appearance caused by model re-initialization or random optimization is eliminated. This provides a strong spatiotemporal prior for subsequent training, accelerates the convergence speed of the model, and enhances Gaussian consistency across windows, thereby improving the training efficiency and training effect of the four-dimensional Gaussian splash model, so as to maintain the high responsiveness of online reconstruction on the client side.

[0128] In one feasible implementation, HexPlane includes a spatial plane and a spatiotemporal plane, and prior to step S20, it further includes:

[0129] Step S204: Delete the high-resolution plane in the spatial plane;

[0130] The spatial planes of HexPlane refer to the planes (XY, XZ, YZ) used to represent the scene spatial information of video frames; in the original structure of HexPlane, each spatial plane includes both single-resolution and high-resolution (2X) versions.

[0131] Understandably, within a short window like a dynamic sliding window, a single resolution is sufficient to represent scene changes. By removing the high-resolution spatial plane, the memory footprint and associated computation of the four-dimensional Gaussian splash model are reduced, effectively improving the model's storage and training efficiency, so as to maintain the high responsiveness of online reconstruction on the client side.

[0132] Step S205: Determine the temporal resolution of the spatiotemporal plane based on the window length of the dynamic sliding window, wherein the temporal resolution is directly proportional to the window length.

[0133] The HexPlane spatiotemporal plane is a plane that contains both spatial and temporal information of a video frame sequence, used to represent the changes of a dynamic scene in time and space.

[0134] Temporal resolution refers to the number of grids that the spatiotemporal plane is divided into along the time dimension (T-axis). The higher the temporal resolution, the denser the sampling on the time axis, and the more training parameters there are on the spatiotemporal plane.

[0135] For example, when constructing the first four-dimensional Gaussian splash model, the temporal resolution of the HexPlane spatiotemporal plane can be calculated based on the window length of the dynamic sliding window and a preset proportional relationship; for example, the proportional relationship can be set as: temporal resolution = k × window length, where k is a proportionality constant (e.g., k = 1 / 2).

[0136] In this embodiment, by setting the temporal resolution of the spatiotemporal plane according to the length of the sliding window, the temporal resolution is significantly reduced compared to the original 4D Gaussian representation where the temporal resolution is proportional to the total number of frames. This reduces the number of parameters to be trained in the spatiotemporal plane, reduces memory usage, and speeds up training, while still having sufficient modeling capabilities to capture dynamic changes within the window.

[0137] To further illustrate the superiority of the scene reconstruction method proposed in this application, this embodiment employs the 3DGStream method, the HiCoM (Hierarchical Coherent Motion) method, and the TGW (Temporal Gaussian Window) method of this application (window length of 8 frames, window sliding step size of 4 frames, only 250 iterations per window, batch size of 2). Qualitative and quantitative experiments are conducted on the widely used N3DV and Meet Room datasets to compare their differences in key performance indicators. Specifically, the peak signal-to-noise ratio (PSNR) from a unified testing perspective is used to evaluate reconstruction quality; average storage per frame is used to evaluate storage efficiency; and average training time per frame is used to evaluate reconstruction speed. As shown in Table 1, which presents the performance indicator results for the 3DGStream method, HiCoM method, and TGW method on the N3DV and Meet Room datasets, respectively.

[0138] Table 1. Examples of performance index results

[0139]

[0140] As shown in Table 1, the TGW method of this application maintains highly competitive rendering quality while significantly reducing single-frame storage and accelerating single-frame training. Please also refer to... Figure 3 , Figure 3 A comparison chart of the effects of different scene reconstruction methods is provided. It can be seen that the TGW method of this application performs well in preserving static details and accurately capturing dynamic targets. It can be on par with the HiCoM method and even better than the 3DGStream method.

[0141] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the scenario reconstruction method of this application. Any simple transformations based on this technical concept are all within the protection scope of this application.

[0142] This application also provides a scene reconstruction device, please refer to... Figure 4 The scene reconstruction device includes:

[0143] Sequence acquisition module 10 is used to acquire video frame sequences;

[0144] The model optimization module 20 is used to reconstruct the scene of the video frames in the preset dynamic sliding window in the video frame sequence through the first four-dimensional Gaussian splash model, obtain multiple restored scene frames, and optimize the first four-dimensional Gaussian splash model in reverse according to each restored scene frame and its corresponding video frame.

[0145] The parameter sending module 30 is used to send the updated parameters in the optimized first four-dimensional Gaussian splash model to the client, so that the client can update the second four-dimensional Gaussian splash model according to the received parameters and reconstruct the scene through the updated second four-dimensional Gaussian splash model.

[0146] The window moving module 40 is used to move the dynamic sliding window according to the preset sliding step size, and repeatedly perform scene reconstruction on the video frames in the preset dynamic sliding window through the first four-dimensional Gaussian splash model to obtain multiple restored scene frames until the dynamic sliding window moves to the end of the video frame sequence.

[0147] The scene reconstruction apparatus provided in this application, employing the scene reconstruction method described in the above embodiments, can solve the technical problem of how to improve the response efficiency of scene reconstruction based on four-dimensional Gaussian splashing. Compared with the prior art, the beneficial effects of the scene reconstruction apparatus provided in this application are the same as those of the scene reconstruction method provided in the above embodiments, and other technical features in the scene reconstruction apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0148] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the scene reconstruction method in the first embodiment described above.

[0149] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0150] like Figure 5 As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0151] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0152] The electronic device provided in this application, employing the scene reconstruction method described in the above embodiments, can solve the technical problem of how to improve the response efficiency of scene reconstruction based on four-dimensional Gaussian splashing. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the scene reconstruction method provided in the above embodiments, and other technical features of the electronic device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0153] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0154] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0155] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the scene reconstruction method in the above embodiments.

[0156] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0157] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0158] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by an electronic device, the electronic device causes the following: to acquire a video frame sequence; to reconstruct a scene from video frames within a preset dynamic sliding window in the video frame sequence using a first four-dimensional Gaussian splash model, obtaining multiple restored scene frames; and to reverse-optimize the first four-dimensional Gaussian splash model based on each restored scene frame and its corresponding video frame; to send the updated parameters in the optimized first four-dimensional Gaussian splash model to a client, so that the client updates a second four-dimensional Gaussian splash model based on the received parameters, and performs scene reconstruction using the updated second four-dimensional Gaussian splash model; and to move the dynamic sliding window according to a preset sliding step size, repeating the steps of reconstructing a scene from video frames within a preset dynamic sliding window using the first four-dimensional Gaussian splash model to obtain multiple restored scene frames, until the dynamic sliding window moves to the end of the video frame sequence.

[0159] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0160] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0161] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0162] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-described scene reconstruction method, and can solve the technical problem of how to improve the response efficiency of scene reconstruction based on four-dimensional Gaussian splashing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the scene reconstruction method provided in the above embodiments, and will not be repeated here.

[0163] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the scene reconstruction method described above.

[0164] The computer program product provided in this application can solve the technical problem of how to improve the response efficiency of scene reconstruction based on four-dimensional Gaussian splashing. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the scene reconstruction method provided in the above embodiments, and will not be repeated here.

[0165] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A scene reconstruction method, characterized in that, The scene reconstruction method includes: Obtain the video frame sequence; Using the first four-dimensional Gaussian splash model, the video frames within the preset dynamic sliding window in the video frame sequence are reconstructed to obtain multiple restored scene frames. Based on each restored scene frame and its corresponding video frame, the first four-dimensional Gaussian splash model is optimized in reverse. The updated parameters in the optimized first four-dimensional Gaussian splash model are sent to the client so that the client can update the second four-dimensional Gaussian splash model according to the received parameters and reconstruct the scene using the updated second four-dimensional Gaussian splash model. According to the preset sliding step size, the dynamic sliding window is moved, and the steps of reconstructing the scene of the video frames within the preset dynamic sliding window through the first four-dimensional Gaussian splash model are repeated to obtain multiple restored scene frames. Based on each restored scene frame and its corresponding video frame, the first four-dimensional Gaussian splash model is optimized in reverse, and the updated parameters in the optimized first four-dimensional Gaussian splash model are sent to the client, until the dynamic sliding window moves to the end of the video frame sequence.

2. The scene reconstruction method as described in claim 1, characterized in that, The first four-dimensional Gaussian splash model includes three-dimensional Gaussian primitives and a deformable network. The step of reconstructing the scene of video frames within a preset dynamic sliding window in the video frame sequence using the first four-dimensional Gaussian splash model to obtain multiple restored scene frames, and then optimizing the first four-dimensional Gaussian splash model in reverse based on each restored scene frame and its corresponding video frame, includes: When the dynamic sliding window is the starting window, the video frames within the dynamic sliding window are reconstructed using the three-dimensional Gaussian primitives to obtain multiple static restored frames. Based on each static restored frame and the corresponding video frame, the three-dimensional Gaussian primitives are optimized in reverse until the number of iterations reaches a preset first iteration threshold. The video frames within the dynamic sliding window are reconstructed using the optimized 3D Gaussian primitives and the deformable network to obtain multiple dynamically restored frames. Based on each dynamically restored frame and the corresponding video frame, the optimized 3D Gaussian primitives and the deformable network are then optimized in reverse.

3. The scene reconstruction method as described in claim 1, characterized in that, The first four-dimensional Gaussian splash model includes three-dimensional Gaussian primitives and a deformable network. The step of reverse-optimizing the first four-dimensional Gaussian splash model based on each of the restored scene frames and their corresponding video frames includes: When the dynamic sliding window is not the starting window and the number of iterations does not exceed the preset second iteration threshold, the deformation network is optimized in reverse according to each restored scene frame and the corresponding video frame; When the dynamic sliding window is not the starting window and the number of iterations exceeds the second iteration threshold, the spatial gradient of the three-dimensional Gaussian unit is determined according to each restored scene frame and the corresponding video frame, and the deformable network and the three-dimensional Gaussian unit with spatial gradients higher than the preset gradient threshold are optimized in reverse.

4. The scene reconstruction method as described in claim 3, characterized in that, The step of determining the spatial gradient of the three-dimensional Gaussian unit based on each of the restored scene frames and the corresponding video frames includes: In each round of iterative training, the gradient values ​​of the three-dimensional Gaussian primitives are determined based on the backpropagation algorithm, each restored scene frame, and the corresponding video frame. The spatial gradient of the three-dimensional Gaussian primitive is obtained by averaging the gradient values ​​corresponding to each round of iterative training.

5. The scene reconstruction method as described in claim 1, characterized in that, The first four-dimensional Gaussian splash model includes HexPlane. Before the step of reconstructing the scene from video frames within a preset dynamic sliding window using the first four-dimensional Gaussian splash model to obtain multiple restored scene frames, the method further includes: When the dynamic sliding window is not the starting window and the iteration number is the first time, the overlapping frame between the current window and the previous window is determined; The HexPlane parameter corresponding to the overlapping frame of the previous window is determined as the HexPlane parameter corresponding to the overlapping frame of the current window, and the HexPlane parameter corresponding to the last frame of the previous window is determined as the HexPlane parameter of the non-overlapping frame of the current window. The parameters of other components trained in the previous window are used as the parameters of other components in the current window, wherein the other components are the components other than HexPlane in the first four-dimensional Gaussian splash model.

6. The scene reconstruction method as described in claim 5, characterized in that, The HexPlane includes a spatial plane and a spatiotemporal plane. Before the step of reconstructing the scene from video frames within a preset dynamic sliding window using a first four-dimensional Gaussian splash model to obtain multiple restored scene frames, the method further includes: Delete the high-resolution plane in the aforementioned spatial plane; The temporal resolution of the spatiotemporal plane is determined based on the window length of the dynamic sliding window, wherein the temporal resolution is directly proportional to the window length.

7. The scene reconstruction method as described in claim 1, characterized in that, Before the step of reconstructing the scene of video frames within a preset dynamic sliding window in the video frame sequence using the first four-dimensional Gaussian splash model, the method further includes: The initial video frames of the video frame sequence are sparsely reconstructed using motion structure recovery technology to obtain sparse point cloud frames. The first four-dimensional Gaussian splash model is initialized based on the sparse point cloud frame.

8. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the scene reconstruction method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the scene reconstruction method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the scene reconstruction method as described in any one of claims 1 to 7.

Images