Dynamic scene reconstruction method and device, equipment and storage medium

Abstract

The invention discloses a dynamic scene reconstruction method and device, equipment and a storage medium, and relates to the technical field of computer vision, and the method comprises the steps: generating a corresponding SFM (motion recovery structure) point cloud according to multiple frames of original images, and carrying out the down-sampling of the SFM point cloud, and obtaining each seed point in the SFM point cloud; for any seed point, calculating a feature vector of the seed point based on the feature information of the seed point, and generating a time Gaussian point corresponding to the seed point according to the feature vector; and constructing a Gaussian point cloud based on each time Gaussian point obtained after traversing each seed point, and generating a target view of the dynamic scene corresponding to the multiple frames of original images according to Gaussian point cloud rendering. According to the method, the complex dynamic scene is divided into a plurality of local space arrays, the problem of dynamic modeling complexity caused by full-time-domain unified modeling in a traditional method is solved, accurate reconstruction of the dynamic motion scene is achieved, and a solution with higher robustness and practicability is provided for dynamic scene reconstruction.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and particularly to a method, device, equipment and storage medium for dynamic scene reconstruction. Background Art

[0002] NVS (Novel View Synthesis) of dynamic scenes has long been a key issue in the fields of computer vision and graphics, and has extensive applications in industries such as AR (Augmented Reality), VR (Virtual Reality) and games. The view synthesis task itself poses many challenges, and synthesizing novel views of dynamic scenes is even more challenging.

[0003] Currently, 3DGS (Gaussian Splatting) represents the scene as a 3D Gaussian point cloud and has achieved new optimal results in terms of rendering quality, training and inference speed through splatting technology. However, this method is limited to static scenes. At the same time, many reconstruction methods are based on NeRF (Neural Radiance Field), and various solutions have been proposed, such as deformation fields, scene flow prediction and multi-plane representation. Among them, the deformation field method maps the Gaussian points in the canonical field to the deformation field to represent the dynamic scene at each timestamp. However, this method is often restricted by complex motions and it is difficult to achieve high-quality rendering; trajectory-tracking-based solutions usually use polynomials or Fourier series to represent the motion trajectories of each Gaussian. However, these methods usually have difficulty accurately simulating the motion of each point, resulting in inconsistent Gaussian points between frames and scene distortion; while the method of extending 3DGS to 4DGS requires a large number of Gaussian points for fitting, resulting in high storage requirements and slow training speed. These methods usually over-expand the Gaussian function to adapt to all training views, ignoring the scene structure and compromising the robustness to a large number of view changes. By uniformly applying global dynamic modeling to all Gaussian points, they also increase the computational complexity and generate dynamic and static blur problems. In addition, they often encounter severe blur or flicker in scenes with large-scale motions.

[0004] Therefore, how to provide a more robust and practical solution for the field of dynamic scene reconstruction is an urgent problem to be solved. Summary of the Invention

[0005] The main purpose of this application is to provide a method, device, equipment and storage medium for dynamic scene reconstruction, aiming to solve the technical problem of providing a more robust and practical solution for the field of dynamic scene reconstruction.

[0006] To achieve the above object, the present application proposes a dynamic scene reconstruction method, and the dynamic scene reconstruction method includes:

[0007] Generating a corresponding Structure from Motion (SFM) point cloud based on multiple frames of original images, and downsampling the SFM point cloud to obtain various sub-points in the SFM point cloud;

[0008] For any seed point, calculating a feature vector of the seed point based on the feature information of the seed point, and generating a temporal Gaussian point corresponding to the seed point according to the feature vector;

[0009] Constructing a Gaussian point cloud based on the temporal Gaussian points obtained after traversing various sub-points, and rendering a target view of the dynamic scene corresponding to the multiple frames of original images according to the Gaussian point cloud.

[0010] In one embodiment, the feature information includes the position of the seed point and the static feature at the seed point, wherein the static feature remains synchronized at each preset query moment corresponding to the seed point, and the preset query moments correspond to the respective frames of the multiple frames of original images;

[0011] The step of calculating the feature vector of the seed point based on the feature information of the seed point includes:

[0012] Determining the dynamic residual feature of the seed point according to the position of the seed point;

[0013] Determining the feature vector of the seed point based on the dynamic residual feature and the static feature.

[0014] In one embodiment, the dynamic scene reconstruction method further includes:

[0015] Supervising the target view based on the multiple frames of original images to obtain a supervision result, and backpropagating and optimizing the static feature according to the supervision result to obtain a new static feature;

[0016] Returning to execute the step of determining the feature vector of the seed point based on the dynamic residual feature and the static feature according to the new static feature.

[0017] In one embodiment, the step of determining the dynamic residual feature of the seed point according to the position of the seed point includes:

[0018] Mapping the seed point to a hash table of a target dimension at different resolutions to obtain hash encodings of the seed point at each resolution, wherein the mapping sources in the mapping process include the position of the seed point and the preset query moment;

[0019] Performing cross-fusion on the hash encodings to obtain the dynamic residual feature of the seed point.

[0020] In one embodiment, the step of determining the feature vector of the seed point based on the dynamic residual feature and the static feature includes:

[0021] Input the seed point position and the preset query time into a preset first multi-layer perceptron to predict, through the first multi-layer perceptron, a first weight corresponding to the dynamic residual feature and a second weight corresponding to the static feature;

[0022] Calculate the feature vector of the seed point according to the first weight, the second weight, the dynamic residual feature and the static feature.

[0023] In one embodiment, the feature information further includes a scale parameter of the seed point, and the step of generating a temporal Gaussian point corresponding to the seed point according to the feature vector includes:

[0024] Input the feature vector, the seed point position and the scale parameter into a preset second multi-layer perceptron to predict, through the second multi-layer perceptron, various temporal Gaussian parameters corresponding to the seed point;

[0025] Construct a temporal Gaussian point corresponding to the seed point based on the various temporal Gaussian parameters.

[0026] In one embodiment, the second multi-layer perceptron includes a plurality of independent multi-layer perceptrons, and the step of inputting the feature vector, the seed point position and the scale parameter into a preset second perceptron includes:

[0027] Input the feature vector, the seed point position and the scale parameter into each independent multi-layer perceptron to predict, through each independent multi-layer perceptron, corresponding temporal Gaussian parameters;

[0028] After traversing each independent multi-layer perceptron, various temporal Gaussian parameters corresponding to the seed point are obtained.

[0029] In addition, to achieve the above object, the present application further provides a dynamic scene reconstruction device, and the dynamic scene reconstruction device includes:

[0030] A spatial partitioning module, configured to generate a corresponding structure from motion (SFM) point cloud according to multiple frames of original images, and downsample the SFM point cloud to obtain various sub-points in the SFM point cloud;

[0031] A feature extraction module, configured to calculate, for any one seed point, a feature vector of the seed point based on the feature information of the seed point, and generate a temporal Gaussian point corresponding to the seed point according to the feature vector;

[0032] A view rendering module, configured to construct a Gaussian point cloud based on various temporal Gaussian points obtained after traversing various sub - points, and render and generate a target view of the dynamic scene corresponding to the multi - frame original images according to the Gaussian point cloud.

[0033] In addition, to achieve the above object, the present application also provides an electronic device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the dynamic scene reconstruction method as described above.

[0034] In addition, to achieve the above object, the present application also provides a storage medium, which is a computer - readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by a processor, it implements the steps of the dynamic scene reconstruction method as described above.

[0035] In addition, to achieve the above object, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the dynamic scene reconstruction method as described above.

[0036] One or more technical solutions proposed by the present application have at least the following technical effects:

[0037] The present application first generates a corresponding Structure from Motion (SFM) point cloud based on multi - frame original images, and downsamples the SFM point cloud to obtain various sub - points in the SFM point cloud, so as to reconstruct a 3D point cloud from multi - frame original images through the SFM (Structure from Motion) technology. Then, the point cloud is downsampled to extract representative seed points, and the dynamic scene is divided into multiple local space arrays corresponding to various sub - points, significantly reducing the complexity of subsequent dynamic modeling; for any seed point, a feature vector of the seed point is calculated based on the feature information of the seed point, and a temporal Gaussian point corresponding to the seed point is generated according to the feature vector to independently model dynamic changes within the local space corresponding to each seed point, thereby achieving accurate restoration of the global scene on a localized basis; a Gaussian point cloud is constructed based on various temporal Gaussian points obtained after traversing various sub - points, and a target view of the dynamic scene corresponding to the multi - frame original images is rendered and generated according to the Gaussian point cloud. Through the construction and rendering of the Gaussian point cloud, high - quality reconstruction of the dynamic scene is achieved, and a target view corresponding to the original image can be generated, solving the problem of rendering quality in dynamic scene reconstruction.

[0038] In summary, by dividing the complex dynamic scene into multiple local spatial arrays, the present application not only effectively resolves the problem of dynamic modeling complexity caused by unified modeling in the entire time domain in traditional methods, but also enhances the adaptability to motions of different scales in the dynamic scene. Whether it is a tiny local motion or a large-scale scene change, this method can handle them flexibly, avoiding the problems of motion blur and insufficient adaptability to large-scale motions caused by unified modeling in existing methods. Thus, it realizes the efficient modeling and precise reconstruction of dynamic motion scenes of various scales, providing a more robust and practical solution for the field of dynamic scene reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings herein are incorporated into and constitute a part of this specification, showing embodiments consistent with the present application and, together with the specification, are used to explain the principles of the present application.

[0040] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0041] Figure 1 It is a schematic flowchart provided for the first embodiment of the dynamic scene reconstruction method of the present application;

[0042] Figure 2 It is a schematic flowchart provided for the second embodiment of the dynamic scene reconstruction method of the present application;

[0043] Figure 3 It is a brief schematic flowchart of the dynamic scene reconstruction method provided for the second embodiment of the present application;

[0044] Figure 4 It is a schematic diagram of local modeling of the dynamic scene reconstruction method provided for the second embodiment of the present application;

[0045] Figure 5 It is a schematic diagram of the module structure of the dynamic scene reconstruction device according to the embodiment of the present application;

[0046] Figure 6 It is a schematic diagram of the device structure of the hardware operating environment involved in the dynamic scene reconstruction method according to the embodiment of the present application.

[0047] The implementation, functional features, and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0049] To better understand the technical solution of this application, the following will be described in detail in conjunction with the accompanying drawings of the specification and specific embodiments.

[0050] The main solution of the embodiments of this application is: generating a corresponding Structure from Motion (SFM) point cloud based on multiple frames of original images, and downsampling the SFM point cloud to obtain various sub-points in the SFM point cloud; for any seed point, calculating the feature vector of the seed point based on the feature information of the seed point, and generating a temporal Gaussian point corresponding to the seed point according to the feature vector; constructing a Gaussian point cloud based on the temporal Gaussian points obtained after traversing various sub-points, and rendering and generating a target view of the dynamic scene corresponding to the multiple frames of original images

[0051] Since current 3D Gaussian Splatting (3DGS) represents the scene as a 3D Gaussian point cloud and has achieved new optimal results in terms of rendering quality, training, and inference speed through splatting techniques, but this method is limited to static scenes. At the same time, many reconstruction methods are based on Neural Radiance Fields (NeRF) and propose various solutions, such as deformation fields, scene flow prediction, and multi-plane representations. Among them, the deformation field method maps the Gaussian points in the canonical field to the deformation field to represent the dynamic scene at each timestamp. However, this method is often limited by complex motions and it is difficult to achieve high-quality rendering; trajectory-tracking-based solutions usually use polynomials or Fourier series to represent the motion trajectories of each Gaussian. However, these methods usually have difficulty accurately simulating the motion of each point, resulting in inconsistent Gaussian points between frames and causing scene distortion; while the method of extending 3DGS to 4DGS requires a large number of Gaussian points for fitting, resulting in high storage requirements and slow training speed. These methods usually over-expand the Gaussian function to adapt to all training views, ignoring the scene structure and compromising the robustness to large variations in views. By uniformly applying global dynamic modeling to all Gaussian points, they also increase the computational complexity and generate dynamic-static blur problems. In addition, they often encounter severe blur or flickering in scenes with large-scale motions. Therefore, how to provide a more robust and practical solution for the field of dynamic scene reconstruction is an urgent problem to be solved

[0052] This application provides a solution. By dividing the complex dynamic scene into multiple local spatial arrays, it not only effectively resolves the problem of dynamic modeling complexity caused by unified modeling in the full time domain in traditional methods, but also enhances the adaptability to motions of different scales in the dynamic scene. Whether it is a tiny local motion or a large-scale scene change, this method can handle them flexibly, avoiding the problems of motion blur and insufficient adaptability to large-scale motions caused by unified modeling in existing methods, thus achieving efficient modeling and accurate reconstruction of dynamic motion scenes of various scales, and providing a more robust and practical solution for the field of dynamic scene reconstruction.

[0053] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of implementing the above functions. Hereinafter, an electronic device will be taken as an example to illustrate this embodiment and the following embodiments.

[0054] Based on this, the embodiment of the present application provides a dynamic scene reconstruction method. Refer to Figure 1 , Figure 1 which is a schematic flowchart of the first embodiment of the dynamic scene reconstruction method of the present application.

[0055] In this embodiment, the dynamic scene reconstruction method includes steps S10 to S30:

[0056] Step S10: Generate a corresponding Structure from Motion (SFM) point cloud according to multiple frames of original images, and downsample the SFM point cloud to obtain various sub-points in the SFM point cloud;

[0057] It should be noted that multiple frames of original images refer to a continuous image sequence captured at different times, and this image sequence contains the dynamic change information of the scene; the SFM point cloud refers to a three-dimensional spatial point cloud recovered from multiple frames of original images through the Structure from Motion algorithm, and this point cloud represents the geometric structure of the scene; the seed points refer to the points selected from the SFM point cloud that represent the key structures of the scene, and these points will be used for subsequent feature calculation and view synthesis.

[0058] It can be understood that since the high-density SFM point cloud directly generated from multiple frames of images contains a large amount of redundant data and noise points, directly using this data will lead to too high computational complexity and storage overhead. Therefore, step S10 is performed. By downsampling the high-density point cloud, it is possible to avoid the computational burden and storage pressure brought by the high-density point cloud, reduce the interference of noise points on the reconstruction result, retain the main structure information of the scene, reduce the complexity of data processing, and provide efficient and high-quality basic data for subsequent dynamic scene reconstruction. At the same time, since the dynamic scene is divided into multiple local space arrays corresponding to various sub-points, the complexity of subsequent dynamic modeling is significantly reduced.

[0059] Exemplarily, the SFM algorithm (such as COLMAP or OpenMVG) is used to extract feature points from multiple frames of raw images and calculate the pose of the camera, generating an initial SFM point cloud. Then, the voxel grid downsampling method is adopted to divide the point cloud into uniform voxel grids, and the point closest to the center in each voxel is retained as the seed point. By adjusting the size of the voxel grid, the downsampling density can be controlled, thereby reducing the number of points in the point cloud while preserving the scene structure. Finally, the set of downsampled seed points is output as the basis for subsequent dynamic scene reconstruction.

[0060] Step S20: For any one seed point, calculate the feature vector of the seed point based on the feature information of the seed point, and generate a temporal Gaussian point corresponding to the seed point according to the feature vector.

[0061] It should be noted that the feature information refers to the attributes such as the position and range scale of the seed point in the dynamic scene corresponding to multiple frames of images, and these information are used to describe the characteristics of the point; the feature vector refers to the vector calculated based on the feature information of the seed point, and this vector encodes the spatio-temporal characteristics of the seed point; the temporal Gaussian point refers to the parametric model generated according to the feature vector of the seed point, and this model can represent the change of the seed point in the time dimension.

[0062] It can be understood that due to the complex and irregular movement of points in the dynamic scene, it is difficult for traditional methods to accurately model the movement trajectory of each point, resulting in inter-frame inconsistency and scene distortion, or the dynamic modeling complexity is too high due to the unified modeling in the entire time domain. Therefore, step S20 is performed. By accurately modeling the feature vector and Gaussian point in the local space, the movement trajectory of each point in the dynamic scene can be captured, avoiding problems such as inaccurate dynamic scene modeling, inter-frame inconsistency, scene distortion, and high modeling complexity caused by complex movement, improving the accuracy and consistency of dynamic scene reconstruction, and enhancing the stability of rendering at the same time.

[0063] Exemplarily, for each seed point, first extract its position information and static features (such as color, normal vector, etc.), then input the seed point position and the preset query time into an MLP (Multi-Layer Perceptron), predict the dynamic residual feature of the point, and then fuse the dynamic residual feature with the static feature to generate a feature vector. Finally, input the feature vector into another MLP to predict the Gaussian parameters (such as mean, covariance, etc.) of the point at the current query time, and generate the corresponding temporal Gaussian point according to these parameters. In this way, each seed point can generate a corresponding Gaussian point at different timestamps, thereby capturing the changes in the dynamic scene.

[0064] In a feasible implementation manner, the feature information includes the seed point position of the seed point and the static features at the seed point, where the static features are synchronized at each preset query moment corresponding to the seed point, and the preset query moments correspond to each frame of the multi-frame original images;

[0065] It should be noted that the static features refer to the inherent attributes of the seed point in the spatial position, such as color, texture, shape, etc., and these features are invariant or change very little over time; the preset query moment refers to a specific time point set during the view synthesis process to maintain the consistency of the static features, and these time points are used to ensure that the static features of the seed point remain consistent in the synthesized views at different time points.

[0066] The step of calculating the feature vector of the seed point based on the feature information of the seed point in step S20 may include steps S21 to S22:

[0067] Step S21, determining the dynamic residual features of the seed point according to the seed point position;

[0068] It should be noted that the dynamic residual features refer to the feature increments (such as displacement, deformation, etc.) that change over time in a dynamic scene, and are used to describe the dynamic change part of the seed point at different timestamps. This feature is calculated through the position of the seed point and the preset query moment, and reflects the motion or deformation difference of this point in the dynamic scene.

[0069] It can be understood that due to the complexity and time-variability of the motion or deformation of points in a dynamic scene, directly modeling the overall dynamic features by traditional methods is likely to bring an excessive load on the dynamic residual field, and at the same time, it is also likely to introduce noise or overfitting. Therefore, in step S21, by separating the dynamic residual features, the local dynamic changes at each timestamp can be accurately captured, avoiding problems such as blurred motion trajectories and inter-frame inconsistencies (such as flickering) caused by mixing the modeling of dynamic and static features. This decoupled design helps to reduce the load on the dynamic residual field and improve the flexibility of motion modeling.

[0070] Exemplarily, for each seed point, its position information in multiple frames of images is obtained through the Structure from Motion (SFM) algorithm. Then, these position information are used to model the motion trajectory of the seed point in the time series. For example, the position of the seed point in non-sampled frames can be predicted by linear interpolation or optical flow method. Then, the predicted position is compared with the position of the seed point actually detected in the non-sampled frame, and the difference between the two is calculated. This difference is the dynamic residual feature. This feature can be a vector, including the motion deviations of the seed point in each direction.

[0071] In a feasible implementation, step S21 may include steps S211 to S212:

[0072] Step S211, mapping the seed points to a hash table in the target dimension at different resolutions to obtain the hash codes of the seed points at each resolution, where the mapping sources in the mapping process include the seed point positions and a preset query time;

[0073] It should be noted that the target dimension refers to the dimensionality of the feature space where the data after mapping the seed points exists. This data is usually predefined to ensure data consistency and processing efficiency; a hash table is a data structure used to quickly search for and store key-value pairs, where the key is the hash code of the seed point and the value is the corresponding seed point information; a hash code refers to a unique identifier obtained by calculating the hash function for the seed point, which encodes the information of the seed point into a fixed-length string or number for quick retrieval and comparison; the mapping source refers to the input data used to generate the hash code, which includes the position information of the seed point and the preset query time in this embodiment.

[0074] It can be understood that since it is often difficult to simultaneously adapt to and process minute local motions and large-scale scene changes in traditional methods, step S211 is performed. By mapping and encoding the seed points at different resolutions, the problem of insufficient adaptability to large-scale motions in the existing solutions is avoided, thereby enhancing the adaptability to different-scale motions.

[0075] Exemplarily, each voxel grid node at different resolutions is mapped to a hash table storing d-dimensional learnable feature vectors. Given the seed point position μ and the preset query time t, (μ, t) ∈ R 3 , its hash code at the resolution level l can be written as: h 4d (μ, t; l) ∈ R m . This encoded feature is a linear interpolation of the feature vectors corresponding to the vertices of the voxel grid around the insertion point. Therefore, the hash-coded feature f h (μ, t) across L resolutions can be expressed as:

[0076] f h (μ, t) = [h 4d (μ, t; 1), h 4d (μ, t; 2), ……, h 4d (μ, t; L)]

[0077] Step S212, performing cross-fusion on the hash codes to obtain the dynamic residual features of the seed points.

[0078] It is understandable that in order to integrate the seed point information at different resolutions to obtain a more comprehensive description of dynamic features, step S212 is performed, which can avoid the problem of inaccurate feature description caused by insufficient information at a single resolution. By cross-fusing the hash codes, the dynamic residual features of the seed points are extracted, thereby improving the accuracy of feature description and the quality of view synthesis.

[0079] Exemplarily, a shallow, fully fused MLP is used to cross-fuse the hash features at each resolution level to obtain the dynamic residual feature f of the seed points d :

[0080] f d = F d (μ,t) = MLP(f h (μ,t))

[0081] In this embodiment, by performing multi-resolution hash mapping and feature fusion, the problems of insufficient adaptability encountered when processing data in different scale spaces and incomplete feature description at a single resolution are avoided, the adaptability to different scale motions is enhanced, efficient data indexing and retrieval are achieved, and the accurate extraction of the dynamic features of the seed points is realized, thereby improving the performance and accuracy of novel view synthesis in dynamic scenes.

[0082] Step S22: Determine the feature vector of the seed point based on the dynamic residual feature and the static feature.

[0083] It is understandable that a dynamic scene needs to express both static (stable) and dynamic (changing) attributes at the same time. Therefore, step S22 is performed, which avoids the problem of losing the static scene structure (such as material and geometric deformation distortion) by relying only on dynamic features or insufficient dynamic modeling ability by relying only on static features, and can not only maintain the stability of the scene structure but also accurately express dynamic changes, improving the consistency and authenticity of rendering.

[0084] Exemplarily, the static features of each seed point are collected, which include attributes such as color, texture, and shape extracted from the original image that are invariant or change very little. Then, these static features are fused with the previously calculated dynamic residual features. The fusion process can be to merge the vectors of the static features and the dynamic residual features to form a longer feature vector, or to combine the two through feature transformation and weighting methods. For example, principal component analysis can be used for dimensionality reduction and extraction of key features, or a machine learning method can be used to train a model that can predict the comprehensive feature vector of the seed point based on the static features and the dynamic residual features. The finally obtained feature vector will comprehensively describe the spatio-temporal characteristics of the seed point and be used in the subsequent view synthesis process.

[0085] In this embodiment, by decoupling the scene features into static features and dynamic features, since most of the static information in the scene is captured using the static features, and these features are shared among all time steps, the dynamic modeling within the local space is simplified, enabling the spatio-temporal field to only provide dynamic residual features to capture a small amount of dynamic information at each time step, and then specifically modeling and optimizing the dynamic part, thereby achieving an accurate characterization of the scene motion.

[0086] Step S30: Construct a Gaussian point cloud based on the Gaussian points at each time obtained after traversing various sub-points, and render the target view of the dynamic scene corresponding to the multi-frame original images according to the Gaussian point cloud.

[0087] It should be noted that the Gaussian point cloud refers to a point cloud model composed of all Gaussian points at each time, and this model can represent the spatio-temporal changes of the entire dynamic scene; the target view refers to an image in a new perspective synthesized according to user requirements, and this image shows what the dynamic scene looks like in the new perspective.

[0088] It can be understood that since the traditional method has deficiencies in Gaussian point expansion and motion trajectory modeling, which easily leads to problems such as rendering blurring, flickering, and dynamic-static blurring, so step S30 is performed to avoid the problems of rendering blurring, flickering, and dynamic-static blurring caused by excessive Gaussian point expansion or inaccurate motion trajectories. Through the construction and rendering of the Gaussian point cloud, high-quality reconstruction of the dynamic scene is achieved, and the generated target view has high rendering quality and temporal consistency, which is suitable for the application requirements of complex dynamic scenes and improves the visual effect and practicality of rendering.

[0089] Exemplarily, after traversing all seed points and generating the corresponding Gaussian points at each time, these Gaussian points are organized into a Gaussian point cloud according to the time stamps, and then the Gaussian point cloud is rendered using differentiable rendering techniques (such as 3D Gaussian splatting rendering). Specifically, for each pixel, the color and depth values are calculated by accumulating the contributions of all visible Gaussian points (based on their means and covariances). Finally, the target view is generated according to the rendering result and optimized by comparing with the multi-frame original images to ensure the consistency of the rendering result with the real scene. This method can efficiently generate high-quality target views and is suitable for the real-time rendering requirements of dynamic scenes.

[0090] In a feasible embodiment, the feature information further includes the scale parameter of the seed points. The step of generating the Gaussian points at each time corresponding to the seed points in step S30 may include steps S31 to S32:

[0091] Step S31: Input the feature vector, the seed point position, and the scale parameter into a preset second multi-layer perceptron to predict each temporal Gaussian parameter corresponding to the seed point through the second multi-layer perceptron.

[0092] It should be noted that the scale parameter of a seed point refers to the spatial scale information related to the seed point, which is used to control the spatial range affected by the seed point and can be initialized as the average distance between any number of nearest seed points. In areas where seed points are sparse, the scale parameter of each seed point will become larger, that is, the corresponding local space of the seed point also becomes larger; each temporal Gaussian parameter refers to the parameter used to describe the distribution characteristics of the seed point in the time dimension, including the mean, opacity, etc. of the Gaussian function, and this parameter determines the relevant attributes of the temporal Gaussian point.

[0093] It can be understood that in order to learn the dynamic behavior of the seed point in the time dimension from the feature and position information of the seed point, step S31 is performed, which can avoid unnatural or distorted motion in view synthesis caused by the inability to accurately model the temporal dynamics of the seed point. By using a multi-layer perceptron for prediction, an accurate model of the temporal dynamics of the seed point is achieved, thereby enabling the generation of smooth and natural dynamic scenes.

[0094] Exemplarily, the feature vector, three-dimensional spatial position information, and corresponding scale parameter of each seed point are normalized and encoded to ensure data consistency. Then, these processed data are used as input and passed to a pre-trained MLP network. The MLP contains multiple hidden layers, and each hidden layer uses a non-linear activation function (such as ReLU or Sigmoid) to extract and transform features. Through training methods such as backpropagation and gradient descent, the MLP learns the complex mapping relationship between the seed point features and their temporal Gaussian parameters. Finally, the output layer of the MLP generates a set of temporal Gaussian parameters for each seed point.

[0095] In a feasible implementation manner, the second multi-layer perceptron includes multiple independent multi-layer perceptrons. The step of inputting the feature vector, the seed point position, and the scale parameter into the preset second multi-layer perceptron in step S31 may include steps S311 to S312:

[0096] Step S311: Input the feature vector, the seed point position, and the scale parameter into each independent multi-layer perceptron to predict the corresponding temporal Gaussian parameter through each independent multi-layer perceptron.

[0097] It should be noted that an independent multi-layer perceptron refers to a multi-layer perceptron model trained separately for each group of features. Each model is independent and specifically used to predict the specific temporal Gaussian parameter of the seed point.

[0098] It is understandable that since Gaussian parameters at different times may have different dynamic characteristics, and using a single MLP to predict all Gaussian parameters over time often leads to problems with low prediction accuracy. Therefore, step S311 is performed. By customizing an independent model for each group of features to accurately predict their temporal changes, it is possible to avoid the problem that a single model cannot adapt to the complex dynamics of all features, enabling each model to better learn the specific dynamic characteristics of the corresponding seed points, thereby providing prediction accuracy and flexibility.

[0099] Exemplarily, each temporal Gaussian point can be parameterized as G t {μ t ,q t ,s t ,σ t ,c t}, where t represents a preset query time, which is a parameter that allows the temporal Gaussian point to vary over time. Each seed generates k temporal Gaussian distributions, and their means are given by:

[0100]

[0101] where μ and v represent the seed point position and scale parameter of the seed point, represents the i-th temporal Gaussian parameter generated from the seed point at time t, and F μ (·) is a shallow independent MLP that outputs a vector of size k×3, and f w is the feature vector of the seed point. The temporal Gaussian parameter: opacity σ t can be expressed as:

[0102]

[0103] where μ c is the center of the observed camera coordinates, and F σ (·) is another shallow independent MLP. In addition, the quaternion and the scale can both be derived similarly:

[0104]

[0105] Step S312, after traversing each independent multi-layer perceptron, the temporal Gaussian parameters corresponding to the seed points are obtained.

[0106] In this embodiment, by customizing an independent deep learning model (multi-layer perceptron) for each group of temporal Gaussian parameters, the problem that a single model cannot accurately capture the unique dynamic features of each seed point in dynamic scene novel view synthesis is avoided, and accurate dynamic modeling of each seed point in the temporal dimension is achieved, thereby improving the authenticity, continuity, and visual quality of the synthesized view.

[0107] Step S32: Based on the respective temporal Gaussian parameters, construct temporal Gaussian points corresponding to the seed points.

[0108] It can be understood that since the predicted temporal Gaussian parameters need to be converted into specific temporal point representations in order to accurately reproduce the dynamic behavior of the seed points when synthesizing new views, performing step S32 can avoid scene motion discontinuity or jitter caused by the lack of fine temporal distribution information in view synthesis. By constructing temporal Gaussian points, a continuous and smooth representation of the seed points in the temporal dimension is achieved, thereby improving the realism and continuity of dynamic scene synthesis.

[0109] In this embodiment, by adopting deep learning (multi-layer perceptron) and temporal Gaussian modeling, the problems of unnatural, discontinuous, or distorted motion caused by the inability to accurately predict and represent the dynamic changes of seed points in the temporal dimension in dynamic scene novel view synthesis are avoided, and accurate dynamic modeling of seed points in the temporal dimension is achieved, thereby improving the authenticity, continuity, and visual quality of the synthesized view.

[0110] This embodiment provides a dynamic scene reconstruction method. By dividing a complex dynamic scene into multiple local spatial arrays, not only is the problem of dynamic modeling complexity caused by unified modeling in the entire time domain effectively resolved in traditional methods, but also the adaptability to different-scale motions in the dynamic scene is enhanced. Whether it is a tiny local motion or a large-scale scene change, this method can flexibly handle them, avoiding the problems of motion blur and insufficient adaptability to large-scale motions caused by unified modeling in existing methods, thereby achieving efficient modeling and accurate reconstruction of dynamic motion scenes of various scales, and providing a more robust and practical solution for the field of dynamic scene reconstruction.

[0111] In a feasible embodiment, step S22 may include steps S221 to S222:

[0112] Step S221: Input the seed point position and a preset query time to a preset first multi-layer perceptron to predict, through the first multi-layer perceptron, a first weight corresponding to the dynamic residual feature and a second weight corresponding to the static feature;

[0113] It should be noted that the first weight refers to the weight associated with the dynamic residual feature, which is used to adjust the influence degree of the dynamic residual feature on the final feature when synthesizing the feature vector; the second weight refers to the weight associated with the static feature, which is used to adjust the influence degree of the static feature on the final feature when synthesizing the feature vector.

[0114] It can be understood that since the traditional method cannot adaptively balance the learning of static features and dynamic residual features, step S221 is performed. By using a shallow MLP to predict the weights of static features and dynamic residual features, it is possible to avoid the problem that the feature description is inaccurate due to fixed weights in the feature vector calculation and the inability to adapt to the changes of seed points at different time and space positions, enabling the model to adaptively balance the learning of static and dynamic residual features and accelerate convergence, thereby improving the adaptability and accuracy of feature description.

[0115] Exemplarily, design a weight field F w , which is implemented by a shallow MLP, to predict the first weight w d and the second weight w s , that is, w s , w d =F w (μ, t), where F w (·) is the first multi-layer perceptron, μ is the seed point position, and t is the preset query time.

[0116] Step S222, calculate the feature vector of the seed point according to the first weight, the second weight, the dynamic residual feature, and the static feature.

[0117] Exemplarily, calculate the feature vector f w of the seed point through the following formula:

[0118] f w =w s f s +w d f d

[0119] where f s is the static feature at the seed point.

[0120] In this embodiment, by using an MLP to design a weight field, the model can adaptively balance the learning of static and dynamic residual features and accelerate convergence, avoiding the problems of inaccurate and unrepresentative feature description caused by fixed weights or single feature representation in dynamic scene novel view synthesis, realizing a dynamic and adaptive feature representation of the spatio-temporal characteristics of seed points, thereby improving the realism, continuity of view synthesis, and the accuracy of scene reconstruction.

[0121] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar content as that in the above-mentioned first embodiment can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 2 , the dynamic scene reconstruction method further includes steps S01 to S02:

[0122] Step S01, supervise the target view based on the multiple frames of original images to obtain a supervision result, and optimize the static features by backpropagation according to the supervision result to obtain new static features;

[0123] It should be noted that the supervision result refers to the result obtained by comparing the synthesized target view with the multiple frames of original images, which can be an error metric, such as a pixel-level loss function value, and this data reflects the difference between the synthesized view and the real image.

[0124] It can be understood that since the directly determined static features often have the problem of low accuracy, so step S01 is performed to optimize the accuracy of the static features by comparing the synthesized view with the real image, which can avoid the problem of inconsistency between the synthesized view and the real scene caused by inaccurate static features, realize the fine adjustment of the static features, thereby reducing the synthesis error and improving the realism of the view synthesis.

[0125] Exemplarily, compare the synthesized target view with the multiple frames of original images, and obtain a loss function value, that is, the supervision result, by calculating the pixel-level difference between them. Then use this supervision result as a feedback signal to optimize the parameters of the static features by backpropagation through the gradient descent algorithm. During the optimization process, adjust the weights and biases related to the static features in the neural network until the loss function value drops to a satisfactory level or reaches the preset number of iterations. Finally, obtain the optimized static feature parameters, which can more accurately describe the static attributes of the seed points.

[0126] Step S02, according to the new static features, return to execute the step of determining the feature vector of the seed point based on the dynamic residual features and the static features.

[0127] It can be understood that in order to recalculate the feature vector of the seed point after optimizing the static features to ensure that the feature vector contains the latest and optimized static feature information, so step S02 is performed, which can avoid the problem of inaccurate feature vectors caused by using unoptimized static features, realize a more accurate description of the seed points, thereby improving the quality of view synthesis and the accuracy of dynamic scene reconstruction.

[0128] Exemplarily, after obtaining the new static features, these features are combined with the previously determined dynamic residual features. Specifically, for each seed point, its new static features and the corresponding dynamic residual features are fused according to a certain weight ratio, which can be fixed or determined through another learning process. The result after fusion is the feature vector of the seed point. This process can be achieved through a vector concatenation operation, that is, the static feature vector and the dynamic residual feature vector are connected into a longer feature vector, and then this feature vector will be used in subsequent view synthesis tasks. In this way, it is ensured that the feature vector of the seed point contains the latest and optimized static feature information, thereby improving the accuracy and realism of view synthesis.

[0129] In this embodiment, through deep learning-based supervised optimization and feature update, the problem of inconsistent synthetic views and real scenes caused by inaccurate or unoptimized static features is avoided, the precision of static features and the optimization of feature vector description are realized, thereby improving the realism and continuity of view synthesis, and achieving a higher quality and more realistic dynamic scene reconstruction effect.

[0130] Exemplarily, to help understand the implementation process of the dynamic scene reconstruction method obtained by combining this embodiment with the above Embodiment 1, please refer to Figure 3 , Figure 3 A brief flow schematic diagram of a dynamic scene reconstruction method is provided. Specifically:

[0131] First, by preprocessing multi-frame images in the time domain (i.e., multi-frame original images), the corresponding SFM point cloud is obtained, and then this point cloud is downsampled to obtain various sub-points G s {μ,f s ,v}, where μ is the seed point position, f s is the static feature at the seed point, and v is the scale parameter of the seed point. Then, based on the seed point position μ and the preset query time t, the dynamic residual feature f d and the weights (w s ,w d ) corresponding to the seed point are determined. Then, in combination with the static feature f s , the feature vector f w of this seed point is calculated. Then, the feature vector is decoded through multiple MLPs, and the observation parameters (θ,φ) of the camera are used in this process to obtain the corresponding temporal Gaussian points G T . Finally, a Gaussian point cloud is constructed based on each temporal Gaussian point, and a rendering image, that is, the target view, is generated by using the splashing technique based on this Gaussian point cloud.

[0132] Furthermore, please refer to Figure 4 , Figure 4A partial modeling schematic diagram of a dynamic scene reconstruction method is provided. In this diagram, various sub - points for local space division in the dynamic scene corresponding to multiple frames of original images and the local spaces corresponding to various sub - points are shown. In addition, it is also shown that each seed generates temporal Gaussian points at different query times to capture the motion within the local space. Therefore, when the seed points cover all regions where moving objects appear, this local space motion modeling method has the potential to handle large - scale dynamic scenes.

[0133] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the dynamic scene reconstruction method of this application. Based on this technical concept, more forms of simple transformations are within the protection scope of this application.

[0134] This application also provides a dynamic scene reconstruction device. Please refer to Figure 5 , the dynamic scene reconstruction device includes:

[0135] A space division module 10, configured to generate a corresponding Structure from Motion (SFM) point cloud according to multiple frames of original images, and down - sample the SFM point cloud to obtain various sub - points in the SFM point cloud;

[0136] A feature extraction module 20, configured to calculate a feature vector of any seed point based on the feature information of the seed point, and generate a temporal Gaussian point corresponding to the seed point according to the feature vector;

[0137] A view rendering module 30, configured to construct a Gaussian point cloud based on the temporal Gaussian points obtained after traversing various sub - points, and render and generate a target view of the dynamic scene corresponding to the multiple frames of original images according to the Gaussian point cloud.

[0138] Optionally, the feature information includes the seed point position of the seed point and the static feature at the seed point, where the static feature remains synchronized at each preset query moment corresponding to the seed point, and the preset query moments correspond to each frame of the multiple frames of original images;

[0139] The feature extraction module 20 is further configured to:

[0140] Determine the dynamic residual feature of the seed point according to the seed point position;

[0141] Determine the feature vector of the seed point based on the dynamic residual feature and the static feature.

[0142] Optionally, the inverse optimization module 40 in the dynamic scene reconstruction device is configured to:

[0143] Supervise the target view based on the multi-frame original images to obtain a supervision result, and optimize the static features through backpropagation according to the supervision result to obtain new static features;

[0144] Return to execute the step of determining the feature vector of the seed point based on the dynamic residual feature and the static feature according to the new static feature.

[0145] Optionally, the feature extraction module 20 is further configured to:

[0146] Map the seed points to a hash table of the target dimension at different resolutions to obtain the hash codes of the seed points at each resolution, where the mapping sources in the mapping process include the seed point positions and a preset query time;

[0147] Perform cross-fusion on the hash codes to obtain the dynamic residual features of the seed points.

[0148] Optionally, the feature extraction module 20 is further configured to:

[0149] Input the seed point positions and the preset query time into a preset first multi-layer perceptron to predict the first weight corresponding to the dynamic residual feature and the second weight corresponding to the static feature through the first multi-layer perceptron;

[0150] Calculate the feature vector of the seed point according to the first weight, the second weight, the dynamic residual feature and the static feature.

[0151] Optionally, the feature information further includes the scale parameter of the seed point, and the feature extraction module 20 is further configured to:

[0152] Input the feature vector, the seed point position and the scale parameter into a preset second multi-layer perceptron to predict the respective temporal Gaussian parameters corresponding to the seed point through the second multi-layer perceptron;

[0153] Construct the temporal Gaussian point corresponding to the seed point based on the respective temporal Gaussian parameters.

[0154] Optionally, the second multi-layer perceptron includes a plurality of independent multi-layer perceptrons, and the feature extraction module 20 is further configured to:

[0155] Input the feature vector, the seed point position and the scale parameter into each independent multi-layer perceptron to predict the corresponding temporal Gaussian parameters through each independent multi-layer perceptron;

[0156] After traversing each independent multi-layer perceptron, obtain the respective temporal Gaussian parameters corresponding to the seed point.

[0157] The dynamic scene reconstruction device provided by this application adopts the dynamic scene reconstruction method in the above-mentioned embodiment, and can solve the technical problem of how to provide a more robust and practical solution for the field of dynamic scene reconstruction. Compared with the prior art, the beneficial effects of the dynamic scene reconstruction device provided by this application are the same as those of the dynamic scene reconstruction method provided by the above-mentioned embodiment, and other technical features in the dynamic scene reconstruction device are the same as the features disclosed in the method of the above-mentioned embodiment, and will not be elaborated here.

[0158] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the dynamic scene reconstruction method in the first embodiment above.

[0159] Refer to the following Figure 6 , which shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of this application.

[0160] As shown in Figure 6As shown, the electronic device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which may perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 into the random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the electronic device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the shown systems. Instead, more or fewer systems may be implemented or had.

[0161] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from the network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are executed.

[0162] The electronic device provided in the present application adopts the dynamic scene reconstruction method in the above-mentioned embodiments, and can solve the technical problem of how to provide a more robust and practical solution for the field of dynamic scene reconstruction. Compared with the prior art, the beneficial effects of the electronic device provided in the present application are the same as those of the dynamic scene reconstruction method provided in the above-mentioned embodiments, and other technical features in the electronic device are the same as those disclosed in the method of the previous embodiment, and will not be elaborated here.

[0163] It should be understood that each part disclosed in this application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0164] As described above, the above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application, and all should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0165] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the dynamic scene reconstruction method in the above embodiments.

[0166] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0167] The above computer-readable storage medium can be included in an electronic device; it can also exist separately without being assembled into the electronic device.

[0168] The above computer-readable storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to: generate a corresponding Structure from Motion (SFM) point cloud based on multiple frames of original images, and downsample the SFM point cloud to obtain various sub-points in the SFM point cloud; for any seed point, calculate the feature vector of the seed point based on the feature information of the seed point, and generate a temporal Gaussian point corresponding to the seed point according to the feature vector; construct a Gaussian point cloud based on the temporal Gaussian points obtained after traversing various sub-points, and render a target view of the dynamic scene corresponding to the multiple frames of original images according to the Gaussian point cloud.

[0169] Computer program code for performing the operations of this application may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, execute as a stand-alone software package, execute partially on the user's computer and partially on a remote computer, or execute entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0170] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0171] The modules involved in the embodiments of the present application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation on the unit itself in some cases.

[0172] The readable storage medium provided by the present application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned dynamic scene reconstruction method, and can solve the technical problem of how to provide a more robust and practical solution for the field of dynamic scene reconstruction. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as those of the dynamic scene reconstruction method provided by the above embodiments, and will not be elaborated here.

[0173] The present application also provides a computer program product, including a computer program, and the steps of the above-mentioned dynamic scene reconstruction method are implemented when the computer program is executed by a processor.

[0174] The computer program product provided by the present application can solve the technical problem of how to provide a more robust and practical solution for the field of dynamic scene reconstruction. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the dynamic scene reconstruction method provided by the above embodiments, and will not be elaborated here.

[0175] The above are only some embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural transformation made by using the content of the specification and drawings of the present application under the technical concept of the present application, or direct / indirect application in other related technical fields, is included in the patent protection scope of the present application.

Claims

1. A dynamic scene reconstruction method, characterized in that: The dynamic scene reconstruction method comprises: Generate a corresponding motion recovery structure SFM point cloud according to multiple frames of original images, and downsample the SFM point cloud to obtain various sub-points in the SFM point cloud; For any seed point, a feature vector of the seed point is calculated based on feature information of the seed point, and a time Gaussian point corresponding to the seed point is generated according to the feature vector; A Gaussian point cloud is constructed based on the Gaussian points at each time obtained after traversing various sub-points, and a target view of the dynamic scene corresponding to the multiple frames of original images is generated according to the Gaussian point cloud rendering.

2. The dynamic scene reconstruction method according to claim 1, characterized in that: The feature information includes a seed point position of the seed point and a static feature at the seed point, wherein the static feature is synchronized in each preset query time corresponding to the seed point, and each preset query time corresponds to each frame of the multiple frames of original images; The step of calculating the feature vector of the seed point based on the feature information of the seed point comprises: Determine the dynamic residual feature of the seed point according to the position of the seed point; A feature vector of the seed point is determined based on the dynamic residual feature and the static feature.

3. The dynamic scene reconstruction method according to claim 2, characterized in that: The dynamic scene reconstruction method also includes: Supervising the target view based on the multiple frames of original images to obtain a supervision result, and performing back-propagation optimization on the static feature according to the supervision result to obtain a new static feature; The step of determining the feature vector of the seed point based on the dynamic residual feature and the static feature is returned according to the new static feature.

4. The dynamic scene reconstruction method according to claim 2, characterized in that: The step of determining the dynamic residual feature of the seed point according to the position of the seed point comprises: Mapping the seed point to a hash table of a target dimension at different resolutions to obtain a hash code of the seed point at each resolution, wherein a mapping source in the mapping process includes a position of the seed point and a preset query time; The hash codes are cross-fused to obtain the dynamic residual features of the seed point.

5. The dynamic scene reconstruction method according to claim 2, characterized in that: The step of determining the feature vector of the seed point based on the dynamic residual feature and the static feature comprises: Inputting the seed point position and the preset query time into a preset first multilayer perceptron, so as to predict the first weight corresponding to the dynamic residual feature and the second weight corresponding to the static feature through the first multilayer perceptron; A feature vector of the seed point is calculated according to the first weight, the second weight, the dynamic residual feature, and the static feature.

6. The dynamic scene reconstruction method according to claim 1, characterized in that: The feature information also includes the scale parameter of the seed point, and the step of generating the time Gaussian point corresponding to the seed point according to the feature vector includes: Inputting the feature vector, the seed point position and the scale parameter into a preset second multilayer perceptron, so as to predict each time Gaussian parameter corresponding to the seed point through the second multilayer perceptron; A time Gaussian point corresponding to the seed point is constructed based on the time Gaussian parameters.

7. The dynamic scene reconstruction method according to claim 6, characterized in that: The second multilayer perceptron includes a plurality of independent multilayer perceptrons, and the step of inputting the feature vector, the seed point position and the scale parameter into a preset second perceptron includes: Inputting the feature vector, the seed point position and the scale parameter into each independent multilayer perceptron to predict the corresponding time Gaussian parameter through each independent multilayer perceptron; After traversing each independent multilayer perceptron, each time Gaussian parameter corresponding to the seed point is obtained.

8. A dynamic scene reconstruction device, characterized in that: The dynamic scene reconstruction device comprises: A space division module, used to generate a corresponding motion recovery structure SFM point cloud according to multiple frames of original images, and downsample the SFM point cloud to obtain various sub-points in the SFM point cloud; A feature extraction module, used for calculating a feature vector of any seed point based on feature information of the seed point, and generating a time Gaussian point corresponding to the seed point according to the feature vector; The view rendering module is used to construct a Gaussian point cloud based on the Gaussian points at each time obtained after traversing various sub-points, and to generate a target view of the dynamic scene corresponding to the multiple frames of original images according to the Gaussian point cloud rendering.

9. An electronic device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the dynamic scene reconstruction method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the dynamic scene reconstruction method according to any one of claims 1 to 7 are implemented.

Images