A dynamic scene reconstruction method, system and related devices

Abstract

The embodiment of the application discloses a dynamic scene reconstruction method and system and related equipment, and is applied to the technical field of information processing. After a dynamic scene reconstruction system obtains point cloud information of multiple frames, the point cloud information can be used to divide a to-be-reconstructed scene into multiple scale voxel grids, and voxel features of each voxel grid are obtained. Then, according to the voxel features, corresponding Gaussian models are determined, multi-dimensional Gaussian features of the Gaussian models are obtained, and the Gaussian models are rendered according to the multi-dimensional Gaussian features, so that the to-be-reconstructed scene can be reconstructed. The main point is to combine the voxel grid and the Gaussian model constructed based on the actual to-be-reconstructed scene to render the to-be-reconstructed scene, and two different feature construction methods are used, so that the expression of details in the reconstruction of the real dynamic scene is enhanced.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and in particular to a dynamic scene reconstruction method, system and related equipment. Background Technology

[0002] Scene reconstruction is generally the process of converting a real-world physical scene into a digital 3D model using computer vision, graphics, or sensor technology. Its applications are wide-ranging, including film and television production, interior design, and autonomous driving. Currently, various methods exist for scene reconstruction, such as traditional computer vision methods, deep learning methods, and sensor fusion methods. Among these, obtaining a scene that accurately reflects the real world is a critical problem that needs to be solved in the scene reconstruction process.

[0003] One scene reconstruction method based on Neural Radiation Field (NeRF) mainly involves meshing the reconstructed scene to construct mesh features, and then training a NeRF network architecture to obtain volume density and color appearance features, thereby rendering a scene consistent with the real image. However, the scene reconstruction methods obtained by this approach still have limited ability to represent the actual scene and cannot better reflect many details. Summary of the Invention

[0004] This invention provides a dynamic scene reconstruction method, system, and related equipment, which enhances the detailed representation in the reconstruction of real-world dynamic scenes.

[0005] One embodiment of the present invention provides a dynamic scene reconstruction method, including:

[0006] Obtain point cloud information from multiple frames of the scene to be reconstructed;

[0007] For point cloud information from multiple frames, the scene to be reconstructed is divided into voxel grids of multiple scales, and the voxel features of each voxel grid are obtained.

[0008] Based on the voxel features, determine the Gaussian model corresponding to each voxel grid, and obtain the multi-dimensional Gaussian features of the Gaussian model;

[0009] The Gaussian model is rendered based on the multi-dimensional Gaussian features to reconstruct the scene to be reconstructed.

[0010] Another aspect of this invention provides a dynamic scene reconstruction system, comprising:

[0011] The point cloud acquisition unit is used to acquire point cloud information of multiple frames of the scene to be reconstructed.

[0012] A voxel grid unit is used to divide the scene to be reconstructed into voxel grids of multiple scales based on point cloud information from multiple frames, and to obtain the voxel features of each voxel grid.

[0013] A Gaussian model unit is used to determine the Gaussian model corresponding to each voxel grid based on the voxel features, and to obtain the multi-dimensional Gaussian features of the Gaussian model.

[0014] The rendering unit is used to render the Gaussian model based on the multi-dimensional Gaussian features and reconstruct the scene to be reconstructed.

[0015] Another aspect of this invention provides a computer-readable storage medium storing a plurality of computer programs adapted for loading by a processor and executing the dynamic scene reconstruction method as described in one aspect of this invention.

[0016] Another embodiment of the present invention provides a terminal device, including a processor and a memory;

[0017] The memory is used to store multiple computer programs, which are loaded and executed by a processor as described in one aspect of the dynamic scene reconstruction method of the present invention; the processor is used to implement each of the multiple computer programs.

[0018] As can be seen, in the method of this embodiment, after acquiring point cloud information from multiple frames, the dynamic scene reconstruction system can divide the scene to be reconstructed into voxel grids of multiple scales based on the point cloud information, acquire the voxel features of each voxel grid, determine the corresponding Gaussian model based on the voxel features, acquire the multi-dimensional Gaussian features of the Gaussian model, and then render the Gaussian model based on the multi-dimensional Gaussian features to reconstruct the scene to be reconstructed. The main point is to combine the voxel grids and Gaussian models constructed based on the actual scene to be reconstructed to render the scene to be reconstructed, employing two different feature construction methods to enhance the detailed expression in the reconstruction of real-world dynamic scenes. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of a dynamic scene reconstruction method provided in an embodiment of the present invention;

[0021] Figure 2This is a flowchart of a dynamic scene reconstruction method provided in an embodiment of the present invention;

[0022] Figure 3a This is a schematic diagram of optimizing a Gaussian model based on a two-branch MLP model in an embodiment of the present invention;

[0023] Figure 3b This is a schematic diagram of another Gaussian model optimization based on a dual-branch MLP model in an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the sampling step size adjustment in an embodiment of the present invention;

[0025] Figure 5 This is a flowchart of a dynamic scene reconstruction method provided in a specific application embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of the logical structure of a dynamic scene reconstruction system provided in an embodiment of the present invention;

[0027] Figure 7 This is a schematic diagram of the logical structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] This invention provides a dynamic scene reconstruction method, primarily used for 3D modeling of actual dynamic scenes, for application in specific fields, such as... Figure 1As shown, the dynamic scene reconstruction system can achieve scene reconstruction through the following methods, including:

[0031] Obtain point cloud information from multiple frames of the scene to be reconstructed;

[0032] For point cloud information from multiple frames, the scene to be reconstructed is divided into voxel grids of multiple scales, and the voxel features of each voxel grid are obtained.

[0033] Based on the voxel features, determine the Gaussian model corresponding to each voxel grid, and obtain the multi-dimensional Gaussian features of the Gaussian model;

[0034] The Gaussian model is rendered based on the multi-dimensional Gaussian features to reconstruct the scene to be reconstructed.

[0035] Thus, in this embodiment, the voxel mesh and Gaussian model constructed based on the actual scene to be reconstructed are combined to render the scene to be reconstructed. Two different feature construction methods are used to enhance the detailed expression in the reconstruction of real dynamic scenes.

[0036] One embodiment of the present invention provides a dynamic scene reconstruction method, the flowchart of which is shown below. Figure 2 As shown, it includes:

[0037] Step 101: Obtain point cloud information of multiple frames of the scene to be reconstructed.

[0038] It is understandable that a dynamic scene reconstruction system can periodically scan a scene to be reconstructed multiple times through sensors at certain time intervals to obtain point cloud information at each moment. The point cloud information acquired at each moment is a frame of point cloud information, and there is a certain temporal sequence between different frames of point cloud information.

[0039] In this embodiment, the multi-frame point cloud information is a temporal point cloud, which can include not only three-dimensional spatial information but also incorporate the time dimension. It can dynamically reflect the evolution of the spatial structure of the scene or target over time, and emphasizes "temporal continuity" and "dynamic change". It is a key data form for understanding the dynamic three-dimensional world.

[0040] Furthermore, in this embodiment, after scanning the point cloud information of each frame, the point cloud density can also be statistically analyzed to describe the degree of aggregation of each point cloud, such as 50 points / m².

[0041] Step 102: For the point cloud information of multiple frames, the scene to be reconstructed is divided into voxel grids of multiple scales, and the voxel features of each voxel grid are obtained.

[0042] Specifically, based on the point cloud density of different regions of the scene to be reconstructed, voxel grids of corresponding scales can be set for different regions of the scene to be reconstructed, resulting in original voxel grids of multiple scales, and the voxel features of the original voxel grids can be obtained.

[0043] In this context, a voxel is the smallest discrete unit in three-dimensional space, analogous to a pixel in a two-dimensional image. A voxel has a fixed spatial size (e.g., 1m × 1m × 1m) and each voxel can store attribute information within that space (e.g., color, density, presence of objects, etc.), used to "mesh" a continuous three-dimensional space. In this embodiment, if a region of the scene to be reconstructed is a high-density area, such as a point cloud density > 50 points / m², a 12.5cm voxel grid can be used; if a region is a medium-density area, such as a point cloud density between 10-50 points / m², a 25cm voxel grid can be used; and if a region is a low-density area, such as a point cloud density < 10 points / m², a 50cm voxel grid can be used. This divides the scene to be reconstructed into multiple scales of voxel grids, i.e., multiple sizes of voxel grids, each reflecting different resolutions. Generally, areas with higher point cloud density use smaller voxel grids to highlight specific details, while areas with lower point cloud density use larger voxel grids.

[0044] The voxel features of each original voxel mesh can include: point cloud color, density, dynamic displacement information, static voxel normal vector information, etc. In this embodiment, dynamic and static attributes are set for each voxel mesh to distinguish whether the same voxel mesh corresponding to multiple frames of point cloud information is in a dynamic or static state.

[0045] The dynamic displacement information refers to the distance the voxel mesh moves between consecutive frames (e.g., 3 frames). Specifically, algorithms such as Iterative Closest Point (ICP) combined with ORB feature matching can be used to register the point clouds of 3 consecutive frames to a unified coordinate system and calculate the displacement Δs of the original voxel mesh. This displacement is the dynamic displacement information. If the displacement is greater than a threshold, such as 0.5m, the dynamic and static attributes of the original voxel mesh can be set to "dynamic", otherwise the dynamic and static attributes of the original voxel mesh can be set to "static".

[0046] The static voxel normal vector information reflects the geometric orientation of the local surface represented by each original voxel mesh and can represent the curvature of the original voxel mesh. Specifically:

[0047] The nearest neighbor point cloud of an original voxel grid K=20 can be extracted. The plane normal vector can be fitted by a combination algorithm of progressive sampling (PROSAC) and maximum likelihood estimation (MLESAC). The PROSAC algorithm selects the initial sample according to probability, giving priority to the sample that is more likely to be a point inside the original voxel grid. The MLESAC algorithm optimizes the fitting result based on maximum likelihood estimation, sets the inlier distance threshold to 0.05m, and quickly and accurately obtains the static voxel geometric features.

[0048] It should be noted that after obtaining the original voxel mesh, step 103 can be performed directly on the original voxel mesh. Alternatively, preferably, the original voxel mesh can be optimized first, and then step 103 can be performed on the optimized voxel mesh. Specifically, when optimizing the original voxel mesh:

[0049] First, based on the voxel characteristics of the original voxel mesh, multiple original voxel meshes can be constructed into a voxel multi-layer structure tree. Then, the original voxel meshes can be adjusted based on the voxel multi-layer structure tree to obtain the adjusted voxel meshes. For the adjusted voxel meshes, the following step 103 can be performed.

[0050] In this embodiment, an octree can be used to construct a multi-level voxel structure tree from the original voxel mesh, specifically a voxel octree. An octree is a tree-like data structure that recursively divides the three-dimensional space into eight equal cubes. Its root node is the entire three-dimensional space, and each non-leaf node contains eight child nodes, corresponding to eight equally divided regions (similar to the eight small cubes of a "Rubik's Cube") formed by dividing the three-dimensional space into eight parts along the x, y, and z axes.

[0051] The constructed voxel octree is a 3D spatial data structure that combines the advantages of voxels and octrees, used for the efficient representation, storage, and processing of discretized information in 3D space. It adaptively subdivides 3D space through a hierarchical recursive partitioning method of the octree, using voxel meshes as the basic unit of spatial discretization. These voxel meshes are organized using the hierarchical structure of the octree, with higher-level nodes representing large-scale voxel meshes and lower-level nodes (child nodes) representing smaller-scale voxel meshes after subdivision, thus forming a "coarse-to-fine" 3D spatial representation. It is a core tool for processing large-scale 3D data (such as point clouds, 3D models, and dynamic scenes).

[0052] In this embodiment, when specifically constructing a voxel multi-layer structure tree, such as a voxel octree:

[0053] The root node of the multi-level voxel structure tree is the smallest bounding box of multiple original voxel meshes, covering the entire scene to be reconstructed.

[0054] Based on the size of the original voxel mesh, each original voxel mesh is mapped to a different level corresponding to the multi-layer structure tree to form an initial level. Each original voxel mesh corresponds to an initial level node, and the resulting pyramid structure includes a root node and an initial level node.

[0055] The characteristics of nodes at each level in a multi-level structure tree of storage voxels.

[0056] For example, when constructing a voxel multi-layer structure tree, such as a voxel octree, the root node covers the entire scene to be reconstructed, with level l=0. Based on the size of the original voxel mesh, initial level nodes are formed. Specifically, larger original voxel meshes (e.g., 1.0m) correspond to the first level node of the voxel octree (l=1), medium-sized original voxel meshes (e.g., 0.5m) correspond to the second level node (l=2), and smaller original voxel meshes (e.g., 0.25m) correspond to the third level node (l=3). The size d of each level node... l =1.0×2 -l m (l represents the level, 0→7 corresponds to 1m→0.0078m), thus forming a pyramid structure, including the root node, initial level nodes, etc.

[0057] Furthermore, the voxel octree can be dynamically adjusted subsequently, such as splitting or merging nodes. When splitting the initial level node, child nodes can be formed. The pyramid structure includes the root node, the initial level node, and child nodes.

[0058] In this embodiment, when storing the features of nodes in each level:

[0059] For nodes of a voxel mesh with corresponding static properties, the normal vector, point density, and color mean of the corresponding voxel mesh can be stored.

[0060] For nodes of a voxel mesh with corresponding dynamic attributes, features such as normal vector, density, color, displacement, and motion confidence can be stored. The motion confidence of a voxel mesh with dynamic attributes can be mainly calculated from the displacement variance of the voxel mesh over three consecutive frames.

[0061]

[0062] Preferably, in a specific embodiment, when adjusting the original voxel mesh based on the voxel multilayer structure tree:

[0063] Based on the density variation and feature similarity of the original voxel mesh, some initial level nodes in the voxel multi-level structure tree are split to generate lower-level child nodes, so that the pyramid structure also includes child nodes, or some initial level nodes in the voxel multi-level structure tree are merged to generate higher-level nodes, so as to split or merge the original voxel mesh related to some initial level nodes.

[0064] The node features between the voxel multi-layer structure trees obtained from the point cloud information of adjacent frames are smoothed.

[0065] In one specific embodiment, when adjusting the voxel multilayer structure tree:

[0066] To accurately capture details, one to two detail levels can be added to the branch containing the first original voxel mesh containing dynamic attributes. For the dynamic target object in the scene to be reconstructed corresponding to the first original voxel mesh, the motion speed and direction of the dynamic target object are determined based on the point cloud information of multiple consecutive frames. Based on the motion speed and direction, the subsequent motion region of the dynamic target object is predicted. Thus, when dividing the voxel mesh based on the point cloud information of subsequent frames, the corresponding motion region is divided into fine-grained voxel meshes.

[0067] Furthermore, when the curvature of the original voxel mesh with static attributes is >0.05, or the motion confidence of the original voxel mesh with dynamic attributes is greater than 0.9, the branch level of these original voxel meshes can be set to l=7 to ensure accurate capture of the details of the target object in the scene to be reconstructed.

[0068] In another specific embodiment, the original voxel mesh is adjusted based on a voxel multi-layer structure tree (such as a voxel octree) to adapt to the density changes and feature similarity of the relevant original voxel mesh. Specifically:

[0069] If the point cloud density in a node of a voxel octree exceeds a threshold, such as 50%, the node is split, that is, a node is split into eight child nodes, so that the voxel mesh corresponding to such a node is split into eight voxel meshes (corresponding to the eight nodes split).

[0070] If the feature similarity between adjacent nodes in a voxel octree is higher than 90%, the adjacent nodes are merged to obtain a merged node. In this way, the voxel grids corresponding to the adjacent nodes are merged into a single voxel grid (corresponding to the merged node).

[0071] When the feature differences between adjacent nodes are too large and the point cloud density within a node is less than a threshold, the original voxel grid remains unchanged.

[0072] Furthermore, for nodes corresponding to voxel meshes with static attributes in the voxel octree, the normal vectors can be recalculated every 10 frames. Generally, an incremental calculation method can be used to quickly update the normal vectors of the nodes in the current frame using the results of the previous frame and the local changes in the current frame. For nodes corresponding to voxel meshes with dynamic attributes in the voxel octree, the displacement and confidence need to be updated in real time in each frame to reflect the motion state of the corresponding target object in a timely manner.

[0073] Furthermore, the node features between the voxel multi-layer structure trees obtained from the point cloud information of adjacent frames can be smoothed. Specifically, the weighted average method can be used to fuse the node feature vectors of the current frame and the corresponding node feature vectors of the previous frame: New feature = confidence score × current frame node feature + (1 - confidence score) × previous frame node feature.

[0074] In the dynamic attribute voxel grid, the confidence of the target object is high, so the proportion of the node features in the current frame is large. For example, when a vehicle brakes suddenly, the latest displacement data is used first. In the static attribute voxel grid, the confidence of the target object is low, so historical node features can be retained, such as roadside buildings, which can reduce inter-frame jitter.

[0075] It should be noted that, in the process of first optimizing the original voxel mesh, before constructing the voxel multi-layer structure tree, it is preferable to preprocess each original voxel mesh, and then construct the voxel multi-layer structure tree based on the preprocessed original voxel mesh. Preprocessing can include, but is not limited to, the following methods:

[0076] To further enhance the detail of the reconstructed scene, upsampling can be used to further subdivide the original voxel mesh, such as by constructing... The voxel grid is progressively upsampled to [value] using trilinear interpolation. The size of the original voxel mesh was reduced from 8cm to 2cm.

[0077] To minimize the amount of data processed later, the original voxel grid in invalid regions can be filtered first. Specifically, the voxel occupancy rate Gσ(X) of the original voxel grid can be calculated as the number of point clouds in the voxel grid / voxel volume. When Gσ(X) > 0.5, the original voxel grid is considered a "valid region" voxel grid; when Gσ(X) is less than or equal to 0.5, the original voxel grid is considered an "invalid region" voxel grid. Filtering is then performed, and only the voxel grid in the valid region is processed later, thus improving computational efficiency.

[0078] Step 103: Based on the voxel features, determine the Gaussian model corresponding to each voxel grid and obtain the multi-dimensional Gaussian features of the Gaussian model.

[0079] It is understandable that in the field of scene reconstruction and rendering, 3D Gaussian modeling, as a typical explicit representation method, constructs a set of Gaussian distributions in 3D space (each Gaussian distribution contains parameters such as position, shape, color, and weight), and projects them onto the image plane for weighted fusion based on the "Gaussian Splatting" rendering method, thereby achieving real-time rendering of high-precision scenes.

[0080] In this embodiment, the formation of the Gaussian model in this step can be achieved through the following steps:

[0081] The original Gaussian model corresponding to each voxel grid can be determined by the voxel features obtained in step 102 above, and the multi-dimensional Gaussian features of the original Gaussian model can be obtained.

[0082] The spatial location and color features in multi-dimensional Gaussian features are compressed to form a compressed Gaussian model;

[0083] Preferably, a two-branch machine learning model can be used to obtain the first optimized value of the spatial location feature and the second optimized value of the color feature of the compressed Gaussian model from the two branches respectively. The spatial location feature and the color feature are optimized according to the first optimized value and the second optimized value respectively to obtain the optimized Gaussian model.

[0084] Specifically, when determining the original Gaussian model, generally at least one Gaussian model is generated within a voxel mesh. This process primarily involves determining the number and radius of the Gaussian models within the voxel mesh.

[0085] The number of Gaussian models corresponding to a voxel grid is determined based on the voxel occupancy rate of the voxel grid. Specifically, the scene to be reconstructed can be horizontally or vertically cut, the point cloud can be projected onto a plane, and the voxel occupancy rate Gσ(X) of each voxel grid can be estimated. When Gσ(X)>0.5, the Gaussian density within the voxel grid is increased, for example, 20 Gaussian models are generated per cubic meter; when Gσ(X)≤0.5, the number of Gaussian models within the voxel grid is reduced, or adjacent Gaussian models are merged (for example, 5~10 per cubic meter) to avoid redundant calculations.

[0086] Preferably, the Gaussian radius can be dynamically adjusted by combining the voxel features (such as curvature and height difference) of the voxel mesh and the semantic information of the point cloud within the voxel mesh. Specifically, the actual category of the target object within the voxel mesh can be obtained through a pre-trained semantic segmentation model. For example, for areas containing high-detail objects such as building corners and vegetation edges, as well as high-curvature regions of the voxel mesh (such as curvature > 0.05), a small-radius Gaussian model (such as 0.1~0.3m) is used to preserve details; for smooth areas such as roads and walls, a large-radius Gaussian model (such as 0.5~1m) is used to reduce computation.

[0087] Once the original Gaussian model is determined, its multi-dimensional Gaussian features can be obtained, including geometric features, radiation features, and spatial features.

[0088] Geometric features can be obtained by fitting the normal vector and curvature of the neighborhood point cloud, and the local surface change rate can be obtained by calculating the changes in normal vector and curvature between adjacent points; radiation features include color variance (characterizing texture richness) and density gradient (detail saliency) based on texture orientation features extracted from Local Binary Patterns (LBP); spatial features are sine and cosine position codes. Enhance location awareness.

[0089] In the embodiment, when compressing spatial location features and color features in multi-dimensional Gaussian features:

[0090] For spatial location feature compression: Different compression methods are used for voxel meshes with different curvatures. For example, when the curvature of the voxel mesh is low (e.g., <0.03), the third-order tensor T formed by the Gaussian position coordinates can be decomposed into 32 sets of basis vectors through CP: When the curvature of the voxel mesh is high (K≥0.03), the Gaussian position coordinate tensor is decomposed using Tucker decomposition, i.e., T=G×1Ux×2Uy×3Uz, where G is the core tensor that stores the interaction information between each dimension, and Ux, Uy, and Uz are the factor matrices of the corresponding dimensions. The decomposition rank is set to 48 to capture complex geometric details.

[0091] For color feature compression: Specifically, the YCbCr color space can be used, with 4 layers of MLP fine encoding for the Y channel (luminance) and 2 layers of MLP coarse encoding for the CbCr channel (chrominance). Combined with the visual attention mechanism: Ccompress=MLPY(Y)⊕MLPCbCr(CbCr)×Att(Y), the chrominance compression intensity is dynamically adjusted, that is, fine encoding is used for the sub-features of the luminance channel, while coarse encoding is used for the sub-features of the chrominance channel.

[0092] In this embodiment, when optimizing the compressed Gaussian model, a two-branch machine learning model, such as a two-branch multilayer perceptron (MLP) model, can be mainly used to obtain the first optimized value of spatial location features and the second optimized value of color features. Furthermore, the two branches of the two-branch machine learning model can not only obtain the first and second optimized values, but also obtain the third optimized value of the shape features of the compressed Gaussian model and the fourth optimized value of the weight information, respectively. For example... Figure 3a As shown:

[0093] Branch 1 of the MLP model is mainly used to optimize the spatial location features of the compressed Gaussian model, such as position and shape. A hierarchical attention mechanism is employed, with each layer of the 8-layer fully connected network (128 dimensions per layer) introducing a hierarchical attention module. This module focuses on the voxel features of important voxel grids (such as high curvature regions and dynamic object boundaries). The 32-dimensional voxel features of the voxel grid are input into Branch 1, and the output is the Gaussian position offset of the compressed Gaussian model corresponding to that voxel grid. and scaling factor The first and third optimization values ​​mentioned above are used to adjust the position and shape of the compressed Gaussian model.

[0094] Branch 2 of the MLP model is mainly used to optimize the color features of the compressed Gaussian model, such as color and weights: Based on the above two fully connected layers, a conditional generative adversarial network (cGAN) and activation function are combined. Specifically, the voxel features of the voxel grid and the semantic information of the scene to be reconstructed within the voxel grid can be used as conditions and input into branch 2. The output can be RGB color correction values ​​and weight coefficients, namely the second and fourth optimization values ​​mentioned above. The activation function in branch 2 can be Swish.

[0095] In this way, the first, second, third, and fourth optimized values ​​can be used to form a joint loss function to optimize the Gaussian model formed above, thus forming an optimized Gaussian model.

[0096] Furthermore, in a specific embodiment, when optimizing the Gaussian model, the following will also be considered:

[0097] For the original voxel mesh with dynamic attributes, the displacement information of the next frame is predicted based on the displacement information of the previous multiple frames; when optimizing the Gaussian model based on the displacement information of the next frame and the first, second, third, and fourth optimized values ​​obtained from the two branches respectively, the following steps are performed: Figure 3b As shown, the Gaussian model can be optimized by forming a joint loss function based on the displacement information of the next frame and the first, second, third, and fourth optimized values ​​obtained from the two branches respectively.

[0098] Step 104: Render a Gaussian model based on multi-dimensional Gaussian features to reconstruct the scene to be reconstructed.

[0099] In a specific embodiment, rendering a Gaussian model can be achieved through the following steps:

[0100] Based on the multi-scale Gaussian feature pyramid and the sparse Gaussian update mechanism, a hierarchical data structure with corresponding levels is constructed, wherein the levels of the multi-scale Gaussian feature pyramid correspond to the levels of the voxel multi-layer structure tree formed by the voxel grid.

[0101] Based on the viewing distance of the scene area to be reconstructed corresponding to the Gaussian model, adjust the hierarchical data structure, such as the level of detail (LOD) hierarchical data structure, and use different levels of LOD hierarchical data structure for different viewing distances;

[0102] The Gaussian model is pruned based on its global importance score, and the hierarchy of the hierarchical data structure is adaptively adjusted to form the adjusted hierarchical data structure.

[0103] The sampling step size for ray tracing is determined based on multi-dimensional Gaussian features. The adjusted hierarchical data structure is then rendered based on the determined sampling step size to form the scene to be reconstructed.

[0104] In this embodiment, when constructing the hierarchical data structure:

[0105] Specifically, when rendering Gaussian models, different methods can be used based on Gaussian models of different resolutions. For example, for coarse-scale Gaussian models (such as 1m resolution), the main focus is on fast global rendering of the scene to be reconstructed. Since the levels of the multi-scale Gaussian feature pyramid correspond to the levels of the voxel multi-layer structure tree formed above, the fast indexing structure of the voxel multi-layer structure tree (such as voxel octree) can generally be used to accelerate the intersection detection of rays and Gaussian models. For fine-scale Gaussian models (such as 0.1m resolution), the main focus is on enhancing local details. Generally, sparsification can be performed using sparse representation learning algorithms to reduce the amount of data.

[0106] When rendering a Gaussian model, the multi-dimensional Gaussian features of Gaussian models at different scales can be fused through trilinear interpolation with adaptive weights. The weights can be dynamically adjusted based on the importance and similarity of Gaussian models at adjacent scales.

[0107] In this embodiment, when constructing a hierarchical data structure based on visual distance:

[0108] The visual distance *d* between different areas of the scene to be reconstructed refers to the straight-line distance from the rendering viewpoint to each area within the rendered scene. When rendering a city street scene, the visual distance is smaller when the virtual camera is closer to nearby pedestrians and vehicles, and larger when it is farther from distant buildings and the sky. This visual distance reflects whether the viewpoint is near or far. Specifically, the relationship between the visual distance *d* and the Level of Distance (LOD) of the hierarchical data structure can be represented by the following function. Thus, a voxel-Gaussian model with a given visual distance can form a hierarchical data structure with corresponding levels:

[0109]

[0110] It can be seen that the scene areas to be reconstructed at a greater visual distance have smaller hierarchical data structures and can be rendered with a lower resolution for blurring, while the scene areas to be reconstructed at a closer visual distance have larger hierarchical data structures and can be rendered with a higher resolution for precision.

[0111] In addition, if the voxel occupancy rate of the voxel mesh with dynamic attributes in the scene to be reconstructed exceeds a certain value, such as 30%, or if the user focuses on key areas, the layer of the hierarchical data structure is increased, for example, by 1, and a Gaussian set of the corresponding scale is selected from the Gaussian pyramid to ensure the rendering accuracy of key areas and dynamic areas.

[0112] Furthermore, in this embodiment, when dynamically adjusting the hierarchical data structure constructed above:

[0113] The global importance score (GSj) of the Gaussian model can be used to adjust the Gaussian model, thereby adaptively adjusting the hierarchy of the corresponding hierarchical data structure. This achieves dynamic adjustment of the constructed hierarchical data structure, such as pruning the last 20% of Gaussian models based on their global importance score and distilling the remaining Gaussian models from order 3 to order 2, reducing rendering computation. The global importance score is a function related to the motion confidence of the voxel mesh, as shown in the following formula:

[0114]

[0115] in, The kernel function measures the correlation between the location of the Gaussian model and the features of the scene to be reconstructed. Gaussian density, Motion confidence score for voxel meshes (the higher the voxel occupancy rate of a dynamic attribute voxel mesh, the higher the score).

[0116] Furthermore, the sampling step size during ray tracing can be adjusted in real time in the embodiment, specifically including:

[0117] Ray tracing technology can be used for high-level detail rendering to provide more realistic visual effects. In this embodiment, after adaptively adjusting the hierarchical data structure based on the Gaussian model, ray tracing can be used for the hierarchical data structures that require high detail, employing high-precision ray tracing to capture the detailed features of the Gaussian model through dense sampling; for hierarchical data structures with low detail requirements, the ray tracing process is simplified.

[0118] In this embodiment, the sampling step size can be dynamically adjusted in real time during ray tracing. This adjustment can be based on multi-dimensional Gaussian features, specifically, as shown below. Figure 4As shown, the adjusted sampling step size is a function related to the texture complexity of the target object corresponding to the Gaussian model, the geometric curvature of the Gaussian model, the corresponding motion confidence, and the Gaussian coverage. Wherein:

[0119] Δt represents the sampling step size in the ray tracing process, which is used to control the sampling interval in the scene during ray tracing. An appropriate sampling step size helps to improve rendering efficiency while ensuring rendering accuracy.

[0120] This represents the texture complexity of the target object corresponding to the Gaussian model. In the formula, this term expresses the influence of texture complexity on the sampling step size. The more complex the texture, the smaller the sampling step size.

[0121] For geometric curvature, this term in the formula expresses the effect of geometric curvature on the sampling step size. The larger the curvature, the smaller the formula and the smaller the step size, with a limit of 0.3. The smaller the curvature, the larger the formula, with a limit of 1.

[0122] This represents the motion confidence level, which reflects the reliability of the voxel grid's motion state. In the formula, this term expresses the influence of the motion confidence level on the sampling step size; the higher the motion confidence level, the smaller the sampling step size.

[0123] This represents the radius of the Gaussian sphere, which measures the coverage of the Gaussian model in space. In the formula, this term expresses the influence of the Gaussian sphere radius on the sampling step size. The larger the Gaussian sphere radius, the larger the sampling step size, avoiding redundancy; the smaller the radius, the smaller the step size.

[0124] The overall formula mainly unifies the object material, geometry, motion confidence, and representation granularity into a continuous control function, forming an adaptive control method for the light sampling step size of dynamic scenes. The coefficients in it can be adjusted according to the actual situation.

[0125] Furthermore, the sampling points obtained during ray sampling in the ray tracing process can be combined with the radiation characteristics of the Gaussian model and the RGB color correction value δ obtained when optimizing the Gaussian model to eliminate the high-frequency detail loss of the corresponding voxel mesh and improve the rendering accuracy: Cfinal = Cvoxel + δ•αv, where Cvoxel is the original color of the voxel mesh.

[0126] Furthermore, during the rendering process, displacement prediction is performed in real time. This predicts the displacement Δs^t+1 of the target object corresponding to each dynamic attribute's voxel mesh in the next frame and offsets the target object in advance, eliminating motion blur. Specifically, displacement prediction can be based on processing temporal data using the LSTM module. For example, the displacement of the next frame can be predicted based on the displacements [Δst,…,Δst-4] of the previous few frames (e.g., the first 5 frames).

[0127] As can be seen, in the method of this embodiment, after acquiring point cloud information of any frame, the dynamic scene reconstruction system can divide the scene to be reconstructed into voxel grids of multiple scales based on the point cloud information, acquire the voxel features of each voxel grid, determine the corresponding Gaussian model based on the voxel features, acquire the multi-dimensional Gaussian features of the Gaussian model, and then render the Gaussian model based on the multi-dimensional Gaussian features to reconstruct the scene to be reconstructed. The main point is to combine the voxel grids and Gaussian models constructed based on the actual scene to be reconstructed to render the scene to be reconstructed, using two different feature construction methods to enhance the detailed expression in the reconstruction of real-world dynamic scenes.

[0128] It should be noted that after steps 101 to 104 above, a single frame of the scene to be reconstructed can be formed. After multiple frames of the scene to be reconstructed are formed, when a new frame of point cloud information is subsequently obtained, it is not necessary to process all the point cloud information to update the subsequently reconstructed scene. Instead, the following steps are performed to update the reconstructed scene:

[0129] For the voxel grid of dynamic attributes in the voxel grid divided from the point cloud information of subsequent frames, and combined with the corresponding voxel grids of the previous multiple frames of subsequent frames, the relative displacement of the voxel grid of dynamic attributes between different frames is calculated.

[0130] Based on the relative displacement and the corresponding voxel grids of adjacent frames, determine the feature information of the voxel grids for dynamic attributes;

[0131] The above two-branch machine learning model is fine-tuned based on the feature information of the voxel grid with dynamic attributes.

[0132] (1) Dynamically update the coordinates and features of the voxel mesh to ensure the continuity of the target object's motion trajectory in the dynamically attributed voxel mesh. Specifically:

[0133] For example, a sliding window can be used to maintain the data of the most recent 20 frames. The first 19 frames form a reconstruction. For the point cloud data acquired in the 20th frame, after dividing it into voxel grids, incremental ICP registration can be started on the voxel grids with potential dynamic attributes. The features of the voxel grid in the current frame are registered with the features of the voxel grid in the previous 19 frames, aligned to a unified coordinate system, and the relative displacement of the voxel grid is calculated.

[0134] By combining the relative displacement (vold) of the voxel grid with potential dynamic attributes and the information of the voxel grid in adjacent frames with a machine learning model (such as an MLP network), the voxel coordinates can be accurately updated: vnew = vold + MLP(vold, adjacent frame voxels), and the accurate coordinates and feature vectors of the voxel grid with dynamic attributes are output.

[0135] (2) Fine-tune the Gaussian model determined for subsequent frames.

[0136] After obtaining the feature information of the voxel grid with dynamic attributes based on multi-frame point cloud information, during the training process of fine-tuning the dual-branch machine learning model, the parameter update threshold can be set to fine-tune only the parameters of the dynamic feature layer with large changes in the dual-branch machine learning model, and freeze the static feature layer.

[0137] Furthermore, the fine-tuning training of the two-branch machine learning model can employ asynchronous incremental training. When a new frame of data arrives, fine-tuning training immediately begins without affecting the real-time optimization of the Gaussian model by the two-branch machine learning model. This real-time dynamic adjustment of the two-branch machine learning model allows for more accurate optimization of the aforementioned Gaussian model.

[0138] Once a dual-branch machine learning model, such as an MLP model, is trained, the position and size of the Gaussian model can be optimized using the first optimization values, such as Δμ and ΔS mentioned above, to adapt to the deformation of dynamic target objects in the dynamically reconstructed scene. A second optimization value can also be output to optimize the color features of the Gaussian model. Furthermore, the color features of the Gaussian model corresponding to the dynamic target object can be updated through interpolation between adjacent frames to reduce flickering.

[0139] The following uses a specific application example to illustrate the dynamic scene reconstruction method of this invention, as follows: Figure 5 As shown, it includes:

[0140] Step 201: Obtain point cloud information of multiple frames (e.g., 2 or 3 frames) of the scene to be reconstructed, including point cloud density.

[0141] Step 202: Based on the point cloud density of different regions of the scene to be reconstructed, set voxel meshes of corresponding scales for different regions of the scene to be reconstructed to obtain original voxel meshes of multiple scales, and obtain the voxel features of the original voxel meshes. In this embodiment, these features may specifically include: point cloud color, density, dynamic displacement information, static voxel normal vector information, and dynamic and static attributes of each voxel mesh.

[0142] Step 203: Preprocess the original voxel mesh to obtain a preprocessed original voxel mesh. Then, perform the following step 204 on the preprocessed original voxel mesh, specifically including:

[0143] For example, on the one hand, upsampling can be used to further subdivide the original voxel grid. On the other hand, in order to minimize the amount of data to be processed later, invalid regions of the original voxel grid can be filtered out first. For example, if the voxel occupancy rate Gσ(X) is less than or equal to 0.5, the original voxel grid is considered an "invalid region" and is filtered out. Only the voxel grids in the valid regions are processed later, which improves computational efficiency.

[0144] Step 204: Optimize the original voxel mesh. For the optimized voxel mesh, proceed to step 205. Specifically, when optimizing the original voxel mesh:

[0145] A1. Based on the voxel characteristics of the original voxel mesh, construct a multi-layer voxel structure tree from multiple original voxel meshes, specifically:

[0146] The root node of the voxel multi-layer structure tree is the smallest bounding box of multiple original voxel meshes, covering the entire scene to be reconstructed. According to the size of the original voxel meshes, each original voxel mesh is mapped to a different level corresponding to the multi-layer structure tree to form an initial level. Each original voxel mesh corresponds to an initial level node. The resulting pyramid structure includes a root node and initial level nodes, etc. The features of the nodes in each level of the voxel multi-layer structure tree are stored.

[0147] Specifically, for nodes of voxel meshes with corresponding static attributes, the normal vector, point density, and mean color of the corresponding voxel mesh can be stored; for nodes of voxel meshes with corresponding dynamic attributes, features such as normal vector, density, color, displacement, and motion confidence can be stored.

[0148] A2. Adjust the original voxel mesh based on the voxel multi-layer structure tree to obtain the adjusted voxel mesh. Specifically, for the adjusted voxel mesh:

[0149] Based on the density variation and feature similarity of the original voxel mesh, some initial-level nodes in the voxel multi-layer structure tree are split to generate lower-level child nodes, thus the pyramid structure also includes child nodes; or some initial-level nodes in the voxel multi-layer structure tree are merged to generate higher-level nodes. For example:

[0150] If the point cloud density within a node in a voxel octree exceeds a threshold, such as 50%, the node is split, meaning it is split into eight child nodes. Thus, the voxel mesh corresponding to one node is split into eight voxel meshes (each corresponding to one of the eight child nodes). If the feature similarity between adjacent nodes in the voxel octree is higher than 90%, the adjacent nodes are merged to obtain a merged node. In this way, the voxel meshes corresponding to the adjacent nodes are merged into one voxel mesh (corresponding to the merged node).

[0151] Furthermore, the node features of each node in the voxel multi-layer structure tree will be adjusted. For example, the node features between the voxel multi-layer structure trees obtained from the point cloud information of adjacent frames can be smoothed. Specifically, the weighted average method can be used to fuse the node feature vectors of the current frame and the corresponding node feature vectors of the previous frame: New feature = confidence × current frame node feature + (1 - confidence) × previous frame node feature.

[0152] Step 205: Based on the voxel features, determine the Gaussian model corresponding to each voxel grid and obtain the multi-dimensional Gaussian features of the Gaussian model. This can be achieved through the following steps:

[0153] B1. Using the voxel features of the optimized voxel meshes from the above steps, determine the original Gaussian model corresponding to each voxel mesh, and obtain the multi-dimensional Gaussian features of the original Gaussian model. These multi-dimensional Gaussian features may include geometric features, radiation features, and spatial features, etc.

[0154] B2. Compress the spatial location features and color features in the multi-dimensional Gaussian features to form a compressed Gaussian model.

[0155] B3. Using a two-branch machine learning model, the two branches obtain the first optimized value of the spatial location feature and the second optimized value of the color feature of the compressed Gaussian model, respectively. Based on the first and second optimized values, the spatial location feature and the color feature are optimized to obtain the optimized Gaussian model.

[0156] Furthermore, the two branches of the dual-branch machine learning model can not only obtain the first and second optimized values, but also the third optimized value of the shape features of the compressed Gaussian model and the fourth optimized value of the weight information. Thus, for the original voxel mesh with dynamic attributes, the displacement information of the next frame is predicted based on the displacement information of the previous multiple frames. Based on the displacement information of the next frame and the first, second, third, and fourth optimized values ​​obtained by the two branches respectively, a joint loss function is formed to optimize the Gaussian model.

[0157] Step 206: Render a Gaussian model based on multi-dimensional Gaussian features to reconstruct the scene to be reconstructed. Specifically, this may include:

[0158] C1. Based on the multi-scale Gaussian feature pyramid and the sparse Gaussian update mechanism, construct a hierarchical data structure with corresponding levels, wherein the levels of the multi-scale Gaussian feature pyramid correspond to the levels of the voxel multi-layer structure tree formed by the voxel grid.

[0159] C2. Based on the viewing distance of the scene region to be reconstructed corresponding to the Gaussian model, adjust the hierarchy of the hierarchical data structure. Different levels of LOD hierarchical data structure are used for different viewing distances. The Gaussian model is pruned according to the global importance score of the Gaussian model, and the hierarchy of the hierarchical data structure is adjusted adaptively to form the adjusted hierarchical data structure.

[0160] C3. Determine the sampling step size for ray tracing based on multi-dimensional Gaussian features, and render the adjusted hierarchical data structure based on the determined sampling step size to form a frame of the scene to be reconstructed.

[0161] It should be noted that through the above steps 201 to 206, a frame of the scene to be reconstructed can be reconstructed. By repeating steps 201 to 206, multiple consecutive frames of the scene to be reconstructed can be formed. Furthermore, after obtaining the point cloud information of a new frame, step 207 can be executed.

[0162] Step 207: For the voxel grids of dynamic attributes in the voxel grids divided from the point cloud information of subsequent frames, and in combination with the corresponding voxel grids of the preceding frames, calculate the relative displacement of the voxel grids of dynamic attributes between different frames; determine the feature information of the voxel grids of dynamic attributes based on the relative displacement and the corresponding voxel grids of adjacent frames; and fine-tune the above dual-branch machine learning model based on the feature information of the voxel grids of dynamic attributes.

[0163] It can be seen that the above dynamic scene reconstruction method can achieve the following technical effects:

[0164] 1. Divide the scene to be reconstructed into regions based on point cloud density, generate voxel meshes of adaptive size, construct multi-scale voxel multi-layer structure trees, such as voxel octrees, and distinguish between dynamic and static voxel meshes and store motion features, such as dynamic displacement information.

[0165] 2. Voxel-Gaussian mapping: The dynamically reconstructed scene is constructed as a multi-scale voxel radiation field. Each voxel grid contains radiation properties and geometric features. The Gaussian model is dynamically adjusted by combining the multi-scale features of the voxel grid to fill the gap of missing geometric features in Gaussian modeling.

[0166] 3. Hybrid data compression: A curvature threshold (0.03) is introduced to switch compression modes to avoid the loss of detail in CP decomposition in high curvature areas. Color parameter compression adopts 4-layer MLP network encoding to solve the memory bottleneck in large scenes.

[0167] 4. Learn dynamic features through dual-branch machine learning models such as MLP models, and optimize for different attributes of Gaussian models;

[0168] 5. The hierarchy of the hierarchical data structure LOD is adaptively adjusted based on visual distance and dynamic weights, and the rendering process is optimized by combining Gaussian model pruning and distillation.

[0169] 6. Ray sampling strategies in the rendering process based on voxel features such as weights, curvature, radiation radius (e.g., Gaussian coverage), and motion compensation;

[0170] 7. By utilizing the detailed residuals output by the dual-branch machine learning model, the voxel radiation properties are corrected, thus overcoming the bottleneck of voxel mesh in representing high-frequency details.

[0171] This invention also provides a dynamic scene reconstruction system, the structural diagram of which is shown below. Figure 6 As shown, it can specifically include:

[0172] The point cloud acquisition unit 10 is used to acquire point cloud information of multiple frames of the scene to be reconstructed.

[0173] The voxel grid unit 11 is used to divide the scene to be reconstructed into voxel grids of multiple scales based on the point cloud information of multiple frames acquired by the point cloud acquisition unit 10, and to acquire the voxel features of each voxel grid.

[0174] The Gaussian model unit 12 is used to determine the Gaussian model corresponding to each voxel grid according to the voxel features obtained by the voxel grid unit 11, and to obtain the multi-dimensional Gaussian features of the Gaussian model.

[0175] The rendering unit 13 is used to render the Gaussian model based on the multi-dimensional Gaussian features obtained by the Gaussian model unit 12, and reconstruct the scene to be reconstructed.

[0176] Wherein, if the point cloud information includes point cloud density, the voxel grid unit 11 is specifically used to set voxel grids of corresponding scales for different regions of the scene to be reconstructed according to the point cloud density of different regions of the scene to be reconstructed, to obtain original voxel grids of multiple scales, and to obtain the voxel features of the original voxel grids; the voxel features include the dynamic and static attributes of the original voxel grids.

[0177] Further, after obtaining multiple original voxel grids, the voxel grid unit 11 is also used to construct a voxel multi-level structure tree based on the voxel features of the original voxel grids, and adjust the original voxel grids based on the voxel multi-level structure tree to obtain an adjusted voxel grid; the step of determining the Gaussian model corresponding to each voxel grid based on the voxel features is performed on the adjusted voxel grid. Specifically, when constructing the voxel multi-level structure tree, the voxel grid unit 11 is used to take the smallest bounding box of the multiple original voxel grids as the root node of the voxel multi-level structure tree, covering the entire scene to be reconstructed; according to the size of the original voxel grids, each original voxel grid is mapped to a different level corresponding to the multi-level structure tree to form an initial level, each original voxel grid corresponds to an initial level node, and the pyramid structure formed includes the root node and the initial level node; the features of the nodes in each level of the voxel multi-level structure tree are stored.

[0178] The voxel grid unit 11 is also used to adjust the original voxel grid, specifically to split certain initial level nodes in the voxel multi-layer structure tree according to the density change and feature similarity of the original voxel grid, to generate lower-level child nodes, and the pyramid structure also includes the child nodes, or to merge certain initial level nodes in the voxel multi-layer structure tree to generate higher-level nodes, so as to split or merge the original voxel grid related to certain nodes; and to smooth the node features between the voxel multi-layer structure trees obtained from the point cloud information of adjacent frames.

[0179] Further, the Gaussian model unit 12 is specifically used to determine the original Gaussian model corresponding to each voxel grid according to the voxel features, and obtain the multi-dimensional Gaussian features of the original Gaussian model; compress the spatial position features and color features in the multi-dimensional Gaussian features to form a compressed Gaussian model; the two branches of the dual-branch machine learning model respectively obtain the first optimized value of the spatial position features and the second optimized value of the color features of the compressed Gaussian model, and optimize the spatial position features and color features according to the first optimized value and the second optimized value to obtain the optimized Gaussian model.

[0180] Furthermore, the two branches also obtain the third optimized value of the shape features of the compressed Gaussian model and the fourth optimized value of the weight information, respectively. Then, the Gaussian model unit 12 is also used to predict the displacement information of the next frame based on the displacement information of the previous multiple frames for the original voxel mesh of dynamic attributes; and optimize the Gaussian model based on the displacement information of the next frame and the first optimized value, second optimized value, third optimized value and fourth optimized value obtained by the two branches respectively.

[0181] Rendering unit 13 is specifically used to construct a hierarchical data structure with appropriate levels based on a multi-scale Gaussian feature pyramid and a sparse Gaussian update mechanism, wherein the levels of the multi-scale Gaussian feature pyramid correspond to the levels of the voxel multi-layer structure tree formed by the voxel mesh; adjust the levels of the hierarchical data structure according to the visual distance of the scene region to be reconstructed corresponding to the Gaussian model; prune the Gaussian model according to the global importance score of the Gaussian model and adaptively adjust the levels of the hierarchical data structure to form an adjusted hierarchical data structure; determine the ray tracing sampling step size according to the multi-dimensional Gaussian features, and render the adjusted hierarchical data structure based on the determined sampling step size to form the scene to be reconstructed.

[0182] Furthermore, the dynamic scene reconstruction system of this embodiment also includes:

[0183] Incremental data unit 14 is used to calculate the relative displacement of the dynamic attribute voxel grid in the voxel grid divided from the point cloud information of subsequent frames, in combination with the corresponding voxel grids of the preceding multiple frames of the subsequent frames; determine the feature information of the dynamic attribute voxel grid based on the relative displacement and the corresponding voxel grids of adjacent frames; and fine-tune the dual-branch machine learning model based on the feature information of the dynamic attribute voxel grid.

[0184] In this embodiment, the system mainly combines a voxel mesh and a Gaussian model constructed based on the actual scene to be reconstructed to render the scene to be reconstructed. It adopts two different feature construction methods to enhance the detailed expression in the reconstruction of real dynamic scenes.

[0185] This invention also provides a terminal device, the structural schematic of which is shown below. Figure 7 As shown, the terminal device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 20 (e.g., one or more processors) and memory 21, and one or more storage media 22 (e.g., one or more mass storage devices) for storing application programs 221 or data 222. The memory 21 and storage media 22 can be temporary or persistent storage. The program stored in the storage media 22 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the terminal device. Furthermore, the CPU 20 may be configured to communicate with the storage media 22 and execute the series of instruction operations in the storage media 22 on the terminal device.

[0186] Specifically, the application 221 stored in storage medium 22 includes a dynamic scene reconstruction application, which may include the point cloud acquisition unit 10, voxel mesh unit 11, Gaussian model unit 12, rendering unit 13, and incremental data unit 14 in the aforementioned dynamic scene reconstruction system, which will not be elaborated here. Furthermore, the central processing unit 20 may be configured to communicate with storage medium 22 and execute a series of operations corresponding to the dynamic scene reconstruction application stored in storage medium 22 on a terminal device.

[0187] The terminal device may also include one or more power supplies 23, one or more wired or wireless network interfaces 24, one or more input / output interfaces 25, and / or one or more operating systems 223, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0188] The steps performed by the dynamic scene reconstruction system in the above method embodiments can be based on this. Figure 7 The structure of the terminal device shown is illustrated.

[0189] Furthermore, in another aspect, the present invention provides a computer-readable storage medium storing a plurality of computer programs adapted for loading by a processor and executing a dynamic scene reconstruction method as described above by the dynamic scene reconstruction system.

[0190] Another embodiment of the present invention provides a terminal device, including a processor and a memory;

[0191] The memory is used to store multiple computer programs, which are loaded by the processor and executed as described in the dynamic scene reconstruction system; the processor is used to implement each of the multiple computer programs.

[0192] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0193] The foregoing has provided a detailed description of a dynamic scene reconstruction method, system, and related equipment provided by embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A dynamic scene reconstruction method, characterized in that, include: Obtain point cloud information of multiple frames of the scene to be reconstructed; The point cloud information includes point cloud density; For point cloud information from multiple frames, the scene to be reconstructed is divided into voxel grids of multiple scales, and the voxel features of each voxel grid are obtained. Based on the voxel features, determine the Gaussian model corresponding to each voxel grid, and obtain the multi-dimensional Gaussian features of the Gaussian model; The Gaussian model is rendered based on the multi-dimensional Gaussian features to reconstruct the scene to be reconstructed. The point cloud information from multiple frames is used to divide the scene to be reconstructed into voxel grids of multiple scales, and the voxel features of each voxel grid are obtained, specifically including: Based on the point cloud density of different regions of the scene to be reconstructed, voxel grids of corresponding scales are set for the scene to be reconstructed in different regions to obtain original voxel grids of multiple scales, and the voxel features of the original voxel grids are obtained. Wherein, for the original voxel mesh with static attributes, the voxel features include the normal vector, point density, and color mean of the original voxel mesh; for the original voxel mesh with dynamic attributes, the voxel features include the normal vector, density, color, displacement, and motion confidence. Based on the voxel characteristics of the original voxel mesh, multiple original voxel meshes are constructed into a voxel multilayer structure tree, and the original voxel meshes are adjusted based on the voxel multilayer structure tree to obtain the adjusted voxel mesh. In this process, 1-2 detail levels are added to the branch where the first original voxel mesh of the dynamic attribute is located. For the dynamic target object corresponding to the first original voxel mesh, the subsequent motion region of the dynamic target object is predicted based on the point cloud information of multiple consecutive frames. When dividing the voxel mesh based on the point cloud information of subsequent frames, the corresponding motion region is divided into a fine-grained voxel mesh. For the adjusted voxel grid, perform the step of determining the Gaussian model corresponding to each voxel grid based on the voxel features.

2. The method as described in claim 1, characterized in that, The step of constructing a multi-layer voxel structure tree from multiple original voxel meshes based on the voxel features of the original voxel meshes specifically includes: The smallest bounding box of the multiple original voxel meshes is used as the root node of the voxel multilayer structure tree, covering the entire scene to be reconstructed; Based on the size of the original voxel grid, each original voxel grid is mapped to a different level corresponding to the multi-layer structure tree to form an initial level. Each original voxel grid corresponds to an initial level node, and the resulting pyramid structure includes the root node and the initial level node. The features of nodes in each level of the voxel multi-layer structure tree are stored.

3. The method as described in claim 2, characterized in that, The adjustment of the original voxel mesh based on the voxel multilayer structure tree specifically includes: Based on the density variation and feature similarity of the original voxel mesh, some initial level nodes in the voxel multi-layer structure tree are split to generate lower-level child nodes. The pyramid structure also includes the child nodes. Alternatively, some initial level nodes in the voxel multi-layer structure tree are merged to generate higher-level nodes, so as to split or merge the original voxel mesh related to the some initial level nodes. The node features between the voxel multi-layer structure trees obtained from the point cloud information of adjacent frames are smoothed.

4. The method according to any one of claims 1 to 3, characterized in that, Based on the voxel features, the Gaussian model corresponding to each voxel grid is determined, and the multi-dimensional Gaussian features of the Gaussian model are obtained: Based on the voxel features, determine the original Gaussian model corresponding to each voxel grid, and obtain the multi-dimensional Gaussian features of the original Gaussian model. The spatial location features and color features in the multi-dimensional Gaussian features are compressed to form a compressed Gaussian model.

5. The method as described in claim 4, characterized in that, After forming the compressed Gaussian model, the process also includes: The two branches of the dual-branch machine learning model obtain the first optimized value of the spatial location feature and the second optimized value of the color feature of the compressed Gaussian model, respectively. Based on the first optimized value and the second optimized value, the spatial location feature and the color feature are optimized to obtain the optimized Gaussian model.

6. The method as described in claim 5, characterized in that, The two branches also respectively obtain the third optimized value of the shape features of the compressed Gaussian model and the fourth optimized value of the weight information. The method further includes: For the original voxel mesh with dynamic properties, the displacement information of the next frame is predicted based on the displacement information of the previous multiple frames. Based on the displacement information of the next frame, and the first, second, third, and fourth optimization values ​​obtained from the two branches respectively, the Gaussian model is optimized.

7. The method according to any one of claims 1 to 3, characterized in that, The step of rendering the Gaussian model based on the multi-dimensional Gaussian features specifically includes: Based on the multi-scale Gaussian feature pyramid and the sparse Gaussian update mechanism, a hierarchical data structure with corresponding levels is constructed, wherein the levels of the multi-scale Gaussian feature pyramid correspond to the levels of the voxel multi-layer structure tree formed by the voxel grid. The hierarchy of the layered data structure is adjusted according to the visual distance of the scene region to be reconstructed corresponding to the Gaussian model. The Gaussian model is pruned based on its global importance score, and the hierarchy of the hierarchical data structure is adaptively adjusted to form an adjusted hierarchical data structure. The sampling step size for ray tracing is determined based on the multi-dimensional Gaussian features, and the adjusted hierarchical data structure is rendered based on the determined sampling step size to form the scene to be reconstructed.

8. The method according to any one of claims 1 to 3, characterized in that, The method further includes: For the voxel grid of dynamic attributes in the voxel grid divided from the point cloud information of the subsequent frame, and in combination with the corresponding voxel grids of the previous multiple frames of the subsequent frame, the relative displacement of the voxel grid of the dynamic attributes between different frames is calculated. Based on the relative displacement and the corresponding voxel grids of adjacent frames, the feature information of the voxel grids of the dynamic attributes is determined; The bi-branch machine learning model is fine-tuned based on the feature information of the voxel grid of the dynamic attributes.

9. A dynamic scene reconstruction system, characterized in that, include: A point cloud acquisition unit is used to acquire point cloud information of multiple frames of the scene to be reconstructed; the point cloud information includes point cloud density. A voxel grid unit is used to divide the scene to be reconstructed into voxel grids of multiple scales based on point cloud information from multiple frames, and to obtain the voxel features of each voxel grid. A Gaussian model unit is used to determine the Gaussian model corresponding to each voxel grid based on the voxel features, and to obtain the multi-dimensional Gaussian features of the Gaussian model. A rendering unit is used to render the Gaussian model based on the multi-dimensional Gaussian features and reconstruct the scene to be reconstructed. The voxel mesh unit is specifically used to set voxel meshes of corresponding scales for different regions of the scene to be reconstructed based on the point cloud density of different regions, thereby obtaining original voxel meshes of multiple scales and acquiring voxel features of the original voxel meshes; wherein, for original voxel meshes with static attributes, the voxel features include the normal vector, point density, and color mean of the original voxel mesh, and for original voxel meshes with dynamic attributes, the voxel features include the normal vector, density, color, displacement, and motion confidence. The voxel grid unit is further configured to construct a voxel multi-layer structure tree from multiple original voxel grids based on the voxel features of the original voxel grids, and adjust the original voxel grids based on the voxel multi-layer structure tree to obtain an adjusted voxel grid; wherein, the branch containing the first original voxel grid of the dynamic attribute is added with 1-2 detail levels, and for the dynamic target object corresponding to the first original voxel grid, the subsequent motion region of the dynamic target object is predicted based on point cloud information of multiple consecutive frames, so that when dividing the voxel grid based on the point cloud information of subsequent frames, the corresponding motion region is divided into fine-grained voxel grids; and the step of notifying the Gaussian model unit to determine the Gaussian model corresponding to each voxel grid based on the voxel features for the adjusted voxel grid.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of computer programs adapted to be loaded by a processor and executed as described in any one of claims 1 to 8.

11. A terminal device, characterized in that, Including processor and memory; The memory is used to store a plurality of computer programs, which are loaded by a processor and executed as described in any one of claims 1 to 8; the processor is used to implement each of the plurality of computer programs.

Images