A Gaussian neural field dynamic scene reconstruction system based on depth consistency constraints

Abstract

This application provides a Gaussian neural field dynamic scene reconstruction system based on depth consistency constraints, relating to the field of computer graphics technology. The system includes: an estimation module that generates an initial depth map of the target frame; a calculation module that reconstructs the point cloud and obtains the point cloud normal and pixel normal; an optimization module that iteratively corrects the initial depth based on the consistency of the two types of normals to obtain an optimized depth map; an alignment module that, using the first target frame as a reference, determines the scale parameters through regression and performs scale transformation on the depths of the remaining frames to form a consistent depth sequence; a reconstruction module that constructs or trains a Gaussian neural field to output a three-dimensional representation based on this sequence; in addition, the calculation module may include multi-dimensional wavelets and sparse reconstruction for multi-scale noise reduction and direction-weighted fitting of normals; the reference frame selection is based on frame-level quality, geometric, and scale stability indices. This system improves intra-frame geometric reliability and cross-frame scale consistency, reduces ghosting and tearing, and enhances the stability and integrity of dynamic scene reconstruction.

Description

Technical Field

[0001] This application relates to the field of computer graphics technology, and more specifically, to a dynamic scene reconstruction system based on a Gaussian neural field with depth consistency constraints. Background Technology

[0002] With the development of computer vision and 3D reconstruction technologies, dynamic scene reconstruction based on video sequences has gradually become a hot topic. In existing technologies, common methods typically rely on frame-by-frame depth estimation or feature extraction, combined with photometric constraints, geometric constraints, or multi-view fusion to model and reconstruct dynamic scenes. In recent years, the introduction of implicit representation methods, especially Gaussian Neural Fields (GNF) and its related extensions such as 3D-GS and Dynamic Gaussian Splatting, has made some progress in the continuous modeling and realistic rendering of dynamic scenes.

[0003] However, existing methods still have shortcomings when dealing with complex dynamic scenes. For example, in scenes with non-rigid deformation or fast-moving objects, the reconstruction results are prone to distortions such as "ghosting" or "tearing." Under multi-view conditions, frame-by-frame depth estimation is easily affected by noise, resulting in an unsmooth and incomplete reconstructed surface. In addition, since the depth generated frame by frame often has scale inconsistencies, the stability and accuracy of cross-frame fusion and neural field modeling are also affected.

[0004] Therefore, the urgent technical problem to be solved in the 3D reconstruction of dynamic video scenes is: how to obtain a set of depth information sequences with high geometric reliability at the single-frame level and uniform scale across frames while preserving dynamic details, so as to support a stable and complete 3D representation of dynamic scenes based on Gaussian neural fields. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this application provides a dynamic scene reconstruction system based on a Gaussian neural field with deep consistency constraints, comprising:

[0006] An estimation module is used to perform depth estimation on target frame images in an input video sequence to generate an initial depth map of the target frame images; wherein, the target frame image is any frame image in the video sequence;

[0007] The calculation module reconstructs the 3D point cloud corresponding to the target frame image based on the initial depth map and the camera's intrinsic and extrinsic parameters, and calculates the point cloud normal vector corresponding to the target frame image from the 3D point cloud, and generates the pixel normal vector corresponding to the target frame image based on the target frame image.

[0008] The optimization module iteratively optimizes the initial depth map of the target frame image based on the consistency constraint between the pixel normal vector and the point cloud normal vector, and outputs the optimized depth map of the target frame image.

[0009] The alignment module is used to set the optimized depth map of the first target frame image as the reference depth map; and for any target frame image in the video sequence other than the first target frame image, to establish a regression model with the optimized depth map of the target frame image and the reference depth map, to determine the scale parameter, and to use the scale parameter to perform scale transformation on the optimized depth map of the target frame image, and output the uniform depth map of the target frame image.

[0010] The reconstruction module generates a Gaussian neural field model based on the uniform depth map sequence and outputs a three-dimensional representation of the dynamic scene corresponding to the video sequence; wherein the uniform depth map sequence places the reference depth map first, and the uniform depth maps of the remaining target frame images are arranged in a preset order.

[0011] Optionally, the calculation module includes a wavelet processing unit;

[0012] The wavelet processing unit is configured as follows:

[0013] Perform at least one layer of discrete wavelet decomposition on the initial depth map to obtain approximate sub-bands and multiple detail sub-bands;

[0014] The detail subbands are thresholded, and a depth map is reconstructed based on the approximate subbands and the thresholded detail subbands. The reconstruction of the 3D point cloud takes the reconstructed depth map as input, and the weighting coefficients are determined according to the energy of the detail subbands in the calculation of the point cloud normal vector. Weighted plane fitting is performed on the neighborhood of the point cloud to calculate the point cloud normal vector.

[0015] Optionally, the calculation module further includes a direction response unit;

[0016] The direction response unit is configured as follows:

[0017] Directional wavelet filtering is performed on the target frame image to obtain a local principal direction response, and the pixel normal vector is calculated based on the local principal direction response.

[0018] Optionally, the first target frame image is the first frame image in the time sequence of the video sequence.

[0019] Optionally, the wavelet processing unit employs a multidimensional wavelet basis, which is at least one of a steerable wavelet basis, a dual-tree complex wavelet basis, and a three-dimensional tensor product wavelet basis.

[0020] The wavelet processing unit is configured as follows:

[0021] Perform multidimensional discrete wavelet decomposition on the initial depth map. The multidimensional discrete wavelet decomposition includes: two-dimensional spatial decomposition or three-dimensional decomposition of the initial depth map sequence stacked in a preset order to obtain detail subbands with orientation index and scale index.

[0022] The directional weighting coefficients of the point cloud neighborhood are calculated based on the amplitude or energy of the detailed subband, and the plane fitting of the point cloud neighborhood is weighted by the directional weighting coefficients to calculate the point cloud normal vector.

[0023] Optionally, the computation module includes a sparse reconstruction unit; the sparse reconstruction unit is configured as follows:

[0024] The initial depth map is divided into blocks according to a preset window size and overlap step size, and each block is flattened into a block vector in a fixed order. Multidimensional wavelet transform is performed on each block vector to obtain a coefficient vector, and the coefficient vector is grouped according to the direction index and scale index to form multiple coefficient groups.

[0025] Under the premise of satisfying data consistency constraints and group sparsity constraints, the reconstruction results are solved by iterative update method for each coefficient group to obtain reconstruction depth blocks. The reconstruction depth blocks are then weighted and stitched together according to the overlap relationship to generate a reconstruction depth map. The reconstruction depth map is used for 3D point cloud reconstruction and point cloud normal vector calculation.

[0026] Optionally, the sparse reconstruction unit further includes a graph construction subunit, a constraint application subunit, and a projection subunit;

[0027] The graph construction subunit is used to construct a weighted adjacency graph with pixels as nodes. The edge weights of the adjacency graph are determined by the spatial distance between pixels and the change in the scale parameter output by the alignment module.

[0028] The constraint application subunit is used to introduce graph smoothing penalty and normal consistency constraint during the iterative update process. The normal consistency constraint is measured by the difference between the point cloud normal vector obtained from the 3D point cloud reconstructed by the reconstructed depth map and the pixel normal vector. Non-negativity constraint is applied to the coefficient elements in the preset index set and boundary gradient upper bound constraint is applied to the reconstructed depth map. The boundary gradient upper bound is determined based on the camera parameters and the edge mask of the target frame image.

[0029] The projection subunit is used to perform the projection operation of the nonnegative constraint and the upper bound constraint of the boundary gradient after each iteration.

[0030] Optionally, the alignment module includes a reference frame selection unit;

[0031] The reference frame selection unit is configured to: before setting the reference depth map, take multiple candidate frame images in the video sequence as objects, and calculate at least two of the following frame-level evaluation metrics respectively:

[0032] The effective pixel ratio or confidence statistics are obtained based on the initial depth map output by the estimation module.

[0033] The quantile statistics of the consistency residual between the pixel normal vector obtained by the calculation module and the point cloud normal vector reconstructed from the initial depth map and the camera's intrinsic and extrinsic parameters;

[0034] Based on the regression model of the alignment module, the dispersion of the scale parameter is obtained by establishing a regression relationship with adjacent frame images within a preset time window.

[0035] High gradient pixel ratio obtained based on the edge mask of the target frame image;

[0036] The evaluation indicators are weighted according to preset weights to obtain a comprehensive evaluation value. Frame images whose comprehensive evaluation values ​​meet preset conditions are selected from the candidate frame images as the first target frame images. The reference depth map is set as the optimized depth map of the first target frame image.

[0037] Optionally, the alignment module includes a stability reference frame selection unit;

[0038] The stability reference frame selection unit is configured to: before setting the reference depth map, for candidate frame images within a preset time window, call the wavelet processing unit and the sparse reconstruction unit respectively to obtain coefficient groups grouped by direction index and scale index, and generate a direction-scale group activation map of the candidate frame based on the coefficient groups.

[0039] Calculate at least one stability metric among the candidate frame and its neighboring frames in the same orientation-scale group, the activation overlap ratio, the number of frames with sustained activation, or the consistency score.

[0040] Candidate frames that meet the stability conditions are selected as the first target frame image according to the preset evaluation rules, and the reference depth map is set as the optimized depth map of the first target frame image.

[0041] Optionally, the alignment module includes a consistency reference frame selection unit;

[0042] The consistency reference frame selection unit is configured to: before setting the reference depth map, for candidate frame images within a preset time window, call the sparse reconstruction unit to obtain the reconstruction depth map, and count at least two of the following solution cost indicators: cumulative value of group sparse penalty, cumulative value of graph smoothing penalty, and residual normal consistency.

[0043] By combining the statistical values ​​of the scale parameter changes or regression residuals of the alignment module within a preset time window, and comparing them according to preset weights or priority order, candidate frames that meet the joint threshold conditions are selected as the first target frame image, and the reference depth map is set as the optimized depth map of the first target frame image.

[0044] Compared with existing technologies, this application introduces a consistency constraint between the image-side pixel normal and the point cloud normal after depth estimation, and iteratively optimizes the frame-by-frame depth map, effectively suppressing single-frame depth noise and local geometric distortion, and improving the geometric reliability of intra-frame depth. Simultaneously, by setting the optimized depth map of the first target frame image as the reference depth map and establishing a regression model for the optimized depth maps of other frames to determine scale parameters and perform scale transformation, a unified depth scale is achieved across frames, reducing ghosting and tearing phenomena caused by scale drift. Based on this, Gaussian neural field modeling is performed using a sequence containing the reference depth map and the unified depth maps of other frames as input, resulting in a 3D representation of dynamic scenes that outperforms existing methods in both integrity and stability. Therefore, this application achieves improved single-frame geometric quality and unified multi-frame scale consistency while ensuring dynamic detail representation, significantly improving the dynamic scene reconstruction effect based on Gaussian neural fields. Attached Figure Description

[0045] Figure 1 A schematic diagram of a Gaussian neural field dynamic scene reconstruction system based on depth consistency constraints provided in this application embodiment;

[0046] Figure 2 A top view of a depth map for recovering geometric features is provided in an embodiment of this application.

[0047] Figure 3 This application provides a front view of a depth map for recovering geometric features and structural diagrams.

[0048] Figure 4 A left view of a depth map recovery geometric feature structure map provided in this application embodiment;

[0049] Figure 5 This is a flowchart illustrating the steps performed by a wavelet processing unit according to an embodiment of this application. Detailed Implementation

[0050] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0051] See Figure 1The diagram shown is a schematic of a Gaussian neural field dynamic scene reconstruction system based on depth consistency constraints provided in an embodiment of this application, including:

[0052] An estimation module is used to perform depth estimation on target frame images in an input video sequence to generate an initial depth map of the target frame images; wherein, the target frame image is any frame image in the video sequence;

[0053] The calculation module reconstructs the 3D point cloud corresponding to the target frame image based on the initial depth map and the camera's intrinsic and extrinsic parameters, and calculates the point cloud normal vector corresponding to the target frame image from the 3D point cloud, and generates the pixel normal vector corresponding to the target frame image based on the target frame image.

[0054] The optimization module iteratively optimizes the initial depth map of the target frame image based on the consistency constraint between the pixel normal vector and the point cloud normal vector, and outputs the optimized depth map of the target frame image.

[0055] The alignment module is used to set the optimized depth map of the first target frame image as the reference depth map; and for any target frame image in the video sequence other than the first target frame image, to establish a regression model with the optimized depth map of the target frame image and the reference depth map, to determine the scale parameter, and to use the scale parameter to perform scale transformation on the optimized depth map of the target frame image, and output the uniform depth map of the target frame image.

[0056] The reconstruction module generates a Gaussian neural field model based on the uniform depth map sequence and outputs a three-dimensional representation of the dynamic scene corresponding to the video sequence; wherein, the uniform depth map sequence is arranged with the reference depth map first, and the uniform depth maps of the remaining target frame images are arranged in a preset order.

[0057] Regarding the estimation module mentioned above:

[0058] In specific implementation, the estimation module receives the input video sequence and performs depth estimation on the target frame image within it. The target frame image is any frame image in the video sequence. The estimation module first performs necessary preprocessing and calibration association on the input frame: geometric distortion correction and principal point and focal length consistency correction are performed based on the camera's intrinsic parameters; brightness and color are normalized; and the frame size is adjusted to the resolution required for inference without destroying the pixel coordinate correspondence. At the same time, the timestamp, resolution, camera identifier, and intrinsic parameter index corresponding to the frame are written into the output metadata so that they can be read and used subsequently according to the same pixel coordinate system.

[0059] After preprocessing, the estimation module calls the depth estimation model to infer the target frame image and generate an initial depth map for that frame. The initial depth map is aligned pixel-by-pixel with the target frame image in terms of resolution and pixel coordinates. Depth is measured linearly and represented in distance units. Abnormal or invalid pixels are identified by invalid values ​​and valid pixel masks.

[0060] Optionally, the estimation module also outputs a confidence map or effective pixel mask of the same size as the depth, used to mark the confidence level of occluded, overexposed, underexposed, or texture-deficient areas; at the same time, it calculates frame-level statistics, such as the effective pixel ratio, the quantile of the depth value distribution, and the mean of the confidence, which are used as inputs for subsequent modules to make quality judgments and select strategies.

[0061] For scenarios with rolling shutter or strong compression, the estimation module can perform frame quality detection before inference and mark or remove frames that clearly do not meet the quality threshold.

[0062] In an optional embodiment, the estimation module uses a monocular depth estimation model to perform inference, thereby reducing reliance on multi-view configurations involving external camera components.

[0063] In another optional embodiment, the estimation module can be connected to a stereo or multi-view matching model to generate an initial depth map by converting disparity to depth. In scenarios where enhanced temporal stability is required, the estimation module can also introduce temporal priors of adjacent frames while keeping the output alignment and interface unchanged, so as to improve the reliability of weak texture regions.

[0064] After preprocessing, the estimation module performs depth estimation on the target frame image, generating an initial depth map aligned pixel-by-pixel with the frame. Simultaneously, it generates and outputs a set of metadata, including at least the frame's timestamp, resolution, camera identifier, and intrinsic parameter index, to ensure subsequent modules can read and use the data within the same pixel coordinate system. Optionally, the estimation module also outputs a confidence map or effective pixel mask of the same size as the depth, and outputs frame-level statistics, such as the effective pixel ratio, depth distribution quantiles, and confidence mean, as inputs for quality and selection strategies.

[0065] The module can be deployed on a general-purpose processor or a graphics processor, and outputs the above results frame by frame in a streaming manner, providing direct input for the computing module to reconstruct the point cloud and for subsequent processing.

[0066] Regarding the aforementioned calculation module:

[0067] In practice, the calculation module receives the initial depth map of the target frame image and the camera intrinsic and extrinsic parameters associated with that frame, completes the 3D point cloud reconstruction and the generation of two types of normal vectors, and outputs the results along with the corresponding metadata for subsequent optimization module calls.

[0068] For example, the initial depth map, effective pixel mask, and confidence statistics associated with the target frame image can be read, invalid or significantly abnormal pixels can be removed, and the pixel coordinate system can be made consistent with the imaging model based on the intrinsic parameters. Then, the point cloud reconstruction process is started: for each effective pixel, according to the imaging geometry determined by the depth value of the pixel and the camera intrinsic parameters, it is back-projected to the camera coordinate system to obtain a spatial point; if the extrinsic parameters are available, the spatial point is further transformed from the camera coordinate system to a unified world coordinate system based on the extrinsic parameters, and attributes such as camera identifier, timestamp, color or grayscale value, and depth confidence are attached to the spatial point to form the three-dimensional point cloud of the frame.

[0069] In addition, to improve the efficiency of subsequent neighborhood queries, the computing module can establish a spatial index structure while generating point clouds, such as a tree-based or raster-based nearest neighbor retrieval index, and perform preliminary filtering of obviously isolated or density-abnormal points to reduce the impact of noise on normal estimation.

[0070] In the point cloud normal vector calculation stage, a combination of neighborhood fitting and robust selection can be used. The calculation module selects a local neighborhood for each point based on the spatial distance between points or a fixed number of nearest neighbors; the neighborhood radius can be adaptively adjusted with depth, allowing for a larger search range in distant regions to obtain sufficient neighborhood samples.

[0071] After determining the neighborhood, the calculation module performs planar fitting on the spatial distribution of neighborhood points to obtain the normal vector representing the local surface orientation of that point. To suppress interference from occlusion boundaries and outliers, the neighborhood is split before fitting using a depth jump threshold, retaining only the main cluster; during the fitting process, random consistency screening can be used to remove a small number of inconsistent samples.

[0072] After fitting, the direction of the normal vector is unified: the direction of the camera or the conventional direction of the world coordinate system is used as a reference to make the normal vectors of the same object surface point in the same direction; then the edge-preserving smoothing process is applied to the normal vector field to preserve boundary details at positions with obvious depth discontinuities and suppress high-frequency noise in continuous surfaces.

[0073] For pixels with unreliable normals due to insufficient neighborhood or geometric degradation, a low-confidence flag is set for reference by the subsequent optimization module when weighting for consistency constraints. The point cloud normal vectors are finally output in a form that corresponds one-to-one with the point cloud and are written into the index structure together with the point cloud of that frame for easy retrieval by spatial location.

[0074] Pixel normal vectors are generated using image-side estimation parallel to the point cloud path. This embodiment provides two alternative implementations, which can be used either one or in combination.

[0075] One approach is analytical estimation based on image and depth: the calculation module performs directional difference on the depth along the main axis of the image in the neighborhood of a pixel, and combines the imaging ratio determined by the camera intrinsic parameters to deduce the tilt direction of the local surface at that pixel, and determines the pixel normal vector accordingly; to reduce the impact of noise, the difference is performed in three windows: small, medium and large, and then multi-scale fusion is performed with edge response and depth confidence as weights to avoid distortion of a single scale in scenes with sparse texture or high-frequency noise.

[0076] Secondly, a learning-based normal estimation network is used: the computation module inputs the target frame image and its initial depth map into the pixel normal estimation network, directly outputting pixel-by-pixel normal vectors and confidence maps; to ensure consistency with the point cloud path, the network output is aligned pixel-by-pixel with the original image in the pixel coordinate system, and obviously unreliable regions are masked using the effective pixel mask from the estimation module. Regardless of the implementation, the computation module performs orientation unification and boundary protection after the pixel normal vectors are generated: orientation unification uses the camera orientation or a preset world orientation as a reference to avoid orientation flipping on the same surface; boundary protection reduces the smoothing intensity in high-gradient regions to preserve the true geometric edges.

[0077] To facilitate direct invocation of subsequent consistency constraints, the calculation module aligns and encapsulates the two types of normal vectors in the output stage: the point cloud normal vector is output together with the world coordinates of the corresponding spatial point, and the pixel normal vector is output as a two-dimensional vector map of the same size as the target frame image; at the same time, it provides a normal confidence map, a depth effective pixel mask, a point cloud spatial index, and frame-level statistical information.

[0078] For multi-camera scenarios, the calculation module generates point clouds and normals according to the camera identifiers, and unifies them to the same world coordinate system when the extrinsic parameters are known. If the extrinsic parameters are temporarily unavailable, they are retained in the camera coordinate system and marked in the metadata so that they can be unified after the extrinsic parameters are completed.

[0079] In this embodiment, spatial points and point cloud normals are first reconstructed from the initial depth map based on intrinsic and extrinsic parameters. Then, pixel normals are generated on the image side. The comparability and weighting of the two information paths are ensured through orientation unification and confidence labeling. Finally, a unified data interface is provided to the optimization module to construct normal consistency constraints.

[0080] Optionally, the estimation module and the calculation module are configured with a depth estimation model and a pixel normal estimation model, respectively. Both types of models are based on the constraints of single-frame input, pixel-by-pixel output, and strict alignment with the pixel coordinates of the frame image, and are connected to other modules of the system through a unified metadata interface.

[0081] In practice, the depth estimation model adopts an encoder-decoder multi-scale structure. The input receives the target frame image, with a resolution consistent with the preprocessed output of the estimation module.

[0082] Optionally, the camera's intrinsic parameters are encoded as vectors and injected into the intermediate feature layer to constrain the imaging geometry; alternatively, adjacent frames can be stacked along the temporal dimension at the input as temporal auxiliary information. The encoder extracts semantic and texture features from low to high, and the decoder restores the spatial resolution step by step through upsampling and skip connections, outputting candidate depths at each scale. The output has two headers: one is an initial depth map header, which generates a linear depth of the same size as the input; the other is a confidence header or effective pixel mask header, which generates a pixel-by-pixel confidence or availability marker. The product of model inference is aligned point-by-point with the frame image on the pixel coordinates and written to metadata, including timestamps, camera identifiers, resolution, and intrinsic parameter indices.

[0083] For training, a two-stage process is adopted. Stage 1 is pre-training on general data, which can use datasets collected by multi-view / depth cameras with ranging references or reference depth obtained by offline reconstruction from multi-view geometry, supplemented by synthetic data to address extreme poses and lighting conditions; on the data side, enhancements such as color perturbation, scale jitter, random cropping, motion blur synthesis, and occlusion synthesis are performed to ensure the model's adaptability to dynamic scenes. Stage 2 is fine-tuning of the target scene, continuing to use the above enhancements, and introducing reprojection consistency and edge preservation class constraints in unlabeled segments, while using scale-invariant depth constraints to weaken the impact of global scale drift on supervision; for data with significant lighting changes, rolling shutter, and compression artifacts, occlusion training is performed in conjunction with frame quality masks to avoid including abnormal regions in supervision.

[0084] The optimized hyperparameters are given in ranges, such as the learning rate, batch size, and number of training epochs, all falling within acceptable ranges for conventional engineering. Specific values ​​can be adjusted based on computing power and data scale. After training, the model outputs an initial depth map, a confidence map / effective mask, and frame-level statistics, such as the effective pixel ratio and depth distribution quantiles. These three types of results are returned to the estimation module through a unified interface for direct use by subsequent calculation and alignment modules.

[0085] In another optional embodiment, a pixel normal estimation model is set in the computation module. The model structure also adopts an encoder-decoder system, with the target frame image as the input; optionally, the initial depth map or its gradient and edge map output by the estimation module are input in parallel to explicitly introduce geometric cues. The output is equipped with a pixel normal head, which outputs a three-dimensional direction vector pixel by pixel, and can be supplemented with a normal confidence head to mark local reliability. During the inference phase, the model output is aligned pixel by pixel with the frame image; the direction is unified with the camera orientation or a preset world direction as a reference, and the smoothing intensity is reduced in the boundary region to preserve the real geometric edges. If the learning model is not enabled, the computation module can also be implemented analytically: the depth is oriented by difference within the pixel neighborhood and the normal is estimated by combining the intrinsic parameter ratio relationship. The result is then fused at multiple scales and the boundary is protected before being equivalently connected to the learning output in subsequent modules.

[0086] The training of this pixel normal estimation model also follows a strategy of pre-training combined with fine-tuning. Supervision signals are derived from reference normals derived from samples with high confidence depth, or from direct normal annotations in synthetic CAD or rendered data. Orientation consistency and smoothness constraints are introduced in unannotated segments, and boundary weights are used to enhance the learning intensity at geometric edges. The model can learn a normal confidence map, which serves as a weight source for subsequent weighting. Training data is selected from dynamic scenes containing non-rigid deformation and rapid motion, and occlusion regions are masked using occlusion masks generated from forward and backward correspondence consistency to avoid including incorrect matches in the supervision.

[0087] The input-output-constraint model described above corresponds one-to-one with the dynamic scene reconstruction requirements in this system: the input side uses the "target frame image" as a unified entry point, and introduces camera intrinsic parameters as conditional variables or control quantities injected into intermediate layers when necessary; the output side ensures that the initial depth map, pixel normal, confidence / mask are aligned pixel by pixel with the frame image, and all carry timestamps and camera identifiers, so as to achieve consistent mapping from pixels to spatial points when the calculation module reconstructs the point cloud and point cloud normal; the constraint side provides frame-level statistics through a unified metadata field, so that reference frame selection and scale regression stability assessment can be directly called without having to traverse the original frame data again.

[0088] Regarding the above-mentioned optimization modules:

[0089] In practice, the optimization module receives the initial depth map, pixel normal vectors, point cloud normal vectors, and corresponding effective pixel masks and confidence information of the target frame image. It iteratively optimizes the initial depth map according to a preset depth update process and outputs the optimized depth map for that frame. To ensure consistency with the aforementioned data link, the optimization module verifies all inputs by pixel coordinates, ensuring that the pixel normal vectors and point cloud normal vectors are comparable point-by-point on the same frame image coordinates. Confidence or neighborhood quality markers from the point cloud path are uniformly mapped to a pixel grid for weighted processing.

[0090] In practice, the optimization module first performs data preparation and weight generation. For each pixel, it reads the pixel normal vector and the point cloud normal vector, calculates the geometric difference metric between them, and generates pixel-level weights for the constraint terms based on this difference metric, the confidence levels of the pixel and the point cloud on both sides, the effective pixel mask, and the neighborhood consistency index. For pixels located at occlusion boundaries, with abrupt depth changes, or with degraded neighborhoods, the weights are reduced according to preset rules; for pixels with significantly low confidence levels or marked as invalid by the mask, the constraints are temporarily ineffective.

[0091] The iterative optimization process then begins. The optimization module uses the initial depth map as the starting value and performs the following steps in each iteration: First, it recalculates the local geometric metrics that match the pixel normal vectors based on the current depth map, updating the pixel-level difference metrics and weights. Second, it weights and aggregates the geometric consistency terms according to their weights to form the update direction for the current iteration. Third, it modifies the update direction by combining smoothing constraints and boundary protection strategies: reducing the smoothing intensity within edge masks to preserve the true geometric boundaries, and moderately increasing the smoothing intensity in flat areas to suppress isolated noise. Fourth, it updates the depth map using an adaptive step-size strategy. If the update results in local depth exceeding limits, non-physical fluctuations, or differences with adjacent pixels exceeding the tolerance range, it backtracks and reduces the step size to retry.

[0092] After each iteration, the optimization module calculates the average difference metric and its decrease rate for the entire frame as part of the convergence criterion; when the decrease rate is below the threshold for several consecutive iterations, or when the iteration limit is reached, the update stops.

[0093] To enhance robustness, this embodiment introduces a multi-scale mechanism during the iteration process. The optimization module first performs several rounds of rapid updates on the initial depth map and the two types of normal vectors at a lower resolution to correct large-scale geometric deviations. Subsequently, it progressively scales up to higher resolutions, using the optimization results from the previous scale as initialization at each level. The multi-scale mechanism effectively avoids getting stuck in local oscillations at high-frequency textures or fine noise, shortening the convergence time. For frames with large-scale non-rigid deformations or rapid motion, the optimization module can appropriately increase the effective range of boundary protection at lower scales and gradually restore it to normal settings at higher scales to achieve a smoother transition.

[0094] For anomalies and occlusion scenarios, the optimization module has a built-in anomaly handling branch. When a large area of ​​invalid pixels, severe compression artifacts, or stripes caused by rolling shutter are detected, the module reduces the update amplitude in that area and prioritizes neighborhood consistency and smoothness constraints to maintain basic coherence. When a low-confidence marker related to neighborhood degradation of the point cloud normal vector source is detected, the module does not force geometric consistency convergence but waits for the next round of re-evaluation after quality improvement from the pixel side or the point cloud side.

[0095] For significantly unstable pixel sets, the optimization module supports block-level updates, which means estimating the update direction and step size for each block region separately, thus avoiding local anomalies from affecting global convergence.

[0096] For example, the optimization module supports operation on general-purpose processors or graphics processors. To adapt to streaming video input, the module processes data frame by frame, while retaining the statistics and boundary mask of the previous frame as heuristic initialization. The module logs the average difference metric, update step size, effective pixel ratio, and backtracking count for each iteration, which is used for quality monitoring and to assist in the selection of subsequent reference frames.

[0097] After optimization, the output optimized depth map is strictly aligned with the original target frame image in pixel coordinates, and is written into metadata along with the updated effective pixel mask, boundary mask and convergence statistics, serving as direct input to the alignment module and reconstruction module.

[0098] In one optional embodiment, the optimization module mainly updates the entire frame and uses alternating row and column subgrid updates in the edge region to reduce oscillations at the boundary; in another optional embodiment, the optimization module adopts a block-pixel hybrid strategy: batch updates are performed on flat areas to accelerate convergence, and pixel-by-pixel updates are performed on high-frequency detail areas to improve local fidelity.

[0099] Understandably, regardless of the variant used, the optimization module follows the same data interface and convergence criteria to ensure that the output optimized depth map can be directly used by the subsequent alignment module to build a regression model, determine scale parameters, and generate a consistent depth map.

[0100] Regarding the alignment module mentioned above:

[0101] In practice, the alignment module receives the optimized depth map of the first target frame image, as well as the optimized depth maps of the remaining target frames, along with corresponding effective pixel masks, confidence statistics, and camera intrinsic and extrinsic parameters, to generate a consistent depth map. To ensure a closed-loop processing mechanism, the alignment module completes data preparation before entering regression modeling: unifying the pixel coordinates and imaging geometric relationships of all frames; removing invalid pixels based on the mask; establishing a pixel-level correspondence between the reference depth map and the depth map to be aligned, and marking areas that may be affected by occlusion, overexposure, or edge abrupt changes as the basis for selecting regression samples.

[0102] In practice, the alignment module sets a reference depth map based on the optimized depth map of the first target frame image; for any target frame image other than the first target frame image, a regression model is established to determine the scale parameters, and the scale transformation is completed accordingly to output the uniform depth map of the frame. To improve robustness, the regression model can adopt a process of "candidate sample screening - weight determination - robust solution - result verification": First, candidate pixels are sampled in the overlapping effective area of ​​the two depth maps, and samples with occlusion boundaries, large depth jumps, and low confidence are removed; then, weights are assigned to candidate samples based on pixel confidence, gradient strength, and neighborhood consistency, so that pixels with stable texture, geometric continuity, and high confidence have a greater influence in the regression; next, an anti-outlier regression solution strategy is adopted. For example, a piecewise penalty loss can be introduced or an excessively high proportion of abnormal residual samples can be iteratively removed to obtain more stable scale parameters; finally, the regression results are verified in two levels: first, local consistency verification, which compares the scale-transformed depth with the reference depth in a small area. If abnormal deviations occur in batches, backtracking and re-sampling are triggered; second, global trend verification, which puts the scale parameters of recent frames into a sliding window for dispersion evaluation. If the dispersion exceeds a preset threshold, the confidence of the current result is reduced and a reference frame is prompted to select a unit for verification.

[0103] For example, the window length can be set to 5 to 15 frames, and the dispersion threshold can be adaptively determined based on the distance range from the scene to the camera.

[0104] After scaling, the alignment module performs lightweight post-processing on the uniform depth map: it fills in small holes while preserving their edges, performs morphological cleanup on isolated noise points, and maintains the original depth values ​​at high-gradient edges to prevent the boundaries from being smoothed out. It then writes the uniform depth map, along with the frame's camera identifier, timestamp, scale parameters, and regression residual statistics, into the metadata. For example, in segments with rapid motion or a high proportion of non-rigid deformation, the alignment module can increase the neighborhood range of candidate samples, tighten the outlier removal threshold, or retain only static background areas as regression samples to prevent moving subjects from causing scale estimation shifts.

[0105] Optionally, in another embodiment, the alignment module can also set a reference frame update strategy: when the average value of the regression residual within the sliding window is continuously higher than the threshold, or the cumulative drift of the scale parameter exceeds the threshold, the first target frame image is replaced with the current target frame image as the new reference frame image, and the reference depth map is updated synchronously; for example, in the case of sudden changes in outdoor illumination or drastic changes in occlusion, this strategy can reduce the persistent deviation caused by the mismatch of the reference frame.

[0106] The reconstruction module receives a sequence of consistent depth maps arranged in a preset order, with the reference depth map at the beginning and the remaining frames added to the reconstruction process in sequence. The module first completes data loading and consistency checks: it reads the consistent depth map and its metadata frame by frame, and verifies the matching relationship with the camera's intrinsic and extrinsic parameters; it marks depth outliers, holes, and low-confidence regions as masks during training and fitting.

[0107] Then, geometric initialization is performed: using the reference depth map as a reference, effective pixels are selected on the image grid at a set sampling interval, and back-projected onto space to obtain an initial three-dimensional point set and set basic attributes for each spatial point; for example, the basic attributes may include spatial location, orientation cue, scale cue, and transparency cue, which will be used as the source of initialization or constraints in subsequent Gaussian neural field modeling.

[0108] For the remaining frames, the reconstruction module introduces the uniform depth map and camera pose frame by frame in a preset order, and uses the spatial projection corresponding to the effective pixels to update the attributes of existing spatial points or supplement new spatial points, so that the geometric prior is gradually enriched as the sequence progresses.

[0109] In the Gaussian neural field modeling phase, the reconstruction module uses frame images and camera parameters as observations and uniform depth maps as geometric priors or supervision sources to drive model parameter updates. To ensure training stability, the module adopts a phased approach: In the initial phase, the geometric center and scale distribution of the model are mainly constrained based on the reference depth map, so that the model forms an initial structure consistent with the reference geometry; in the expansion phase, the remaining uniform depth maps and camera poses are introduced frame by frame, and updates are accumulated according to frame-level weights. The frame-level weights can be set with reference to statistics such as the effective pixel ratio, scale parameter dispersion, and occlusion ratio of the uniform depth map; in the convergence phase, the weight of the geometric prior is gradually reduced, and the detail fitting strength driven by image observation is increased to restore texture and small-scale structure. For example, in indoor weak texture environments, the number of iterations in the initial phase can be appropriately extended to ensure that the geometric skeleton is fully formed; in outdoor strong texture environments, the entry into the expansion and convergence phases can be accelerated to improve the speed of texture detail restoration.

[0110] To adapt to dynamic scenes, the reconstruction module employs a sliding window processing strategy for temporally continuous segments: it retains only the parameters and updates of the most recent frames, using earlier frames as read-only references to control computational consumption and avoid long-term drift accumulation; the window length can be adaptively adjusted according to the scene's movement speed and available computing power. The module also records key quality indicators, including frame-level reprojection error, statistical differences between the uniformity depth and model depth, spatial point density, and coverage, to determine whether reference frame verification or geometric initialization fine-tuning is needed. For example, when a long-term increase in scale deviation or an increase in reconstruction gaps in contiguous regions is detected, the module can request the alignment module to output the latest scale parameter statistics and stability indicators to decide whether to temporarily freeze updates, reweight geometric priors, or shorten the sliding window.

[0111] During the output phase, the reconstruction module provides multiple result formats based on application requirements: it can directly output the parameter representation of the Gaussian neural field for subsequent rendering or analysis; it can also perform regular sampling and filtering on the model to export dense point clouds or meshed 3D surfaces; for scenes requiring new perspective synthesis, the module renders the corresponding image sequence according to the given camera pose. All outputs include key metadata and quality statistics at the time of generation, facilitating system-level traceability and comparative evaluation.

[0112] For example, in robot navigation or AR scenarios, passable areas and obstacle boundaries estimated by consistent depth can be output for direct consumption by the business side.

[0113] For example, see Figures 2-4 ,in, Figure 4 The left view of a depth map recovery geometric feature structure map provided in this application embodiment; the three-dimensional reconstruction result observed from the left side of the scene shows the processed point cloud / scattered point distribution on the outer contour of the main object; the superimposed arcs and thin lines are used to indicate the observation path or coordinate reference, which is convenient for comparing the shape of people and equipment with the local geometric continuity.

[0114] Figure 2 This is a top view of a depth map for restoring geometric features, provided in an embodiment of this application. The top-down perspective shows the reconstructed scene, highlighting the spatial relationship between the desktop, devices, and the outlines above the person. Point cloud density and void areas are more intuitive from this perspective, and circular lines indicate the camera or view trajectory, making it easy to observe the scale and alignment effect.

[0115] Figure 3 This is a front view of a depth map for restoring geometric features and structural diagrams, provided in an embodiment of this application. It shows a reconstructed scene from a frontal perspective, with two seated people in the foreground and workstations and ceiling structures in the background. Colored textures and point clouds are superimposed to make the geometric shapes of the plane and complex structures visible at the same time.

[0116] Through the above implementation, the alignment module completes the depth scale uniformity processing before entering reconstruction, and the reconstruction module completes the modeling and output of the Gaussian neural field with the support of the uniform depth map sequence. The two form a clear interface in terms of data and control: the uniform depth, scale parameters, and regression residual statistics output by the alignment module are used as priors and weights by the reconstruction module; the quality indicators and coverage statistics of the reconstruction module can provide feedback on whether the reference frame needs to be updated, and whether the window length and weights need to be adjusted. This collaborative relationship enables the system to maintain stable geometric consistency and usable reconstruction quality under different dynamic scenarios.

[0117] Optionally, the calculation module includes a wavelet processing unit;

[0118] The wavelet processing unit is configured as follows:

[0119] Perform at least one layer of discrete wavelet decomposition on the initial depth map to obtain approximate sub-bands and multiple detail sub-bands;

[0120] The detail subbands are thresholded, and a depth map is reconstructed based on the approximate subbands and the thresholded detail subbands. The reconstruction of the 3D point cloud takes the reconstructed depth map as input, and the weighting coefficients are determined according to the energy of the detail subbands in the calculation of the point cloud normal vector. Weighted plane fitting is performed on the neighborhood of the point cloud to calculate the point cloud normal vector.

[0121] In practical implementation, the computation module is equipped with a wavelet processing unit to perform multi-scale noise suppression and boundary preservation processing on the initial depth map before entering point cloud reconstruction and normal calculation, and to use the multi-scale detail intensity to guide the subsequent weighted plane fitting. The technical problem it solves is that the initial depth map often contains high-frequency noise, ringing, and compression artifacts. Direct back projection will introduce outliers and step effects in the point cloud neighborhood, resulting in unstable normal estimation and smoothed geometric boundaries. Through wavelet decomposition and thresholding, it can suppress fragmented noise while preserving the true boundaries, and characterize local uncertainties with detail subband energy, thereby improving the robustness of point cloud fitting and normal estimation.

[0122] For example, the wavelet processing unit includes a decomposition subunit, a thresholding subunit, a reconstruction subunit, and an energy mapping subunit, and its workflow is as follows:

[0123] The decomposition sub-unit performs at least one layer of discrete wavelet decomposition on the initial depth map, obtaining an approximate sub-band and detail sub-bands in several directions. The wavelet basis used can be selected according to the scene and computing power; for example, an orthogonal or bioorthogonal basis with good edge response can be used. The number of decomposition layers is preferably 1 to 3 layers to avoid loss of detail due to excessive decomposition. To ensure pixel coordinate alignment, this unit retains the same boundary extension strategy as the input and records the wavelet basis and number of layers used in the metadata.

[0124] The thresholding sub-unit thresholds each detail subband, for example, using either a soft or hard thresholding strategy. The threshold can be given by global noise estimation, adaptive noise estimation, or a block-based statistical approach. To accommodate geometric boundaries, the upper threshold for edge regions is moderately increased, while the threshold for flat regions is correspondingly decreased. When strong textures or high gradients are detected, only high-frequency subbands are lightly suppressed to avoid weakening the true structure. The thresholded detail subbands retain their orientation information for subsequent energy assessment.

[0125] The reconstruction sub-unit reconstructs the thresholded detail subbands and approximate subbands to obtain a reconstructed depth map. The reconstruction process follows the same boundary treatment as during decomposition to ensure that the output and input are aligned point-by-point on the pixel grid. For example, when fringe artifacts appear at far distances indoors, a gentle spatial smoothing can be added to the detail subbands in the corresponding direction before reconstruction; in areas with localized pops caused by strong outdoor light reflection, the detail gain in that area can be temporarily reduced to decrease small holes after reconstruction.

[0126] The energy mapping subunit statistically analyzes the energy or amplitude of each detail subband within the pixel neighborhood, resulting in a "detail energy map" of the same size. This energy map guides the calculation of point cloud normals: high-energy pixels are typically located at boundaries or abrupt texture changes, indicating higher local geometric uncertainty; low-energy pixels are mostly located in flat regions, suitable as main samples for fitting. For example, this unit can map energy to a weighted interval, setting a lower weight limit to suppress the influence of boundary outliers and an upper weight limit to increase the contribution of samples in flat regions; the mapping function can be set segmentally based on engineering experience, and specific coefficients can be given within recommended ranges in the embodiments of the specification.

[0127] After the above processing is completed, the wavelet processing unit outputs two types of results: a reconstructed depth map and a detail energy map. The calculation module then performs subsequent steps based on these results.

[0128] The first step is point cloud reconstruction, which involves backprojecting a reconstructed depth map into the original depth map to generate the point cloud for that frame. Since high-frequency noise has been suppressed, holes and outliers in the point cloud are significantly reduced.

[0129] Further weighted plane fitting is performed, introducing weights into the neighborhood fitting of the normal calculation. These weights are derived from the detailed energy map and depth confidence. For example, the weights are reduced at higher energy boundaries to decrease the impact of unstable samples on the fitting; the weights are increased in flat regions (lower energy) to stabilize the local plane orientation. If the energy distribution within the neighborhood is highly uneven, the neighborhood radius can be adaptively expanded to ensure a sufficient number of fitted samples.

[0130] Further, we perform orientation consistency and boundary protection by unifying the orientation of the fitted normal and reducing the intensity of secondary smoothing at high gradient boundaries in conjunction with the energy map to prevent the true boundary from being smoothed out.

[0131] Understandably, to adapt to different application scenarios, wavelet processing units can be parameterized and adaptively configured in the following ways:

[0132] For example, in indoor scenes with weak textures and uniform lighting, the number of decomposition layers should be 1 to 2, and the thresholding intensity should be moderate.

[0133] For example, in outdoor high-contrast or significant glass reflection scenarios, the upper threshold of the high-frequency directional subband is increased first, and the weight suppression range of the boundary region is expanded in the energy mapping.

[0134] For example, in fast-moving scenarios on mobile platforms, the number of decomposition layers can be reduced and the threshold increased in exchange for real-time performance, while time stability can be balanced through a sliding window during the reconstruction phase.

[0135] The wavelet processing unit maintains consistency with other modules in the system: the input is the initial depth map and its metadata, and the output is the reconstructed depth map and detail energy map. The wavelet basis type, decomposition layer number, threshold setting and energy mapping range are recorded in the metadata, which facilitates the tracking and reuse of the optimization module and the alignment module.

[0136] Optionally, the calculation module further includes a direction response unit;

[0137] The direction response unit is configured as follows:

[0138] Directional wavelet filtering is performed on the target frame image to obtain a local principal direction response, and the pixel normal vector is calculated based on the local principal direction response.

[0139] Optionally, the first target frame image is the first frame image in the time sequence of the video sequence.

[0140] In practical implementation, the computation module also includes a direction response unit, which performs directional analysis on the target frame image and generates pixel normal vectors accordingly. This unit aims to solve the following technical problems: in areas with weak texture, strong reflection, or repetitive texture, the directional information obtained by directly relying on gray-level gradients or single-scale differencing is unstable, leading to jitter in pixel normal estimation or smoothing of boundaries; by obtaining local principal directions through multi-scale, steerable directional filtering, the consistency of normal estimation can be improved while preserving geometric boundaries.

[0141] In practical implementation, the directional response unit pre-configures a library of directional wavelets or equivalent steerable directional filters, covering several discrete directions and two to three scale levels. The input is the target frame image and its metadata (timestamp, camera identifier, resolution), and optionally receives a detail energy map from the wavelet processing unit and an effective pixel mask from the estimation module for subsequent weighting and occlusion.

[0142] For example, the number of directions can be 8 to 16, and the number of scales can be 2 to 3. In indoor environments with weak textures, the number of directions can be reduced to improve real-time performance. In outdoor scenes with significant high-frequency structures, the number of directions can be appropriately increased to improve resolution.

[0143] Furthermore, directional wavelet filtering is performed on the target frame image in each direction and at each scale to obtain a set of response maps. After amplitude normalization and noise suppression, the response maps are weighted and aggregated by scale to obtain the comprehensive response intensity in each direction; the weights can be set empirically or provided by the scene adaptation module to make the near scale more sensitive to boundaries and the far scale more robust to texture noise.

[0144] For example, near-scale weights can be increased at high-contrast edges, while far-scale weights can be increased in large flat areas to enhance stability.

[0145] Furthermore, at each pixel location, the largest response from all directions is selected as the "local principal direction," and the "principal direction confidence" is obtained by combining indicators such as the ratio of peak to sub-peak values, cross-scale consistency, and neighborhood coherence. Locations where the effective pixel mask is invalid are directly set to low confidence and skipped from subsequent estimation.

[0146] For example, when the difference between the peak and the subpeak is small and the direction is scattered in the neighborhood, the confidence level decreases; when there is cross-scale consistency and high neighborhood consistency, the confidence level increases.

[0147] Furthermore, coherence enhancement and fracture repair are performed on the initial principal direction field: small-scale smoothing is performed within the continuous surface, and the smoothing intensity is reduced at significant boundaries to preserve the true edges; isolated direction anomalies caused by reflection or shadow are regionally replaced, with the direction with the highest built-in confidence in the neighborhood being used as the reference.

[0148] For example, in glass or metal reflective regions, the orientation field often abruptly changes, so the neighborhood search radius can be appropriately expanded and the smoothing weight of the region reduced.

[0149] Furthermore, the orientation response unit regards the "local principal direction" as a principal axis of the surface tangency at the pixel, and then combines it with the auxiliary direction orthogonal to it (which can be obtained by the orthogonal response of the orientation filter or by the image principal axis) to determine the local tangent plane at the pixel, and uses the direction perpendicular to the tangent plane as the pixel normal vector; the direction of the normal vector is uniformly referenced by the camera orientation or the conventional direction of world coordinates to avoid orientation flipping on the same object surface.

[0150] For example, at strong geometric boundaries such as stair treads and cabinet edges, the principal direction is consistent with the boundary direction, and the generated normals exhibit stable directional differences on both sides of the boundary rather than random jittering.

[0151] Furthermore, the pixel normal vector and the main direction confidence map output by the directional response unit are aligned pixel by pixel with the target frame image and output along with the metadata. The confidence map and the detail energy map can be used together as the weight source for consistency constraints in the optimization module, so that the weight at the boundary is reasonably reduced and the weight in the flat area is moderately increased, thereby improving the stability of deep iterative optimization.

[0152] For example, in fast motion segments, the near-scale weight can be reduced, the far-scale weight can be increased, and a stricter threshold can be set for confidence to reduce orientation misjudgment caused by motion blur.

[0153] In another embodiment, the implementation of the first target frame image being the first frame image in the time sequence of the video sequence is as follows: During the input stage, the system sorts the video sequence in ascending order based on timestamps, directly selects the first frame in the time sequence as the "first target frame image," and sets the optimized depth map obtained by the optimization module for this frame as the reference depth map. To ensure consistency with the overall system flow, this reference depth map is placed at the beginning of the consistent depth map sequence, and the remaining frames are added to the alignment and reconstruction process in a preset order, such as time sequence.

[0154] For example, in indoor monitoring scenarios where a static background dominates, the first frame can be directly used as a reference, facilitating the establishment of a stable baseline for subsequent residual statistics during alignment. For instance, in outdoor scenarios with drastic lighting changes or significant initial occlusion, although the reference frame remains the first frame in chronological order, the system can enable a reference frame update strategy in subsequent alignment stages. This could involve triggering replacement when regression residuals continuously exceed thresholds or scale drift accumulates beyond limits, thereby reducing persistent bias caused by the initial reference frame's mismatch. This implementation satisfies the constraint of "first frame in chronological order" while also being compatible with the robustness mechanism of the alignment module, ensuring stable progress in sequence organization and subsequent modeling.

[0155] Optionally, the wavelet processing unit employs a multidimensional wavelet basis, which is at least one of a steerable wavelet basis, a dual-tree complex wavelet basis, and a three-dimensional tensor product wavelet basis.

[0156] See Figure 5 The flowchart illustrates the steps performed by a wavelet processing unit according to an embodiment of this application, including steps S101 to S102, wherein:

[0157] S101: Perform multidimensional discrete wavelet decomposition on the initial depth map. The multidimensional discrete wavelet decomposition includes: two-dimensional spatial decomposition or three-dimensional decomposition on the initial depth map sequence stacked in a preset order to obtain detail subbands with orientation index and scale index.

[0158] S102: Calculate the directional weighting coefficients of the point cloud neighborhood based on the amplitude or energy of the detailed subband, and use the directional weighting coefficients to weight the plane fitting of the point cloud neighborhood to calculate the point cloud normal vector.

[0159] In this embodiment, the wavelet processing unit of the calculation module adopts a multi-dimensional wavelet basis to perform multi-scale analysis with direction discrimination on the initial depth map before entering point cloud reconstruction and normal calculation, and transforms the "direction-scale" information into directional weights for point cloud neighborhood fitting.

[0160] The technical problems addressed are: conventional denoising lacks anisotropy discrimination and is prone to over-smoothing at edges; frame-by-frame independent processing is insufficient for the stability of temporally continuous segments, easily resulting in flickering and normal jitter. By obtaining detail subbands with direction and scale indices through multidimensional wavelet decomposition, noise can be suppressed while preserving the true geometric boundaries, and the robustness of normal estimation is improved by directional consistency.

[0161] In practical implementation, the wavelet processing unit incorporates multiple multidimensional wavelet bases, preferably selecting at least one from turnable wavelet bases, dual-tree complex wavelet bases, or three-dimensional tensor product wavelet bases. Base selection is performed by the "base selection subunit," and the base type and parameters are recorded in the metadata to ensure traceability.

[0162] For example, to obtain better displacement invariance and orientation resolution, a dual-tree complex wavelet basis can be used; when the scene has rich slender edges and continuous orientation distribution, a steerable wavelet basis can be used; when it is necessary to consider both spatial and temporal correlations, a three-dimensional tensor product wavelet basis can be used to decompose the stacked sequence.

[0163] The number of decomposition layers and directions can be adaptively set according to the scene. For example, 8 to 16 directions and 2 to 3 scale layers can cover common indoor and outdoor scenes.

[0164] For example, at the edge of a strong contrast rooftop outdoors, the sub-band energy is high along the rooftop direction. After weighting the samples across the rooftop direction, the fitting normal is stable and the boundary is clear. At the junction of the indoor ground and the column, the energy map guides the neighborhood segmentation. The fitting is performed in the two main clusters of ground and plane, respectively, to avoid averaging.

[0165] For example, when real-time requirements are high, the number of orientations can be 8, the number of scale layers can be 2, and only two-dimensional decomposition can be enabled; in offline reconstruction that emphasizes quality, the number of orientations can be 16, the number of scale layers can be 3, and three-dimensional decomposition can be enabled to improve temporal stability.

[0166] In strong light reflection scenarios, increase the threshold of high-frequency directional subbands and expand the boundary weight reduction range; in indoor scenarios with significant low-light noise, appropriately increase the base threshold of flat areas and reduce the weight of far-scale areas.

[0167] For example, the input to the wavelet processing unit is an initial depth map and its metadata (frame identifier, camera intrinsic index), and the output is a reconstructed depth map, a direction-scale energy map, and a processing record (wavelet basis type, number of directions, number of scale layers, threshold setting, and weight mapping range). The output is aligned point-by-point with the input on the pixel grid to facilitate point cloud reconstruction and normal fitting; the processing record is written into the metadata for reuse or traceability by the optimization and alignment modules.

[0168] In this way, the wavelet processing unit explicitly introduces direction and scale information through multidimensional wavelet decomposition. On the one hand, it suppresses noise and maintains the true boundary at the depth map reconstruction level. On the other hand, it transforms the direction-scale energy into directional weights for fitting the point cloud neighborhood, making the point cloud normal calculation more robust to boundary and temporal perturbations, thereby providing a more reliable geometric basis for subsequent consistency constraints and neural field modeling.

[0169] Optionally, the computation module includes a sparse reconstruction unit; the sparse reconstruction unit is configured as follows:

[0170] The initial depth map is divided into blocks according to a preset window size and overlap step size, and each block is flattened into a block vector in a fixed order. Multidimensional wavelet transform is performed on each block vector to obtain a coefficient vector, and the coefficient vector is grouped according to the direction index and scale index to form multiple coefficient groups.

[0171] Under the premise of satisfying data consistency constraints and group sparsity constraints, the reconstruction results are solved by iterative update method for each coefficient group to obtain reconstruction depth blocks. The reconstruction depth blocks are then weighted and stitched together according to the overlap relationship to generate a reconstruction depth map. The reconstruction depth map is used for 3D point cloud reconstruction and point cloud normal vector calculation.

[0172] In this embodiment, the computation module is equipped with a sparse reconstruction unit, which performs block-level sparse representation and reconstruction of the initial depth map before entering point cloud reconstruction and normal calculation. This is to suppress high-frequency noise, ringing and compression artifacts while preserving the real geometric boundaries, reduce outliers and step effects after back projection, and provide a smoother and more reliable depth field for subsequent neighborhood fitting.

[0173] To this end, the sparse reconstruction unit first divides the initial depth map into several overlapping depth blocks according to the preset window size and overlap step size, and then flattens each depth block into a block vector in a fixed order to ensure that the processing order between different implementations is consistent and that overlapping areas can be seamlessly stitched together.

[0174] For example, the window can be set to an odd number of side lengths, such as 7 to 11 pixels, and the overlap step can be between half a window and one-third of a window, in order to achieve a balance between reconstruction quality and computational overhead; in weak texture or distant scenes, the window can be appropriately enlarged to increase stable samples.

[0175] After completing the block partitioning, the sparse reconstruction unit performs a multidimensional wavelet transform on each block vector to obtain a set of coefficient vectors with direction and scale indices. The coefficients are then grouped according to the "direction-scale" relationship to form multiple coefficient groups. The purpose of grouping is to treat the detail coefficients of the same direction and adjacent scales as a whole for constraint, thereby better reflecting the consistency of the real surface at different scales.

[0176] The process then proceeds to the sparse solution phase: Under the premise of satisfying the data consistency constraint and the group sparsity constraint, the unit iteratively solves for the values ​​of each coefficient group using an alternating update method, and generates the corresponding reconstruction depth block after each iteration. The data consistency constraint is used to ensure that the reconstruction depth and the original block remain consistent within the effective pixel range, while the group sparsity constraint encourages groups in the same direction to present a consistent sparse structure at multiple scales, so as to suppress random noise and retain meaningful structural lines.

[0177] Furthermore, to improve boundary performance, the sparse reconstruction unit references the detail energy map from the wavelet processing unit and the effective pixel mask from the estimation module, and sets different solution weights for each set of coefficients and each pixel position: for example, reducing the penalty of constraints on high-frequency coefficients at the boundary to avoid excessive weakening of real edges; and increasing the sparse constraint weight in flat areas to more effectively suppress isolated noise.

[0178] For example, in glass or highly reflective areas, the caution of the corresponding directional group can be increased in conjunction to limit the diffusion of abnormal high-frequency components; in fast-moving segments, the number of iterations can be reduced and the window can be appropriately increased to meet real-time requirements.

[0179] The iterative process terminates using convergence criteria, including the magnitude of the reduction in reconstruction error, the magnitude of the change in group sparse cost, and the maximum number of iterations. Once the termination criteria are met, the sparse reconstruction unit outputs the reconstructed depth block of the current iteration as the result and enters the overlapping and stitching stage. Stitching uses a weighted superposition method to merge overlapping regions. The weights are related to the distance from the block center, detail energy, and local confidence: pixels closer to the block center, with lower energy and higher confidence, are given greater weight to reduce seams and bright / dark bands at block edges; in areas with higher boundary energy, the original gradient trend is maintained to prevent the boundaries from being flattened. After stitching, the entire reconstructed depth map is obtained, with pixel coordinates aligned point-by-point with the initial depth map. Block-level and frame-level statistics, such as reconstruction error, effective pixel ratio, and iteration count, are output and written to metadata for subsequent module weighting and tracking.

[0180] The reconstructed depth map then replaces the initial depth map in the point cloud reconstruction process. The point cloud generated by backprojection shows a significant decrease in the number of holes and outlier density, thereby improving the stability of the normal fit. Simultaneously, the orientation-scale information obtained by the sparse reconstruction unit during the solution process can serve as an additional reference, guiding the weighted plane fitting of the point cloud neighborhood: the weight of samples in flat regions is appropriately increased, while the weight of samples along boundary directions is appropriately decreased. If necessary, the search radius is expanded or partitioned fitting is performed for neighborhoods with uneven energy levels to further reduce the interference of cross-boundary samples on the normal.

[0181] For example, at the junction of a weakly textured interior wall and a column, the depth after sparse reconstruction can keep the wall smooth without sacrificing the column's edges; near the high-contrast boundary where the outdoor eaves meet the facade, weighted fitting can avoid "flattening" the eaves, so that the normals present a stable and reasonable directional difference on both sides.

[0182] To accommodate different application scenarios and computing power conditions, the sparse reconstruction unit supports parameter adaptation and degraded operation: on mobile platforms with high real-time requirements, the number of iteration rounds can be shortened, the number of direction groups can be reduced, and the convergence threshold can be relaxed; in offline high-precision reconstruction, the number of iteration rounds and direction group subdivision can be increased, and stricter group sparsity control can be enabled to obtain a cleaner geometric surface.

[0183] Understandably, regardless of the configuration, the input and output interfaces of the unit remain consistent: the input is the initial depth map and its metadata, and the output is the reconstructed depth map and statistical records, all of which are strictly aligned with the image pixel grid and directly used for point cloud reconstruction and point cloud normal vector calculation, thereby improving the stability and completeness of subsequent reconstruction.

[0184] Optionally, the sparse reconstruction unit further includes a graph construction subunit, a constraint application subunit, and a projection subunit;

[0185] The graph construction subunit is used to construct a weighted adjacency graph with pixels as nodes. The edge weights of the adjacency graph are determined by the spatial distance between pixels and the change in the scale parameter output by the alignment module.

[0186] The constraint application subunit is used to introduce graph smoothing penalty and normal consistency constraint during the iterative update process. The normal consistency constraint is measured by the difference between the point cloud normal vector obtained from the 3D point cloud reconstructed by the reconstructed depth map and the pixel normal vector. Non-negativity constraint is applied to the coefficient elements in the preset index set and boundary gradient upper bound constraint is applied to the reconstructed depth map. The boundary gradient upper bound is determined based on the camera parameters and the edge mask of the target frame image.

[0187] The projection subunit is used to perform the projection operation of the nonnegative constraint and the upper bound constraint of the boundary gradient after each iteration.

[0188] In this embodiment, the sparse reconstruction unit adds a graph construction subunit, a constraint application subunit, and a projection subunit, forming a closed loop between block-level sparse reconstruction and full-graph geometric constraints. This addresses engineering problems such as initial depth distortion in the boundary region, temporal instability caused by cross-frame scale perturbations, and local update out-of-bounds issues during iterative solutions. This unit operates within an iterative main loop of "coefficients - reconstructed depth - point cloud / normal - update." The inputs are the reconstructed depth map and its coefficients obtained in the previous stage, the scale parameter change statistics from the alignment module, and the pixel normal and point cloud normal and their confidence levels from the calculation module. The output is a new set of coefficients and a reconstructed depth map that satisfies the projection constraints, and key statistics are recorded in metadata.

[0189] The graph construction subunit establishes a weighted adjacency graph on the image grid with pixels as nodes. It is preferred to connect by eight neighborhoods and introduce an adaptive radius so that distant regions still have enough adjacent edges when sampling is sparse.

[0190] The weight of each edge is determined by two parts: one is the static component derived from the spatial distance or color / depth similarity between pixels, which is used to express local geometric coherence; the other is the temporal component corresponding to the scale parameter change output by the alignment module within the sliding time window, which is used to express the cross-frame stability of the current segment.

[0191] For example, in segments with small scale variations, edge weights within the same image block are increased to enhance smoothness; in segments with large scale variations, cross-block connections are appropriately weakened while preserving directionality at boundaries to prevent excessive flattening. To improve efficiency, the graph construction sub-unit stores adjacency relationships in a sparse structure and automatically breaks edges for invalid pixels and low-confidence regions, ensuring that subsequent constraints only apply to reliable samples. For high-contrast edges, anisotropic adjustment of edge weights can be performed by referring to wavelet direction-scale energy maps: connectivity is preserved along the boundary tangent, and connectivity is reduced along the normal, thereby naturally forming "weak coupling" on both sides of the boundary.

[0192] The constraint-applying subunit synchronously injects three types of constraints in each round of coefficient iteration. The first is a graph smoothing constraint: a consistency penalty is applied to strongly connected nodes along the weighted adjacency graph, promoting smooth changes in coefficients or reconstruction depth within the same surface. At boundaries, due to the different weights of tangential and normal edges, the constraint is anisotropic, preserving true edges while suppressing step effects between blocks. The second is a normal consistency constraint: a local point cloud is back-projected onto the current reconstruction depth map, and the point cloud normal is estimated and compared pixel-by-pixel with the pixel normal. The difference is used as a metric to guide coefficient updates towards "geometric consistency." This constraint is weighted by confidence level; when the confidence level is low on either the pixel side or the point cloud side, the constraint strength automatically decreases to prevent noise from misleading updates.

[0193] For example, in areas of glass reflection or strong shadow, where pixel normal confidence is low, sub-cells will weaken consistency constraints in that area and allocate more weight to graph smoothing.

[0194] Thirdly, there are two explicit feasible region constraints: applying a non-negativity constraint to the coefficient elements within the preset index set, and applying a boundary gradient upper bound constraint to the reconstructed depth map. The former is applicable to "energy-type" components such as coefficient groups or approximate subbands stored in amplitude form, preventing non-physical negative amplitude values ​​from appearing during iteration; the latter is used to limit the maximum slope of the reconstructed depth at strong boundaries. The boundary gradient upper bound is jointly determined by the camera's intrinsic parameters (the spatial scale corresponding to pixel size and focal length) and the edge mask of the target frame image: within the edge mask, larger tangential changes are allowed, and excessively steep rises in the normal direction are limited; in flat regions, the upper bound is tightened to avoid local "bumps / concaves".

[0195] For example, at the junction of an indoor wall and a column, the sub-unit allows for rapid changes along the column direction while suppressing abrupt changes across the boundary direction; under strong outdoor light conditions, for highlight bursts, the upper boundary of the boundary will automatically tighten with the effective pixel mask to reduce pseudo gradient propagation.

[0196] The projection subunit is responsible for projecting the solution back to the feasible region after each round of coefficient updates, which includes two steps. The first step is coefficient projection: non-negative pruning and intra-group consistency convergence are performed on coefficient elements belonging to the preset index set to ensure that energy components are not negative and the intra-group distribution is reasonable; for direction groups that are not in the set but have abnormal spikes due to numerical fluctuations, a threshold is applied to gently suppress them to prevent a single direction from becoming overly dominant. The second step is depth projection: local gradients are calculated on the reconstructed depth map and compared pixel by pixel with the upper bound of the boundary gradient. For pixels that exceed the upper bound, the update magnitude is reduced along the normal direction or the pixel reverts to the previous valid state, while ensuring that the true changes in the tangential direction are not smoothed out; for non-boundary regions, if the gradient exceeds the limit for two consecutive rounds, the step size of the region is reduced and the graph smoothing weight is increased until it is restored to within the upper bound.

[0197] For example, in fast motion segments, the projection subunit can temporarily tighten the upper bound and shorten the step size in exchange for stability; in offline high-precision reconstruction, the upper bound and step size can be relaxed, but the boundary protection of directional anisotropy is still effectively guaranteed.

[0198] For example, the graph construction subunit outputs the sparse adjacency structure and edge weights at the beginning of the iteration, the constraint application subunit queries the edge weights, normal differences and confidence as needed during the update process, and the projection subunit returns the clipping and rollback results at the end of each round, and writes the statistics such as the proportion of pixels that violate the constraints and the number of rollbacks into the metadata for reference by the alignment module and the reconstruction module.

[0199] For example, when the number of backoffs is detected to be increasing or the violation rate is concentrated in the boundary band, the system can prompt the alignment module to check the scale parameters or trigger the reference frame update, thereby forming a stable closed loop at the system level.

[0200] This not only improves the smoothness and boundary fidelity of block-level reconstruction, but also explicitly introduces the scale stability of the alignment stage into the reconstruction process, which helps the convergence and completeness of subsequent normal estimation and neural field modeling.

[0201] Optionally, the alignment module includes a reference frame selection unit;

[0202] In this embodiment, the alignment module includes a reference frame selection unit. This unit performs frame-level quality and stability evaluations on multiple candidate frame images in the video sequence before setting the reference depth map. A comprehensive evaluation value is generated based on preset weights to select the first target frame image, and the optimized depth map output by the optimization module for this frame is set as the reference depth map. The technical problem addressed by this unit is that improper reference frame selection can amplify cross-frame scale drift and introduce geometric instability, thereby affecting the convergence and completeness of subsequent uniform alignment and neural field modeling.

[0203] In practical implementation, the reference frame selection unit first determines the candidate set and time window. The system indexes the input video sequence by timestamp and, without changing the description of the set of "any target frame images other than the first target frame" above, sets a sliding time window for the selection action; the window length can be adaptively determined according to the scene motion speed and computing power, for example, 5 to 15 frames. Subsequently, candidate pre-screening is performed: frames with too low effective pixel ratio, too high overexposure / underexposure ratio, or frame quality detection failure are removed, while consecutive highly similar frames are merged to avoid redundant calculations.

[0204] After the candidate set is determined, the reference frame selection unit independently calculates at least two frame-level evaluation metrics for each candidate frame and temporarily stores the results along with metadata.

[0205] The first type of metric comes from the estimation module: it reads the initial depth map and confidence map of the frame to obtain the statistical value of the effective pixel ratio or confidence level, which is used to reflect the usable depth area and overall confidence level.

[0206] The second type of metric comes from the consistency between the pixel normal and the point cloud normal in the calculation module: the pixel normal of the frame is compared pixel by pixel with the point cloud normal reconstructed from its initial depth map and the camera's intrinsic and extrinsic parameters, and the quantile values ​​of the consistency residual are statistically analyzed to characterize the geometric self-consistency within the frame. For example, the median and upper quantile can be considered to balance robustness.

[0207] The third type of metric comes from the alignment module itself: by establishing regression relationships between the candidate frame and its adjacent frames within the window, a sequence of scale parameters is obtained, and the dispersion of this sequence is calculated to measure the scale stability when aligned with adjacent frames.

[0208] The fourth type of metric comes from the image side edges: based on the edge mask of the frame, the proportion of high-gradient pixels is statistically analyzed to reflect the richness of geometric structure and the amount of potential boundary information. Of the above four types of metrics, the system must enable at least two; if a data source is missing, the reference frame selection unit automatically resets the weights according to the enabled metrics and continues execution.

[0209] To achieve comparability of indicators with different dimensions and polarities, the reference frame selection unit standardizes and unifies the polarity of each indicator: the direction of "the larger the value, the better / the smaller the value, the better" is adjusted to be compared in the same direction through interval mapping, and preset weights are loaded according to the scene level to generate a comprehensive evaluation value.

[0210] For example, in indoor weak texture scenes, the weight of "effective pixel ratio / confidence" is increased to ensure that the reference frame has a sufficient usable area; in outdoor strong texture and fast motion scenes, the weight of "scale parameter dispersion" is increased to prioritize frames that are stably aligned with neighboring frames; in industrial scenes with a high proportion of reflections or complex boundaries, the weight of "consistency residual quantile" is increased to avoid frames with poor geometric self-consistency becoming the benchmark.

[0211] Understandably, adaptive weighting can be enabled when necessary: ​​when the overall scale dispersion within the window increases, the weight of scale stability is automatically increased; when the overall effective pixel ratio decreases, the weight of available area is automatically increased. To avoid random values ​​dominating, a "bottom-line constraint" can be introduced for each cell, meaning that if any key indicator falls below the bottom line, it will not be included in the final comparison.

[0212] After the comprehensive evaluation is completed, the reference frame selection unit sorts the candidate frames according to the evaluation values ​​and performs hierarchical decision-making.

[0213] First, select the frame with the highest comprehensive evaluation value that meets the bottom line constraint as the first target frame image; if there is a tie, then compare the frame with lower scale dispersion, higher effective pixel ratio, and better consistency residual in turn; if it is still impossible to distinguish, backtrack to the candidate frame that is earlier in time to ensure the determinism of the process.

[0214] For example, in indoor static background surveillance footage, the selection of a geometrically stable reference is often driven by the "effective pixel ratio / confidence" and "residual quantiles"; while in outdoor footage shot while moving, the "scale dispersion" metric is more likely to dominate in order to suppress scale drift in subsequent alignment.

[0215] After selecting the first target frame image, the unit sets the optimized depth map output by the optimization module of the frame as the reference depth map, and writes information such as "frame identifier, window range, values ​​of each indicator, normalization parameters, weights and bottom line thresholds used, comprehensive evaluation value and sorting results" into the metadata for subsequent auditing and reuse.

[0216] To enhance operational robustness, the reference frame selection unit and the alignment module maintain online linkage. During subsequent alignment stages, the system continuously monitors the sliding window statistics of the regression residuals and scale parameters. When it detects that the residuals continuously exceed the threshold or the scale drift accumulates beyond the limit, it triggers the reference frame review process. This allows for the re-execution of candidate evaluation and replacement of the reference frame without altering the structure described above. For example, in segments with drastic changes in sunlight conditions, scenes entering large occlusions, or sudden changes in sensor exposure strategies, this mechanism can significantly reduce the cascading effects of long-term deviations.

[0217] It should be noted that regardless of whether the reference frame is updated, the reference depth map is always placed at the beginning of the consistent depth map sequence, and the remaining frames are arranged in a preset order to ensure that the reconstruction module can carry out Gaussian neural field modeling and output on a stable sequence organization.

[0218] Optionally, the alignment module includes a stability reference frame selection unit;

[0219] In this embodiment, the stability reference frame selection unit is used to perform orientation-scale level stability evaluation on candidate frames within a preset time window before setting the reference depth map, thereby selecting a stable frame as the first target frame image, and setting the optimized depth map of this frame output by the optimization module as the reference depth map. This unit revolves around the output of the wavelet processing unit and the sparse reconstruction unit, and addresses the technical problem that: relying solely on image quality or single-frame geometric self-consistency to select a reference frame is difficult to reflect the continuity and consistency of cross-frame structural patterns, and is prone to introducing unstable benchmarks under rapid motion or non-rigid deformation conditions, leading to subsequent scale regression drift.

[0220] In practical implementation, the stability reference frame selection unit first determines the evaluation window and candidate set. The system indexes the input sequence based on the timestamp and selects a time window of appropriate length without changing the overall process; for example, 5 to 15 frames can be selected. Frames that are obviously underexposed, overexposed, or have an effective pixel ratio lower than the threshold are first removed to avoid invalid samples interfering with the evaluation.

[0221] Subsequently, the unit calls the wavelet processing unit and the sparse reconstruction unit respectively to obtain the coefficient groups and their amplitude information grouped by direction index and scale index for each candidate frame in the window, and generates a "direction-scale group activation map" accordingly.

[0222] The generation of activation maps follows a unified threshold and connectivity rule: the coefficient magnitudes in each direction and scale group are compared with the adaptive threshold to obtain binary or probabilistic activation masks; lightweight connectivity filtering and outlier cleanup are performed on the activation masks on the pixel grid; when necessary, effective pixel masks and detail energy maps are referenced to preserve slender, continuous real structures within the boundary bands to avoid being misjudged as noise.

[0223] In this way, each candidate frame has a set of spatially aligned and uniformly regulated orientation-scale group activation maps, which serve as the basis for cross-frame comparison.

[0224] During the stability measurement phase, the unit takes the candidate frame as the center and compares it with its neighboring frames one by one with the same direction-scale group to calculate at least one stability index.

[0225] For example, the first type of metric is the activation overlap ratio: between the activation maps of the same direction-scale group of the candidate frame and the adjacent frame, the proportion of the overlapping area to the activation area of ​​the candidate frame is counted; in order to improve the adaptability to motion, lightweight alignment based on optical flow or feature matching can be used before comparison to compensate for local displacement.

[0226] The second type of metric is the number of active frames: track the active direction-scale groups of candidate frames along the time axis, and count the number of frames that remain active continuously within the window. Groups that flicker briefly will be considered unstable.

[0227] The third type of indicator is the consistency score: This involves statistically analyzing the response amplitude and spatial distribution of the same direction-scale group within the window. Groups with smaller response amplitude fluctuations and higher spatial distribution coherence receive higher scores. In areas with significant deformation or rapid motion, the sensitivity of the score to the absolute magnitude of the amplitude can be appropriately reduced, relying more on coherence measurements. At least one of these three types of indicators should be used, preferably two or more simultaneously, to improve the robustness of the judgment.

[0228] After obtaining the stability metrics, the stability reference frame selection unit comprehensively evaluates and ranks the candidate frames. During the evaluation, metrics of different dimensions and polarities are first normalized and then weighted and summed according to preset evaluation rules to obtain the comprehensive stability score of the candidate frames. Evaluation rules can be loaded according to scene level: for example, in indoor scenes with weak textures, the weight of activation duration is increased to ensure that the reference frame comes from a stable background; in outdoor scenes with strong textures and fast camera movement, the weight of activation overlap ratio and consistency score is increased to reduce occasional misjudgments caused by rapid changes in perspective; in scenes where non-rigid areas such as people or clothing account for a large proportion, region masks can be enabled, and only groups of static backgrounds or low-deformation areas participate in the evaluation.

[0229] To avoid outliers dominating the rankings, a baseline constraint is set for each unit: when the activation overlap ratio or consistency score is below the baseline, the candidate frame is not included in the final comparison. After sorting, candidate frames whose comprehensive scores meet the preset conditions are selected as the first target frame images, and their optimized depth maps are set as the reference depth maps. In case of ties, the frame with the higher activation duration, higher overlap ratio, and better consistency score are compared in that order. If they still cannot be distinguished, the candidate frame with the earlier time frame is selected to ensure the determinism of the process.

[0230] The stability reference frame selection unit maintains interface consistency with other modules during execution: the input consists of the target frame image of each candidate frame within the window, the direction-scale group coefficients and detail energy map derived by the wavelet processing unit and the sparse reconstruction unit, the effective pixel mask, and the necessary optical flow or feature correspondence; the output consists of the frame identifier and stability evaluation record of the first target frame image, including the window range, the indicators used, the normalization parameters, the statistical values ​​of each indicator, the comprehensive score, and the ranking results.

[0231] For example, in indoor monitoring scenarios, reference frames are often dominated by background walls and the ground, and their orientation-scale groups maintain a high degree of overlap and long duration within the window, resulting in a higher overall score. In outdoor mobile shooting, reference frames are more likely to come from building facades or ground areas, and their orientation-scale groups show a stable banded distribution over time, resulting in a better consistency score.

[0232] In this way, the selection of reference frames no longer depends solely on the quality of a single frame or local geometric self-consistency, but is based on the mode stability across frames, scales, and directions, thereby reducing the instability factors in subsequent scale regression and reconstruction from the source.

[0233] Optionally, the alignment module includes a consistency reference frame selection unit, used to evaluate the consistency of candidate frame images within a preset time window based on the solution cost of the sparse reconstruction process and the scale stability statistics of the alignment stage before setting the reference depth map, thereby selecting the first target frame image and setting the optimized depth map output by the optimization module of this frame as the reference depth map. The technical problem addressed by this unit is that relying solely on image quality or geometric consistency is insufficient to reflect the overall difficulty of sparse reconstruction and cross-frame scale regression, and it is easy to select frames with "abnormally high cost and unstable scale" as the benchmark, leading to cascading errors in subsequent consistency and reconstruction.

[0234] In practice, the consistency reference frame selection unit first determines the evaluation window and candidate set. The system establishes an index for the input sequence based on the timestamp, selects a sliding time window of appropriate length, for example, 5 to 15 frames; frames with significantly low effective pixel ratios, severely overexposed / underexposed images, or substandard quality detection are first eliminated, and the remaining frames are used as candidate sets for consistency evaluation.

[0235] Subsequently, the sparse reconstruction unit is called one by one to generate a reconstruction depth map for each candidate frame, and the solution cost index corresponding to this reconstruction is extracted simultaneously, including at least two of the following: the cumulative value of group sparsity penalty, the cumulative value of graph smoothing penalty, and the normal consistency residual.

[0236] Among them, the cumulative value of the group sparsity penalty reflects the sparsity difficulty of the frame in the orientation-scale group. The larger the value, the more high-frequency structures need to be preserved or the more difficult it is to suppress noise. The cumulative value of the graph smoothing penalty reflects the degree of harmony of the frame in local geometric coherence. The larger the value, the more surface undulations or breaks there are, and more smoothing is needed to maintain usable geometry. The normal consistency residual reflects the residual between the point cloud normal and the pixel normal obtained by backprojection of the reconstructed depth map. The larger the value, the worse the geometric self-consistency of the frame and the higher the risk of using it as a reference.

[0237] To enhance comparability, the cost indicators are standardized and their polarities are unified so that they can participate in comprehensive judgment under the same dimension.

[0238] After obtaining the solution cost, the consistency reference frame selection unit simultaneously reads the scale stability statistics maintained by the alignment module within the time window, including the magnitude statistics of scale parameter changes and the central tendency of the regression residuals.

[0239] By combining "sparse reconstruction cost" and "scale stability statistics," candidate frames are compared and sorted according to preset weights or priorities. For example, in indoor scenes with weak textures and slow camera movement, the weights of group sparsity penalty and normal consistency remainder can be increased to prioritize frames with low sparsity difficulty and good geometric consistency. In outdoor scenes with strong textures or fast camera movement, the weights of scale parameter changes and regression residuals can be increased to prioritize frames with more stable cross-frame scale relationships. In scenes with significant non-rigid deformation, the impact of graph smoothing penalty can be appropriately reduced to avoid misjudging natural undulations in deformed regions as poor geometry.

[0240] In addition, to avoid occasional anomalies dominating the results, a joint threshold condition is set: for example, if any adopted solution cost index exceeds its preset upper limit, or if the scale parameter change is continuously higher than the threshold within the window, then the candidate frame will not enter the final comparison; when multiple candidate frames have similar comprehensive evaluations, the decision can be made in the order of "scale stability first, then normal consistency comparison, and then group sparsity cost comparison".

[0241] To improve operational robustness, the consistency reference frame selection unit supports adaptive weighting and backoff strategies. When an increase in overall scale dispersion is detected within a window, the weight of scale stability statistics is automatically increased; when the overall solution cost is generally high, the weight of normal consistency remainder is automatically increased to prioritize the exclusion of frames with poor geometric consistency.

[0242] For example, in segments with significant illumination changes or strong occlusion, the cell can trigger a "conservative mode," shortening the evaluation window and increasing the stringency of the joint threshold to avoid short-term disturbances affecting the determination of the reference frame. In offline high-precision reconstruction, the window can be lengthened and a smoother weight curve can be used to obtain a more stable comprehensive score. After sorting, the candidate frame that meets the joint threshold condition and has the best ranking is selected as the first target frame image, and the optimized depth map of this frame is set as the reference depth map. At the same time, the window range, various costs and statistics, normalization method, weight setting, joint threshold, and final ranking used in this decision are recorded in the metadata for subsequent alignment and reconstruction processes to be traced and reused.

[0243] The consistent reference frame selection unit maintains an online closed loop with the sparse reconstruction unit and alignment module: when the reconstruction module reports that the reprojection error and gap rate continue to deteriorate over time, it can actively request this unit to re-execute the evaluation within the latest window and replace the reference frame when the trigger condition is met; after the reference frame is updated, this unit places the new reference depth map at the beginning of the consistent depth map sequence, and the remaining frames are arranged in a preset order to ensure that the reconstruction module can continue to model and output the Gaussian neural field on a stable sequence organization.

[0244] In this way, the consistency reference frame selection unit selects reference frames by combining the solution cost with the scale stability criterion, avoiding the use of frames that are "difficult to reconstruct and scale unstable" as the benchmark, and effectively reducing the cascading risk of subsequent alignment and reconstruction.

[0245] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A dynamic scene reconstruction system based on a Gaussian neural field with depth consistency constraints, characterized in that, include: An estimation module is used to perform depth estimation on a target frame image in an input video sequence to generate an initial depth map of the target frame image; wherein, the target frame image is any frame image in the video sequence; The calculation module reconstructs the 3D point cloud corresponding to the target frame image based on the initial depth map and the camera's intrinsic and extrinsic parameters, and calculates the point cloud normal vector corresponding to the target frame image from the 3D point cloud, and generates the pixel normal vector corresponding to the target frame image based on the target frame image. The optimization module iteratively optimizes the initial depth map of the target frame image based on the consistency constraint between the pixel normal vector and the point cloud normal vector, and outputs the optimized depth map of the target frame image. The alignment module is used to set the optimized depth map of the first target frame image as the reference depth map; and for any target frame image in the video sequence other than the first target frame image, to establish a regression model with the optimized depth map of the target frame image and the reference depth map, to determine the scale parameter, and to use the scale parameter to perform scale transformation on the optimized depth map of the target frame image, and output the uniform depth map of the target frame image. The reconstruction module generates a Gaussian neural field model based on the uniform depth map sequence and outputs a three-dimensional representation of the dynamic scene corresponding to the video sequence; wherein the uniform depth map sequence places the reference depth map first, and the uniform depth maps of the remaining target frame images are arranged in a preset order.

2. The dynamic scene reconstruction system based on a Gaussian neural field with depth consistency constraints according to claim 1, characterized in that, The calculation module includes a wavelet processing unit; The wavelet processing unit is configured as follows: Perform at least one layer of discrete wavelet decomposition on the initial depth map to obtain approximate sub-bands and multiple detail sub-bands; The detail subbands are thresholded, and a depth map is reconstructed based on the approximate subbands and the thresholded detail subbands. The reconstruction of the 3D point cloud takes the reconstructed depth map as input, and the weighting coefficients are determined according to the energy of the detail subbands in the calculation of the point cloud normal vector. Weighted plane fitting is performed on the neighborhood of the point cloud to calculate the point cloud normal vector.

3. The dynamic scene reconstruction system based on a Gaussian neural field with depth consistency constraints according to claim 2, characterized in that, The calculation module also includes a direction response unit; The direction response unit is configured as follows: Directional wavelet filtering is performed on the target frame image to obtain a local principal direction response, and the pixel normal vector is calculated based on the local principal direction response.

4. The dynamic scene reconstruction system based on a Gaussian neural field with depth consistency constraints according to claim 1, characterized in that, The first target frame image is the first frame image in the time sequence of the video sequence.

5. The dynamic scene reconstruction system based on a Gaussian neural field with depth consistency constraints according to claim 2, characterized in that, The wavelet processing unit employs a multidimensional wavelet basis, which is at least one of a steerable wavelet basis, a dual-tree complex wavelet basis, and a three-dimensional tensor product wavelet basis. The wavelet processing unit is configured as follows: Perform multidimensional discrete wavelet decomposition on the initial depth map. The multidimensional discrete wavelet decomposition includes: two-dimensional spatial decomposition or three-dimensional decomposition of the initial depth map sequence stacked in a preset order to obtain detail subbands with orientation index and scale index. The directional weighting coefficients of the point cloud neighborhood are calculated based on the amplitude or energy of the detailed subband, and the plane fitting of the point cloud neighborhood is weighted by the directional weighting coefficients to calculate the point cloud normal vector.

6. The dynamic scene reconstruction system based on a Gaussian neural field with depth consistency constraints according to claim 5, characterized in that, The computation module includes a sparse reconstruction unit; the sparse reconstruction unit is configured as follows: The initial depth map is divided into blocks according to a preset window size and overlap step size, and each block is flattened into a block vector in a fixed order. Multidimensional wavelet transform is performed on each block vector to obtain a coefficient vector, and the coefficient vector is grouped according to the direction index and scale index to form multiple coefficient groups. Under the premise of satisfying data consistency constraints and group sparsity constraints, the reconstruction results are solved by iterative update method for each coefficient group to obtain reconstruction depth blocks. The reconstruction depth blocks are then weighted and stitched together according to the overlap relationship to generate a reconstruction depth map. The reconstruction depth map is used for 3D point cloud reconstruction and point cloud normal vector calculation.

7. The dynamic scene reconstruction system based on a Gaussian neural field with depth consistency constraints according to claim 6, characterized in that, The sparse reconstruction unit further includes a graph construction subunit, a constraint application subunit, and a projection subunit; The graph construction subunit is used to construct a weighted adjacency graph with pixels as nodes. The edge weights of the adjacency graph are determined by the spatial distance between pixels and the change in the scale parameter output by the alignment module. The constraint application subunit is used to introduce graph smoothing penalty and normal consistency constraint during the iterative update process. The normal consistency constraint is measured by the difference between the point cloud normal vector obtained from the 3D point cloud reconstructed by the reconstructed depth map and the pixel normal vector. Non-negativity constraint is applied to the coefficient elements in the preset index set and boundary gradient upper bound constraint is applied to the reconstructed depth map. The boundary gradient upper bound is determined based on the camera parameters and the edge mask of the target frame image. The projection subunit is used to perform the projection operation of the nonnegative constraint and the upper bound constraint of the boundary gradient after each iteration.

8. The dynamic scene reconstruction system based on a Gaussian neural field with depth consistency constraints according to claim 1, characterized in that, The alignment module includes a reference frame selection unit; The reference frame selection unit is configured to: before setting the reference depth map, take multiple candidate frame images in the video sequence as objects, and calculate at least two of the following frame-level evaluation metrics respectively: The effective pixel ratio or confidence statistics are obtained based on the initial depth map output by the estimation module. The quantile statistics of the consistency residual between the pixel normal vector obtained by the calculation module and the point cloud normal vector reconstructed from the initial depth map and the camera's intrinsic and extrinsic parameters; Based on the regression model of the alignment module, the dispersion of the scale parameter is obtained by establishing a regression relationship with adjacent frame images within a preset time window. High gradient pixel ratio obtained based on the edge mask of the target frame image; The evaluation indicators are weighted according to preset weights to obtain a comprehensive evaluation value. Frame images whose comprehensive evaluation values ​​meet preset conditions are selected from the candidate frame images as the first target frame images. The reference depth map is set as the optimized depth map of the first target frame image.

9. The dynamic scene reconstruction system based on a Gaussian neural field with depth consistency constraints according to claim 6, characterized in that, The alignment module includes a stability reference frame selection unit; The stability reference frame selection unit is configured to: before setting the reference depth map, for candidate frame images within a preset time window, call the wavelet processing unit and the sparse reconstruction unit respectively to obtain coefficient groups grouped by direction index and scale index, and generate a direction-scale group activation map of the candidate frame based on the coefficient groups. Calculate at least one stability metric among the candidate frame and its neighboring frames in the same orientation-scale group, the activation overlap ratio, the number of frames with sustained activation, or the consistency score. Candidate frames that meet the stability conditions are selected as the first target frame image according to the preset evaluation rules, and the reference depth map is set as the optimized depth map of the first target frame image.

10. A dynamic scene reconstruction system based on a Gaussian neural field with depth consistency constraints according to claim 6, characterized in that, The alignment module includes a consistency reference frame selection unit; The consistency reference frame selection unit is configured to: before setting the reference depth map, for candidate frame images within a preset time window, call the sparse reconstruction unit to obtain the reconstruction depth map, and count at least two of the following solution cost indicators: cumulative value of group sparse penalty, cumulative value of graph smoothing penalty, and residual value of normal consistency. By combining the statistical values ​​of the scale parameter changes or regression residuals of the alignment module within a preset time window, and comparing them according to preset weights or priority order, candidate frames that meet the joint threshold conditions are selected as the first target frame image, and the reference depth map is set as the optimized depth map of the first target frame image.

Images