A hybrid expert gaussian three-dimensional modeling and compression method for robot dynamic scene perception

Abstract

The application discloses a kind of hybrid expert Gauss three-dimensional modeling and compression method for robot dynamic scene perception, comprising the following steps, obtaining multi-view video sequence;Multi-view video sequence is preprocessed, and multiple frames of continuous 3D point cloud data are obtained;Multi-view image rendering model is constructed, and the image under any view is rendered;Multi-view image rendering model is trained, and the trained multi-view image rendering model is obtained;The specified view is input into the trained multi-view image rendering model, and the rendering of the image under any view is realized.The method can flexibly trade off rendering fidelity and bit rate according to application requirements, realizing true end-to-end compression modeling.The scheme proposed by the application significantly reduces the model size while maintaining high-fidelity real-time rendering performance, providing reliable technical support for robot dynamic perception and long-term mapping in resource-constrained environments.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, 3D reconstruction and robot perception technology, specifically involving a dynamic 3D Gaussian representation, training strategy and end-to-end compression method based on a hybrid time expert gating mechanism and anchor-driven structure. Background Technology

[0002] With the rise of 3D Gaussian Splatting (3DGS) technology, significant breakthroughs have been achieved in the real-time performance and rendering quality of 3D scene reconstruction. However, existing technologies face enormous challenges in dynamic scene perception tasks for robots. Dynamic scenes not only need to represent the geometry and appearance of space, but also require accurate modeling of object deformation and motion in the temporal dimension. Existing deformation field-based methods typically assign independent deformation parameters to each Gaussian primitive, which leads to a linear explosion in the size of the 3D reconstruction model with scene complexity and temporal length, greatly limiting its application on mobile robot platforms with limited storage resources.

[0003] Traditional dynamic modeling methods often employ a single, small multilayer perceptron (MLP) to capture motion throughout the entire timeframe. Due to the limited capacity of a single network, it is often difficult to handle dynamic scenes with complex nonlinear motion or long-term variations. While introducing a Mixture of Experts (MoE) model can theoretically improve model capacity, when processing continuous time streams, the traditional hard gating mechanism is highly susceptible to "expert collapse," where a few experts are overactivated while others fail to be effectively trained, leading to instability in dynamic modeling.

[0004] In dynamic scene reconstruction, background or quasi-static regions are often incorrectly introduced with minute deformations, resulting in visual flickering and geometric jitter. Furthermore, how to quantify and optimize the bitrate used by the model through mathematical means while ensuring rendering accuracy (distortion), achieving the optimal rate-distortion trade-off, is a common challenge that urgently needs to be solved in the field of efficient robotic perception.

[0005] Most existing 3DGS compression technologies are designed for static scenes, ignoring the structured redundancy of dynamic deformation parameters; while existing dynamic modeling methods lack an end-to-end compression framework that integrates with information theory, resulting in high bit rates for 3D reconstruction models that cannot meet the needs of robots for real-time transmission and storage. Summary of the Invention

[0006] To address the issues that existing 3DGS compression technologies mostly target static scenes and neglect the structured redundancy of dynamic deformation parameters, and that existing dynamic modeling methods lack an end-to-end compression framework integrated with information theory, resulting in high bit rates in 3D reconstruction models that cannot meet the real-time transmission and storage requirements of robots, this invention provides a hybrid expert Gaussian 3D modeling and compression method for robot dynamic scene perception, comprising the following steps:

[0007] Acquire multi-view video sequences; Preprocessing of multi-view video sequences yields multiple frames of continuous 3D point cloud data; Construct a multi-view image rendering model to render images from any viewpoint; The multi-view image rendering model is trained to obtain a trained multi-view image rendering model. By inputting a specified camera viewpoint into a trained multi-view image rendering model, the rendering of images from any viewpoint can be achieved.

[0008] Furthermore, the multi-view image rendering model includes: Preprocessing module: preprocesses multi-view video sequences, extracts scene features, and obtains multi-frame continuous 3D point cloud data; First prediction network: Based on the multi-frame continuous 3D point cloud data output by the preprocessing module, it constructs a canonical space containing sparse motion anchors. Each anchor is associated with several Gaussian elements, and each anchor is parameterized by explicit Gaussian attributes and latent feature vectors. It is used to predict the residual attributes of the Gaussian elements associated with the anchor. The second prediction network is used to predict the motion increment of the anchor points transmitted by the first prediction module, which have explicit Gaussian properties and latent eigenvectors, and output deformation residuals to drive the motion of the anchor points.

[0009] Furthermore, the second prediction network includes a prediction network of multiple parallel time expert networks. A normalized Gaussian kernel time function is used to calculate the mixed weights of each expert at continuous time t. The predictions of the motion increment of the anchor point are aggregated by weighted aggregation of each expert network. Then, a lightweight multilayer perceptron is used in combination with position embedding and temporal embedding to predict color and opacity residuals at the Gaussian level, so as to correct the detail deviations caused by the coarse-scale deformation stage.

[0010] Furthermore, a Gaussian kernel bandwidth annealing mechanism is introduced to train the multi-view image rendering model, thereby adjusting the Gaussian kernel width parameter. It gradually decreases as the number of iterations increases; In the Gaussian kernel bandwidth annealing mechanism, the first... Bandwidth at the next iteration satisfy:

[0011] in, > ,in Indicates the starting bandwidth. Indicates the end of bandwidth. This represents the total number of iterations. Used to control the annealing rate; Training the multi-view image rendering model also includes dynamic anchor point growth and pruning steps: calculating the cumulative positional gradient value of Gaussian elements across frames as a temporal importance indicator to guide anchor points to grow in areas of intense motion or prune in areas of extremely low transparency.

[0012] Furthermore, in the canonical space initialization, the properties of the Gaussian meta-elements are derived in the following way: Using the first prediction network Based on the reference features and residual features of the anchor point, the residual terms of position, covariance, color, and opacity are predicted. The Gaussian meta-attributes are obtained by applying the predicted residuals to the explicit attributes of the anchor point.

[0013] Furthermore, the hybrid weights The calculation formula is:

[0014] in, For the input timestamp or time code, For the first The central position of a time expert The total number of time experts, To control the Gaussian kernel bandwidth of the mixed breadth, j is just a sequence number used to indicate the quantity M.

[0015] Furthermore, the objective function of the multi-view image rendering model includes: Static consistency constraint loss function: Identify the set of static anchor points whose motion amplitude is less than the amplitude threshold by calculating the exponential moving average of the anchor point displacement, and apply displacement regularization constraint to the set of static anchor points; Rate-distortion joint compression loss function: Establish a rate-distortion joint optimization objective function, use a multidimensional entropy model to estimate the bit rate of quantization parameters, and collaboratively optimize reconstruction quality and model bit rate.

[0016] Furthermore, the objective function of the multi-view image rendering model The expression is as follows:

[0017] in, Based on Compared to the reconstruction loss of SSIM, The expected code length is calculated based on the multidimensional entropy model. and These are the weighting coefficients. Loss function for static consistency constraints

[0018] and The objective function is a joint optimization of rate distortion.

[0019] A hybrid expert Gaussian 3D modeling and compression system for robot dynamic scene perception includes: Acquisition module: Used to acquire multi-view video sequences; Builder module: Used to build multi-view image rendering models and render images from any viewpoint; Training module: Used to train the multi-view image rendering model to obtain a trained multi-view image rendering model; Implementation module: Used to input the specified camera viewpoint into the trained multi-view image rendering model to render images from any viewpoint.

[0020] A robot includes a stored program module that runs in a processor to implement the method as described in any one of the above.

[0021] The invention provides a hybrid expert Gaussian 3D modeling and compression method and system for robot dynamic scene perception. While ensuring high-quality reconstruction of dynamic scenes, the system significantly reduces the storage and transmission bit rate and effectively suppresses timing jitter in dynamic modeling, making it suitable for resource-constrained robot platforms. Compared with the prior art, the present invention has achieved beneficial effects in several aspects: First, by introducing an anchor-driven representation structure, it is possible to reduce massive Gaussian parameters to a sparse anchor level, thereby eliminating the redundancy of dynamic scene representation from a structural perspective and greatly improving the compression potential.

[0022] Secondly, the hybrid time expert network based on normalized Gaussian kernel gating effectively solves the problem of insufficient modeling capability of a single network. Combined with the annealing strategy, it not only enhances the capture accuracy of complex nonlinear motion, but also avoids the expert crash phenomenon commonly found in traditional MoE architectures.

[0023] Furthermore, by optimizing the coupling rate-distortion objective in the training expert network, this method can flexibly balance rendering fidelity and bit rate according to application requirements, achieving true end-to-end compression modeling.

[0024] Furthermore, by utilizing the static consistency constraint driven by exponential moving average, this invention fundamentally solves the common problems of visual flickering and background jitter in dynamic representation, significantly improving the stability of the robot's perceived map.

[0025] Finally, the proposed solution significantly reduces the model size while maintaining high-fidelity real-time rendering performance, providing reliable technical support for robot dynamic perception and long-term mapping in resource-constrained environments. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of the method described in this application; Figure 2 This is a schematic diagram of the hybrid expert Gaussian 3D modeling and compression system for dynamic scene perception as described in Embodiment 1 of the present invention; Figure 3 This is a flowchart of the hybrid expert Gaussian 3D modeling and compression method for dynamic scene perception as described in Embodiment 2 of the present invention; Figure 4 This is a diagram showing the overall network structure of the hybrid time expert (MoE) deformation module and the anchor-driven representation in the method described in Embodiment 2 of the present invention. Figure 5 The images show a comparison of the dynamic rendering effects of the present invention on the HyperNeRF dataset; where (a) is the image processed by the method of this application, and (b) is the actual image value. Figure 6 The images show a comparison of the dynamic rendering effects of the present invention on the Neu3D dataset, where (a) is the image processed by the method of this application, and (b) is the actual image value. Detailed Implementation

[0028] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Figure 1 This is a flowchart of the method described in this application; A hybrid expert Gaussian 3D modeling and compression method for robot dynamic scene perception includes the following steps: S1: Obtain multi-view video sequences; S2: Preprocess the multi-view video sequence to obtain multi-frame continuous 3D point cloud data; S3: Construct a multi-view image rendering model to render images from any viewpoint; S4: Train the multi-view image rendering model to obtain a trained multi-view image rendering model; S5: Input the specified camera viewpoint into the trained multi-view image rendering model to render images from any viewpoint.

[0031] Furthermore, the multi-view image rendering model includes: Preprocessing module: preprocesses multi-view video sequences, extracts scene features, and obtains multi-frame continuous 3D point cloud data; First prediction network: Based on the multi-frame continuous 3D point cloud data output by the preprocessing module, it constructs a canonical space containing sparse motion anchors. Each anchor is associated with several Gaussian elements, and each anchor is parameterized by explicit Gaussian attributes and latent feature vectors. It is used to predict the residual attributes of the Gaussian elements associated with the anchor. The second prediction network is used to predict the motion increment of the anchor points transmitted by the first prediction module, which have explicit Gaussian properties and latent eigenvectors, and output deformation residuals to drive the motion of the anchor points.

[0032] Furthermore, in the canonical space initialization, the properties of the Gaussian meta-elements are derived in the following way: Using the first prediction network Based on the reference features and residual features of the anchor point, the residual terms of position, covariance, color, and opacity are predicted. The Gaussian meta-attributes are obtained by applying the predicted residuals to the explicit attributes of the anchor point.

[0033] A joint approach combining sparse motion anchors and neural Gaussian regression is employed for scene representation. The first prediction network constructs a set of sparse, learnable anchors within a canonical space, significantly reducing storage overhead through parameter sharing at the anchor level. First, the latent feature vectors associated with the anchors are input into the regression network. Based on reference and residual features, this network uses a multilayer perceptron (MLP) to predict the positional offset ΔX of the associated primitives relative to the anchors, the covariance matrix increment ΔΣ, the color residual ΔC, and the opacity O. Finally, the absolute geometric properties of the primitives are obtained by linearly superimposing the explicit anchor properties with the aforementioned regression residuals. This design compresses a massive number of Gaussian regressor properties into a very small number of anchor features, achieving an efficient and compact representation of scene features.

[0034] The second prediction network includes multiple parallel time expert networks. A normalized Gaussian kernel time function is used to calculate the mixed weights of each expert at continuous time t. The predictions of the motion increment of the anchor point are aggregated by weighted aggregation of each expert network. Then, a lightweight multilayer perceptron is used in combination with position embedding and temporal embedding to predict color and opacity residuals at the Gaussian level, so as to correct the detail deviations caused by the coarse-scale deformation stage.

[0035] Furthermore, a Gaussian kernel bandwidth annealing mechanism is introduced to train the multi-view image rendering model, thereby adjusting the Gaussian kernel width parameter. It gradually decreases as the number of iterations increases; In the Gaussian kernel bandwidth annealing mechanism, the first... Bandwidth at the next iteration satisfy:

[0036] in, > ,in Indicates the starting bandwidth. Indicates the end bandwidth. This represents the total number of iterations. Used to control the annealing rate; Training the multi-view image rendering model also includes dynamic anchor point growth and pruning steps: calculating the cumulative positional gradient value of Gaussian elements across frames as a temporal importance indicator to guide anchor points to grow in areas of intense motion or prune in areas of extremely low transparency.

[0037] Furthermore, the hybrid weights The calculation formula is:

[0038] in, For the input timestamp or time code, For the first The central position of a time expert The total number of time experts, To control the Gaussian kernel bandwidth of the mixing breadth.

[0039] Furthermore, the objective function of the multi-view image rendering model includes: Static consistency constraint loss function: Identify the set of static anchor points whose motion amplitude is less than the amplitude threshold by calculating the exponential moving average of the anchor point displacement, and apply displacement regularization constraint to the set of static anchor points; Rate-distortion joint compression loss function: Establish a rate-distortion joint optimization objective function, use a multidimensional entropy model to estimate the bit rate of quantization parameters, and collaboratively optimize reconstruction quality and model bit rate.

[0040] Furthermore: the objective function of the multi-view image rendering model The expression is as follows:

[0041] in, Based on Compared to the reconstruction loss of SSIM, The expected code length is calculated based on the multidimensional entropy model. and These are the weighting coefficients. Loss function for static consistency constraints

[0042] and The objective function is a joint optimization of rate distortion.

[0043] A hybrid expert Gaussian 3D modeling and compression system for robot dynamic scene perception includes: Acquisition module: Used to acquire multi-view video sequences; Builder module: Used to build multi-view image rendering models and render images from any viewpoint; Training module: Used to train the multi-view image rendering model to obtain a trained multi-view image rendering model; Implementation module: Used to input the specified camera viewpoint into the trained multi-view image rendering model to render images from any viewpoint.

[0044] A robot includes a stored program module that runs in a processor to implement any of the methods described above.

[0045] Example 1, as Figure 2As shown, a hybrid expert Gaussian 3D modeling and compression system for robot dynamic scene perception is mainly designed to meet the real-time perception and efficient mapping needs of service robots, inspection robots, and intelligent sensing terminals in complex dynamic environments. It can achieve high-precision non-rigid motion modeling and high-fidelity dynamic view synthesis while ensuring extremely low data storage and transmission bandwidth.

[0046] At the hardware level, this embodiment uses multiple handheld monocular cameras or multi-camera arrays as the main sensors (corresponding to the acquisition modes of HyperNeRF and NEU3D datasets).

[0047] For handheld camera scenarios, the system processes multi-view single-channel RGB video streams with complex motion trajectories; For camera array scenarios, the system synchronously processes multiple video streams with overlapping fields of view.

[0048] The raw image sequence output by the sensor is timestamped by a preprocessing engine, mapping it to a continuous time domain. Furthermore, the camera intrinsic and extrinsic parameters and the initial sparse point cloud are solved using the Structure for Motion Restoration (SfM) algorithm, providing the system with spatiotemporally consistent multimodal input.

[0049] A hybrid expert Gaussian 3D modeling and compression method for robot dynamic scene perception, characterized by the following steps: S1: Obtain multi-view video sequences; S2: Preprocess the multi-view video sequence to obtain multi-frame continuous 3D point cloud data; S3: Construct a multi-view image rendering model to render images from any viewpoint; S4: Train the multi-view image rendering model to obtain a trained multi-view image rendering model; S5: Input the camera viewpoint to be tested into the trained multi-view image rendering model to achieve rendering of images from any viewpoint.

[0050] At the software level, in this embodiment, the system includes: a multi-view image rendering model whose input is a normalized image sequence and pose data, and whose output is a dynamic three-dimensional Gaussian bitstream that has been compressed end-to-end.

[0051] The multi-view image rendering model includes: Preprocessing module: preprocesses multi-view video sequences, extracts scene features, and obtains multi-frame continuous 3D point cloud data; First prediction network: Based on the multi-frame continuous 3D point cloud data output by the preprocessing module, it constructs a canonical space containing sparse motion anchors. Each anchor is associated with several Gaussian elements, and each anchor is parameterized by explicit Gaussian attributes and latent feature vectors. It is used to predict the residual attributes of the Gaussian elements associated with the anchor. The second prediction network is used to predict the motion increment of the anchor points transmitted by the first prediction module, which have explicit Gaussian properties and latent eigenvectors, and output deformation residuals to drive the motion of the anchor points.

[0052] The first prediction network uses a joint approach of sparse motion anchors and neural Gaussian regression to represent the scene. A set of sparse learnable anchors is constructed in the normal space. By sharing parameters at the anchor level, the storage overhead is significantly reduced. The specific process is as follows: First, the latent feature vector associated with the anchor point is input into a regression network. This network, based on reference features and residual features, uses a multilayer perceptron (MLP) to predict the positional offset of the associated primitives relative to the anchor point. Increment of covariance matrix Color Residual and opacity .

[0053] Ultimately, the absolute geometric properties of the primitives are obtained by linearly superimposing the explicit properties of the anchor points with the regression residuals mentioned above. This design compresses a massive number of Gaussian primitive properties into a very small number of anchor point features, achieving an efficient and compact expression of scene features.

[0054] The second prediction network predicts the time-varying deformation parameters of the anchor point based on a normalized Gaussian kernel gating mechanism. System instantiation. A parallel time expert network, where each expert is responsible for modeling a specific motion pattern. For the input time... The system calculates the radial basis function distance between the anchor point and the time center of each expert, generates Gaussian kernel weights, and obtains mixed weights after Softmax normalization. Finally, the motion displacement and geometric deformation of the anchor point are obtained by aggregating the deformation residuals predicted by each expert according to the weights, thus achieving smooth and accurate capture of nonlinear and non-rigid motion.

[0055] The multi-view image rendering model dynamically adjusts the response range of the hybrid experts during training. In the early stages of training, a large initial bandwidth is assigned to the Gaussian kernel to create strong coupling between experts and learn global coherent motion. As the iteration progresses, the bandwidth gradually decreases according to a preset function, making the experts tend to be time-localized, thereby enabling the model to depict fine dynamic details in the HyperNeRF dataset, such as object deformation and topological changes.

[0056] The objective function of the multi-view image rendering model includes: Static consistency constraint loss function: Identify a set of static anchor points with low motion intensity by calculating the exponential moving average of anchor point displacements and apply displacement regularization constraints to them; Rate-distortion joint compression loss function: Establish a rate-distortion joint optimization objective function, use a multidimensional entropy model to estimate the bit rate of quantization parameters, and collaboratively optimize reconstruction quality and model bit rate.

[0057] Rate-distortion optimization and compression employ a joint optimization strategy based on a multidimensional entropy model and differentiability. Model parameters (such as anchor features and expert network weights) are input into the entropy model to estimate their probability distribution, and the estimated negative log-likelihood is used as the bitrate loss term. By introducing a differentiable operator, storage costs can be perceived while optimizing reconstruction quality, achieving end-to-end model compression.

[0058] Static consistency constraints employ a displacement statistics strategy based on exponential moving average (EMA) to ensure the stability of the background region during dynamic modeling. By monitoring the cumulative displacement of each anchor point in real time, the system automatically identifies quasi-static anchor points belonging to the background and applies position regularization constraints to them, thereby eliminating the background "jelly effect" and visual flicker commonly seen when shooting with a handheld camera.

[0059] like Figure 3 As shown, the objective function of the multi-view image rendering model includes two main threads: a dynamic modeling thread and a rate-distortion optimization thread, to realize the complete processing flow from the original video to the compressed bitstream.

[0060] In the scene modeling phase, the anchor-driven process first performs voxel downsampling on the initial point cloud to determine the anchor point distribution within the canonical space. Each anchor point is assigned a high-dimensional latent feature tensor. To further reduce overhead, the anchor point attributes are mapped to specific neural Gaussian units through a first prediction network. This hierarchical representation not only captures large-scale global structures but also characterizes fine-grained surface textures through primitive residuals, while ensuring the compactness of the representation.

[0061] In the dynamic deformation stage, the second prediction network employs multiple expert networks to perform deformation prediction at anchor points. Through a normalized Gaussian kernel gating mechanism, the system can adaptively adjust the expert contribution based on the current timestamp. This nonlinear fusion approach effectively overcomes the capacity limitations of traditional single deformation networks in handling long-term, complex motions, significantly improving the reconstruction accuracy of dynamic scenes.

[0062] During the training optimization phase, Gaussian kernel bandwidth annealing is performed. In the early stages of training, a large bandwidth helps stabilize the camera trajectory and scene overlap under complex handheld camera trajectories; in the later stages of training, a smaller bandwidth is used to focus on minor corrections to human movements, such as those found in the NEU3D dataset. Simultaneously, by utilizing differentiable quantization techniques, the system generates high-fidelity rendering results within a limited bit budget, improving detail reproduction while compressing the model size.

[0063] In the static constraint phase, the EMA motion index of each anchor point is maintained. The optimization objective includes a static consistency loss, which is refined layer by layer through a pyramid weighting strategy, forcing the displacement of background anchor points on the time axis to tend to zero. This geometrically and statistically driven constraint approach can maintain the visual coherence of the background in noisy and occluded environments, significantly reducing visual redundancy.

[0064] Experimental results show that the proposed method achieves superior performance compared to baseline methods such as 3DGS, 4DGaussian, HyperReel, and DN-4DGS on publicly available dynamic scene datasets such as HyperNeRF and Neu3D. In terms of reconstruction completeness, accuracy, and rendering fidelity, the proposed method achieves a PSNR of 25.61 dB on the HyperNeRF dataset, a significant improvement over 3DGS; and a PSNR of 31.67 dB on the Neu3D dataset. Regarding model lightweighting, the proposed method demonstrates strong compression capabilities: on the HyperNeRF dataset, the model size is only 8.70 MB, reducing storage space by approximately 5-15 times compared to the deformed 4DGS method; while on the Neu3D dataset, the model size is further optimized to 8.13 MB, achieving a volume compression of 123-288 times compared to earlier dynamic modeling methods. Furthermore, it also demonstrates excellent rendering efficiency, achieving an average frame rate of 79-114 FPS, which is more than three times that of traditional deformation methods. It maintains stable camera tracking performance even in noisy and occluded real-world environments, effectively recovering fine-grained geometric details. In summary, this invention balances global consistency with efficiency and reconstruction accuracy, exhibiting good robustness and generalization ability, and possesses broad potential for practical applications.

[0065] In summary, the system proposed in this embodiment organically combines a hybrid expert architecture with anchor-driven Gaussian modeling, and innovatively introduces key technologies such as normalized Gaussian kernel gating, bandwidth annealing strategy, rate-distortion optimization, and EMA static constraints. The system can operate stably in dynamic scenes from either a handheld or array perspective, generating dense dynamic 3D models with high compression ratios and high fidelity, and has broad application prospects in video transmission, virtual reality, and mobile robot environmental perception.

[0066] Example 2, as Figure 4 As shown, a hybrid expert Gaussian 3D modeling and compression method for dynamic scene perception includes the following steps: S1. Obtain the RGB image sequence and associated parameters of the current scene, and preprocess the obtained data.

[0067] In practice, a continuous image sequence of the current scene is acquired using a handheld multi-view monocular camera or a multi-view camera array. For the HyperNeRF dataset, the system focuses on processing single-channel video streams acquired by the handheld device under uncontrolled trajectories; for the Neu3D dataset, high-throughput video streams from multiple perspectives are processed simultaneously. The acquired raw RGB image sequences are first spatiotemporally aligned. A linear mapping function is then used... Original shooting timestamp Normalization to continuous time domain Subsequently, the Structure for Motion Reconstruction (SfM) algorithm was used to calculate the camera's intrinsic and extrinsic trajectory and generate an initial sparse point cloud. The preprocessing included noise filtering, optical flow consistency detection, and voxel grid-based downsampling to extract key sparse anchor point sets while preserving the scene's topology. This provides a highly reliable input for establishing a hierarchical representation that combines implicit and explicit representations in the canonical space.

[0068] S2. Input the preprocessed data into the first prediction network, anchor-driven representation, and use a joint scheme of sparse motion anchors and neural Gaussian regression to model the scene.

[0069] The core of this step lies in constructing a hierarchical geometric representation that "represents surfaces from points," enabling the reconstruction of complex surfaces with extremely low parameter counts. Within the canonical space, each anchor point is associated with a latent feature vector that integrates local geometry and texture priors.

[0070] The first prediction network uses a lightweight multilayer perceptron (MLP) as the attribute regressor. The latent features of the anchor points are input into the regression network, which then predicts attributes through a nonlinear mapping. The positional offset of each associated sub-primitive relative to the anchor point Increment of covariance matrix Color Residual and opacity .

[0071] In practice, the final covariance matrix of the primitive is determined by the basic form of the anchor point and the regression result. The covariance matrix is ​​obtained by adding the results through exponential mapping to ensure its positive definite physical constraint. This anchor-driven mechanism avoids redundant independent storage of massive Gaussian points, efficiently reducing the essential features of dynamic scenes to a sparse anchor level, thereby supporting subsequent high-rate reduction at the underlying architecture.

[0072] S3. Construct a second prediction network for hybrid time-based expert deformation and use the hybrid expert network to predict the non-rigid deformation of the anchor point at different times.

[0073] This embodiment instantiates a method for handling highly nonlinear motion in dynamic scenes (such as violent rotation of human joints or dynamic evolution of topology). A parallel time expert network This module employs a gating mechanism based on a normalized Gaussian kernel. The system first calculates the normalized time step. With the time center of various experts The radial basis distance between them. To ensure extremely high continuity of the deformation field in the time domain, the response value is passed through a Gaussian kernel function. The mapping is performed, and the normalized weights are obtained through the Softmax operator. The final anchor point displacement is generated by a weighted aggregation of all expert predictions. This design effectively enhances the system's model representation capacity when handling object deformation in HyperNeRF scenes and long-term non-rigid motion in Neu3D scenes, achieving refined segmentation and robust fitting of motion laws.

[0074] S4. During the rendering stage, differentiable rasterization technology is used for view composition based on the deformed Gaussian properties, and a bandwidth annealing strategy is introduced to optimize the training process.

[0075] Bandwidth annealing is crucial for ensuring a smooth transition of expert networks from "global learning" to "local focus." During the training period, the bandwidth parameter... Follow the polynomial annealing curve Dynamic adjustments are made. In the early stages of training, larger... The value motivates all experts to participate in modeling, learning the shared rigid tone of the scenario; as the number of iterations increases, By gradually shrinking the pixel size, the gating mechanism is forced to select the expert most relevant to the current moment, thereby greatly improving the rendering fidelity of high-frequency dynamic details (such as fine folds in clothing or subtle manipulations of facial expressions). The system projects the deformed primitives onto a two-dimensional plane using a CUDA-accelerated differentiable rasterizer, generates a rendered image, compares it with the actual observation frame, and drives the entire network to perform end-to-end gradient updates.

[0076] S5. Execution rate distortion joint optimization: The code rate is estimated using a multidimensional entropy model and the model parameters are refined by combining a differentiable quantization strategy.

[0077] In practice, the system uses rate-distortion (RD) theory to perform extreme model compression. A multidimensional entropy model is also introduced. The probability density of anchor point latent features and expert network parameters is estimated online, and the estimated bitrate is then used. An explicit loss function is introduced. To address the issue of non-differentiability in quantization operations, this step introduces uniformly distributed noise into the forward propagation of training. To simulate quantization error, a straight-through estimator is used during backpropagation to maintain gradient flow. This is achieved by adjusting the Lagrange multipliers in the loss function. The system can compress the model size to an extremely small range of 8.13MB - 8.70MB while ensuring high-quality reconstruction with HyperNeRF achieving 25.61dB PSNR or Neu3D achieving 31.67dB PSNR. The compression ratio is more than 123 times higher than that of traditional methods.

[0078] S6. Introduce static consistency constraints and use exponential moving average statistics to refine the pose and geometry stability of the background region.

[0079] To address the common background region "drift" or "jelly effect" in dynamic reconstruction, this step introduces physical prior constraints. The system maintains an exponential moving average (EMA) statistic of displacement intensity for each anchor point. Through analysis Based on the distribution of data, a quantile algorithm is used to automatically identify quasi-static anchor points in the background region. A penalty term is then applied to these anchor points in the total loss function. This forces the predicted motion offset to approach zero. This mechanism effectively counteracts the non-physical deformation introduced by handheld cameras due to minor shake or sensor noise, ensuring the absolute stillness of the background in temporal rendering and eliminating visual flicker.

[0080] S7. Employ a dynamic growth and pruning strategy for anchor points to guide them to grow in areas of intense movement or prune in areas of extremely low transparency.

[0081] In practice, the system performs dynamic anchor point lifecycle management based on scene complexity. In geometrically complex regions or regions with drastic gradient changes, the system triggers an anchor point growth mechanism based on the accumulated positional gradient value, splitting to generate new anchor points to supplement details; in regions with extremely low opacity contribution, anchor point pruning is performed to maintain the compactness of the model.

[0082] S8. Output the current scene compressed bitstream, camera pose trajectory, and corresponding high-fidelity dynamic 3D model of the scene.

[0083] The method described in this embodiment can operate stably in indoor environments with complex dynamic disturbances. In different scenarios, whether it is a dense environment with a small spatial scale and frequent human activity, or an environment with drastic changes in lighting and various non-rigid deformation objects in an open area, the system can maintain a continuous and stable operating state.

[0084] In actual data acquisition using handheld cameras or camera arrays, this method enables real-time parallel processing of massive amounts of sensor data. A second prediction network ensures high-precision estimation of the deformation field, while dynamically updating the anchor-driven scene representation, thereby generating a dense, dynamic 3D map with extremely high temporal coherence. The resulting dynamic map possesses physical integrity in its overall structure and realistically reflects the geometric evolution and surface texture in the dynamic environment in local details (such as clothing wrinkles, facial expressions, and object topological changes). It can provide accurate four-dimensional spatial references for robot path planning, obstacle avoidance decisions, task execution, and human-robot interaction in dynamic environments.

[0085] The hybrid expert Gaussian modeling method proposed in this invention has been systematically validated and tested on authoritative dynamic datasets such as HyperNeRF and Neu3D. Experimental results on the HyperNeRF dataset show that this method can achieve stable pose tracking and dense dynamic reconstruction in challenging non-rigid scenes captured by handheld cameras, and exhibits excellent performance in both rendering quality and geometric accuracy. Overall, this method achieves a PSNR of 25.61dB, an SSIM of 0.784, and an LPIPS of 0.230 on the HyperNeRF dataset, with a model storage size of only 8.70MB. Figure 5 As shown, (a) is the image processed by the method of this application, and (b) is the actual image value; the generated rendering result can accurately restore the texture information of the scene at the level of detail, such as... Peel-Banana Continuous deformation during the peeling process in the scene Chicken The details of joint motion in the scene are clearly represented, which fully demonstrates that the method effectively improves the model capacity through the hybrid temporal expert module and maintains a high level of image generation and dynamic expression capability with extremely low bandwidth overhead.

[0086] like Figure 6As shown, (a) is the image processed by the method of this application, and (b) is the ground truth image. The reconstruction results of this method on the Neu3D dataset are compared with the ground truth. The results show that the system can effectively recover the overall structure and complex motion details of long-term scenes. cook_spinach and sear_steak In scenarios involving complex hand-eye interactions and appearance changes, this method achieves a PSNR of up to 31.67 dB and a real-time rendering speed of 114 FPS, compressing the file size by approximately 123 to 288 times compared to traditional methods. The spatial layout of the room, the motion trajectory of human joints, and the detailed evolution during the cooking process are all highly consistent with the real values, and the reconstructed map exhibits excellent dynamic consistency and temporal stability. These results demonstrate that this method can not only achieve high-precision scene modeling in virtual and standard dynamic environments, but also maintain the robustness of pose estimation and the global stability of the map background during long-term operation through bandwidth annealing and static consistency constraints.

[0087] In real-world testing, the system can operate stably under various lighting conditions and rapid motion disturbances. The generated dynamic dense map fully describes the evolution of geometric structures and the shapes of major moving objects in the environment. Its compact bitstream of approximately 8.13MB greatly reduces the storage and transmission threshold, providing reliable input data for the robot's subsequent dynamic path planning, flexible interactive operations, and real-time semantic understanding.

[0088] In summary, the hybrid expert Gaussian 3D modeling and compression method proposed in this invention demonstrates top-notch mapping results, extreme compression efficiency, and operational stability in both standard dynamic datasets and real-world environments. It can effectively achieve high-fidelity reconstruction and real-time rendering of complex dynamic scenes, providing solid support for robots' autonomous navigation, environmental perception, and remote digital interaction tasks in dynamic environments.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hybrid expert Gaussian 3D modeling and compression method for robot dynamic scene perception, characterized in that, Includes the following steps: S1: Obtain multi-view video sequences; S2: Preprocess the multi-view video sequence to obtain multi-frame continuous 3D point cloud data; S3: Construct a multi-view image rendering model to render images from any viewpoint; S4: Train the multi-view image rendering model to obtain a trained multi-view image rendering model; S5: Input the specified camera viewpoint into the trained multi-view image rendering model to render images from any viewpoint.

2. The hybrid expert Gaussian 3D modeling and compression method for robot dynamic scene perception according to claim 1, characterized in that, The multi-view image rendering model includes: Preprocessing module: preprocesses multi-view video sequences, extracts scene features, and obtains multi-frame continuous 3D point cloud data; First prediction network: Based on the multi-frame continuous 3D point cloud data output by the preprocessing module, it constructs a canonical space containing sparse motion anchors. Each anchor is associated with several Gaussian elements, and each anchor is parameterized by explicit Gaussian attributes and latent feature vectors. It is used to predict the residual attributes of the Gaussian elements associated with the anchor. The second prediction network is used to predict the motion increment of the anchor points transmitted by the first prediction module, which have explicit Gaussian properties and latent eigenvectors, and output deformation residuals to drive the motion of the anchor points.

3. The hybrid expert Gaussian 3D modeling and compression method for robot dynamic scene perception according to claim 1, characterized in that, The second prediction network includes multiple parallel time expert networks. A normalized Gaussian kernel time function is used to calculate the mixed weights of each expert at continuous time t. The predictions of the motion increment of the anchor point are aggregated by weighted aggregation of each expert network. Then, a lightweight multilayer perceptron is used in combination with position embedding and temporal embedding to predict color and opacity residuals at the Gaussian level, so as to correct the detail deviations caused by the coarse-scale deformation stage.

4. The hybrid expert Gaussian 3D modeling and compression method for robot dynamic scene perception according to claim 1, characterized in that, To train a multi-view image rendering model, a Gaussian kernel bandwidth annealing mechanism is introduced to adjust the Gaussian kernel width parameter. It gradually decreases as the number of iterations increases; In the Gaussian kernel bandwidth annealing mechanism, the first... Bandwidth at the next iteration satisfy: in, > ,in Indicates the starting bandwidth. Indicates the end of bandwidth. This represents the total number of iterations. Used to control the annealing rate; Training the multi-view image rendering model also includes dynamic anchor point growth and pruning steps: calculating the cumulative positional gradient value of Gaussian elements across frames as a temporal importance indicator to guide anchor points to grow in areas of intense motion or prune in areas of extremely low transparency.

5. The hybrid expert Gaussian 3D modeling and compression method for robot dynamic scene perception according to claim 1, characterized in that, During the initialization of the canonical space, the properties of Gaussian elements are derived in the following manner: Using the first prediction network Based on the reference features and residual features of the anchor point, the residual terms of position, covariance, color, and opacity are predicted. The Gaussian meta-attributes are obtained by applying the predicted residuals to the explicit attributes of the anchor point.

6. The hybrid expert Gaussian 3D modeling and compression method for robot dynamic scene perception according to claim 1, characterized in that, The mixed weight The calculation formula is: in, For the input timestamp or time code, For the first The central position of a time expert The total number of time experts, To control the Gaussian kernel bandwidth of the mixed breadth, j is just a sequence number used to indicate the quantity M.

7. The hybrid expert Gaussian 3D modeling and compression method for robot dynamic scene perception according to claim 1, characterized in that, The objective function of the multi-view image rendering model includes: Static consistency constraint loss function: Identify the set of static anchor points whose motion amplitude is less than the amplitude threshold by calculating the exponential moving average of the anchor point displacement, and apply displacement regularization constraint to the set of static anchor points; Rate-distortion joint compression loss function: Establish a rate-distortion joint optimization objective function, use a multidimensional entropy model to estimate the bit rate of quantization parameters, and collaboratively optimize reconstruction quality and model bit rate.

8. The hybrid expert Gaussian 3D modeling and compression method for robot dynamic scene perception according to claim 7, characterized in that: The objective function of the multi-view image rendering model The expression is as follows: in, Based on Compared to the reconstruction loss of SSIM, The expected code length is calculated based on the multidimensional entropy model. and These are the weighting coefficients. The loss function for static consistency constraints. and The objective function is a joint optimization of rate distortion.

9. A hybrid expert Gaussian 3D modeling and compression system for robot dynamic scene perception, characterized in that, include: Acquisition module: Used to acquire multi-view video sequences; Builder module: Used to build multi-view image rendering models and render images from any viewpoint; Training module: Used to train the multi-view image rendering model to obtain a trained multi-view image rendering model; Implementation module: Used to input the specified camera viewpoint into the trained multi-view image rendering model to render images from any viewpoint.

10. A robot, comprising a stored program module, characterized in that, The program module, when run in a processor, can implement the method as described in any one of claims 1-8.

Images