Method for constructing an explorable native three-dimensional world model based on a point cloud implicit representation

Abstract

The application discloses a method for constructing an explorable native three-dimensional world model based on point cloud implicit representation, which comprises the following steps: acquiring three-dimensional scene data, and generating point cloud data containing geometric information and appearance information based on the three-dimensional scene data; performing down-sampling processing and feature extraction on the point cloud data through an encoder to obtain point cloud implicit representation; in the space formed by the point cloud implicit representation, based on a given condition signal and noise samples, performing denoising processing through a denoising network to generate new potential points corresponding to the condition signal; taking the potential points corresponding to the known area as the condition signal to generate potential points of the unknown area, thereby expanding the three-dimensional world representation, and rendering based on the expanded potential points through the decoder to obtain multi-view consistent images. The application can fundamentally guarantee the multi-view consistency of the three-dimensional scene, realize high-quality rendering from any view angle in a single generation, and support progressive scene expansion.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a method for constructing an explorable native 3D world model based on point cloud hidden representation. Background Technology

[0002] World models, as a core artificial intelligence technology for simulating and understanding the operating mechanisms of the physical world, have significant application value in fields such as autonomous driving, robot interaction, and virtual reality. Among related technologies, a complete technical system from environmental perception to dynamic prediction has been constructed through the synergy of scene representation, generative models, and rendering techniques. Specifically, this system encompasses scene modeling based on 2D images / videos, 2.5D depth-image pairs, and native 3D representations (such as point clouds and 3D Gaussians), and utilizes generative methods such as diffusion models and flow matching to simulate world evolution. With technological advancements, its path has evolved from representations based on 2D pixels to methods attempting to introduce explicit 3D geometric constraints.

[0003] However, existing technical solutions all have significant limitations. Methods based on 2D representations, such as DriveDreamer and Sora, while capable of generating high-quality frames, suffer from a representation paradigm that contradicts the 3D nature of the physical world. The models can only implicitly learn 3D cues from training data, leading to object deformation and spatial distortion during multi-view generation. Furthermore, generating multiple views requires multiple independent forward computations, resulting in low efficiency. Methods based on 2.5D pixel-aligned representations, such as ViewCrafter, introduce geometric constraints by jointly predicting RGB and depth maps. However, depth, pose, and texture are deeply coupled through perspective transformations, making it difficult for neural networks to accurately learn this complex correspondence, resulting in inherent defects in 3D consistency. While native 3D generation models directly process 3D data, they are either limited to object-level generation and lack realistic textures, or can only output a coarse scene-occupying mesh, failing to support high-quality rendering. They also generally lack the ability to progressively expand the scene based on local observations, making it difficult to meet the needs of large-scale, coherent world simulations. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose a method for constructing an explorable native 3D world model based on point cloud implicit representation, aiming to overcome the inherent defects of existing world models in terms of 3D consistency, generation efficiency, and scene explorability. Specifically, existing technologies mainly suffer from the following problems: 1) Methods based on 2D pixel or 2.5D pixel aligned representations, because their representation paradigms contradict the 3D nature of the physical world, make it difficult to guarantee the 3D consistency of multi-view generation results; 2) Existing native 3D generation models are either limited to the object level and cannot handle large-scale scenes, or the generated scene representations are coarse and cannot support high-quality rendering; 3) Most models lack the ability to progressively explore and expand, and cannot coherently deduce and generate unknown large-scale environments based on local observations.

[0006] To achieve the above objectives, a first aspect of the present invention proposes a method for constructing an explorable native 3D world model based on point cloud latent representation, comprising: S1, acquiring 3D scene data and generating point cloud data containing geometric and appearance information based on the 3D scene data; S2, performing downsampling processing and feature extraction on the point cloud data through an encoder to obtain a point cloud latent representation, and upsampling and decoding the point cloud latent representation into renderable 3D primitives through a decoder; S3, in the space constituted by the point cloud latent representation, performing denoising processing through a denoising network based on a given conditional signal and noise samples to generate new potential points corresponding to the conditional signal; S4, using the potential points corresponding to known regions as the conditional signal to generate potential points in unknown regions, thereby expanding the 3D world representation, and rendering based on the expanded potential points through the decoder to obtain a multi-view consistent image.

[0007] In one embodiment of the present invention, S1 includes: S11, acquiring an RGB image sequence, a depth map sequence, and corresponding camera pose parameters of a three-dimensional scene; S12, based on the camera pose parameters, transforming the RGB image and depth map of each frame to a unified world coordinate system through back projection; S13, fusing the back projection results of all frames to generate color point cloud data containing three-dimensional coordinates and RGB colors.

[0008] In one embodiment of the present invention, S2 includes: S21, performing downsampling and feature extraction on the point cloud data by an encoder to obtain initial latent points, wherein the initial latent points contain position feature pairs; S22, applying predefined noise to the coordinates of the initial latent points and regularizing the features of the initial latent points to obtain a regularized latent representation of the point cloud; S23, performing adaptive upsampling and refinement on the latent representation of the point cloud by a decoder to upsample and decode the sparse latent points into three-dimensional Gaussian primitives with geometric and appearance attributes.

[0009] In one embodiment of the present invention, step S23 includes: S231, using an adaptive upsampling module, splitting each parent potential point into K child points using K learnable query vectors, and predicting the position residual and feature residual of each child point, thereby increasing the point cloud density; S232, using an adaptive fine-tuning module, predicting the position offset based on the features of the upsampled points, and fine-tuning the position of the points to restore the fine geometric structure; S233, directly converting the fine-tuned points into three-dimensional Gaussian units, each point corresponding to a Gaussian unit, whose position is determined by the fine-tuned coordinates, and other attributes are predicted by the point features through a lightweight network.

[0010] In one embodiment of the present invention, in step S231, the query vector first interacts with the parent point features to obtain context, and then predicts the relative displacement of each child point. and characteristic residuals : in, This indicates the point-to-query interaction module. and These are the coordinates and features of the parent potential point, respectively. and The first The coordinates and features of each sub-point.

[0011] In one embodiment of the present invention, S3 includes: S31, constructing a UNet network based on three-dimensional sparse convolution as a denoising network; S32, during the diffusion process, adding noise to the position and features of the potential points corresponding to the conditional signal simultaneously to obtain the denoised potential points; S33, during the reverse denoising process, inputting the denoised potential points and time steps into the denoising network to predict the velocity field, and training by minimizing the error between the predicted velocity and the target velocity to achieve joint denoising of position and features.

[0012] In one embodiment of the present invention, in S32, a given time step ,noise Potential points after adding noise Defined as: ,in The potential point is the one corresponding to the conditional signal.

[0013] In one embodiment of the present invention, S3 further includes: S34, at the start of training, optimizing the potential point positions by solving a linear assignment problem. With noise samples Matching relationship between To minimize the total Euclidean distance: ,in Indicates assignment to the first The location portion of the noise sample for each potential point.

[0014] In one embodiment of the present invention, in step S3, the conditional signal is obtained from the point cloud hidden representation by at least one of the following methods: randomly cropping a connected three-dimensional region from the point cloud hidden representation; randomly and uniformly sampling a portion of points in the entire point cloud hidden representation; first cropping the connected region, and then uniformly sampling within the connected region.

[0015] In one embodiment of the present invention, S4 includes: S41, taking a portion of the set of potential points corresponding to the currently known world as a condition signal; S42, concatenating the condition signal with noisy potential points, and fixing the portion corresponding to the condition signal throughout the denoising process; S43, executing S3 to generate a new potential point corresponding to the condition signal, the new potential point representing an unknown region; S44, concatenating the new potential point with the set of historically known potential points to expand the representation of the three-dimensional world.

[0016] The method of this invention can ensure three-dimensional consistency from multiple perspectives, support high-quality rendering from any perspective, and achieve explorable expansion of large-scale scenes through progressive generation.

[0017] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0018] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0019] Figure 1 A flowchart illustrating a method for constructing an explorable native 3D world model based on point cloud implicit representation, provided in an embodiment of the present invention; Figure 2 A schematic diagram of the overall framework of an explorable native 3D world model based on point cloud hidden representation provided in an embodiment of the present invention; Figure 3 The data flow diagram is provided for the method of constructing an explorable native 3D world model based on point cloud hidden representation in the embodiments of the present invention. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] Figure 1 This is a flowchart of an embodiment of the present invention for constructing an explorable native 3D world model based on point cloud hidden representation.

[0023] like Figure 1 As shown, the method for constructing an explorable native 3D world model based on point cloud hidden representation includes the following steps: S1, acquire 3D scene data, and generate point cloud data containing geometric and appearance information based on the 3D scene data. To construct an explorable native 3D world model, the first step is to acquire foundational 3D scene data for model training and inference, and then generate a point cloud representation suitable for subsequent processing. Specifically, this step involves acquiring 3D scene data describing the physical or virtual environment from external sources. This data typically includes both geometric and visual information about the scene. The 3D scene data can originate from various sensing devices (such as depth cameras, LiDAR, multi-view camera arrays, etc.), or from a pre-built 3D model database or a synthetic data generation process. After acquiring the data, it needs to be processed to form a point cloud that simultaneously encodes geometric and visual information. This point cloud consists of a series of discrete points in space, where each point is associated with at least a positional attribute (e.g., 3D coordinates) representing its 3D spatial location and a visual attribute (e.g., color, reflectivity) representing its visual appearance. In this way, the original 3D scene is transformed into a structured, discrete point set representation containing rich geometric and textural information, providing direct and complete input for subsequent compression, generation, and rendering processes.

[0024] As a specific implementation method, by fusing color images, depth maps, and corresponding camera pose parameters from an RGB-D sensor sequence, each frame of data can be transformed to a unified world coordinate system through back projection, thereby generating a fused coordinate system. With color The colored point cloud of information can be represented as ,in The number of representative points.

[0025] This step generates point cloud data with both geometric and appearance attributes by acquiring and processing 3D scene information from multiple possible data sources. This provides an accurate and complete input foundation for the entire world model, ensuring the reliability and richness of the data upon which the subsequent representation learning and generation processes depend.

[0026] S2, the point cloud data is downsampled and feature extracted by the encoder to obtain the point cloud hidden representation, and the point cloud hidden representation is upsampled and decoded into renderable 3D primitives by the decoder.

[0027] This step involves transforming point cloud data containing geometric and appearance information into a compact and renderable 3D representation. Specifically, the input point cloud data undergoes dimensionality reduction and feature encoding via an encoder configured to perform downsampling to reduce the number of points and extract key features, outputting a structured latent representation of the point cloud that captures the geometric structure and appearance attributes of the original scene in a low-dimensional form. Subsequently, a decoder processes this latent representation, configured to perform upsampling to increase the point density and predict the necessary geometric and appearance attributes for rendering based on the upsampled points, ultimately decoding to generate a set of renderable 3D primitives. This encoder-decoder framework aims to achieve efficient compression from high-dimensional point clouds to compact latent representations, and high-fidelity reconstruction from latent representations to high-quality, renderable 3D assets. The encoding process may include regularization of the latent representation to improve its robustness and generalization ability, while the decoding process may include fine-grained position adjustments and attribute predictions to recover details.

[0028] One implementation approach is to employ a point-Gaussian variational autoencoder architecture, where the encoder performs multi-level downsampling based on an improved point Transformer network and applies perturbations to the coordinates of potential points for regularization; the decoder progressively increases the point density through an adaptive upsampling module containing learnable query vectors and adjusts the point positions using a fine-tuning module, ultimately converting each point into a 3D Gaussian primitive with attributes of position, rotation, scaling, opacity, and spherical harmonic coefficients. These attributes are predicted by a lightweight network based on point features, and the training process can integrate multiple supervisory signals such as rendering loss and geometric alignment loss.

[0029] This step provides an efficient and structured potential space for subsequent 3D generative modeling by generating a compact point cloud hidden representation. At the same time, by decoding it into renderable 3D primitives, it ensures high-quality and highly consistent image synthesis from any viewpoint, thus laying a key representational foundation for building native 3D world models.

[0030] S3, in the space formed by the hidden representation of the point cloud, based on the given condition signal and noise samples, a denoising network is used to perform denoising processing to generate new potential points corresponding to the condition signal.

[0031] In the space formed by the hidden representation of the point cloud, based on a given conditional signal and noise samples, a denoising network is used to denoise and generate new potential points corresponding to the conditional signal. This step aims to learn and model the probability distribution of the compact point cloud hidden space, enabling the controllable generation of structured potential points from random noise within this hidden space.

[0032] Specifically, this technical solution involves constructing a generative model that learns the mapping relationship from noise distribution to target latent point distribution. The conditional signal guides the generation process and can be in the form of a partial latent point representation of a known region, thereby constraining the generated result to maintain geometric and semantic coherence with the known portion. The denoising network is configured to receive an initial state consisting of noisy samples and the conditional signal, and through an iterative or continuous denoising process, gradually remove noise and recover a new set of latent points that conforms to the target distribution and is consistent with the conditional signal. This process essentially performs a conditional generation task in the latent space, and its training objective is to enable the network to accurately predict the transition from a noisy state to a clean target state. In one specific implementation, a flow matching framework can be used to construct a UNet network based on three-dimensional sparse convolution as the denoising network. The network simultaneously performs joint denoising on the location and features of potential points. During training, the target potential points... and noise Linear interpolation is used to obtain noisy samples. The Internet and time The input is a target velocity, and the training objective is to minimize the difference between the predicted velocity and the target velocity. The difference between them, i.e., the optimization loss function Furthermore, to optimize the training trajectory, a distance-aware matching strategy can be introduced to align noise with target points. The conditional signal may include, but is not limited to, connected 3D regions cropped from the point cloud hidden representation, a set of points uniformly sampled in the scene, or a combination thereof. During generation, the conditional signal portion is fixed to guide the generation of unknown regions.

[0033] Through the aforementioned technical means, this step achieves high-quality and controllable 3D content generation within a highly compact point cloud latent space. It enables the model to infer and generate coherent unknown regions based on locally known information (conditional signals), providing a crucial foundation for the core explorable world simulation mechanism, while simultaneously ensuring the structural consistency of the generated results at the latent space level.

[0034] S4, using the potential points corresponding to the known region as the condition signal to generate potential points in the unknown region, thereby expanding the representation of the three-dimensional world, and rendering based on the expanded potential points through the decoder to obtain a consistent image from multiple perspectives.

[0035] This method enables explorable simulation of a 3D world, involving iterative expansion based on point cloud latent representations and final rendering output. Specifically, the method progressively expands the representation range of the 3D world through an iterative conditional generation process. In each iteration, at least a portion of the point cloud latent representation corresponding to the currently known 3D world is selected as a conditional signal. Subsequently, guided by the aforementioned conditional signal, a pre-trained conditional generation model generates new potential points in the point cloud latent representation space that are geometrically and texturally coherent with the conditional signal. These new potential points correspond to the unknown regions to be expanded. Then, the newly generated potential points are fused with known potential points, thereby updating and expanding the overall 3D world representation. After expanding the world representation, based on the final obtained point cloud latent representation, it is converted into a renderable set of 3D primitives by a decoder. Then, using this set of 3D primitives, rendering can be performed from any specified viewpoint to generate an image with strict 3D consistency. For example, in one specific implementation, we can start with an initially generated set of potential points, which is considered the known world. When it is necessary to explore in a specific direction, a portion of the potential points in the current known world is selected as conditions through cropping or sampling, and input into a trained sparse point flow matching network to generate potential points for the next region. The newly generated set of points is merged with the historical set of points through splicing, thereby continuously expanding the world. For the expanded set of potential points, the decoder of the point-Gaussian variational autoencoder is converted into three-dimensional Gaussian primitives, and the rasterizer is used for rendering to obtain a multi-view consistent RGB image and depth map.

[0036] Through the aforementioned technical means, this method achieves a progressive and coherent extended simulation of the three-dimensional world, enabling the model to deduce unknown large-scale environments based on locally known information. Furthermore, since the entire extension process takes place in the native three-dimensional point cloud latent space, and images from any viewpoint can be obtained through a single decoding and rendering, the multi-view three-dimensional consistency of the generated results is fundamentally guaranteed, and flexible and efficient scene exploration and visualization are supported.

[0037] Example 2 Based on the above embodiments, this embodiment provides a detailed description of the specific implementation of step S1, "acquiring three-dimensional scene data and generating point cloud data containing geometric and appearance information based on the three-dimensional scene data," in the method for constructing an explorable native 3D world model based on point cloud implicit representation.

[0038] In this embodiment, the 3D data preparation and preprocessing process described in step S1 is specifically detailed into the following sub-steps. First, sub-step S11 is executed to obtain the RGB image sequence, depth map sequence, and corresponding camera pose parameters of the 3D scene. Specifically, the input comes from labeled data provided by public datasets such as ScanNet v2. For a given 3D scene, the system reads multiple frames of RGB images acquired from it, the strictly aligned depth map, and the camera pose parameters (usually including rotation matrices and translation vectors) corresponding to each frame of the image, which are pre-calculated using sensor calibration and structure-of-motion reconstruction techniques. These data together constitute the original observation set for 3D reconstruction of the scene.

[0039] Next, sub-step S12 is executed, whereby, based on the camera pose parameters, the RGB image and depth map of each frame are transformed to a unified world coordinate system through back projection. The processing action of this sub-step is as follows: for each frame of data, the image pixel coordinates are transformed using the camera intrinsic parameter matrix. Its corresponding depth value By combining these methods, the three-dimensional coordinates of the pixel in the camera coordinate system can be calculated. Subsequently, using the camera pose parameters corresponding to that frame (i.e., the transformation matrix from world coordinates to camera coordinates), the points in the camera coordinate system are transformed through coordinate transformation. Transform to a predefined world coordinate system to obtain its world coordinates. At the same time, the RGB color value of this pixel The data is preserved and associated with the 3D coordinates. The output of this process is a color point cloud fragment generated for a single frame image, where each point contains 3D position and color information in world coordinates.

[0040] Finally, sub-step S13 is executed to fuse the back projection results of all frames, generating color point cloud data containing 3D coordinates and RGB colors. The input to this sub-step is all the single-frame point cloud fragments output from sub-step S12. During processing, the system aggregates the point cloud fragments generated from all frames into a common data structure. Since observations from different frames may overlap, a direct fusion method is to directly stitch all points together to form an initial dense point cloud. In practice, voxelization downsampling and other operations can be further employed to remove redundant points and control the data size, ultimately outputting a unified color point cloud covering the visible portion of the scene. .in, This represents the total number of points after fusion; the 6-dimensional feature corresponds to the 3D coordinates. and RGB colors This colorful dotted cloud This serves as the input for the subsequent point-Gaussian variational autoencoder (P2G-VAE), providing the model with an initial 3D representation that combines geometric and appearance information.

[0041] Through the specific implementation methods described above, high-quality, accurately aligned color point cloud data can be reliably acquired and constructed from standardized 3D datasets. This lays a precise and consistent data foundation for subsequent learning of compact point cloud hidden representations, ensuring the reliability and standardization of the input sources throughout the entire world model construction process.

[0042] Example 3 Based on the above embodiments, this embodiment provides a detailed description of the specific implementation of step S2 in the method for constructing an explorable native 3D world model based on point cloud hidden representation: "The point cloud data is downsampled and features are extracted by an encoder to obtain the point cloud hidden representation, and the point cloud hidden representation is upsampled and decoded into a renderable 3D primitive by a decoder."

[0043] In this embodiment, step S2 specifically converts the input point cloud data into renderable 3D Gaussian primitives through three sub-steps: encoding, regularization, and decoding. First, sub-step S21 is executed, which involves downsampling and feature extraction of the point cloud data using an encoder to obtain initial latent points. Specifically, the encoder input comes from the color point cloud containing geometric coordinates and RGB colors generated in step S1. This encoder is built on an improved Point Transformer v3 architecture, removing residual connections and performing three downsampling operations with a stride of 2. Its processing involves hierarchical feature extraction and point reduction of the input point cloud, downsampling the number of points from approximately one million to approximately five thousand. The output of this process is an initial set of latent points, where each latent point contains a 3D coordinate. and a feature vector , constitute positional feature pairs .

[0044] Next, sub-step S22 is executed, where predefined noise is applied to the coordinates of the initial potential points, and the features are regularized to obtain the regularized point cloud latent representation. The input to this sub-step is the initial potential point output from sub-step S21. The processing involves introducing a robust position perturbation technique: at the bottleneck layer of the encoder, the coordinates of the potential points are... Apply a predefined noise At the same time, for features Standard variational autoencoder regularization is performed. This operation enables the decoder to adapt to small perturbations in the latent point locations. The output of this sub-step is a regularized, more robust latent representation of the point cloud, providing a stable latent space basis for subsequent generation processes.

[0045] Then, sub-step S23 is executed, whereby the point cloud hidden representation is adaptively upsampled and refined using a decoder to decode it into three-dimensional Gaussian primitives. The input to this sub-step is the regularized point cloud hidden representation output from sub-step S22. The processing action specifically includes three stages, corresponding to the content of the next claim. First, through the adaptive upsampling module (corresponding to S231), K learnable query vectors are used... Each parent potential point It splits into K child points. Specifically, the query vector and the parent point features are exchanged through a point-query interaction module. Interact with the system to obtain contextual information and then predict the relative displacement of each sub-point. and characteristic residuals The process can be described by the following formula: In this embodiment, the decoder performs upsampling three times, with the corresponding magnification K set to 7, 3, and 3 respectively, to gradually restore the point cloud density. Next, through the adaptive fine-tuning module (corresponding to S232), based on the features of the upsampled points... Further predict the position offset, and the processing action is as follows: This allows for fine-tuning of the point positions to restore a more accurate geometric structure. Finally (corresponding to S233), the finely tuned points are directly converted into three-dimensional Gaussian elements, with each point corresponding to a Gaussian element whose position is determined by the finely tuned coordinates. The other attributes, including rotation, scaling, opacity, and spherical harmonics used to represent appearance, are determined by predicting the features of the point using a lightweight multilayer perceptron network. The final output of substep S23 is a renderable set of 3D Gaussian primitives with complete geometric and appearance attributes.

[0046] Through the above specific implementation methods, the encoding, regularization and decoding process of point cloud latent representation has been refined. The robust position perturbation technology enhances the stability of the generated model, the adaptive upsampling and refining module effectively recovers the dense and geometrically accurate 3D structure, and efficiently decodes the compact latent representation into a 3D Gaussian representation that supports high-quality rendering, laying a solid foundation for subsequent generation and exploration in the latent space.

[0047] Example 4 Based on the above embodiments, this embodiment provides a detailed description of the specific implementation of step S4 in the method for constructing an explorable native 3D world model based on point cloud hidden representation: "Using the potential points corresponding to the known region as the condition signal, executing step S3 to generate potential points in the unknown region, thereby expanding the 3D world representation, and rendering based on the expanded potential points through the decoder to obtain a multi-view consistent image."

[0048] In this embodiment, the specific implementation of latent point generation in the latent space of the point cloud described in step S3 is as follows. First, a UNet network based on three-dimensional sparse convolution is constructed as a denoising network. Specifically, the network uses 3D sparse convolution as the basic operator to construct the encoder and decoder. The encoder extracts multi-scale geometric context features by downsampling layer by layer, while the decoder fuses deep semantic information and shallow detail information by upsampling and skip connections, and finally outputs a velocity field prediction with the same dimension as the input point set.

[0049] During the diffusion process, for the potential points corresponding to the conditional signals (Including location coordinates and feature vectors) Noise is also added. Given a time step obtained by uniform sampling within the interval [0, 1]. and from the standard normal distribution noise samples in the middle The noise addition process is performed according to the linear interpolation formula to generate noisy latent points. Its calculation formula is This potential source of noise. As one of the inputs to the denoising network.

[0050] During the training process of reverse denoising, the noisy potential points are... With the corresponding time step The UNet network based on 3D sparse convolution is input together. The network uses the aforementioned input as its source, performs forward propagation calculations, and outputs a predicted value for the velocity field at the current moment. The goal of training is to make the predicted value approximate the true target velocity field. That is, by minimizing the mean square error between the two. To optimize network parameters This mechanism enables joint denoising modeling of latent point locations and texture features. To further optimize training, a distance-aware trajectory smoothing technique is introduced in the initial stage of training iterations: by solving a linear assignment problem (LAP) for each target latent point location... Assign a nearest neighbor noise sample location component to minimize the total Euclidean distance. ,in Indicates based on matching relationship Assigned to the The location of noise samples for each potential point; this optimization problem is efficiently solved using the Jonker-Volgenant algorithm to obtain the optimal matching relationship. This is then applied to subsequent loss calculations, thereby straightening the denoised trajectory and improving training stability and generation quality.

[0051] This specific implementation constructs a dedicated 3D sparse convolutional UNet structure and implements joint noise perturbation and denoising learning of position and features, while supplementing it with initial trajectory smoothing optimization. This enables the model to efficiently and stably learn complex geometric and appearance joint distributions in the unstructured point cloud latent space, laying a solid foundation for subsequent high-quality and coherent conditional generation.

[0052] In this embodiment, the explorable world simulation and rendering process described in step S4 is implemented through the following specific sub-steps. First, sub-step S41 is executed, using a portion of the potential point set corresponding to the currently known world as a condition signal. Specifically, the generation methods of the condition signal include, but are not limited to, the following: a clipping method, which randomly selects all potential points within a connected 3D cube region from the known potential point set; a uniform sampling method, which randomly and uniformly selects a fixed proportion of points from the entire known potential point set; or a combined strategy, which first performs region clipping and then performs uniform sampling within the clipped region to simulate the observation data of an actual sensor (such as an RGB-D camera). The input to this step is the complete known potential point set, the processing action is to sample or clip according to one of the above strategies, and the output is a selected portion of potential points, which serves as the condition signal guiding subsequent generation.

[0053] Next, in sub-step S42, the generated conditional signal is concatenated with randomly sampled noisy latent points to form the initial input to the denoising network. The number of noisy latent points is consistent with the expected number of new latent points, and their positions and features are sampled from a standard Gaussian distribution. Specifically, the conditional latent points and noisy latent points are directly concatenated along the point dimension to form a larger point set. Throughout the subsequent denoising process, the coordinates and feature values ​​of the latent points corresponding to the conditional signal will be fixed and will not participate in the network prediction update, thus ensuring that the generated result maintains strict geometric and textural consistency with the known region. The input of this step is the conditional signal and noise samples; the processing action is to concatenate and set the state of some points to be fixed; the output is a complete set of noisy latent points with some points fixed, which serves as the input to the denoising network.

[0054] Then, sub-step S43 is executed, inputting the concatenated noisy latent points into the trained sparse point flow matching network (SPFlow) to perform a denoising process, generating new latent points corresponding to the conditional signals. This process corresponds to the flow matching reverse process defined in step S3. The network uses the aforementioned concatenated point set... (in , For the target point, (for noise) and time step As input, the velocity field is predicted through multiple iterations, and the positions and features of the variable noise points are updated to obtain denoised potential points. These newly generated potential points represent unknown regions that are spatially adjacent or logically connected to the known conditional regions. The input of this step is the concatenated noisy potential points, the processing action is flow matching denoising generation, and the output is a batch of new potential points that are connected to the conditional regions.

[0055] Finally, in sub-step S44, the new potential points generated in sub-step S43 are concatenated with the set of historically known potential points. This is a simple set merging operation, adding the new point set to the existing world representation's point set, thereby expanding the 3D world representation. The expanded point cloud implicit representation constitutes an updated, larger-scale known world model, which can be used in the next exploration loop or directly rendered. The input to this step is the new potential points and the set of historical potential points; the processing action is point set merging; and the output is the expanded, complete 3D world point cloud implicit representation.

[0056] Through the specific implementation methods described above, the model can use known regions as anchor points to gradually and coherently generate the geometry and texture of unknown regions, achieving controllable and progressive expansion of large-scale scenes. This directly brings about technical effects: enabling the 3D world model to possess a dynamic expansion capability similar to that of humans exploring their environment, allowing it to simulate and generate infinitely coherent large-scale scenes starting from local observations, greatly enhancing the practicality and realism of the world model in tasks requiring long-term environmental interaction, such as robot navigation and digital twins.

[0057] Example 6 This embodiment proposes another method for constructing an explorable native 3D world model based on point cloud hidden representation, and provides a detailed description of the complete implementation process.

[0058] This invention proposes an explorable native 3D world model (Terra) based on point cloud hidden representation, with appended... Figure 2 The diagram below shows the overall framework of the invention. The specific implementation steps of the invention will be described in detail below with reference to the accompanying drawings.

[0059] S100: 3D data preparation and preprocessing.

[0060] First, 3D scene data needs to be acquired and preprocessed as input to the model. For each 3D scene (e.g., an indoor scene in the ScanNet v2 dataset), this invention utilizes its provided RGB image sequence, depth map sequence, and corresponding camera pose parameters. Through backprojection, the RGB-D data of each frame is transformed to a unified world coordinate system and fused to generate a colored 3D point cloud. ,in This represents the number of points, where 6 represents the three-dimensional coordinates. and RGB colors The total dimensions of this colored point cloud. It serves as input to the subsequent point-Gaussian variational autoencoder (P2G-VAE), providing initial geometric and appearance information to the model.

[0061] S200: Learns compact point cloud hidden representations through a point-Gaussian variational autoencoder (P2G-VAE), such as... Figure 3 As shown.

[0062] This step aims to compress a high-dimensional, dense input point cloud into a low-dimensional, compact, and structured latent representation of the point cloud, and then decode it into renderable 3D Gaussian primitives. This process is accomplished jointly by the encoder and decoder.

[0063] S201: Encoding and Latent Space Regularization The encoder is built on an improved Point Transformer v3 (PTv3) architecture, removing residual connections from the original architecture to adapt to the generation task. The encoder processes the input point cloud... Downsampling is performed, specifically three times with a step size of 2, reducing the number of points from approximately 1 million to approximately 5,000, forming the initial potential points. Next, to address the challenge of regularizing the latent space of unstructured point clouds, this invention introduces robust position perturbation technology. Unlike traditional VAEs that only regularize features, this invention performs latent representation of the point cloud... Split into Location feature pairs At the bottleneck layer, this invention addresses the coordinates. Apply predefined noise At the same time, for features Standard regularization is performed. This step enables the decoder to adapt to small perturbations in position, thereby significantly improving the robustness of the generation stage.

[0064] S202: Adaptive Upsampling and Refining The decoder is responsible for upsampling the sparse latent points and decoding them into 3D Gaussians. It includes an adaptive upsampling module and an adaptive fine-tuning module.

[0065] The upsampling module splits each potential point into multiple sub-points using learnable query vectors and predicts the location and feature residuals for each sub-point, thus controllably increasing the number of points. This module utilizes K learnable query vectors. Each parent potential point The system splits into K child points. Specifically, the query vector first interacts with the parent point's features to obtain context, and then predicts the relative displacement of each child point. and characteristic residuals :

[0066] in, This represents the point-query interaction module. In a specific implementation, the present invention performs three upsampling operations in the decoder, with the corresponding magnification K set to 7, 3, and 3 respectively, to gradually restore the point cloud density.

[0067] The fine-tuning module further predicts the positional offset based on the point features and fine-tunes the upsampled position of the point to restore a more refined geometric structure. .

[0068] S203: 3D Gaussian Decoding and Supervised Training The upsampled points are directly converted into 3D Gaussian primitives, with each point corresponding to a Gaussian primitive, and its position determined by the finely adjusted coordinates. Other attributes (rotation, scaling, opacity, spherical harmonics) are predicted by point features through a lightweight network. Training of P2G-VAE is supervised using a comprehensive loss function, including rendering loss (L2, SSIM, LPIPS), geometric loss (Chamfer distance), Gaussian attribute regularization (normal vector, effective rank), and explicit color supervision loss (aligning the color of each Gaussian with the color of its nearest neighbor in the input point cloud).

[0069] S300: Potential point generation is performed using a sparse point flow matching network (SPFlow), such as... Figure 3 As shown.

[0070] This step aims to learn the probability distribution of the latent space of compact point clouds, enabling the generation of structured potential points from random noise. This invention employs a flow matching framework and optimizes it for the characteristics of point data.

[0071] S301: Combined noise reduction During the diffusion process, this invention simultaneously adds noise to the location and features of potential points. Given a time step t∈[0,1], the noise N N(0,I), the latent points after adding noise Defined as: .

[0072] In the reverse denoising process, this invention constructs a UNet network based on three-dimensional sparse convolution. To predict the velocity field. The network also accepts noisy potential points. Using time step t as input, the training objective is to minimize the difference between the predicted velocity and the target velocity (i.e., ...). The mean square error between ) This joint denoising mechanism, which combines location and features, leverages the complementarity of geometry and texture to promote mutual enhancement between the two.

[0073] S302: Distance-aware trajectory smoothing To address the trajectory chaos caused by index mismatch in flow matching of unstructured point clouds, this invention proposes a distance-aware trajectory smoothing technique. At the start of training, this invention optimizes point positions by solving a linear assignment problem (LAP). With noise samples Matching relationship between To minimize the total Euclidean distance: ,in This represents the location portion of the noise sample assigned to the m-th potential point. This invention employs the Jonker-Volgenant algorithm to efficiently solve this formula. This optimal matching effectively "straightens" the transport trajectory from noise to the target point, significantly accelerating model convergence and improving generation quality.

[0074] S303: Condition Generation Mechanism To achieve explorable generation, this invention employs a multi-stage training strategy: 1) reconstruction stage; 2) unconditional generation pre-training stage; 3) mask conditional generation stage.

[0075] In the condition generation stage, this invention designs multiple condition signals. Cropping: A connected 3D region is randomly cropped from the point cloud hidden representation as a known condition, used to train the model's ability to "imagine" and fill unknown regions. Uniform Sampling: A subset of points is randomly and uniformly sampled throughout the scene as conditions, used to train the model's ability to refine known regions. Combination Strategy: First, connected regions are cropped, then uniform sampling is performed within those regions, simulating RGB-D sensor data as conditions.

[0076] During inference, this invention concatenates conditional latent points with noise latent points and fixes the conditional point portion throughout the denoising process, thereby guiding the model to generate unknown regions that are consistent with the geometry and texture of known regions.

[0077] S400: Explorable World Simulation and Rendering After model training is complete, large-scale, coherent world simulations can be performed. Starting with an initial generation step, the generated set of potential points is treated as the known world. When exploration in a certain direction is needed, a portion of the currently known world (selected through the aforementioned conditional mechanism) is used as a condition and input into the trained SPFlow network to generate potential points for the next region. The newly generated point set is simply concatenated with the historical point set, continuously expanding the known world. Any final set of potential points can be converted into 3D Gaussian primitives using the P2G-VAE decoder, and then rendered in real-time, with high quality from any given camera viewpoint using a standard rasterizer, resulting in RGB images and depth maps with strict 3D consistency.

[0078] In summary, this invention proposes a native 3D world model based on point cloud latent representation. This invention represents a scene as a set of spatially sparse but semantically compact latent points. Each latent point contains 3D coordinates and a feature vector. This design completely abandons traditional pixel-aligned representations, directly learning and generating the environment from the most natural 3D form, fundamentally ensuring multi-view consistency at the architectural level. Simultaneously, this representation supports flexible rendering from any viewpoint, requiring only a single generation process, eliminating the need to repeatedly run the model for different viewpoints. A point-Gaussian variational autoencoder is designed to achieve efficient conversion between RGB point clouds and renderable 3D Gaussian primitives. This invention proposes a P2G-VAE, where the encoder compresses the input 3D point cloud into a compact point cloud latent representation, while the decoder upsamples these latent points and decodes them into 3D Gaussian primitives with geometric and appearance attributes. This module introduces robust positional perturbation techniques to enhance generation robustness, and adaptive upsampling and refining modules to recover dense and accurate 3D structures, thereby achieving efficient reconstruction from a low-dimensional latent space to a high-quality, renderable 3D scene. This paper proposes a sparse point flow matching network for native 3D generative modeling in the latent space of point clouds. The network employs flow matching technology to learn the trajectory from noise distribution to the distribution of target potential points. It simultaneously denoises the location and features of potential points, learns the joint distribution of geometry and texture, and leverages their complementarity to enhance each other. Furthermore, a distance-aware trajectory smoothing technique is introduced to straighten the denoised trajectory by optimizing the matching relationship between points and noise, significantly improving the convergence and generation quality of the model on unstructured point data. An explorable world modeling mechanism based on progressive extrapolation of the latent space of point clouds is constructed. This paper defines explorable world modeling as an extrapolation task in the latent space of point clouds. Through a multi-stage training strategy including reconstruction, unconditional generation pre-training, and masked conditional generation, the model can learn to coherently generate and expand unknown regions based on known regions (conditionally constructed through cropping, uniform sampling, or combinations thereof), thereby achieving large-scale, infinite, and coherent world simulation.

[0079] The method of this invention, by constructing a native 3D world representation based on point cloud hidden representation and combining it with a P2G-VAE and a sparse point flow matching network, successfully achieves significant improvements in 3D consistency, rendering quality, and scene explorability. This model can generate a complete 3D scene from a single forward pass, supporting high-quality rendering from any viewpoint, and can progressively expand the boundaries of the known world. Experiments on public datasets such as ScanNet v2 demonstrate that this invention achieves leading-edge geometric accuracy and visual fidelity in 3D scene reconstruction and generation, providing a solid technical foundation for building intelligent systems that truly understand and simulate the physical world.

[0080] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0081] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "joining," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0082] In the description of this specification, the references to "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0083] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for constructing an explorable native 3D world model based on point cloud hidden representation, characterized in that, Includes the following steps: S1, acquire three-dimensional scene data, and generate point cloud data containing geometric and appearance information based on the three-dimensional scene data; S2, the point cloud data is downsampled and feature extracted by the encoder to obtain the hidden representation of the point cloud; S3, In the space formed by the hidden representation of the point cloud, based on the given conditional signal and noise samples, a denoising network is used to perform denoising processing to generate new potential points corresponding to the conditional signal. S4, using the potential points corresponding to the known region as the condition signal to generate potential points in the unknown region, thereby expanding the representation of the three-dimensional world, and rendering based on the expanded potential points through the decoder to obtain a consistent image from multiple perspectives.

2. The method according to claim 1, characterized in that, S1 includes: S11, acquire the RGB image sequence, depth map sequence and corresponding camera pose parameters of the 3D scene; S12, based on the camera pose parameters, transform the RGB image and depth map of each frame into a unified world coordinate system through back projection; S13, fuses the back projection results of all frames to generate color point cloud data containing 3D coordinates and RGB colors.

3. The method according to claim 1, characterized in that, S2 includes: S21, the point cloud data is downsampled and features are extracted by the encoder to obtain initial potential points, wherein the initial potential points contain position feature pairs; S22, apply predefined noise to the coordinates of the initial potential points, and at the same time regularize the features of the initial potential points to obtain the regularized point cloud hidden representation; S23, the decoder performs adaptive upsampling and refinement on the hidden representation of the point cloud, upsampling and decoding the sparse latent points into three-dimensional Gaussian primitives with geometric and appearance attributes.

4. The method according to claim 3, characterized in that, S23 includes: S231, through the adaptive upsampling module, uses K learnable query vectors to split each parent potential point into K child points, and predicts the position residual and feature residual of each child point, thereby increasing the point cloud density; S232, through the adaptive fine-tuning module, predicts the position offset of the point based on the features of the upsampled point, and fine-tunes the position of the point to restore the fine geometric structure; S233 directly converts the fine-tuned points into three-dimensional Gaussian units, with each point corresponding to a Gaussian unit. The position of each point is determined by the fine-tuned coordinates, and other attributes are predicted by the point features through a lightweight network.

5. The method according to claim 4, characterized in that, In step S231, the query vector first interacts with the parent point features to obtain context, and then predicts the relative displacement of each child point. and characteristic residuals : in, This indicates the point-to-query interaction module. and These are the coordinates and features of the parent potential point, respectively. and The first The coordinates and features of each sub-point.

6. The method according to claim 1, characterized in that, S3 includes: S31, Construct a UNet network based on three-dimensional sparse convolution as a denoising network; S32, During the diffusion process, noise is added to both the position and features of the potential points corresponding to the conditional signal to obtain the noisy potential points; S33, In the reverse denoising process, the noisy latent points and time steps are input into the denoising network to predict the velocity field. The network is trained by minimizing the error between the predicted velocity and the target velocity to achieve joint denoising of position and features.

7. The method according to claim 6, characterized in that, In S32, a given time step ,noise Potential points after adding noise Defined as: ,in The potential point is the one corresponding to the conditional signal.

8. The method according to claim 6, characterized in that, S3 further includes: S34, at the start of training, optimizes the potential point locations by solving a linear assignment problem. With noise samples Matching relationship between To minimize the total Euclidean distance: ,in Indicates assignment to the first The location portion of the noise sample for each potential point.

9. The method according to claim 1, characterized in that, In step S3, the conditional signal is obtained from the point cloud hidden representation in at least one of the following ways: randomly cropping a connected three-dimensional region from the point cloud hidden representation; randomly and uniformly sampling a portion of points in the entire point cloud hidden representation; first cropping the connected region, and then uniformly sampling within the connected region.

10. The method according to claim 1, characterized in that, S4 includes: S41, take a portion of the set of potential points corresponding to the currently known world as a condition signal; S42, the conditional signal is spliced ​​with the potential noise points, and the part corresponding to the conditional signal is fixed throughout the denoising process; S43, execute S3 to generate a new potential point corresponding to the condition signal, the new potential point representing an unknown region; S44, the new potential points are spliced ​​with the set of historically known potential points to expand the representation of the three-dimensional world.

Images