Real-time interactive three-dimensional (3D) scene reconstruction and simulation using neural representations

Abstract

The disclosure relates to real-time interactive three-dimensional (3D) scene reconstruction and simulation with neural representations. Various examples, systems, and methods related to reconstructing, segmenting, and / or simulating pipelines are disclosed herein. A first computing system may obtain video data including a depth map of a scene. The first computing system may reconstruct a three-dimensional (3D) representation of the scene using at least one or more Gaussian sputtering representations of one or more objects in the scene and the depth map. The first computing system may segment at least one object in the 3D representation. The first computing system may update at least one region of the plurality of regions of the 3D representation within a threshold distance of the at least one object in the scene. The first computing system may generate, for display, at least one image depicting a 3D representation of at least a portion of the at least one object.

Description

BACKGROUND

[0001] Three-dimensional (3D) scene reconstruction and interaction often involves using neural representations, such as neural radiance fields (NeRF) or mesh-based methods, to create 3D environments from image and video data. These existing methods have limitations in terms of accuracy and efficiency, especially when applied to interactive or real-time applications. For example, mesh-based representations can introduce inaccuracies due to discretization errors when approximating continuous surfaces. This presents a challenge for accurately extracting surface data for simulating physical interactions. Unlike traditional mesh models, neural reconstruction methods, such as NeRF or 3D Gaussian splats, inherently lack an explicit surface definition. As a result, generating meshes from these neural representations is computationally intensive and can result in geometric inaccuracies that impact simulation fidelity. Furthermore, simulating deformations or managing object interactions using these neural representations often requires algorithms that can accurately handle volumetric data. These limitations reduce the realism and effectiveness of such simulations in augmented reality (AR) or virtual reality (VR) environments, where precise real-time interaction models are critical. Moreover, while certain methods can perform simulations on static meshes, neural representations such as NeRF or 3D Gaussian splats can provide more detailed, realistic 3D reconstructions from multi-view images or videos. However, simulating these neural representations can be challenging because these neural representations often do not have explicit surfaces like traditional meshes. At least one approach to address this challenge is to extract meshes from these representations and then sample within the mesh volumes, but this conversion process can introduce errors and reduce fidelity. SUMMARY

[0002] Embodiments of the present disclosure relate to systems and methods for 3D scene reconstruction, segmentation, and / or simulation using neural representations in conjunction with segmentation models and volume densification techniques. The disclosed systems and methods can generate 3D representations depicting scenes using depth maps and video data. Segmentation models can be used to isolate objects in 3D environments for manipulation and simulation. These embodiments can further refine these 3D representations by performing operations such as inpainting or artifact removal to address inconsistencies or inaccuracies, thereby improving the quality of the reconstructed scenes. For example, systems and methods according to the present disclosure provide pipelines for physical simulations by generating volumetric representations from 3D data, updating these volumes based at least in part on additional data inputs, and merging volume elements to perform realistic simulations of rigid and elastic objects.

[0003] Some embodiments relate to a system comprising one or more processors to perform one or more operations comprising obtaining, from a video source, video data comprising a depth map of a scene. The one or more processors perform one or more operations to reconstruct the scene into a three-dimensional (3D) representation using at least one or more Gaussian splatting representations and the depth map. The one or more processors perform one or more operations to segment at least one object in the 3D representation. The one or more processors perform one or more operations to generate a two-dimensional (2D) segmentation mask of a reference view of the video data. The one or more processors perform one or more operations to interpolate the 2D segmentation mask over a plurality of frames of the video data. The one or more processors perform one or more operations to map the 2D segmentation mask over the plurality of frames onto at least one corresponding region of a plurality of regions of the 3D representation in order to segment at least one object in the 3D representation from the scene. The one or more processors perform one or more operations to update at least one region of the plurality of regions of the 3D representation within a distance of the at least one object in the scene. The one or more processors perform one or more operations to display the at least one object.

[0004] In some embodiments, the one or more processors are to perform one or more operations to densify the at least one object. In some embodiments, the densifying comprises sampling a plurality of points on or approximately around the at least one object. In some embodiments, the densifying comprises generating a voxelized body of the at least one object based at least in part on the plurality of points. In some embodiments, the densifying comprises updating the voxelized body based at least in part on an occupancy status of at least one voxel of the voxelized body, the occupancy status being based at least in part on the at least one rendered depth map.

[0005] In some embodiments, the one or more processors are to perform one or more operations to fill an interior of the voxelized body based at least in part on injecting a plurality of volume elements in the interior of the at least one object comprising a plurality of interior regions. In some embodiments, the one or more processors perform one or more operations to simulate one or more interactions of a voxelized body of the at least one densified object, the at least one densified object corresponding to the volumetric representation, to update at least one physical property of the at least one object.

[0006] In some embodiments, simulating the one or more interactions includes performing a rigid simulation, the rigid simulation including: applying, using the first physics model, a first plurality of transforms to the at least one densified object to obtain a plurality of rigid motions of the at least one densified object. In some embodiments, the applying includes: determining an energy function using at least one of the plurality of scene parameters or the plurality of object parameters. In some embodiments, the applying includes: minimizing the energy function to determine a plurality of rigid states of the at least one densified object. In some embodiments, the applying includes: applying the plurality of rigid states to simulate the plurality of rigid motions of the at least one densified object over time.

[0007] In some embodiments, simulating the one or more interactions includes performing an elastic simulation, the elastic simulation including: applying, using the second physics model, a second plurality of transforms to the at least one densified object to obtain a plurality of deformed states of the at least one densified object. In some embodiments, the applying includes: determining an energy function using at least one of the plurality of scene parameters or the plurality of object parameters. In some embodiments, the applying includes: minimizing the energy function to determine a plurality of updates to the plurality of control points. In some embodiments, the applying includes: computing one or more deformations of the at least one densified object based at least in part on the plurality of updates to the plurality of control points and a plurality of corresponding skinning fields.

[0008] In some embodiments, the reconstructing is further based on at least one of: (i) the at least one refined pose of the video source and the plurality of two-dimensional (2D) frames of the video data. In some embodiments, the reconstructing further includes: updating the at least one initial pose of the video source to the at least one refined pose based at least in part on aligning the 3D representation with the plurality of 2D frames of the video data.

[0009] In some embodiments, the one or more processors are to perform one or more operations including: generating an initial Gaussian distribution based at least in part on the depth data of the depth map and the at least one initial pose of the video source. In some embodiments, the one or more processors are to perform one or more operations including: generating the 3D reconstruction based at least in part on the initial Gaussian distribution, the at least one refined pose of the video source, and the plurality of 2D frames.

[0010] In some embodiments, the segmenting includes using a segmentation model. In some embodiments, the reference view is based at least in part on user input selecting the at least one object. In some embodiments, the reference view corresponds to a frame of the plurality of frames of the video data.

[0011] In some embodiments, updating the at least one region of the plurality of regions of the 3D representation within the distance is based at least in part on populating the at least one region of the plurality of regions within the distance based at least in part on sampling data of one or more neighboring regions. In some embodiments, updating the at least one region of the plurality of regions of the 3D representation within the distance is based at least in part on removing one or more elements in the at least one region of the plurality of regions within the distance. In some embodiments, updating the at least one region of the plurality of regions of the 3D representation within the distance is based at least in part on updating the at least one region of the plurality of regions based at least in part on sampling data of one or more regions of the plurality of regions of the 3D representation.

[0012] Some embodiments relate to one or more processors comprising one or more circuits to receive video data comprising a depth map of a scene. The one or more circuits are to reconstruct the scene into a three-dimensional (3D) representation using at least one or more Gaussian splatting representations and the depth map. The one or more circuits are to segment at least one object in the 3D representation based at least in part on mapping a two-dimensional (2D) segmentation mask of a reference view of the video data over a plurality of frames onto at least one corresponding region of a plurality of regions of the 3D representation. The one or more circuits are to update at least one region of the plurality of regions of the 3D representation within a distance of at least one object in the scene. The one or more circuits are to generate at least one image depicting at least a portion of the at least one object for display.

[0013] In some embodiments, the one or more circuits are to densify the at least one object. In some embodiments, the one or more circuits are to sample a plurality of points on or approximately around the at least one object. In some embodiments, the one or more circuits are to generate a voxelized volume of the at least one object based at least in part on the plurality of points. In some embodiments, the one or more circuits are to update the voxelized volume based at least in part on an occupancy state of at least one voxel of the voxelized volume, the occupancy state being based at least in part on at least one rendered depth map.

[0014] In some embodiments, the one or more circuits are to populate an interior of the voxelized volume based at least in part on injecting a plurality of elemental volumes inside the at least one object comprising a plurality of interior regions. In some embodiments, the one or more circuits are to simulate one or more interactions of the voxelized volume of the at least one densified object to update at least one physical property of the at least one object. In some embodiments, the at least one densified object corresponds to a volume representation.

[0015] In some embodiments, simulating the one or more interactions includes performing a rigid simulation including applying, using a first physical model, a first plurality of transforms to the at least one densified object to obtain a plurality of rigid motions of the at least one densified object. In some embodiments, applying includes determining an energy function using at least one of the plurality of scene parameters or the plurality of object parameters. In some embodiments, applying includes minimizing the energy function to determine a plurality of rigid states of the at least one densified object. In some embodiments, applying includes applying the plurality of rigid states to simulate the plurality of rigid motions of the at least one densified object over time.

[0016] In some embodiments, simulating the one or more interactions includes performing an elastic simulation including applying, using a second physical model, a second plurality of transforms to the at least one densified object to obtain a plurality of deformed states of the at least one densified object. In some embodiments, applying includes determining an energy function using at least one of the plurality of scene parameters or the plurality of object parameters. In some embodiments, applying includes minimizing the energy function to determine a plurality of updates to the plurality of control points. In some embodiments, applying includes computing one or more deformations of the at least one densified object based at least in part on the plurality of updates to the plurality of control points and the plurality of corresponding skin fields.

[0017] In some embodiments, the reconstructing is further based on at least one of (i) the at least one refined pose of the video source, or (ii) a plurality of two-dimensional (2D) frames of the video data. In some embodiments, the reconstructing further includes updating the at least one initial pose of the video source to the at least one refined pose based at least in part on aligning the 3D representation with the plurality of 2D frames of the video data.

[0018] Some embodiments relate to a method. The method includes receiving, by one or more processors, video data comprising a depth map of a scene. The method includes reconstructing, by the one or more processors, the scene into a three-dimensional (3D) representation using at least a Gaussian splatting and the depth map. The method includes segmenting, by the one or more processors, at least one object in the 3D representation. The method includes updating, by the one or more processors, at least one region of a plurality of regions of the 3D representation within a distance of at least one object in the scene. The method includes densifying, by the one or more processors, the at least one object by generating a voxelized body of the at least one object and updating the voxelized body based at least in part on an occupancy status of at least one voxel of a plurality of voxels of the voxelized body, the occupancy status based at least in part on at least one rendered depth map. The method includes simulating, by the one or more processors, one or more interactions of the voxelized body of the at least one densified object. The method includes displaying, by the one or more processors, at least one rendered image depicting at least a portion of the at least one object using a display device.

[0019] In some embodiments, the method further includes populating, by the one or more processors, an interior of the voxelized body based at least in part on injecting a plurality of volume elements into the interior of the at least one object comprising a plurality of interior regions. In some embodiments, simulating the one or more interactions includes performing a rigid simulation. In some embodiments, the rigid simulation includes applying, by the one or more processors, a first plurality of transformations to the at least one densified object using a first physical model to obtain a plurality of rigid motions of the at least one densified object. In some embodiments, the applying includes determining an energy function using at least one of a plurality of scene parameters or a plurality of object parameters. In some embodiments, the applying includes minimizing the energy function to determine a plurality of rigid states of the at least one densified object. In some embodiments, the applying includes applying the plurality of rigid states to simulate the plurality of rigid motions of the at least one densified object over time.

[0020] In some embodiments, simulating the one or more interactions includes performing an elastic simulation. In some embodiments, the elastic simulation includes applying, by the one or more processors, a second plurality of transformations to the at least one densified object using a second physical model to obtain a plurality of deformed states of the at least one densified object. In some embodiments, the applying includes determining an energy function using at least one of a plurality of scene parameters or a plurality of object parameters. In some embodiments, the applying includes minimizing the energy function to determine a plurality of updates of a plurality of control points. In some embodiments, the applying includes computing one or more deformations of the at least one densified object based at least in part on the plurality of updates of the plurality of control points and a plurality of corresponding skinning fields.

[0021] Some embodiments relate to a system comprising one or more processors to perform one or more operations. The one or more operations include at least one operation to receive and / or obtain at least one object segmented from video data. The one or more operations include at least one operation to densify the at least one object. In some embodiments, densifying includes sampling a plurality of points on or approximately around the at least one object. In some embodiments, densifying includes generating a voxelized volume of the at least one object based at least in part on the plurality of points. In some embodiments, densifying includes updating the voxelized volume based at least in part on an occupancy state of at least one voxel of the voxelized volume, the occupancy state based at least in part on at least one rendered depth map. The one or more operations include at least one operation to simulate one or more interactions of the voxelized volume of the at least one densified object to update at least one physical property of the at least one object. The one or more operations include at least one operation to generate an image depicting at least a portion of the at least one object for display using a display device.

[0022] In some embodiments, the one or more operations include at least one operation to fill an interior of the voxelized volume based at least in part on injecting a plurality of elemental volumes into an interior of the at least one object comprising a plurality of interior regions. In some embodiments, the at least one densified object corresponds to a volume representation.

[0023] In some embodiments, simulating the one or more interactions includes performing a rigid simulation comprising applying a first plurality of transformations to the at least one densified object using a first physical model to obtain a plurality of rigid motions of the at least one densified object. In some embodiments, obtaining the plurality of rigid motions of the at least one densified object includes determining an energy function using at least one of a plurality of scene parameters or a plurality of object parameters. In some embodiments, obtaining the plurality of rigid motions of the at least one densified object includes minimizing the energy function to determine a plurality of rigid states of the at least one densified object. In some embodiments, obtaining the plurality of rigid motions of the at least one densified object includes applying the plurality of rigid states to simulate the plurality of rigid motions of the at least one densified object over time.

[0024] In some embodiments, simulating one or more interactions includes performing an elasticity simulation including applying a second plurality of transformations to the at least one densified object using a second physics model to obtain a plurality of post-deformation states of the at least one densified object. In some embodiments, obtaining the plurality of post-deformation states of the at least one densified object includes determining an energy function using at least one of a plurality of scene parameters or a plurality of object parameters. In some embodiments, obtaining the plurality of post-deformation states of the at least one densified object includes minimizing the energy function to determine a plurality of updates to the plurality of control points. In some embodiments, obtaining the plurality of post-deformation states of the at least one densified object includes computing one or more deformations of the at least one densified object based at least in part on the plurality of updates to the plurality of control points and the plurality of corresponding skin fields.

[0025] Some embodiments relate to one or more processors including one or more circuits to receive and / or obtain at least one object segmented from video data. The one or more circuits are to densify the at least one object. In some embodiments, densifying includes sampling a plurality of points on or approximately around the at least one object. In some embodiments, densifying includes generating a voxelized volume of the at least one object based at least in part on the plurality of points. In some embodiments, densifying includes updating the voxelized volume based at least in part on an occupancy state of at least one voxel of the voxelized volume, the occupancy state based at least in part on at least one rendered depth map. The one or more circuits are to simulate one or more interactions of the voxelized volume of the at least one densified object to update at least one physical property of the at least one object. The one or more circuits are to generate at least one image of the at least one object using the at least one updated physical property.

[0026] In some embodiments, the one or more circuits are to fill an interior of the voxelized volume based at least in part on injecting a plurality of volume elements into an interior of the at least one object including a plurality of interior regions. In some embodiments, the at least one densified object corresponds to a volume representation.

[0027] In some embodiments, simulating the one or more interactions includes performing a rigid simulation, the rigid simulation including applying a first plurality of transforms to the at least one densified object using a first physics model to obtain a plurality of rigid motions of the at least one densified object. In some embodiments, obtaining the plurality of rigid motions of the at least one densified object includes determining an energy function using at least one of the plurality of scene parameters or the plurality of object parameters. In some embodiments, obtaining the plurality of rigid motions of the at least one densified object includes minimizing the energy function to determine a plurality of rigid states of the at least one densified object. In some embodiments, obtaining the plurality of rigid motions of the at least one densified object includes applying the plurality of rigid states to simulate the plurality of rigid motions of the at least one densified object over time.

[0028] In some embodiments, simulating the one or more interactions includes performing an elastic simulation, the elastic simulation including applying a second plurality of transforms to the at least one densified object using a second physics model to obtain a plurality of deformed states of the at least one densified object. In some embodiments, obtaining the plurality of deformed states of the at least one densified object includes determining an energy function using at least one of the plurality of scene parameters or the plurality of object parameters. In some embodiments, obtaining the plurality of deformed states of the at least one densified object includes minimizing the energy function to determine a plurality of updates to the plurality of control points. In some embodiments, obtaining the plurality of deformed states of the at least one densified object includes computing one or more deformations of the at least one densified object based at least in part on the plurality of updates to the plurality of control points and the plurality of corresponding skin fields.

[0029] Some embodiments relate to a method. The method includes receiving, by one or more processors, at least one object segmented from video data. The method includes densifying, by the one or more processors, the at least one object. In some embodiments, densifying includes sampling a plurality of points on or approximately around the at least one object. In some embodiments, densifying includes generating a voxelized volume of the at least one object based at least in part on the plurality of points. In some embodiments, densifying includes updating the voxelized volume based at least in part on an occupancy state of at least one voxel of the voxelized volume, the occupancy state based at least in part on at least one rendered depth map. The method includes simulating, by the one or more processors, one or more interactions of the voxelized volume of the at least one densified object to update at least one physical property of the at least one object. The method further includes displaying, by the one or more processors, using a display device, at least one rendered image depicting at least a portion of the at least one object.

[0030] In some embodiments, the method further includes: populating, by the one or more processors, an interior of the voxelized volume based at least in part on injecting a plurality of volume elements into an interior of at least one object that includes a plurality of interior regions. In some embodiments, the at least one densified object corresponds to the volumetric representation.

[0031] In some embodiments, simulating the one or more interactions includes: performing a rigid simulation, the rigid simulation including: applying, by the one or more processors, a first plurality of transformations to the at least one densified object using a first physical model to obtain a plurality of rigid motions of the at least one densified object. In some embodiments, simulating the one or more interactions includes: performing an elastic simulation, the elastic simulation including applying, by the one or more processors, a second plurality of transformations to the at least one densified object using a second physical model to obtain a plurality of post-deformation states of the at least one densified object.

[0032] The processors, systems, and / or methods described herein can be implemented by or included in at least one system. The system can include a system for executing a game. The system can include a system for executing content streaming. The system can include a system for executing collaborative content creation. The system can include a system for executing simulation operations. The system can include a system for executing collaborative content creation of 3D assets. The system can include a system for generating synthetic data. The system can include a system including one or more visual language models (VLMs). The system can include a system including one or more large language models (LLMs). The system can include a system for executing conversational AI operations. The system can include a system for executing light transport simulation. The system can include a system for executing deep learning operations. The system can include a system for executing digital twin operations. The system can include a control system for autonomous or semi-autonomous machines. The system can include a perception system for autonomous or semi-autonomous machines. The system can include a system including one or more virtual machines (VMs). The system can include a system implemented using robots. The system can include a system implemented using edge devices. The system can include a system implemented at least partially in a data center. The system can include a system implemented at least partially using cloud computing resources. The system can include a system for generating interactive 3D visualizations. The system can include a system implemented at least partially using augmented reality (AR) or virtual reality (VR) platforms. BRIEF DESCRIPTION OF DRAWINGS

[0033] The present systems and methods for reconstructing and interacting with 3D environments will be described in detail below with reference to the attached drawings, in which:

[0034] Figure 1 is a block diagram of an example of a system in accordance with some embodiments of the present disclosure;

[0035] Figure 2 is a block diagram of an example reconstruction stage in an example pipeline according to some embodiments of the present disclosure;

[0036] Figure 3A is a block diagram of an example segmentation stage in an example pipeline according to some embodiments of the present disclosure;

[0037] Figure 3B is a block diagram of another example segmentation stage in an example pipeline according to some embodiments of the present disclosure;

[0038] Figure 4 is a block diagram of an example pre-processing stage in an example pipeline according to some embodiments of the present disclosure;

[0039] Figure 5 is a block diagram of an example densification stage in an example pipeline according to some embodiments of the present disclosure;

[0040] Figure 6A is a block diagram of an example simulation stage in an example pipeline according to some embodiments of the present disclosure;

[0041] Figure 6B is a block diagram of an example rigid simulation stage in an example pipeline according to some embodiments of the present disclosure;

[0042] Figure 6C is a block diagram of an example elastic simulation stage in an example pipeline according to some embodiments of the present disclosure;

[0043] Figure 7 is a block diagram of an example object simulation in an example pipeline according to some embodiments of the present disclosure;

[0044] Figure 8A is a flow diagram of an example of a method for scene reconstruction, segmentation, pre-processing, densification, and / or simulation in an example pipeline according to some embodiments of the present disclosure;

[0045] Figure 8B is a flow diagram of an example of a method for object densification and / or simulation in an example pipeline according to some embodiments of the present disclosure;

[0046] Figure 9A is a block diagram of an example generative language model system for implementing at least some embodiments of the present disclosure;

[0047] Figure 9B is a block diagram of an example generative language model including a transformer encoder-decoder for implementing at least some embodiments of the present disclosure;

[0048] Figure 9Cis a block diagram of an example generative language model including a decoder-only transformer architecture for implementing at least some embodiments of the present disclosure;

[0049] Figure 10 is a block diagram of an example computing device for implementing at least some embodiments of the present disclosure; and

[0050] Figure 11 is a block diagram of an example data center for implementing at least some embodiments of the present disclosure. DETAILED DESCRIPTION

[0051] The present disclosure relates to systems and methods for reconstructing three-dimensional (3D) environments, segmenting 3D environments, and / or interacting with 3D environments using volumetric representations (e.g., Gaussian splats), which leverage improved implementations for segmenting, densifying, and simulating objects in a scene. For example, systems and methods in accordance with the present disclosure involve generating 3D representations from video data and depth information, which can be used for object manipulation, simulation, and visualization in augmented reality (AR) and virtual reality (VR) platforms. That is, due to limitations in object segmentation, volumetric representations, and physical simulation, existing systems often fail to provide accurate real-time interaction and simulation capabilities. In contrast, implementations described herein can use 3D representations, segmentation models, and volumetric densification to create more accurate, more efficient real-time 3D reconstruction, thereby supporting object manipulation, realistic physical simulation, and interactive experiences in AR and VR.

[0052] Further, generating mesh-based representations from neural data (e.g., NeRF or 3D Gaussian) can be computationally intensive and can result in geometric inaccuracies, as these neural representations inherently lack explicit surface definitions. That is, while traditional mesh-based approaches can suffer from discretization errors, the problem with neural-based reconstruction is obtaining a usable mesh representation. For example, one approach can involve extracting a mesh from a neural representation and using that mesh for simulation, which often uses a multi-part conversion process and can degrade the fidelity of the resulting model. In another example, a neural representation can be directly simulated, avoiding surface extraction, and instead using methods for volumetric interaction within the simulation data. Accordingly, these systems and methods address these challenges by simulating neural representations, managing the complexity of volumetric densification and interaction modeling, thereby improving the accuracy and efficiency of real-time 3D simulation in augmented reality (AR) and virtual reality (VR) environments.

[0053] Embodiments of the present disclosure provide systems and methods for simulating three-dimensional (3D) environments using neural representations (e.g., NeRF and 3D Gaussian Splats) that can generate high-quality 3D reconstructions from multi-view images or videos. Unlike traditional mesh-based approaches, neural representations can present challenges for simulation as they often lack clearly defined surfaces. The disclosed systems and methods sample within the volumetric data of these representations to facilitate accurate simulation of physical interactions without relying on conversion to mesh form. This technical solution reduces potential errors associated with traditional mesh extraction approaches and supports efficient, realistic simulation for various applications in dynamic environments.

[0054] Some techniques for 3D scene reconstruction, segmentation, and / or interaction rely on neural radiance fields (NeRF) or mesh representations, which often result in inaccurate or inefficient representations of object segmentation, interaction, and physical simulation. These techniques often fail to provide high-quality interactive 3D reconstructions as they are unable to accommodate real-time object manipulation or accurately manage physical forces and deformations. Limitations include ineffective segmentation, inaccurate transformations, and insufficient volumetric representations. For example, mesh-based approaches can result in inaccurate representations of object deformations and interactions under physical forces, leading to reduced realism and usability. Furthermore, segmentation and densification methods can hinder the processing of AR and VR applications under real-time constraints, resulting in inefficient rendering and interaction.

[0055] Systems and methods in accordance with the present disclosure can improve the accuracy and efficiency of 3D scene reconstruction, segmentation, and / or simulation by providing a framework that uses neural representations and volumetric densification. For example, a plurality of neural representations (e.g., Gaussian Splats, collectively referred to herein as “3D representations”) representing a 3D environment can be generated based at least in part on depth maps (e.g., low-resolution depth maps captured from a LiDAR sensor) and video data (e.g., RGB frames under camera intrinsics and poses). Furthermore, one or more segmentation models (e.g., Segment Anything Model (SAM)) can be used to isolate objects for manipulation and simulation. In some embodiments, parameters such as depth maps, camera poses, and / or 2D segmentation masks (e.g., binary masks generated for different object views) can be used to represent the relevance and importance of features of 3D content. These embodiments can further refine the 3D representations by updating regions of the neural representations within a given distance threshold (e.g., selection based on proximity) to repair inconsistencies or inaccuracies (e.g., artifacts or missing data). For example, refining the 3D representations can include performing inpainting (e.g., filling gaps or holes with data from adjacent regions) and artifact removal (e.g., discarding or replacing regions with poor reconstruction).

[0056] In some embodiments, a densification process can be performed by generating a voxelized body from a sampling point (e.g., converting a neural representation to a voxel grid) and updating it based at least in part on a rendered depth map (e.g., depth carve to remove unoccupied regions) to provide accurate volume for simulation. Generally, the densification process can include voxelizing a 3D Gaussian body to create a voxelized shell (e.g., where only voxels on the approximate shape surface are occupied). Further, the depth map can be used to carve out unoccupied regions around the voxelized shell, resulting in a dense body representing the interior of the shape. This dense body can then be used to sample an isotropic 3D Gaussian body, which can be used for physically realistic interactions within the object. Once densified, these embodiments can fill the interior of the voxelized body with additional volume elements (e.g., inject an isotropic Gaussian body) to facilitate realistic physical simulations, such as rigid body (e.g., simulate solid objects) and elastic simulations (e.g., model deformation under forces) to predict the behavior of the object under different forces. These improvements provide increased accuracy and interaction framework for 3D scene reconstruction, enhancing the realism and usability of AR, VR environments, and other applications by reducing computational inefficiencies and improving the quality of object representation and simulation.

[0057] In some embodiments, video data captured from a device can include RGB frames and camera information (e.g., intrinsics and poses). For example, a low resolution depth map can be converted to a point cloud, which can be used to generate neural representations (e.g., Gaussian splats, collectively forming or creating a 3D representation) that can reconstruct a 3D scene. A segmentation model can be used to generate 2D segmentation masks that can be interpolated over multiple frames, and these masks can be propagated over time using video tracking. That is, a segmentation process can be used to map 2D masks to corresponding 3D neural representations, facilitating object segmentation within 3D space. In some embodiments, densification can be used to refine the properties of the 3D representation. For example, points can be sampled on the surface of an object to generate a voxelized body. The voxelized body (e.g., voxelized shell) can be updated based at least in part on a rendered depth map. Further, volume elements (e.g., isotropic Gaussian bodies) can be injected into the interior of the object to facilitate realistic physical simulations based at least in part on carving space around the voxelized shell using the depth map (e.g., preserving a dense body).

[0058] The systems and methods described herein can be used for a variety of purposes, including but not limited to 3D environment reconstruction, object manipulation in AR / VR, simulation-based training applications, digital twin creation, and interactive content development. These methods can improve the efficiency of tasks involving 3D visualization, such as gaming, robotics, and autonomous driving simulation.

[0059] References Figure 1 , Figure 1is an example block diagram of a system 100 in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be wholly omitted or consolidated. In addition, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any combination and location. The various functions described herein as being performed by the entities can be stored in a memory or memories and executed by a processor or processors (e.g., as a series of computer-readable instructions). In some embodiments, the systems, methods, and processes described herein can be implemented using hardware that is purpose-built for the functions described, using a combination of hardware and software, using software alone, etc. Figure 9A example generative language model system 900 of Figures 9B-9C example generative LM 930 of Figure 10 example computing device 1000 of and / or Figure 11 components, features, and / or functions of the example data center 1100 of

[0060] The system 100 can implement at least a portion of a 3D reconstruction, segmentation, and / or simulation (RCS) pipeline. For example, the system 100 can process video data and depth maps to generate a three-dimensional (3D) representation for object segmentation, manipulation, and physical simulation. The system 100 can be used to perform real-time 3D reconstruction, object interaction, and simulation by any of the various systems described herein, including but not limited to AR and VR systems, autonomous driving systems, robotic systems, gaming systems, and / or digital twin systems.

[0061] Generally, a 3D RCS pipeline can include operations performed by the system 100. For example, the 3D RCS pipeline can include one or more of a video reception stage, a reconstruction stage, a segmentation stage, a pre-processing stage, a simulation stage, and / or a display stage.

[0062] The system 100 (e.g., which implements a 3D RCS pipeline) can receive and / or obtain video data and depth information to reconstruct a three-dimensional (3D) environment using neural representations and volume densification. Further, the system 100 can process and segment objects in the 3D space using a generated 2D segmentation mask, which can be interpolated over multiple frames. In some embodiments, the system 100 can perform inpainting and artifact removal (e.g., prior to simulation) to refine particular regions of the 3D representation (e.g., within a distance of segmented objects). Thus, the 3D RCS pipeline can improve the quality of 3D environment reconstruction and facilitate accurate physical simulation, reducing inconsistencies in object representations and enhancing the fidelity of object interactions.

[0063] In some embodiments, a video reception stage can be a stage in the 3D RCS pipeline in which the system 100 prepares captured video data (e.g., RGB frames with camera intrinsics and poses) and depth information for initial processing and / or alignment evaluation. For example, the video source 104 can provide data in a format such as raw RGB and / or depth maps, and the reconstructor 108 can process the data to extract pixel-level information for reconstructing a 3D environment. In some embodiments, the video reception stage can perform operations to prepare depth maps by correcting any discrepancies in camera poses that can affect segmentation and / or simulation processes.

[0064] The system 100 can include or be coupled with at least one data source 104. The data source 104 can include data such as video data, sensor data, and / or image data. The data source 104 can include data from one or more sensors (or implemented by one or more sensors), such as any one or more cameras (e.g., RGB-D cameras), LiDAR sensors, and / or depth sensors. For example, the data source 104 can include data structured as image frames and / or video frames, which can contain a plurality of pixels for representing information captured by a respective sensor outputting the data. The data source 104 can include two-dimensional and / or three-dimensional image data and / or video data.

[0065] In some embodiments, the data source 104 includes training data (e.g., for training a segmentation model and / or a simulation model). For example, the data source 104 can include one or more example frames each assigned a label. The label can indicate at least one identifier of an object represented in the example frame, such as a region of interest, a segmentation mask, or a classification (e.g., a type, a class). The label can include object data such as a 3D region, a volume density, or metadata. In some embodiments, the segmentation model and / or the simulation model can be configured based at least in part on at least some data other than data of the data source 104. The system 100 can retrieve data from the data source 104 as one or more data streams. For example, the data can be retrieved according to a streaming protocol. Data from the data source 104 can be encoded, such as according to one or more encoding parameters.

[0066] In some embodiments, the system 100 includes at least one reconstructor 108. At the reconstruction stage, the reconstructor 108 can apply any of various reconstruction operations to data from the data source 104, e.g., to perform reconstruction based at least in part on Gaussian splats (e.g., one or more Gaussian splat representations) and depth maps. The reconstructor 108 can generate an initial set of one or more neural representations (e.g., as Gaussian splats of a 3D distribution) based at least in part on depth data of the depth maps and at least one initial pose of the video source. The reconstructor 108 can further refine the 3D representation by aligning the initial neural representations with two-dimensional (2D) video frames using updated camera poses. The refined 3D representation can be provided to or used by subsequent stages for further processing.

[0067] At the reconstruction stage, the reconstructor 108 can generate a 3D representation (e.g., one or more neural representations) of the scene using the video data and associated depth information. For example, the reconstructor 108 can convert a low-resolution depth map (e.g., 192x256 resolution) obtained from a depth sensor (e.g., a LiDAR sensor on a mobile phone, tablet, and / or other smart device) into at least one point cloud (e.g., which represents the scene as a collection of 3D points). The reconstructor 108 can generate a volumetric neural representation (e.g., Gaussian splats, voxel grid, multi-resolution grid) using the point cloud. For example, a Gaussian splat can be a 3D Gaussian distribution that models the spatial properties of the scene. That is, the reconstructor 108 can align the splats with 2D frames by refining parameters (e.g., mean, covariance, orientation, and / or other shape information) based at least in part on feedback from camera intrinsics and extrinsics.

[0068] In some embodiments, the reconstructor 108 can obtain (e.g., virtual) camera parameters, such as intrinsics (e.g., focal length, optical center) and extrinsics (e.g., position, orientation), using an auxiliary data source (e.g., ARKit obtained through the NVIDIA iOS application). For example, the reconstructor 108 can receive these parameters as initial estimates (which can be inaccurate) and perform optimization to refine these parameters. For example, the reconstructor 108 can iteratively adjust 3D point positions and camera parameters to reduce the difference between projected 3D points and observed 2D image points. The reconstructor 108 can iteratively update camera poses and 3D points using bundle adjustment to minimize re-projection error between observed 2D video frames and projected 3D splats. In another example, the reconstructor 108 can apply a non-linear least squares optimization to adjust Gaussian splats and camera parameters simultaneously, thereby ensuring more accurate alignment with the video frames.

[0069] Additionally, the reconstructor 108 can perform reconstruction of different types of 3D representations based at least in part on a particular use case. For example, if the application includes detailed volume rendering of a scene, the reconstructor 108 can generate a Neural Radiance Field (NeRF). In another example, the reconstructor 108 can generate a mesh representation by converting a point cloud to a polygonal surface model. In some embodiments, the reconstructor 108 can determine which representation to use based at least in part on various characteristics (e.g., computational resources, desired fidelity, and specific requirements of downstream processes (e.g., rendering, object manipulation)).

[0070] In some embodiments, the reconstructor 108 can partially optimize (also referred to herein as “reconstruct”) a scene and provide intermediate outputs to subsequent stages in the pipeline. That is, at the segmentation stage, the segmenter 112 can begin processing the partially optimized scene while additional optimization (or reconstruction) is still being performed in the background. For example, the segmenter 112 can begin identifying and classifying objects in the scene based at least in part on initial reconstruction data. Additionally, the simulation stage can run asynchronously, simulating interactions and behaviors within the scene using segmented data while visual quality continues to improve as optimizations are applied to the reconstruction output. Reference will be made to the following figures to describe the reconstruction in more detail. Figure 2 Reconstruction is described in more detail.

[0071] In some embodiments, the segmentation stage can refer to a stage in the 3D RCS pipeline in which the system 100 isolates an object from a 3D representation. That is, the segmenter 112 can generate a two-dimensional (2D) segmentation mask (e.g., a binary mask that identifies particular regions corresponding to an object) for a reference view of the video data. For example, the reference view can be based at least in part on a user selecting an object of interest in a particular video frame. In this example, the reference view can correspond to a frame (e.g., a video snapshot) used for object segmentation. The segmentation stage can interpolate the 2D segmentation mask across multiple frames of the video data (e.g., to maintain consistency across frames). The segmentation stage can map the 2D segmentation mask across the multiple frames to at least one corresponding region of the 3D representation to segment at least one object in the 3D representation from the scene.

[0072] In some embodiments, the segmentation stage can include a semi-interactive process in which the segmenter 112 can generate 2D segmentation masks of objects in the video data. That is, the segmenter 112 can allow a user to select a reference view corresponding to a frame of the video data and provide one or more selections (e.g., mouse clicks, taps, etc.) to guide a segmentation model (e.g., an image segmentation model that can generate a pixel-wise mask from an input image, a model that can use user-provided points to delineate an object, and / or any region-based model that can refine boundaries based at least in part on iterative user input) to identify a foreground object. For example, a user can click on different parts of an object (e.g., a doll) on a surface (e.g., a table) to guide the algorithm in determining which regions represent the object. In this example, the segmentation model can create a 2D mask that delineates the object from surrounding objects or the background in the reference view.

[0073] At the segmentation stage, the segmenter 112 can perform additional operations by allowing the user to change views and provide more selections for refining the segmentation masks. For example, the user can rotate the camera to view the back of the object, providing additional selections that can help the segmentation model adjust the segmentation mask based at least in part on this new perspective. In another example, the user can select different features for distinguishing the back of the object (e.g., a doll) from the rest of the scene, enabling the segmenter 112 to capture details not visible in the front view. The segmenter 112 can use these multiple views to further refine the segmentation masks.

[0074] At the segmentation stage, the segmenter 112 can generate a series of 2D segmentation masks for at least one (e.g., each) view for which the user provided input. That is, the segmenter 112 can compare these segmented views to the original video data to determine points at which the segmented masks align with the captured trajectory. For example, the segmenter 112 can identify frames in the video that correspond to each segmented view and inject the segmentation masks into the video data at that point. In another example, the segmenter 112 can facilitate aligning the segmentation masks with the spatial properties of the 3D representation to maintain consistency.

[0075] In some embodiments, the segmenter 112 can use a video tracker model to interpolate 2D segmentation masks across a sequence of frames in the video data (e.g., a temporal propagation model that can maintain object consistency across frames, a recurrent network-based model that can use memory to recall individual frames, a feature matching model that can align segmented regions over time, or any model that applies learned tracking algorithms to interpolate 2D segmentation masks across a sequence of frames in the video data). That is, the video tracker of the segmenter 112 can use a reference view as a permanent memory input to maintain recognition of segmented objects throughout a video to propagate segmentation masks across frames (e.g., in time). For example, the segmenter 112 can input a segmented reference view and apply interpolation to project the segmentation onto the remaining video frames. In another example, the segmentation masks can be dynamically adjusted to accommodate changes in object appearance across frames.

[0076] Further, the segmenter 112 can propagate the segmentation masks to neural representations (e.g., 3D Gaussian splats) to facilitate segmentation of objects of interest. That is, the segmenter 112 can freeze a 3D representation (e.g., a Gaussian splat model) and update the 3D representation using newly obtained segmentation masks to classify whether at least one (e.g., each) neural representation is part of a foreground object. For example, the updated Gaussian splats can carry binary values indicating the presence of a segmented object, allowing for further processing at subsequent stages. In another example, the segmenter 112 can repeat this process for multiple foreground objects in a scene, thereby generating different segmentation outputs for at least one (e.g., each) object.

[0077] In some embodiments, the segmentation stage can include a manual mode that uses the intersection of 2D bounding box queries to define regions of interest in a scene. That is, a user can define bounding boxes in 2D views that select all 3D Gaussian bodies whose centers project within a defined region for a current camera view (e.g., regardless of their depth in 3D space). For example, a bounding box drawn around one or more objects (e.g., dolls, flower pots, trees) in one view can select all Gaussian splats representing parts of the objects as well as splats from behind the objects (e.g., dolls). In this example, subsequent queries can be performed by the segmenter 112 by rotating the camera to new views and defining additional bounding boxes to refine the selection. The segmenter 112 can use the intersection of these bounding boxes across different views to isolate foreground objects by removing background objects and retaining only the desired neural representations (e.g., Gaussian splats) that remain within the bounding box regions across multiple views.

[0078] In some embodiments, the segmentation stage can support iterative refinement by allowing the user to add, remove, or retain selections through multiple interaction steps. That is, at least one (e.g., each) interaction step can include defining a new bounding box query or adjusting an existing bounding box query, after which the segmenter 112 recomputes the intersection of the selected 3D Gaussian blobs across the various views. For example, the user can first select a coarse region that includes the doll and its surrounding objects, then rotate the view to draw additional bounding boxes that exclude unwanted objects. In another example, the segmenter 112 can retain selected neural representations that consistently remain within the refined bounding boxes in all views, while removing neural representations outside any updated regions. That is, the iterative process can continue until the segmentation accurately isolates the object of interest based at least in part on the user-defined queries and intersection terms.

[0079] In some embodiments, the segmentation stage can include a semi-automatic mode that leverages a combination of image segmentation models and video tracker models to provide more efficient, more accurate segmentation under user guidance. That is, the semi-automatic mode can allow the user to provide cues (e.g., clicks or selections) in an interactive manner to guide the segmenter 112 (e.g., which implements a segmentation model) to distinguish the object of interest from its background in the 2D views. For example, the segmentation model can process the user input to generate an initial segmentation mask that identifies the desired object within the frame. In another example, the user can alter the view or perspective and provide additional input to further refine the segmentation mask, which can be used to account for variations in the appearance of the object under different angles. Further, the video tracker model can subsequently use the reference segmentation mask to propagate the segmentation to subsequent frames.

[0080] The segmenter 112 can include one or more artificial intelligence models (e.g., machine learning models, supervised models, neural network models, deep neural network models), rules, heuristics, algorithms, functions, or any of various combinations thereof to perform operations including segmenting one or more objects or features of one or more objects from data (e.g., from one or more frames of data). In some embodiments, the segmenter 112 can use these models to generate a segmented mask or outline the boundaries of an object based at least in part on input data. For example, the segmenter 112 can employ various segmentation models to identify and isolate the object of interest in different frames or views, refining the segmentation boundaries as needed across multiple perspectives. The segmenter 112 can leverage user input (e.g., clicks or bounding boxes) to guide the segmentation process.

[0081] In some embodiments, the segmenter 112 can maintain, execute, train, and / or update one or more machine learning models during the segmentation stage. In some embodiments, the machine learning models can include any type of image segmentation model configured to process frame data (e.g., image frames) to identify and segment objects. For example, the machine learning models can be trained and / or updated to process image frame inputs while accounting for variations in object appearance or perspective. The machine learning models can be or include transformer-based models (e.g., encoder-decoder models) or other segmentation architectures for high-precision object delineation. The segmenter 112 can execute the machine learning models to generate segmented outputs from provided data. Reference will be made to FIGS. 2-4 below for further details on the segmentation. Figures 3A-3B Segmentation is described in further detail.

[0082] Further reference is made to Figure 1 The system 100 can perform any of a variety of pre-processing operations on the 3D representation output by the reconstructor 108 and segmented by the segmenter 112. For example, but not limited to, during the pre-processing stage, the system 100 can perform inpainting, artifact removal, point sampling, or various combinations thereof on the 3D representation (e.g., a neural representation such as a Gaussian splash). That is, inpainting can include filling gaps or holes in an object model by sampling data from nearby regions. For example, the pre-processor 116 can perform an inpainting operation by sampling regions within a defined distance threshold from the segmented object. Further, artifact removal can include replacing regions of reconstruction disparity. For example, the pre-processor 116 can discard or replace artifacts to improve the visual and structural integrity of the 3D representation.

[0083] In some embodiments, the pre-processing stage can employ artifact processing based at least in part on artificial artifacts present in the 3D representation generated by the reconstructor 108 and segmented by the segmenter 112. That is, the pre-processing stage can include operations such as filling gaps, correcting regions of reconstruction disparity, and / or removing incorrect or unnecessary shadows or artifacts. For example, the pre-processor 116 can perform inpainting to fill gaps in an object model (e.g., represented as a Gaussian splash or other neural representation) by sampling from nearby well-reconstructed regions within a defined distance threshold and / or using Gaussian splashes from neighboring surfaces with similar texture and lighting. In this example, the inpainting can include the pre-processor 116 selecting Gaussian splashes from a region (e.g., a clean region represented by a flat surface or uniform background) to cover areas not visible or poorly captured in the original training view. Further, the pre-processing stage can also include the pre-processor 116 performing artifact removal, where regions of reconstruction disparity are replaced with sampled data from nearby regions to improve the visual and structural integrity of the 3D representation.

[0084] In some embodiments, the pre-processing stage can use a warping technique (e.g., an affine transformation, non-rigid morphing, or any geometric transformation) to manipulate the Gaussian splats, which can expose or reveal previously unseen areas to be corrected. That is, the pre-processor 116 can modify the position or covariance of the Gaussian splats using transformations such as translation, rotation, or scaling (e.g., affine transformations), which can expose areas in the training images that were not visible. For example, translating an object upwards can expose surface portions underneath that were poorly reconstructed due to lack of visibility during the training stage. In another example, the pre-processor 116 can perform a rotation on the object to discover hidden artifacts that need immediate attention in the pre-processing to maintain visual consistency.

[0085] In some embodiments, the training stage can refer to a process in which the system 100 is provided with a series of images or video frames of a scene from various viewpoints to construct a 3D representation of the environment. During this stage, the system 100 can process the training images to create a neural representation, such as Gaussian splats, which can capture the spatial and visual properties of objects and surfaces in the scene. The training stage can include using the images to compute the parameters of Gaussian volumes that make up the 3D scene, such as position, orientation, color, and depth information. After the training stage, the system 100 can perform stages of refinement, preparation, and application of the pre-constructed 3D object models.

[0086] In some embodiments, the pre-processor 116 can facilitate user-guided repair by allowing the user to mark or select an area to use as a sample for covering the area of the reconstruction error. That is, the user can interactively select well-reconstructed areas and instruct the pre-processor 116 to clone and paste Gaussian splats from these areas to the exposed areas that need repair. For example, the user can identify a flat and textured area near a region of the table with visible artifacts and use it as a source for repair. In another example, the pre-processor 116 can automatically identify Gaussian splats within a certain distance threshold around an object and use these splats to fill in gaps or replace erroneous areas.

[0087] In some embodiments, the pre-processing stage can perform shadow removal on Gaussian splats that were captured with the object during the initial training views. That is, shadowing or color distortions that appear as artifacts in the 3D representation can be modified, updated, and / or removed. For example, if a segmented object (e.g., a doll) has shadows in the underlying surface Gaussian volumes due to lighting conditions during capture, the pre-processor 116 can change the color of these Gaussian volumes to remove the shadows. In another example, the pre-processing can include sampling color data from nearby shadow-free areas to provide consistent lighting throughout the 3D representation.

[0088] In some embodiments, preprocessor 116 can support multiple pre-processing actions and / or tasks in order to prepare the 3D representation for subsequent stages (e.g., densification, simulation, and / or rendering). For example, preprocessor 116 can first apply a repair to fix poorly reconstructed areas, then perform shadow removal to ensure consistent lighting, and finally perform artifact removal to address any remaining visual distortions. In another example, pre-processing can be prioritized based at least in part on requirements of downstream processes, e.g., a smooth and artifact-free surface is needed in order to perform accurate physical simulation. That is, by providing a 3D representation that is free (or nearly free) of artifacts and visually consistent, system 100 can facilitate more accurate interactions and simulations of objects within a scene. For example, a pre-processed 3D model can improve physics-based simulations in which collisions and interactions are computed based at least in part on accurate geometry. The pre-processing will be described in more detail below with reference to Figure 4 Pre-processing is described in more detail.

[0089] In some embodiments, the densification stage can refer to a stage in the 3D RCS pipeline in which system 100 densifies the 3D representation to improve volume quality accuracy. That is, simulator 120 can sample a plurality of points on or around segmented objects to generate a voxelized volume based at least in part on the plurality of points. The densification stage can update the voxelized volume based at least in part on a rendered depth map. For example, simulator 120 can perform depth carving to determine an occupancy status of voxels.

[0090] In some embodiments, the densification stage can include converting a 3D Gaussian Spray (3D GS) representation of an object into a dense voxel grid to simulate volume quality. That is, the densification stage can be implemented by simulator 120 voxelizing the space around segmented objects to determine an occupancy of each voxel based at least in part on the presence of a Gaussian Spray. For example, simulator 120 can use a CUDA-based octree algorithm (e.g., to accelerate space subdivision by leveraging GPU processing power to create a hierarchical voxel grid from a 3D Gaussian Spray) to subdivide the space around an object into finer voxels, thereby creating a hierarchical structure that efficiently represents 3D occupancy. In this example, an axis-aligned bounding box of the Gaussian Spray can be enclosed within a root node of an octree, which can be recursively subdivided into smaller nodes while maintaining a list of overlapping Gaussian Sprays for each child node. In some embodiments, simulator 120 can use a uniform grid-based framework and / or voxel hashing techniques. For example, a uniform grid-based approach can be used to divide the space into fixed-size voxels. In another example, voxel hashing can be used to dynamically allocate voxels in sparse regions.

[0091] In some embodiments, the simulator 120 can use a voxelization process to output a high-resolution representation of the interior of the object. That is, the voxelization process can include subdividing nodes containing Gaussian splats until a desired resolution is reached, creating a rasterized representation (e.g., a sparse point cloud (SPC), a dense occupancy raster, or any hierarchical voxel raster) of voxels occupied by the splats. For example, nodes at the edges of an octree can form a voxelized shell of the object (e.g., a voxel raster covering the approximate surface occupied by the voxels), capturing surface properties represented by the neural representation. In another example, the voxelized shell does not include interior voxels of the object, which can be further processed to provide a volumetric representation for accurate physical simulation.

[0092] In some embodiments, at the densification stage, the simulator 120 can perform deep carving to fill the voxelized shell with volumetric mass that approximates a solid interior. That is, deep carving can include determining an occupancy state of each voxel within the shell using depth maps rendered from multiple viewpoints (e.g., depth maps rendered according to an arrangement of virtual cameras). Further, the depth maps can also be used to carve space around the voxelized shell, preserving a dense volume of the object. For example, the simulator 120 can perform ray tracing of a sparse point cloud (SPC) from a set of viewpoints to generate depth maps that can capture distances (e.g., a threshold distance) to the surface of the object from different angles. In another example, the depth maps can be fused together to form a second sparse SPC that can record an occupancy state of each voxel, such as empty, occupied, or invisible.

[0093] In some embodiments, the simulator 120 can use the fused SPC to refine the voxelized volume by carving out unoccupied space and preserving solid regions. That is, the simulator 120 can update an occupancy state of at least one (e.g., each) voxel based at least in part on the depth maps to create a volumetric dense representation of the object. For example, the carving process can start from a fully occupied voxel raster and iteratively remove voxels that are determined to be empty based at least in part on their visibility in the depth maps. In this example, the carving can continue until only occupied voxels representing a solid shape of the object are left (e.g., filling the interior volume).

[0094] In some embodiments, the densification stage can be used to ensure that the 3D representation is suitable for physics-based simulation (e.g., in cases where accurate mass properties can be important). That is, the densified object representation can provide a realistic basis for simulating interactions, collisions, and physical behavior. For example, once the densification stage is complete, the simulator 120 can accurately compute forces, torques, and deformations based at least in part on the solidified voxelized body. In another example, the provided mass properties approximation can facilitate stable and realistic simulation. In some embodiments, the densification stage can be optimized for performance and integrated as a specialized component into a software framework, such as Kaolin by NVIDIA. That is, the body densification block can be implemented as a CUDA kernel that is used to perform the voxelization and depth carving processes. Reference will be made to the following figures in describing the densification in more detail. Figure 5 Densification is described in more detail.

[0095] In some embodiments, the simulation stage can refer to a stage in the 3D RCS pipeline in which the system 100 simulates interactions of the voxelized body. That is, the simulator 120 can inject a plurality of body elements (e.g., isotropic Gaussian bodies) within the interior of the voxelized body to fill the interior. The simulator 120 can simulate one or more interactions of the voxelized body of the densified object to update at least one physical property of the object, such as rigidity or elasticity. The system 100 can include at least one simulator 120. The simulator 120 can include one or more physics-based models, rules, heuristics, algorithms, functions, or various combinations thereof, for performing operations including simulating one or more physical interactions of the object (e.g., rigid body dynamics, elasticity). That is,

[0096] To simulate one or more interactions of the voxelized body, one or more processors will perform these operations. For example, the simulator 120 can apply a first physics model to obtain a plurality of rigid motions of the object and a second physics model to obtain a plurality of post-deformation states of the object. In some embodiments, the simulator 120 can maintain, execute, train, and / or update one or more simulation models during the simulation stage. In some embodiments, the simulation model can include any type of physics-based simulation model configured to process 3D representations to simulate physical behavior. For example, the simulation model can be trained and / or updated to process voxelized inputs. The simulation model can be or include a physics engine model. The simulation model can be configured to predict physical properties, such as deformations under forces.

[0097] The simulator 120 can include one or more physics-based models (e.g., mass-spring model, finite element model, neural network-based physics model), rules, heuristics, algorithms, functions, or any of various combinations thereof, for performing operations including simulating physical interactions of objects within a 3D scene (e.g., rigid body dynamics, elasticity, fluid dynamics). In some embodiments, the simulator 120 can simulate object behavior based at least in part on various physical properties (e.g., mass, density, stiffness, and elasticity). For example, the simulator 120 can apply physics-based rules to compute interactions, deformations, and forces acting on objects. In another example, the simulator 120 can use a neural network model to predict physical behavior of objects and generate a simulation based at least in part on training data. In some embodiments, the simulator 120 can be trained independently of the model used by the segmenter 112. In some embodiments, the simulator 120 can be jointly trained with the segmenter 112. The simulator 120 can be configured to perform both rigid and elastic simulations to model physical behavior of objects in a scene.

[0098] The simulator 120 can include at least one physics model. The physics model can include input parameters (e.g., object properties), transformation parameters, and / or one or more intermediate layers (e.g., skinning field), where at least one (e.g., each) intermediate layer can have respective control points. The system 100 can configure (e.g., train, update) the physics model by modifying or updating one or more parameters (e.g., weights and / or biases of various nodes of the physics model) based at least in part on an evaluation of the estimated output. The simulator 120 can be or include various physics-based models effective for operations or generating data, including but not limited to object deformation, collision detection, or various combinations thereof. In some embodiments, the simulator 120 can be configured (e.g., trained, updated, fine-tuned) based at least in part on training data derived from 3D representations and segmentation results. For example, one or more example scenes of training data can be applied as input to the simulator 120 to generate an estimated output. The estimated output can be evaluated and / or compared to one or more example outputs (e.g., using a cost function, an objective function, a scoring function), and the simulator 120 can be updated based at least in part on the evaluation and / or comparison. For example, one or more parameters (e.g., weights) of the simulator 120 can be updated based at least in part on an output of the objective function.

[0099] Further reference is made to Figure 1The simulator 120 can receive and / or obtain one or more voxelized bodies of data (e.g., by performing densification), and can perform simulation operations (e.g., rigid or elastic simulation) on the voxelized bodies. For example, the simulator 120 can determine, based at least on a given voxelized body, one or more representations of interactions (or simulation results) of the voxelized body. The simulation representations can provide information related to the physical properties and / or behavior of the segmented objects.

[0100] In some embodiments, the simulation stage can include simulating physical interactions of voxelized bodies representing the interior of segmented objects. That is, the simulator 120 can perform physics-based simulation by injecting a plurality of volume elements (e.g., isotropic Gaussian volumes, cubature points, particle-based elements, or any volume representation) into the interior of a voxelized body (e.g., created during densification), thereby filling the space with material properties for simulation. For example, the simulator 120 can simulate rigid body dynamics by treating an object as a rigid body with control handles, allowing the entire object to move as a whole under external forces. In another example, the simulator 120 can simulate elastic deformation by generating a plurality of control handles that guide the elastic properties and deformation of an object using some method (e.g., deformation-based modeling, finite element analysis, or any physics-based simulation technique).

[0101] In some embodiments, the simulator 120 can distinguish between (at least) two types of simulatable objects: rigid and elastic. That is, the simulator 120 can simulate a rigid object after segmentation and treat the object as a single entity with defined mass and inertia. For example, the simulator 120 can apply an external force (e.g., gravity) to a rigid object and compute the resulting motion based at least in part on the mass and other properties of the object. In another example, the simulator 120 can perform collision detection and response computation to determine how a rigid object interacts with other objects in a scene. In some embodiments, in elastic simulation, the simulator 120 can employ techniques such as energy minimization, deformation field optimization, and / or machine learning-based skinning methods to model an object using a plurality of control points and their associated weights.

[0102] In some embodiments, the simulator 120 can utilize object parameters and scene parameters as inputs to simulate the physical interaction. That is, the object parameters can include initial states, rest positions, and physical material properties, such as stiffness, density, and / or elastic modulus, of the sampled volume points. For example, the volume points can represent the internal bodies of the objects and provide a basis for simulating deformation and interaction. In another example, the scene parameters can include external forces, such as gravity, wind, and / or contact forces, which can be modeled as constraints that affect the potential energy of each volume point. The simulator 120 can minimize the potential energy of the system by solving a Newton optimization problem (e.g., iterative gradient descent) to determine how the objects deform under the applied forces.

[0103] In some embodiments, the simulator 120 can output a transformation that defines how the object changes over time. For example, the simulator 120 can compute a 12 degrees of freedom (DoF) affine transformation that determines how the plurality of (e.g., all, some) Gaussian body positions and covariances of the object are transformed at each time frame. For example, for a rigid object, the transformation can represent a combination of translation and rotation, which can be uniformly applied to the neural representation (e.g., Gaussian bodies) within the object. In another example, for an elastic object, the transformation can vary across different parts of the object, providing non-uniform deformation, such as bending, stretching, or twisting. In some embodiments, the simulator 120 can perform elastic simulation to animate the Gaussian splats according to the computed transformation. That is, the elastic simulation can be performed using techniques such as linear blend skinning (LBS), dual quaternion skinning (DQS), skeleton-based deformation, and / or any mesh-based deformation. For example, the simulator 120 can transform both the Gaussian mean and covariance using deformation gradients obtained from LBS. For example, a skinning function can introduce weights for each Gaussian body, determining the strength of its reaction to control point movement. In another example, the transformations can be used to simulate larger elastic deformations, facilitating movement of the object, such as squeezing, stretching, and twisting.

[0104] In some embodiments, simulator 120 can operate in an interactive manner, allowing a user to influence the simulation by providing inputs (e.g., clicks, drags, selections). For example, a user can interact with a simulated object in real-time on an application (e.g., application 124), applying forces directly to the nearest neural representation (e.g., 3D GS) to simulate pulling and / or pushing forces (e.g., gravity, wind, poking objects, etc.), rolling, jumping, or other interactions. In another example, a user can click and drag a specific part of a flexible object to simulate a pulling motion, causing the object to deform accordingly. In yet another example, the interactive simulation can provide visual feedback in real-time (or near real-time). In some embodiments, simulator 120 can improve simulation runtime by using pre-computed data and efficient algorithms. That is, simulator 120 can use techniques such as cached Hessians (e.g., pre-computed second-order derivatives for energy minimization) to reduce the time of complex physical calculations. For example, using NVIDIA Warp, simulator 120 can reduce the training time of morphing modeling methods to 30-90 seconds per object, facilitating fast setup for elastic simulation. In another example, the optimization can enable the simulation to run online in interactive mode, providing a better experience to the user when manipulating and testing different physical scenarios. The simulation will be described in more detail below with reference to Figures 6A-6C The simulation is described in more detail.

[0105] In some embodiments, the display stage can refer to a stage in the 3D RCS pipeline where data, including simulation output, is prepared for visualization or interaction. That is, system 100 can generate at least one image of a 3D representation depicting at least a portion of at least one object for display. Typically, the at least one image can be a rendered frame showing the geometry, morphing, and simulated behavior of the object, and can be generated by applying a rendering technique such as rasterization, ray tracing, or neural rendering to the 3D representation. For example, system 100 can use ray tracing to generate photo-realistic images of the object under various lighting conditions or camera angles. That is, system 100 can create visual outputs of different physical properties or interactions of the object as simulated in the previous stages. Furthermore, to generate the images, system 100 can process data from simulator 120, applying shaders and materials to enhance visual fidelity. For example, system 100 can render the surface of the object with detailed textures, reflections, and shadows, providing a realistic view of the simulated interaction and morphing. That is, the rendered images can be displayed on a graphical user interface, allowing a user to interact with or analyze the simulated object under various states and perspectives.

[0106] The system 100 can include or be coupled with at least one application 124. That is, the at least one application 124 can manage the generation, rendering, and display of objects output from the pipeline. For example, the application 124 can facilitate the arrangement of output data from the segmentation model 112 and / or the simulation model 120 into a structured format for rendering and presentation. In some embodiments, the application 124 can function as a simulation management and visualization system for rendering and presenting output from the 3D RCS pipeline. That is, the application 124 can generate (or render) and display simulation results (at least one image of a 3D representation) from the simulator 120 and translate the results into a visual representation on a user interface (e.g., a 3D visualization tool, an interactive display panel, a simulation control panel, and / or any graphical user interface environment). For example, the application 124 can use GPU-accelerated rendering techniques to process transformation matrices and deformation gradients generated during elastomeric or rigid body simulation. In this example, the application 124 can maintain a real-time data stream between the simulator 120 and a rendering system or device, providing visual updates as simulation parameters are modified.

[0107] Further, the application 124 can perform interpolation and blending of simulation frames. In some embodiments, the application 124 can provide a programmable environment that supports custom user input and real-time adjustments to simulation parameters. That is, the application 124 can expose an API or scripting layer that allows users to programmatically control simulation behavior, modify physical properties, and / or introduce new force fields or constraints. For example, the application 124 can use shader programming and parallel computing techniques to adjust the rendering of a neural representation based at least in part on deformation gradients computed by the simulator 120. In another example, the application 124 can perform collision detection and response calculations in parallel with the simulator 120. Reference will be made to the following examples for further details. Figure 7 The display is described in more detail.

[0108] Reference is made to the following examples. Figure 2FIG. 2 depicts a block diagram of an example initialization stage 200 in an example pipeline according to some embodiments of the present disclosure. For example, the initialization stage 200 can refer to the beginning of the reconstruction phase of the 3D RCS pipeline, in which the reconstructor 108 can generate an initial representation of the scene using inaccurate camera poses and a low-resolution depth map at step 210. That is, the reconstructor 108 can receive and / or obtain inaccurate camera poses (e.g., providing a coarse estimate of the camera’s position and orientation) and combine them with a low-resolution depth map to create an initial Gaussian spray representation 220. For example, the low-resolution depth map can provide sparse information about the depth of the scene, thereby facilitating the reconstructor 108 to approximate the spatial structure using the initial Gaussian spray representation 220. In another example, the inaccurate camera poses at step 210 can be later refined during the training stage 250 to improve the accuracy of the reconstruction.

[0109] In some embodiments, the reconstruction can also be based at least in part on at least one of: (i) at least one refined pose of the video source and a plurality of two-dimensional (2D) frames of the video data. For example, the reconstruction can also include updating at least one initial pose (e.g., inaccurate camera poses) of the video source to at least one refined pose based at least in part on aligning the 3D representation with the plurality of 2D frames of the video data. In this example, the aligning can include determining corresponding points between the 3D representation and the 2D frames to improve (or optimize) the camera position. Further, the reconstructor 108 can generate a Gaussian spray representation based at least in part on the depth data of the depth map and at least one initial pose of the video source (e.g., initialization stage - input: inaccurate camera poses, low-resolution depth map; output: initial Gaussian spray). That is, the reconstructor 108 can compute initial spray positions by projecting the depth values in the depth map to 3D space using the initial camera poses. In some embodiments, the reconstructor 108 can generate a 3D reconstruction aligned with the 2D video frames using updated camera poses based at least in part on the initial Gaussian spray representation, at least one refined pose of the video source, and the plurality of 2D frames (e.g., training stage - input: initial Gaussian spray, RGB frames, and initial poses; output: refined 3D Gaussian spray). That is, the reconstructor 108 can iteratively improve (or optimize) the camera poses by minimizing the difference between the projected spray positions and corresponding 2D features in the video frames.

[0110] In some embodiments, the training stage 250 can refer to a stage in the reconstruction stage of the 3D RCS pipeline in which the reconstructor 108 can use additional data (e.g., RGB frames 260 and updated camera poses) to refine the initial Gaussian spray representation 220 generated during the initialization stage 200. That is, the reconstructor 108 can use these RGB frames 260 containing detailed color and texture information of the scene and refined camera poses to update the position and orientation of the Gaussian volumes. For example, during the training stage 250, the reconstructor 108 can optimize the Gaussian volumes to better align with the RGB frames 260, resulting in a more accurate reconstruction 270. In another example, the updated camera poses can provide improved geometric information to refine the placement of the Gaussian volumes, resulting in a 3D reconstruction 270 that can more accurately capture the appearance and structure of the scene. The training stage 250 enhances the initial output by leveraging color data and refined geometric information, ultimately generating a high-fidelity 3D reconstruction 270 suitable for subsequent processing and simulation stages in the pipeline.

[0111] Reference is made to Figure 3A FIG. 3B illustrates a block diagram of an example segmentation stage in an example pipeline, in accordance with some embodiments of the present disclosure. For example, the segmenter 112 can segment (e.g., identify and isolate objects from a 3D representation) at least one object in a 3D representation. This segmentation can include generating a two-dimensional (2D) segmentation mask (e.g., a binary mask identifying particular regions corresponding to objects) of a reference view of the video data. Further, the segmenter 112 can interpolate (e.g., using the tracker 322) this 2D segmentation mask over multiple frames of the video data. Moreover, the segmenter 112 can map into the 3D representation of the scene (e.g., associating 2D pixels in the mask with particular 3D regions of the scene).

[0112] In some embodiments, the segmenter 112 can implement and / or use a segmentation model (e.g., a segmentation all model (SAM)). Further, the reference view can be based at least in part on user input selecting at least one object. For example, a user can guide the segmentation process by selecting an object of interest in a particular frame (e.g., which is used as the reference view). In this example, the reference view can correspond to a frame (e.g., a video snapshot used as the reference view) in the multiple frames of the video data.

[0113] In some embodiments, the segmentation stage can refer to a stage in the 3D RCS pipeline where the segmenter 112 generates segmented views from video frames by combining both user input and a segmentation model. That is, the segmenter 112 can perform a single view process 310 (e.g., an image or frame) where the user selects an object within the frame. The segmentation model 312 can use these user selections to output a mask that identifies a part of the object (e.g., the head). After this first segmentation, the user can perform another selection (or the system can automatically perform a refinement) to further refine the segmentation. For example, selecting an additional part of the object (e.g., the body). That is, once the segmentation model 312 outputs this initial mask, another segmentation model 314 (or the same segmentation model 312) can be applied to segment the entire object. Further, the single view process 310 can segment a single image or frame, allowing the user to make selections and apply the segmentation to that particular image. In some embodiments, a video process 320 can be performed on multiple frames, where the reference view can be selected by the user or automatically selected by the segmenter 112, and the tracker 322 can be applied to the entire video sequence. In some embodiments, the frame 330 depicts the object before any user selections, while the frame 332 depicts the object highlighted after the user selects the object to be segmented.

[0114] Referring to Figure 3B FIG. 4B illustrates a block diagram of another example segmentation stage in an example pipeline, according to some embodiments of the present disclosure. In some embodiments, the segmentation stage can refer to a stage in the 3D RCS pipeline where the segmenter 112 uses bounding box queries and 2D to 3D projection techniques to separate objects in different scenes to maintain segmentation across frames. That is, the segmentation can receive a view 340 of an object (e.g., a pineapple item) where the user defines a bounding box around the object to guide the segmentation model. The segmenter 112 can use the initial bounding box selection to create a mask 342 that segments the object within the defined area. For example, the segmenter 112 can utilize multiple bounding boxes from different views to assist with occlusions or changes in perspective. In another example, the segmenter 112 can refine the segmentation by dynamically adjusting the bounding box 344 in different frames, thereby maintaining consistency even if the object appears in different contexts or angles.

[0115] Referring to Figure 4FIG. 4 shows a block diagram of an example pre-processing stage in an example pipeline, in accordance with some embodiments of the present disclosure. In some embodiments, the pre-processor 116 can update at least one region of the 3D representation within a threshold distance of at least one object in the scene. That is, updating at least one region of the 3D representation within the threshold distance can be based at least in part on filling (e.g., repairing) at least one region of the plurality of regions within the threshold distance based at least in part on sampling data of one or more neighboring regions. Further, updating at least one region of the 3D representation within the threshold distance can also be based at least in part on removing one or more elements in at least one region of the plurality of regions within the threshold distance and updating at least one region of the plurality of regions based at least in part on sampling data of one or more regions of the 3D representation.

[0116] In general, the pre-processing stage refers to a stage in the 3D RCS pipeline where the pre-processor 116 can modify and enhance the segmented objects for further densification, simulation, and / or display. Further, the pre-processing stage can include identifying a segmented object (e.g., a doll) in the rest view 400 and applying a transform operation to adjust its position or orientation. For example, the pre-processor 116 can use a transform tool to manipulate (e.g., or allow a user to manipulate using a user interface) an object in the scene, thereby exposing previously unseen or poorly reconstructed areas, as shown in the adjusted view 410. In another example, after repositioning or scaling the object, the pre-processor 116 can perform a repair or artifact removal to clean up any visual inconsistencies or artifacts that have emerged in the new position. The result can be a refined representation of the object in the updated view 420, where the pre-processor 116 has corrected any visual errors and prepared the object for subsequent densification or simulation stages in the 3D RCS pipeline.

[0117] Referring to FIG. 4, the pre-processing stage can include a fill operation 402, a repair operation 404, and a transform operation 406. The fill operation 402 can be performed to fill in any missing data in the 3D representation of the scene. For example, the pre-processor 116 can identify a region of the 3D representation that is missing data (e.g., a region that is empty or has no data). The pre-processor 116 can then fill in the missing data by sampling data from one or more neighboring regions of the 3D representation. In some embodiments, the pre-processor 116 can fill in the missing data by sampling data from one or more neighboring regions of the 3D representation that are within a threshold distance of the missing region. In some embodiments, the pre-processor 116 can fill in the missing data by sampling data from one or more neighboring regions of the 3D representation that are within a threshold distance of the missing region and that have a similar density to the missing region. Figure 5FIG. 1 illustrates a block diagram of an example pipeline for rendering a scene, in accordance with some embodiments of the present disclosure. In some embodiments, the pipeline can include a pre-processing stage 102, a rendering stage 104, a densification stage 106, a post-processing stage 108, and a rendering stage 110. In some embodiments, the pre-processing stage 102 can include a meshing module 112, a mesh optimization module 114, and a mesh decimation module 116. In some embodiments, the rendering stage 104 can include a ray tracing module 118. In some embodiments, the densification stage 106 can include a simulator 120. In some embodiments, the post-processing stage 108 can include a meshing module 122, a mesh optimization module 124, and a mesh decimation module 126. In some embodiments, the rendering stage 110 can include a ray tracing module 128.

[0118] In some embodiments, the densification stage in the 3D RCS pipeline can include the simulator 120 converting a set of 3D Gaussian splats into a dense voxel grid to simulate a volumetric representation. That is, the simulator 120 can voxelize the Gaussian splats by subdividing the space around the object using a hierarchical algorithm (e.g., CUDA-based octree, SPC of Kaolin). For example, the simulator 120 can fit the axis-aligned bounding boxes of the Gaussian splats inside a cubic root node, which can be recursively subdivided (e.g., 12-way, 8-way splits, and / or 4-way splits to create smaller nodes in other recursive processes). At least one (e.g., each) child node can maintain a list of overlapping Gaussian bodies to represent the surface of the object. This subdivision can continue until the simulator 120 reaches a desired voxel resolution (e.g., 64x64x64, 128x128x128, or any user-defined resolution), resulting in a voxelized shell that captures the outer surface of the object.

[0119] In some embodiments, the simulator 120 can use the depth maps generated from multiple viewpoints to fill the interior of the voxelized shell to generate a filled object in voxelized form. That is, the simulator 120 can perform ray tracing from various viewpoints (e.g., icosahedral arrangement) to create depth maps of the object. These depth maps can be fused together to represent the internal structure of the object. For example, the simulator 120 can fuse these depth maps into a second sparse point cloud (SPC) and classify each voxel as empty, occupied, or invisible based at least in part on the depth information. Then, the simulator 120 can carve away the unoccupied voxels to refine the volumetric representation, resulting in a filled object in voxelized form. Figure 5 Figure 5 ​The filled object displayed in voxelized form. That is, the simulator 120 refines the interior by removing unoccupied space, creating a volume representation where the remaining voxels reflect the mass and structure of the object. As shown, the interior of the object can be densely populated with voxels that represent the volumetric properties of the object. For example, the densely populated voxels can represent the internal structure of the object, where each voxel corresponds to depth information from multiple viewpoints.

[0120] Referring to Figure 6A depicts a block diagram of example simulation stages in an example pipeline, in accordance with some embodiments of the present disclosure. In some embodiments, the simulator 120 can simulate one or more interactions of a voxelized volume of at least one densified object to update at least one physical property of the at least one object. For example, the at least one densified object can correspond to a volume representation. The reconstruction, segmentation, pre-processing, densification, and / or simulation process 300 can include the simulator 120 performing the training and simulation stages of the 3D RCS pipeline after the 3D Gaussian Splatting block 602 is processed by the Segmentation block 604 (and pre-processing), separating the objects for further operations. In some embodiments, the simulator 120 can rig the objects at the Training block 606 for simulation. That is, during the Training block 606, the simulator 120 can assign control points and neural weights to the objects based at least in part on the geometry and material properties of the objects. In some examples, the simulator 120 can determine the influence of each control point on the surrounding Gaussian volume. The rigged objects can then be output at the Simulation block 608 for simulation.

[0121] In some embodiments, simulator 120 can apply physics-based simulation to the rigged object and / or directly to the segmented object. That is, after segmentation block 604 (and / or after pre-processing), simulator 120 can perform training on the object (e.g., where control points are rigged for more complex simulation) and / or can perform simulation directly on the object. For example, when rigging the object during training block 606, simulator 120 can use the control points and neural weights to provide elastic deformation or rigid body simulation (e.g., based at least in part on the material properties of the object). In some embodiments, elastic simulation can include employing a deformable model, simulating movement of the object (e.g., bending, stretching, compressing) under applied forces. In some embodiments, rigid body simulation can include preserving the structural integrity of the object and simulating movement as a single solid unit. In both simulations, whether the object is rigged or not, simulator 120 can compute physics-based transformations, applying input forces (e.g., gravity or user interaction) to produce motion. For objects provided from segmentation block 604 to simulation block 608, simulator 120 can apply rigid body simulation, where the object is treated as a single entity. For rigged objects, the control points can allow for more deformation and elastic simulation. The output of the simulation can provide realistic movement and interaction, preparing the object for rendering in the next stage or additional physical manipulation in the 3D RCS pipeline.

[0122] Referring to Figure 6B FIG. 19 shows a block diagram of an example rigid simulation stage in an example pipeline, in accordance with some embodiments of the present disclosure. In some embodiments, simulating one or more interactions can include performing rigid simulation. Rigid simulation can include simulator 120 applying a first plurality of transformations to the at least one densified object using a first physics model (e.g., rigid body dynamics, mass-spring system, collision detection algorithm) to obtain a plurality of rigid motions of the at least one densified object. That is, the transformations can be applied based at least in part on determining an energy function using at least one of a plurality of scene parameters (e.g., external forces such as gravity and constraints such as boundary conditions) or a plurality of object parameters (e.g., physical properties of the object (i.e., initial states of the sampled volume points)). Further, the transformations can include minimizing the energy function (e.g., minimizing potential energy) to determine a plurality of rigid states of the at least one densified object. In some embodiments, the transformations can include applying the plurality of rigid states to simulate a plurality of rigid motions of the at least one densified object over time.

[0123] In some embodiments, the preparation stage 610 can include determining and / or defining parameters for rigid simulation. That is, the simulator 120 can receive and / or obtain input parameters representing physical properties of the object, such as volume points, material properties, and external forces. For example, volume points can capture properties such as position, stiffness, and density (e.g., initial rest position, density values), and can be used to represent discrete points on the object. Scene forces can include a plurality of constraints and influencing factors that affect the object, such as gravity and boundary conditions (e.g., gravitational field, collision boundaries, static surfaces). Further, at least one (e.g., each) volume point can be assigned a value representing an undisturbed configuration of the object prior to the application of forces.

[0124] The simulator 620 can model and / or determine transformations of the rigid object based at least in part on the prepared parameters. For example, the simulator 620 can employ methods such as optimization algorithms (e.g., Newton optimization, gradient descent, constraint satisfaction) to minimize potential energy within the system. That is, the simulator 120 can compute transformations for at least one (e.g., each) control handle to represent movement such as translation, rotation, or scaling (e.g., 12 degrees of freedom (DoF), 6 DoF transformations, affine transformations). For example, the simulator 120 can apply one or more transformations to simulate motion of mechanical components such as robotic arms or articulated machinery. In this example, motion can be simulated independently on one or more individual segments while maintaining overall rigidity of the object. In some embodiments, the simulator 620 can use a per-point deformation formula (shown below) that combines neural weights trained during the preparation stage with computed affine transformations in the simulation (per-point deformation formula):

[0125]

[0126] In some embodiments, the simulator output 630 can include a combined set of transformations for the object. This combined set of transformations can be animated using linear blend skinning (LBS) techniques. That is, LBS can interpolate the transformations for each handle on the volume points Z j For example, LBS can be used to animate rigid assemblies in various examples such as industrial machinery, articulated vehicles, and / or interconnected components. The formula for per-point deformation integrates the influence of at least one (e.g., each) control handle by applying a weighted sum of affine transformations, facilitated by neural weights obtained during training. That is, the per-point deformation formula is used so that the object can maintain its structural coherence (e.g., points that react to movements indicated by control handles).

[0127] In general, training in the simulation can include computing neural weights for control handles associated with volume points to establish deformation and movement characteristics of the object. During training, the simulator 120 can iteratively adjust the neural weights based at least in part on perturbations applied to the control points using optimization techniques (e.g., gradient descent, Newton’s method) to minimize a predefined objective function, such as a deformation error or potential energy. The influence of at least one (e.g., each) control handle on surrounding volume points can be quantified by the neural weights (e.g., which define how a movement or force applied to a handle will propagate in the geometry of the object). For example, in a rigid simulation, the training phase can be used to ensure that a transformation applied to a particular handle (e.g., moving or rotating a robotic arm) is reflected throughout the connected region while maintaining structural integrity. That is, the training can output a set of neural weights that can be applied during the simulation phase, allowing the simulator 120 to animate the object with high fidelity based at least in part on the learned control points and their corresponding influence regions.

[0128] The simulator 120 can also provide interactive simulation, allowing a user to influence the simulation by manipulating control handles or adjusting parameters in real-time (or near real-time). That is, a user can update or modify inputs such as external forces or time sliders (e.g., apply a directional force, set a constraint, modify simulation speed) to observe the response of the object under various conditions. For example, a user can simulate independent movement of a particular part of a rigid object (e.g., adjust the arm of an excavator while the body remains stationary to observe the distribution of the transformation on the object). That is, the simulator 120 can use the per-point deformation formula, in conjunction with the neural weights and affine transformations, to apply the computed transformations to accurately animate the object.

[0129] Reference Figure 6C, shows a block diagram of an example elastic simulation stage in an example pipeline, in accordance with some embodiments of the present disclosure. In some embodiments, simulating one or more interactions of the objects can include performing elastic simulation (e.g., elastic simulation: simulating movement and behavior of densified objects). The elastic simulation can include: applying, by the simulator 120, a plurality of transformations to the at least one densified object using a second physical model (e.g., finite element model, mass-spring system, or any meshless method) to obtain a plurality of post-deformation states of the at least one densified object. That is, the transformation can include: determining an energy function using at least one of the plurality of scene parameters or the plurality of object parameters. Further, the transformation can include: minimizing the energy (e.g., potential energy) function to determine a plurality of updates to the plurality of control points. The transformation can also include: computing one or more deformations of the at least one densified object (e.g., animating the object Gaussians) based at least in part on the plurality of updates to the plurality of control points (e.g., points that control deformation) and a plurality of corresponding skin fields (e.g., learned weights that determine how control points affect deformation of the object - skin fields can be cleaned from deformation gradients).

[0130] In some embodiments, the simulator 120 can perform elastic simulation by modeling and / or computing a set of neural weights using a neural network (e.g., feedforward neural network, convolutional neural network, recurrent neural network) that define the influence of each control handle on the deformation of the object. That is, at least one (e.g., each) control handle can correspond to a Gaussian point or a set of points, and the simulator 120 can use a neural skinning function to determine the strength of the influence (e.g., magnitude, influence radius, degree of deformation) of the handle on the movement of the surrounding Gaussians. For example, the neural weights may be optimized to minimize an objective function that includes an elastic loss term and an orthogonality loss term. In another example, the neural weights can be applied to various elastic models (e.g., linear blend skinning, dual quaternion skinning, spline function-based deformation) to simulate different material properties (e.g., rubbery elasticity or flexible fabric behavior).

[0131] The training phase of these neural weights can include the simulator 120 implementing self-supervision. For example, small perturbations can be applied to the control points, and the resulting deformations are utilized to refine the weight values. For example, the simulator can perform small perturbations to determine the optimal weight field W * :

[0132]

[0133] where represents the energy required to elastically deform the object, denotes a constraint that ensures deformations are orthogonal to each other (e.g., to prevent unnatural movement), λ elastic denotes the target deformed position of an object point under the elastic force, λ ortho denotes a constraint that maintains orthogonality between deformation modes (e.g., one deformation does not interfere with other deformations). In some embodiments, numerical gradient computation or NVIDIA Warp can be used to speed up the training process (e.g., reducing the training time per object to 30-90 seconds). That is, after training the neural skinning functions, these functions can define a rigging handle for each Gaussian body, facilitating accurate and efficient simulation of elastic behavior.

[0134] Referring to Figure 7 , a block diagram illustrating example simulation of objects in an example pipeline according to some embodiments of the present disclosure is shown. Figure 7 An interactive simulation mode 700 is depicted, in which the application 124 and / or the simulator 120 can be used to simulate interactions between a user and an object represented by Gaussian splats. That is, the application 124 can process user inputs (e.g., clicks and drags) to apply local forces (e.g., pulling, twisting, pushing) to the nearest Gaussian body, causing the object to deform or move in response to the applied forces. For example, as shown in Figure 7 , a user can interactively manipulate a doll by clicking and dragging different parts of the doll’s body, and the application 124 and / or the simulator 120 apply corresponding forces to generate realistic movement and deformation of the doll in the scene. The interactive simulation mode 700 can provide real-time feedback using the computational configuration of the application 124 and / or the simulator 120.

[0135] Referring to Figure 8A , an example flow diagram illustrating methods for scene reconstruction, segmentation, pre-processing, densification, and / or simulation in an example pipeline according to some embodiments of the present disclosure is depicted. It should be understood that this arrangement and other arrangements described herein are set forth merely as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of the ones shown, and certain elements can be wholly omitted. Moreover, many of the elements described herein are functional entities, which can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any combination and at any location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, the various functions can be implemented using one or more processors executing instructions stored in one or more memories. For example, in some embodiments, the systems and methods described herein can use one or more generative language models (e.g., as described in Figures 9A-9Cone or more computing devices or components thereof (e.g., as described in Figure 10 one or more data centers or components thereof (e.g., as described in Figure 11 ).

[0136] Referring now to Figure 8A , each block of the method 800 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by one or more processors executing stored instructions as Figure 1 described with respect to the system of . However, the method can additionally or alternatively be performed by any one system or any combination of systems, including but not limited to the systems described herein.

[0137] Figure 8A is a flowchart showing a method 800 for scene reconstruction, segmentation, pre-processing, densification, and / or simulation in an example pipeline, in accordance with some embodiments of the present disclosure. The method 800 includes receiving (e.g., by the reconstructor 108) video data including a depth map of a scene from a video source (e.g., the video source 104) at block 810. For example, the video data can be captured RGB frames that include camera information, such as intrinsics and poses. Further, the depth map can provide depth information for one or more (e.g., each) pixels.

[0138] The method 800 includes, at block 820, reconstructing (e.g., by the reconstructor 108) the scene as a three-dimensional (3D) representation (e.g., neural representation, 3D Gaussian splat). For example, the scene can be reconstructed as a series of Gaussian distributions (splat) using at least one or more Gaussian splat representations. In another example, a depth map can be converted to a point cloud to generate a 3D Gaussian splat. In some embodiments, during an initialization phase of the reconstruction, the processing circuitry can generate an initial Gaussian splat representation based at least in part on depth data of the depth map and at least one initial pose of the video source. For example, an initial Gaussian splat can be output using inaccurate camera poses and low resolution depth maps. In some embodiments, during a training phase of the reconstruction (after initialization), the initial Gaussian splat representation and RGB frames and optimized camera poses can be input to obtain (or receive) a 3D GS reconstruction. For example, the 3D GS reconstruction can be based on at least one of: (i) at least one refined pose of the video source and a plurality of two-dimensional (2D) frames of the video data. That is, the 3D reconstruction can be aligned with the 2D video frames using updated camera poses based at least in part on the initial Gaussian splat representation (initial Gaussian body), at least one refined pose of the video source, and / or the plurality of 2D frames. Further, the processing circuitry can update the at least one initial pose (e.g., inaccurate camera poses) of the video source to the at least one refined pose based at least in part on aligning the 3D representation with the plurality of 2D frames of the video data.

[0139] The method 800 includes segmenting (e.g., by the segmenter 112) at least one object in the 3D representation and / or segmentation data corresponding to the at least one object at block 830. That is, the processing circuitry can identify and isolate objects from the 3D representation. The segmentation can include generating a two-dimensional (2D) segmentation mask (e.g., a binary mask identifying particular regions corresponding to the object) of a reference view of the video data. Further, the segmentation can include interpolating the 2D segmentation mask over multiple frames of the video data. For example, the processing circuitry can maintain consistency across frames by applying the 2D segmentation mask to multiple views. In this example, the mask can indicate which pixels in the image are the object or background. In some embodiments, the segmentation can include mapping the 2D segmentation mask over the multiple frames to at least one corresponding region of a plurality of regions of the 3D representation to segment the at least one object (e.g., in the 3D representation of the scene) and / or the segmentation data (e.g., corresponding to the at least one object). For example, the processing circuitry can associate 2D pixels in the mask with particular 3D regions of the scene (e.g., each pixel can correspond to a location in 3D space). In this example, the processing circuitry can identify which Gaussian splats correspond to the object in 3D space. In some embodiments, the segmentation can include using a segmentation model (e.g., a segmentation all model (SAM)). Further, the reference view can be based at least in part on user input. For example, the user can guide the segmentation process by selecting (or performing multiple selections) objects of interest (or certain portions of the object) in a particular frame. In this example, the reference view can correspond to a frame (e.g., a video snapshot used as the reference view) of the multiple frames of the video data.

[0140] The method 800 includes updating (e.g., by the preprocessor 116) at least one region of the plurality of regions of the 3D representation within a threshold distance of the at least one object in the scene at block 840. For example, the 3D Gaussian splat regions can be preprocessed by performing inpainting and artifact removal. In some embodiments, the processing circuitry can fill at least one region of the plurality of regions within the threshold distance based at least in part on sampling data of one or more neighboring regions. For example, the processing circuitry can perform inpainting by sampling data from nearby regions to fill gaps and / or holes in the object model. In some embodiments, the processing circuitry can remove one or more elements in at least one region of the plurality of regions within the threshold distance. That is, regions of poor reconstruction can be discarded or replaced. Further, the removing can include updating at least one region of the plurality of regions based at least in part on sampling data of one or more regions of the 3D representation.

[0141] The method 800 includes densifying (e.g., by the simulator 120) the at least one object by sampling a plurality of points on or approximately around the at least one object at block 850. For example, the processing circuitry can convert the sampled Gaussian spray into a voxel grid to represent the structure of the object. Further, the densifying can include generating (e.g., Step 1: Gaussian to Voxel) a voxelized body of the at least one object based at least in part on the plurality of points. That is, the processing circuitry can subdivide the space around the object into voxels, creating a structured representation of the 3D space. In some embodiments, the densifying can further include updating (e.g., Step 2: Deep Carving) the voxelized body based at least in part on an occupancy status of at least one of the plurality of voxels in the voxelized body, the occupancy status based at least in part on the at least one rendered depth map. For example, the processing circuitry can refine the body to match the geometry of the object by comparing voxel positions to depth map values, removing voxels that are not occupied. The processing circuitry can fill the interior of the voxelized body based at least in part on injecting a plurality of body elements (e.g., isotropic Gaussian bodies, particles, or any discrete sampling) inside the at least one object that includes a plurality of interior regions. That is, the body elements can be injected to perform the simulation at block 860.

[0142] The method 800 includes simulating (e.g., by the simulator 120) one or more interactions of the voxelized body of the at least one densified object to update at least one physical property of the at least one object at block 860. That is, the at least one densified object can correspond to a volumetric representation (e.g., provided during the densifying at block 850). In some embodiments, simulating the one or more interactions can include performing a rigid simulation (e.g., a rigid simulation that simulates movement and behavior of the densified object). That is, the simulation can include the processing circuitry applying a first plurality of transformations to the at least one densified object using a first physical model to obtain a plurality of rigid motions of the at least one densified object. For example, applying the transformations can be based at least in part on the processing circuitry determining an energy function using at least one of a plurality of scene parameters (e.g., external forces such as gravity, and constraints such as boundary conditions) or a plurality of object parameters (e.g., physical properties of the object, such as initial states of the sampled volume points). In this example, applying the transformations can be further based at least in part on the processing circuitry minimizing the energy function (e.g., minimizing potential energy) to determine a plurality of rigid states of the at least one densified object. Further, applying the transformations can be further based at least in part on the processing circuitry applying the plurality of rigid states to simulate a plurality of rigid motions of the at least one densified object over time.

[0143] In some embodiments, simulating the one or more interactions can include performing an elastic simulation (e.g., an elastic simulation that simulates movement and behavior of the densified object). That is, the one or more operations of simulating the one or more interactions include at least one operation for performing an elastic simulation that includes applying, using the second physical model, a second plurality of transformations to the at least one densified object to obtain a plurality of deformed states of the at least one densified object. For example, the simulation can include the processing circuitry applying, using the second physical model, a second plurality of transformations to the at least one densified object to obtain a plurality of deformed states of the at least one densified object. For instance, applying the transformations can be based at least in part on the processing circuitry determining an energy function using at least one of a plurality of scene parameters or a plurality of object parameters. In this example, applying the transformations can also be based at least in part on the processing circuitry minimizing the energy (e.g., potential energy) function to determine a plurality of updates to a plurality of control points. Further, applying the transformations can also be based at least in part on the processing circuitry computing one or more deformations of the at least one densified object (e.g., animating the object Gaussian volume) based at least in part on the plurality of updates to the plurality of control points (e.g., points that can control deformation) and a plurality of corresponding skin fields (e.g., learned weights for determining how control points affect object deformation).

[0144] The method 800 includes, at block 870, generating at least one image depicting the 3D representation of at least a portion of the at least one object for display. In some embodiments, the processing circuitry can generate the at least one object for display (e.g., display on the application 124). For example, the generation can include rendering the voxelized volume or 3D Gaussian spray representation of the object as visual output from a plurality of viewpoints that capture different angles and aspects of the structure and behavior of the object. In this example, the at least one image can be a sequence of frames displaying the deformation and interaction of the object over time, while the 3D representation can be a model that includes physical properties and simulated effects. That is, the processing circuitry can generate (e.g., render) the simulated object for visualization or interaction on a user interface. For example, the application 124 can generate the object in various states for display based at least in part on the simulation results, enabling the user to observe the behavior of the object under different conditions. In some embodiments, the generation can include using rendering techniques such as shadow mapping or global illumination to enhance the realism of the visual output. Further, the display can facilitate the user to make updates or modifications. For example, the user can modify the simulation parameters or exert new forces on the object to see the reaction of the object. That is, the processing circuitry can update the visual representation in real-time based at least in part on the input of the user.

[0145] The disclosed implementations can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots or robotic platforms, aviation systems, medical systems, marine systems, smart regional monitoring systems, systems for performing deep learning operations, systems for performing simulation operations (e.g., in driving or vehicle simulation, in robotics simulation, in smart city or surveillance simulation, etc.), systems for performing digital twin operations (e.g., in conjunction with a collaborative content creation platform or system, such as but not limited to NVIDIA’s OMNIVERSE and / or another platform, system, or service using USD or OpenUSD data types), systems implemented using edge devices, systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations (e.g., using one or more neural radiance fields (NERF), neural representation techniques, diffusion models, transformer models, etc.), systems implemented at least partially in a data center, systems for performing conversational AI operations, systems implementing one or more language models (e.g., one or more large language models (LLMs), one or more visual language models (VLMs), one or more multi-modal language models, etc.), systems for performing light transport simulation, systems for performing collaborative content creation of 3D assets (e.g., using universal scene description (USD) data (e.g., OpenUSD), computer-aided design (CAD) data, 2D and / or 3D graphics or design data, and / or other data types), systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0146] Referring to Figure 8B , an example flow diagram is shown that illustrates a method for object densification and / or simulation in an example pipeline, in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be omitted altogether. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any combination, as well as in any location. Various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For instance, various functions can be implemented using one or more processors that execute instructions stored in one or more memories. For example, in some embodiments, the systems and methods described herein can use one or more generative language models (e.g., as described in Figures 9A-9C , one or more computing devices or components thereof (e.g., as described in Figure 10one or more data centers or components thereof (e.g., as described above in connection with Figure 11 implemented using the systems described above in connection with

[0147] Referring now to the drawings of the embodiments herein, Figure 8B Each block of the method 880 described herein includes a computational procedure that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by one or more processors executing stored instructions. The method can also be embodied as computer-usable instructions stored on computer storage media. The method can be provided by a standalone application, a service, or a hosted service (either independently or in combination with another hosted service), or offered as a micro-service via an application programming interface (API), or as a plug-in to another product, to name a few examples. Further, the method 880 is described with respect to the system of Figure 1 However, the method can additionally or alternatively be performed by any one system or any combination of systems, including but not limited to the systems described herein.

[0148] Figure 8B is a flow diagram illustrating a method 880 for object densification and / or simulation in an example pipeline, in accordance with some embodiments of the present disclosure. The method 880 includes receiving, at block 882, at least one object segmented from video data (e.g., segmentation data corresponding to the at least one object). That is, the processing circuitry can receive and / or obtain (or identify) a 3D representation of the segmented object (e.g., a Gaussian spray representation and / or a neural representation). For example, the segmented object can be represented by a set of Gaussian spray representations corresponding to its surface structure. In some embodiments, the segmented object can include metadata, such as object boundary and pose information. For example, the object metadata can include camera poses and depth information used during reconstruction and segmentation.

[0149] Method 880 includes, at block 884, densifying the at least one object (e.g., at least one object in the segmentation data corresponding to the at least one object). Generally, densification can include converting a set of 3D Gaussian splats to a dense voxel grid to simulate a volume mass. That is, processing circuitry can inject a plurality of volume elements (e.g., isotropic Gaussian volumes) into the interior of a voxelized volume, filling previously hollowed regions to create a solid representation. For example, processing circuitry can use a hierarchical voxelization method (e.g., CUDA-based octree, SPC of Kaolin) to subdivide the space around an object into a structured voxel grid, thereby capturing the surface and interior of the object. In some embodiments, processing circuitry can initialize this voxelization by packing the axis-aligned bounding boxes of Gaussian splats into cubic root nodes (e.g., which can be recursively subdivided into finer nodes) within. Further, the voxel grid can be updated based at least in part on an occupancy state determined from the depth maps rendered from multiple viewpoints. For example, processing circuitry can perform depth carving to refine the voxel grid, thereby removing voxels that are not occupied based at least in part on the rendered depth map information. In some embodiments, the output voxelized volume can provide a high-resolution representation of the object (e.g., to be provided for physical simulation).

[0150] Method 880 includes, at block 886, sampling a plurality of points on or approximately around the at least one object. That is, processing circuitry can sample points on or around an object to generate a voxel grid representing the structure of the object. For example, processing circuitry can distribute the sampled points evenly on the surface of the object and its interior, thereby creating a voxelized representation that can capture the geometry and volumetric properties of the object. In some embodiments, processing circuitry can use Gaussian splats as sampled points to generate a voxel grid, thereby converting the splats to voxels based at least in part on the location and covariance of the splats. Further, the sampling process can be refined to achieve a desired resolution of the voxelized volume. For example, processing circuitry can adjust the sampling density to ensure that the voxel grid accurately represents the surface and internal features of the object. In some embodiments, the voxelized volume can be further refined by aligning the sampled points with depth maps rendered from different viewpoints.

[0151] Method 880 includes, at block 888, generating a voxelized volume of the at least one object based at least in part on the plurality of points. That is, the processing circuitry can generate a structured voxel grid representing the geometry of the object based at least in part on the sampled points. For example, the processing circuitry can assign at least one (e.g., each) point to a voxel based at least in part on the location and covariance of the at least one (e.g., each) point, thereby creating a dense grid that captures surface and internal features of the object. In some embodiments, the processing circuitry can generate a voxelized shell of the object by defining a boundary for each voxel based at least in part on the location of the Gaussian spray. Further, the processing circuitry can adjust the resolution of the voxel grid to achieve a desired level of detail. For example, the processing circuitry can subdivide the voxel grid into smaller nodes to refine the representation of certain regions (e.g., complex, highly pixelated regions). In some embodiments, the processing circuitry can store the generated volume using a hierarchical voxel grid.

[0152] Method 880 includes, at block 890, updating the voxelized volume based at least in part on an occupancy status of at least one voxel of the plurality of voxels in the voxelized volume, the occupancy status based at least in part on the at least one rendered depth map. That is, the processing circuitry can determine an occupancy status for each voxel based at least in part on the depth information obtained from the plurality of viewpoints. For example, the processing circuitry can perform ray tracing from the viewpoints of the icosahedral arrangement to generate depth maps that capture to a threshold distance of the surface of the object from different angles. In some embodiments, the processing circuitry can fuse one or more depth maps to classify each voxel as occupied, empty, or invisible, thereby refining the voxelized volume to match the geometry of the object. Further, the processing circuitry can update the voxelized volume by removing unoccupied voxels based at least in part on the depth map values. For example, the processing circuitry can store the occupancy status of the voxels using a sparse point cloud (SPC) representation. In this example, the SPC can enable efficient querying and manipulation of the voxelized volume.

[0153] Method 880 includes, at block 892, simulating one or more interactions of a voxelized body of at least one densified object to update at least one physical property of the at least one object. That is, the processing circuitry can simulate interactions such as deformation, collision, or movement based at least in part on the material properties of the object and external forces. For example, for an elastic object, the processing circuitry can use control points and / or neural weights to simulate deformation to animate the object based at least in part on input forces (e.g., gravity or user interaction). In another example, for a rigid object, the processing circuitry can simulate rigid body dynamics by applying a set of affine transformations to a voxelized representation of the object (e.g., a solid object). In some embodiments, the processing circuitry can use different physical models for rigid and elastic simulation, adjusting parameters such as stiffness, density, or external forces. Moreover, the simulation results can also include updates to the object position, orientation, and shape over time. For example, the processing circuitry can animate the object using methods such as linear blend skinning or dual quaternion skinning to visualize the simulated interactions. In another example, the processing circuitry can generate visual output depicting the simulated behavior of the object under different conditions.

[0154] Method 880 includes, at block 894, generating at least one image depicting a 3D representation of at least a portion of the at least one object for display. That is, the at least one object can be displayed. For example, the generating can include rendering the voxelized body or 3D Gaussian spray representation of the object as one or more images from different viewpoints. In this example, the at least one image can be a rendered frame depicting the geometry, material properties, and any simulated interactions of the object, while the 3D representation can be a visual model of the object with textures and lighting effects. That is, the processing circuitry can generate (or render) and display the simulated object within a graphical user interface or visualization platform. In some embodiments, the generating can include using photo-realistic rendering techniques, such as ray tracing or path tracing. For example, the processing circuitry can visualize the state of the object at one or more time frames, depicting deformation, movement, or other physical changes. In some embodiments, the display can include interactive features that allow a user to manipulate the object, change simulation parameters, or view the simulation from different perspectives. Moreover, the processing circuitry can update the visualization in real-time as the one or more simulations progress. For example, the display can depict a comparative view of different simulation results, highlighting changes in the object properties or behavior under different conditions.

[0155] Example language model

[0156] In at least some embodiments, language models, such as large language models (LLMs), visual language models (VLMs), multi-modal language models (MMLMs), and / or other types of generative artificial intelligence (AI) can be implemented. Generally, language models can be used to process, analyze, and generate multi-modal content (e.g., text, images, video, 3D models) in various applications, such as in the 3D RCS pipeline described above. That is, these models can interpret and produce outputs that conform to the specific requirements of the reconstruction, segmentation, densification, and / or simulation stages. These models are capable of understanding, summarizing, translating, and / or otherwise generating text (e.g., natural language text, code, etc.), images, video, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., in USD format, such as OpenUSD), etc., based at least in part on the context provided in an input prompt or query. In embodiments, these language models can be considered “large” based at least in part on the fact that these models are trained on massive datasets and their architecture includes a large number of learnable network parameters (weights and biases), such as millions or billions of parameters. LLMs / VLMs / MMLMs, etc., can be implemented to summarize textual data, analyze and extract insights in data (e.g., text, images, video, etc.), and generate new text / images / video, etc., in a user-specified style, tone, and / or format. In embodiments, the LLMs / VLMs / MMLMs, etc., of the present disclosure can be specialized for text processing, while in other embodiments, multi-modal LLMs can be implemented to accept, understand, and / or generate text and / or other types of content, such as images, audio, 2D and / or 3D data (e.g., in USD format), and / or video. For example, a visual language model (VLM) or more generally a multi-modal language model (MMLM) can be implemented for accepting image, video, audio, text, 3D designs (e.g., CAD), and / or other types of input data, and / or for generating or outputting image, video, audio, text, 3D designs, and / or other types of output data.

[0157] Various types of LLM / VLM / MMLM / etc. architectures can be implemented in various embodiments. For example, different architectures can be implemented that use different techniques to understand and generate output (e.g., text, audio, video, images, 2D and / or 3D design or asset data, etc.). In some embodiments, LLM / VLM / MMLM / etc. architectures (e.g., recurrent neural networks (RNNs) or long short-term memory networks (LSTMs)) can be used, while in other embodiments, transformer architectures (e.g., architectures that rely on self-attention and / or cross-attention (e.g., between contextual data and textual data) mechanisms) can be used to understand and recognize relationships between words or tokens and / or contextual data (e.g., other text, video, images, design data, USD, etc.). One or more generative processing pipelines that include LLM / VLM / MMLM / etc. can also include one or more diffusion blocks (e.g., denoisers). LLM / VLM / MMLM / etc. of the present disclosure can include encoder and / or decoder blocks. For example, discriminative or encoder-only models (e.g., BERT (Bidirectional Encoder Representations from Transformers)) can be implemented for tasks involving language understanding (e.g., classification, sentiment analysis, question answering, and named entity recognition). As another example, generative or decoder-only models (e.g., GPT (Generative Pretrained Transformer)) can be implemented for tasks involving language and content generation (e.g., text completion, story generation, and dialogue generation). LLM / VLM / MMLM / etc. that include both encoder and decoder components (e.g., T5 (Text-to-Text Transformer)) can be implemented to understand and generate content, such as for translation and summarization. These examples are not intended to be limiting, and any architecture type (including, but not limited to, the architecture types described herein) can be implemented depending on the particular embodiment and the tasks being performed using the LLM / VLM / MMLM / etc.

[0158] In various implementations, an LLM / VLM / MMLM / etc. can be trained using unsupervised learning, where the LLM / VLM / MMLM / etc. learns patterns from a large amount of unlabeled text / audio / video / image / design / USD / etc. data. Due to extensive training, in implementations, the model can not require task-specific or domain-specific training. An LLM / VLM / MMLM / etc. that has been extensively pre-trained on a large amount of unlabeled data can be referred to as a base model, and can be good at a variety of tasks, such as question answering, summarization, filling in missing information, translation, image / video / design / USD / data generation. Some LLMs / VLMs / MMLMs / etc. can be customized for specific use cases using techniques such as prompt tuning, fine-tuning, retrieval-augmented generation (RAG), adding adapters (e.g., custom neural networks and / or neural network layers for tuning or adjusting prompts or tokens to bias the language model toward a particular task or domain), and / or using optimization models for specific tasks and / or for other fine-tuning or customization techniques within a particular domain.

[0159] In some implementations, LLMs / VLMs / MMLMs / etc. of the present disclosure can be implemented using various model alignment techniques. For example, in some implementations, guardrails can be implemented to identify incorrect or unwanted inputs (e.g., prompts) and / or outputs of the model. In this process, the system can use guardrails and / or other model alignment techniques to prevent particular unwanted inputs from being processed using the LLM / VLM / MMLM / etc., and / or to prevent outputs or presentations (e.g., displays, audio outputs, etc.) of information generated using the LLM / VLM / MMLM / etc. In some implementations, one or more additional models (or layers thereof) can be implemented to identify problems with inputs and / or outputs of the model. For example, these “guard” models can be trained to identify “safe” or otherwise ok or wanted inputs and / or outputs and / or “unsafe” or otherwise unwanted inputs and / or outputs for a particular application / implementation. Thus, LLMs / VLMs / MMLMs / etc. of the present disclosure can be less likely to output language / text / audio / video / design data / USD data / etc. that can be offensive, vulgar, inappropriate, unsafe, out of scope, and / or otherwise unwanted for a particular application / implementation.

[0160] In some implementations, the LLM / VLM / etc. can be configured to or have access to or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations for which the model is not ideally suited, the model can have instructions for accessing one or more plugins (e.g., third-party plugins) to assist in processing the current input (e.g., as a result of training, and / or based at least in part on instructions in a given prompt). In such examples, when at least a portion of the prompt is related to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. Another example is if at least a portion of the response requires mathematical calculations, the model can access one or more mathematical plugins or APIs to assist in solving the problem, which can then be used in the output of the model from the response of the plugin and / or API. This process can repeat (e.g., recursively) for any number of iterations and using any number of plugins and / or APIs until a response to the input prompt can be generated that addresses each inquiry / question / request / process / operation / etc. Thus, the model can rely not only on its own knowledge gained from training on large datasets, but also on the expertise or optimized properties of one or more external resources (e.g., APIs, plugins, etc.).

[0161] In some implementations, multiple language models (e.g., LLMs / VLMs / MMLMs / etc., multiple instances of the same language model, and / or multiple prompts provided to the same language model or instance of the same language model) can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide outputs responsive to the same query or responsive to separate portions of the query. In at least one implementation, the same input query and prompts (e.g., a set of constraints, condition generators, etc.) can be provided to multiple language models (e.g., language models having different architectures, language models trained on different (e.g., updated) data corpora). In one or more implementations, the language models can be different versions of the same base model. In one or more implementations, at least one language model can be instantiated as multiple agents, e.g., more than one prompt can be provided to constrain, guide, or otherwise influence the style, content, or character of the output provided, etc. In one or more example non-limiting implementations, the same language model can be required to provide outputs corresponding to different roles, perspectives, characters, or having different knowledge bases, etc. as defined by the prompts provided.

[0162] In any of such implementations, the outputs of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiations of at least one language model, and / or two or more prompts provided to at least one language model can be further processed, e.g., aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more implementations, the output from one language model (or version, instantiation, or agent) can be provided as input to another language model for further processing and / or validation. In one or more implementations, a language model can be required to generate or otherwise obtain an output about an input source material, where the output is associated with the input source material. Such association can include, for example, generating an embedding (e.g., as metadata) a caption or text portion within an input source text or image. In one or more implementations, the output of a language model can be used to determine the validity of an input source material for further processing or inclusion in a dataset. For example, a language model can be used to assess the presence (or absence) of a target word in a text portion or the presence (or absence) of an object in an image, where the text or image is annotated to indicate such presence (or lack thereof). Alternatively, the determination from a language model can be used to determine whether a source material should be included in a curated dataset, for example, but not limited to.

[0163] Figure 9A is a block diagram of an example generative language model system 900 suitable for implementing at least some embodiments of the present disclosure. Generally, the example generative language model system 900 can be used at different stages of a 3D RCS pipeline. That is, the system can generate parameters, refine segmentation outputs, and simulate object behavior during the reconstruction, segmentation, densification, and / or simulation stages. In Figure 9A In the example shown, the generative language model system 900 includes a retrieval-augmented generation (RAG) component 992, an input processor 905, a tokenizer 910, an embedding component 920, a plug-in / API 995, and a generative language model (LM) 930 (which can include a LLM, a VLM, a multi-modal LM, etc.).

[0164] At a high level, input processor 905 can receive input 901 that includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, Universal Scene Description (USD) data (e.g., OpenUSD, etc.), depending on the architecture of generative LM 930 (e.g., LLM / VLM / MMLM / etc.). In some implementations, input 901 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 901 can include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., table format, JSON, or XML). In some implementations where generative LM 930 is capable of processing multi-modal input, input 901 can combine text (or can omit text) with image data, audio data, video data, design data, USD data, and / or other types of input data such as, but not limited to, the data described herein. As an example with raw input text, input processor 905 can prepare the raw input text in various ways. For example, input processor 905 can perform various types of text filtering to remove noise (e.g., special characters, punctuation, HTML tags, stop words, portions of images, portions of audio, etc.) from the relevant textual content. In examples involving stop words (commonly used words that tend to have little semantic meaning), input processor 905 can remove stop words to reduce noise and cause generative LM 930 to focus on more meaningful content. Input processor 905 can apply text normalization, e.g., by converting all characters to lowercase, removing diacritics, and / or handling special cases (such as abbreviations or contractions) to ensure consistency. These are just a few examples, and other types of input processing can be applied.

[0165] In some implementations, RAG component 992 (which can include one or more RAG models, and / or which can use generative LM 930 itself to perform) can be used to retrieve additional information to be used as part of input 901 or a prompt. RAG can be used to augment input to an LLM / VLM / MMLM / etc. with external knowledge in order to make answers to particular questions or queries or requests more relevant, e.g., in cases where specific knowledge is needed. RAG component 992 can obtain this additional information (e.g., underlying information such as underlying text / image / video / audio / USD / CAD / etc.) from one or more external sources, which can then be fed to the LLM / VLM / MMLM / etc. along with a prompt to improve the accuracy of the model’s response or output.

[0166] For example, in some implementations, input 901 can be generated using a query or model input (e.g., a question, a request, etc.) in addition to data retrieved using RAG component 992. In some implementations, input processor 905 can analyze input 901 and communicate with RAG component 992 (or in implementations, RAG component 992 can be part of input processor 905) in order to identify relevant text and / or other data to provide to generative LM 930 as additional context or information sources, typically from which to identify a response, answer, or output 990. For example, when input indicates that a user is interested in the required tire pressure for a particular make and model of vehicle, RAG component 992 can use a RAG model to perform a vector search, e.g., in an embedding space, to retrieve tire pressure information or text corresponding thereto from a digital (embedded) version of the user manual for that particular vehicle make and model. Similarly, when a user revisits a chatbot related to a particular product sale or service, RAG component 992 can retrieve a previously stored dialog history (or at least an abridged version thereof) and provide the previous dialog history along with the current inquiry / request as part of input 901 to generative LM 930.

[0167] RAG component 992 can use various RAG techniques. For example, a naive RAG (RAG) can be used in which documents are indexed, chunked, and applied to an embedding model to generate embeddings corresponding to the chunks. A user query can also be applied to the embedding model and / or another embedding model of RAG component 992, and the embeddings of the chunks can be compared to the embedding of the query to identify the most similar / most relevant embeddings to the query, which can be provided to generative LM 930 to generate an output.

[0168] In some implementations, more advanced RAG techniques can be used. For example, chunks can undergo pre-retrieval processes (e.g., routing, rewriting, metadata analysis, expansion, etc.) before being passed to an embedding model. Furthermore, outputs of the embedding model can be subjected to post-retrieval processes (e.g., re-ranking, prompt compression, etc.) before the final embeddings are used as comparisons to an input query.

[0169] As a further example, modular RAG techniques can be used, e.g., techniques similar to naive RAG and / or advanced RAG but also including features such as hybrid search, recursive retrieval and query engine, StepBack method, subquery and hypothetical document embeddings, etc.

[0170] As another example, a Graph RAG can use a knowledge graph as a source of contextual or factual information. The Graph RAG can be implemented using a graph database as a source of contextual information sent to the LLM / VLM / MMLM / etc. Rather than (or in addition to) providing the model with data chunks extracted from larger documents (which can result in a lack of context, factual accuracy, language accuracy, etc.), the Graph RAG can also provide structured entity information to the LLM / VLM / MMLM / etc. by combining structured entity text descriptions with their many attributes and relationships, allowing the model to gain deeper insights. In implementing a Graph RAG, the systems and methods described herein use graphs as content stores and extract relevant document chunks and ask the LLM / VLM / MMLM / etc. to use them to answer. In such implementations, the knowledge graph can contain relevant textual content and metadata about the knowledge graph, and can also be integrated with a vector database. In some implementations, the Graph RAG can use graphs as subject matter experts, where descriptions of concepts and entities relevant to the query / prompt can be extracted and passed to the model as semantic context. These descriptions can include relationships between concepts. In other examples, the graph can be used as a database, where a portion of the query / prompt can be mapped to a graph query, the graph query can be executed, and the LLM / VLM / MMLM / etc. can summarize the results. In such examples, the graph can store relevant factual information, and can use querying (natural language querying) and entity linking to a graph querying tool (NL-to-graph querying tool). In some implementations, the Graph RAG (e.g., using a graph database) can be combined with standard (e.g., vector database) RAGs and / or other RAG types to benefit from multiple approaches.

[0171] In any implementation, the RAG component 992 can implement plugins, APIs, user interfaces, and / or other functionality to perform RAGs. For example, the LLM / VLM / MMLM / etc. can use a Graph RAG plugin to run queries against a knowledge graph to extract relevant information to feed into the model, and can use a standard or vector RAG plugin to run queries against a vector database. For example, the graph database can interact with a REST interface of the plugin, such that the graph database can be decoupled from the vector database and / or embedding model.

[0172] The tokenizer 910 can segment (e.g., processed) textual data into smaller units (tokens) for subsequent analysis and processing. Depending on the implementation, a token can represent an individual word, subword, character, portion of audio / video / image / etc. Word-based tokenization divides text into individual words, treating each word as a separate token. Subword tokenization breaks down words into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 930 to understand morphological variations and more effectively handle out-of-vocabulary words. Character-based tokenization represents each character as a separate token, enabling the generative LM 930 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the task at hand, and / or characteristics of the training dataset. Accordingly, the tokenizer 910 can transform (e.g., processed) text into a structured format according to a tokenization scheme implemented in a particular embodiment.

[0173] The embedding component 920 can transform discrete tokens into a semantic (e.g., dense, continuous vector) representation using any known embedding technique. For example, the embedding component 920 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or others.

[0174] In some implementations in which input 901 includes image data / video data / etc., input processor 901 can resize the data to a standard size compatible with the format of the respective input channel and / or can normalize the pixel values to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 920 can encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations in which input 901 includes audio data, input processor 901 can resample the audio files to a consistent sampling rate for uniform processing, and embedding component 920 can extract and encode audio features using any known technique, e.g., in the form of a spectrogram (e.g., a mel-spectrogram). In some implementations in which input 901 includes video data, input processor 901 can extract frames or apply resizing to extracted frames, and embedding component 920 can extract features such as optical flow embeddings or video embeddings and / or can encode temporal information or sequences of frames. In some implementations in which input 901 includes multi-modal data, embedding component 920 can fuse representations of different types of data (e.g., text, image, audio, USD, video, design, etc.) using techniques such as early fusion (concatenation), late fusion (sequential processing), attention-based fusion (e.g., self-attention, cross-attention), etc.

[0175] The generative LM 930 and / or other components of the generative LM system 900 can use different types of neural network architectures depending on the implementation. For example, a transformer-based architecture (such as used in GPT, etc. models) can be implemented, and it can include a self-attention mechanism that weights the importance of different words or tokens in the input sequence and / or a feed-forward network that processes the output of the self-attention layers, which applies a non-linear transformation to the input representation and extracts higher-level features. Some non-limiting example architectures include transformers (e.g., encoder-decoder, decoder-only, multi-modal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of architectures adversarial networks such as generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning, etc. Thus, depending on the implementation and architecture, the embedding component 920 can apply the encoded representation of the input 901 to the generative LM 930, and the generative LM 930 can process the encoded representation of the input 901 to generate an output 990, which can include response text and / or other types of data.

[0176] As described herein, in some implementations, generative LM 930 can be configured to access or use (or be able to access or use) plugins / APIs 995 (which can include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations that are not ideally suited for generative LM 930, the model can have instructions (e.g., as a result of training, and / or based at least in part on instructions in a given prompt, e.g., instructions retrieved using RAG component 992) for accessing one or more plugins / APIs 995 (e.g., third-party plugins) to assist in processing the current input. In such examples, when at least a portion of the prompt is related to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs), send at least a portion of the prompt related to the particular plugin / API 995 to the plugin / API 995, the plugin / API 995 can process that information and return an answer to generative LM 930, which can use that response to generate output 990. This process can repeat (e.g., recursively) any number of iterations and with any number of plugins / APIs 995 until output 990 can be generated that addresses each query / question / request / process / operation / etc. from input 901. Thus, the model can rely not only on its own knowledge obtained from training on large datasets and / or from data retrieved using RAG component 992, but also on the specialized knowledge or optimized properties of one or more external resources (e.g., plugins / APIs 995).

[0177] Figure 9B is a block diagram of an example implementation in which generative LM 930 includes a transformer encoder-decoder. Generally, generative LM 930 can generate model parameters and processing rules for stages of the 3D RCS pipeline. That is, generative LM 930 can generate segmentation masks, update camera poses, and adjust simulation parameters based at least in part on input data. For example, assume that input text (e.g., “Who discovered gravity”) is tokenized (e.g., by tokenizer 910 of Figure 9A is a block diagram of an example implementation in which generative LM 930 includes a transformer encoder-decoder. Generally, generative LM 930 can generate model parameters and processing rules for stages of the 3D RCS pipeline. That is, generative LM 930 can generate segmentation masks, update camera poses, and adjust simulation parameters based at least in part on input data. For example, assume that input text (e.g., “Who discovered gravity”) is tokenized (e.g., by tokenizer 910 of Figure 9A is a block diagram of an example implementation in which generative LM 930 includes a transformer encoder-decoder. Generally, generative LM 930 can generate model parameters and processing rules for stages of the 3D RCS pipeline. That is, generative LM 930 can generate segmentation masks, update camera poses, and adjust simulation parameters based at least in part on input data. For example, assume that input text (e.g., “Who discovered gravity”) is tokenized (e.g., by tokenizer 910 of is a block diagram of an example implementation in which generative LM 930 includes a transformer encoder-decoder. Generally, generative LM 930 can generate model parameters and processing rules for stages of the 3D RCS pipeline. That is, generative LM 930 can generate segmentation masks, update camera poses, and adjust simulation parameters based at least in part on input data. For example, assume that input text (e.g., “Who discovered gravity”) is tokenized (e.g., by tokenizer 910 of

[0178] In example implementations, the encoder 935 forms an encoder stack, where each encoder includes a self-attention layer and a feedforward network. In an example transducer architecture, each token (e.g., word) flows through a separate path. Thus, each encoder can accept a sequence of vectors, pass each vector through a self-attention layer, then through a feedforward network, and then pass up to the next encoder in the stack. Any known self-attention technique can be used. For example, to compute a self-attention score for each token (word), a query vector, a key vector, and a value vector can be created for each token, a self-attention score for a pair of tokens can be computed by taking the dot product of the query vector and the corresponding key vector, normalizing the resulting score, multiplying by the corresponding value vector, and summing the weighted value vectors. The encoder can apply multi-head attention, in which the attention mechanism is applied multiple times in parallel with different learned weight matrices. Any number of encoders can be concatenated to generate a context vector that encodes the input. The attention projection layer 940 can transform the context vector into an attention vector (keys and values) for the decoder 945.

[0179] In example implementations, the decoder 945 forms a decoder stack, where each decoder includes a self-attention layer, an encoder-decoder self-attention layer that attends to relevant portions of the input sequence using attention vectors (keys and values) from the encoder, and a feedforward network. As with the encoder 935, in an example transducer architecture, each token (e.g., word) flows through a separate path in the decoder 945. During a first pass, the decoder 945, the classifier 950, and the generation mechanism 955 can generate a first token, and the generation mechanism 955 can apply the generated token as input during a second pass. This process can be repeated iteratively, generating tokens (e.g., words) one at a time and adding them to the output of the previous pass, and applying the token embedding of the composite sequence with positional encodings as input to the decoder 945 in subsequent passes, iteratively generating one token at a time (known as autoregressive) until a symbol or token is predicted that indicates the end of the response. In each decoder, the self-attention layer is typically restricted to attending only to previous positions in the output sequence by applying a masking technique (e.g., setting future positions to negative infinity) prior to the softmax operation. In example implementations, the operation of the encoder-decoder attention layer is similar to that of the (e.g., multi-head) self-attention in the encoder 935, except that it creates its queries from the layer below it and takes its keys and values (e.g., matrices) from the output of the encoder 935.

[0180] Accordingly, the decoder 945 can output some decoded (e.g., vector) representation of the input applied during a particular pass. The classifier 950 can include a multi-class classifier that includes one or more neural network layers that project the decoded (e.g., vector) representation to respective dimensions (e.g., one dimension for each supported word or token in the output vocabulary) and a softmax operation that converts logits to probabilities. Accordingly, the generation mechanism 955 can select or sample a word or token based at least in part on the respective predicted probabilities (e.g., select the word with the highest predicted probability) and append it to the output of the previous pass, generating each word or token in order. The generation mechanism 955 can repeat this process, triggering successive decoder inputs and respective predictions, until a symbol or token is selected or sampled that represents the end of the response, at which point the generation mechanism 955 can output the generated response.

[0181] Figure 9C is a block diagram of an example implementation in which the generative LM 930 includes a decoder-only transducer architecture. For example, Figure 9C The decoders 960 of the decoder-only transducer architecture can operate similarly to the decoders 945 of the encoder-decoder transducer architecture, except that Figure 9B Each decoder 960 of the decoder-only transducer architecture omits the encoder-decoder self-attention layer (as there is no encoder in this implementation). Accordingly, the decoders 960 can form a stack of decoders, where each decoder includes a self-attention layer and a feedforward network. Further, a symbol or token representing the end of the input sequence (or the beginning of the output sequence) can be appended to the input sequence, rather than encoding the input sequence, and the resulting sequence (e.g., respective embeddings with position encodings) can be applied to the decoders 960. Like the decoders 945 of the encoder-decoder transducer architecture, each token (e.g., word) can flow through a separate path in the decoders 960, and the decoders 960, classifier 965, and generation mechanism 970 can use autoregression to generate tokens one at a time in order until a symbol or token is predicted that represents the end of the response. The classifier 965 and generation mechanism 970 can operate similarly to the classifier 950 and generation mechanism 955 of the encoder-decoder transducer architecture, where the generation mechanism 970 selects or samples each successive output token based at least in part on the respective predicted probabilities and appends it to the output of the previous pass, generating each token in order until a symbol or token is selected or sampled that represents the end of the response. These and other architectures described herein are merely examples, and other suitable architectures can be implemented within the scope of the present disclosure. Figure 9C Figure 9B Figure 9B

[0182] Example computing device

[0183] Figure 10 ​​​is a block diagram of an example computing device 1000 suitable for implementing some embodiments of the present disclosure. Generally, the example computing device 1000 can perform various stages of a 3D RCS pipeline, such as reconstruction, segmentation, pre-processing, densification, and / or simulation. That is, the computing device 1000 can perform computations to generate a 3D representation, segment objects, process volume data, densify objects, and / or simulate interactions of objects with the pipeline. The computing device 1000 can include an interconnect system 1002 that directly or indirectly couples the following devices: memory 1004, one or more central processing units (CPUs) 1006, one or more graphics processing units (GPUs) 1008, a communication interface 1010, input / output (I / O) ports 1012, I / O components 1014, a power supply 1016, one or more presentation components 1018 (e.g., one or more displays), and one or more logic units 1020. In at least one embodiment, one or more computing devices 1000 can include one or more virtual machines (VMs), and / or any component thereof can include a virtual component (e.g., a virtual hardware component). For a non-limiting example, one or more of the GPUs 1008 can include one or more vGPUs, one or more of the CPUs 1006 can include one or more vCPUs, and / or one or more of the logic units 1020 can include one or more virtual logic units. As such, one or more computing devices 1000 can include discrete components (e.g., a full GPU dedicated to the computing device 1000), virtual components (e.g., a portion of a GPU dedicated to the computing device 1000), or a combination thereof.

[0184] Although Figure 10 various blocks of the computing device are shown as being connected to the line via the interconnect system 1002, this is not intended to be limiting, and is merely for clarity. For example, in some embodiments, the presentation components 1018 (such as a display device) can be considered I / O components 1014 (e.g., if the display is a touchscreen). As another example, the CPUs 1006 and / or GPUs 1008 can include memory (e.g., in addition to the memory of the GPUs 1008, the memory 1004 can represent a storage device). Thus, Figure 10 the computing device is merely illustrative. Distinction is not made between a “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “hand-held device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and other device or system types as all are contemplated within the scope of the computing device. Figure 10 ​

[0185] The interconnection system 1002 can represent one or more links or buses linking the various components, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 1002 can include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. For example, the CPU 1006 can be directly connected to the memory 1004. Further, the CPU 1006 can be directly connected to the GPU 1008. Where there are direct connections or point-to-point connections between components, the interconnection system 1002 can include a PCIe link to perform the connection. In these examples, the computing device 1000 need not include a PCI bus.

[0186] The memory 1004 can include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 1000. Computer-readable media can include both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media.

[0187] Computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and / or other data types. For example, the memory 1004 can store computer readable instructions (e.g., representing a program and / or program elements, such as an operating system). Computer storage media can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 1000. Computer storage media, as used herein, does not include signals per se.

[0188] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.

[0189] The CPU(s) 1006 can be configured to execute computer-readable instructions to control one or more components of the computing device 1000 to perform one or more of the methods and / or procedures described herein. The CPU(s) 1006 can each include one or more cores (e.g., 1, 2, 4, 8, 28, 72, etc.) capable of processing multiple software threads simultaneously. The CPU(s) 1006 can include any type of processors and can include different types of processors depending on the type of computing device 1000 being implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 1000, the processor can be an Advanced RISC Machines (ARM) processor implemented using a Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 1000 can include one or more CPUs 1006 in addition to one or more microprocessors or supplemental co-processors such as math co-processors.

[0190] In addition or alternatively to CPU(s) 1006, GPU(s) 1008 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. One or more of GPU(s) 1008 can be integrated GPUs (e.g., with one or more of CPU(s) 1006) and / or one or more of GPU(s) 1008 can be discrete GPUs. In implementations, one or more of GPU(s) 1008 can be a co-processor of one or more of CPU(s) 1006. GPU(s) 1008 can be used by computing device 1000 to render graphics (e.g., 3D graphics) or to perform general-purpose computing. For example, GPU(s) 1008 can be used for general-purpose computing on GPUs (GPGPU). GPU(s) 1008 can include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPU(s) 1008 can generate pixel data for an output image in response to rendering commands (e.g., rendering commands received from CPU(s) 1006 via a host interface). GPU(s) 1008 can include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory can be included as part of memory 1004. GPU(s) 1008 can include two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or can connect the GPUs through a switch (e.g., using NVSwitch). When combined together, each GPU 1008 can generate pixel data or GPGPU data for different portions of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can include its own memory or can share memory with other GPUs.

[0191] In addition or alternative to CPU(s) 1006 and / or GPU(s) 1008, one or more logic units 1020 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. In implementations, CPU(s) 1006, GPU(s) 1008, and / or logic unit(s) 1020 can perform any combination of methods, processes, and / or portions thereof, discretely or jointly. One or more of logic units 1020 can be part of and / or integrated within one or more of CPU(s) 1006 and / or GPU(s) 1008, and / or one or more of logic units 1020 can be discrete components or otherwise external to CPU(s) 1006 and / or GPU(s) 1008. In implementations, one or more of logic units 1020 can be a co-processor of one or more of CPU(s) 1006 and / or one or more of GPU(s) 1008.

[0192] Examples of logic units 1020 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multi-processor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), a programmable vision accelerator (PVA) which can include one or more direct memory access (DMA) systems, one or more vision or vector processing units (VPUs), one or more pixel processing engines (PPEs) — e.g., including a 2D array of processing elements each in north, south, east, and west communication with one or more other processing elements in the array, one or more decoupled accelerators or units (e.g., decoupled lookup table (DLUT) accelerators or units), etc., a vision processing unit (VPU), an optical flow accelerator (OFA), a field programmable gate array (FPGA), a neuromorphic chip, a quantum processing unit (QPU), an associative processing unit (APU), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating-point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) element, etc.

[0193] The communication interface 1010 can include one or more receivers, transmitters, and / or transceivers that allow the computing device 1000 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. The communication interface 1010 can include components and functionality to allow communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., through Ethernet or InfiniBand communications), low power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 1020 and / or the communication interface 1010 can include one or more data processing units (DPUs) for sending data received over a network and / or through the interconnection system 1002 directly to one or more GPUs 1008 (e.g., memory of the one or more GPUs 1008).

[0194] The I / O ports 1012 can allow the computing device 1000 to be logically coupled to other devices including the I / O components 1014, the presentation components 1018, and / or other components, some of which can be built-in (e.g., integrated in) the computing device 1000. Illustrative I / O components 1014 include a microphone, mouse, keyboard, joystick, game pad, game controller, disc-type satellite antenna, scanner, printer, wireless device, etc. The I / O components 1014 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs can be transmitted to an appropriate network element for further processing. A NUI can implement any combination of speech recognition, gesture recognition, facial recognition, biometric recognition, posture recognition, gesture recognition within, and on, a display, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with a display of the computing device 1000. The computing device 1000 can include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touch screen technology, and combinations of these, for instance, to capture motion and / or still images, and can further include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit, or IMU) to detect motion. In some examples, the output of the accelerometers or gyroscopes can be used by an application to render immersive augmented reality or virtual reality.

[0195] The power supply 1016 can include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply 1016 can provide power to the computing device 1000 to enable the components of the computing device 1000 to operate.

[0196] One or more presentation components 1018 may include displays (e.g., monitors, touchscreens, television screens, head-up displays (HUDs), other display types, or combinations thereof), speakers, and / or other presentation components. Presentation component 1018 may receive data from other components (e.g., GPU 1008, CPU 1006, DPU, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0197] Example Data Center

[0198] Figure 11 An example data center 1100 that can be used in at least one embodiment of this disclosure is illustrated. Typically, the example data center 1100 can support the execution of computation and storage within a 3D RCS pipeline. That is, the data center 1100 can process and store data for stages such as reconstruction, partitioning, densification, and / or simulation. The data center 1100 may include a data center infrastructure layer 1110, a framework layer 1120, a software layer 1130, and / or an application layer 1140.

[0199] like Figure 11 As shown, the data center infrastructure layer 1110 may include a resource coordinator 1112, grouped computing resources 1114, and node computing resources (“nodes CR”) 1116(1)-1116(N), where “N” represents any integer, a positive integer. In at least one embodiment, the nodes CR 1116(1)-1116(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (“NW”) devices, and network network interfaces (“NW”). I / O devices, network switches, virtual machines ("VMs"), power modules and / or cooling modules, etc. In some embodiments, one or more nodes CR1116(1)-1116(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR1116(1)-1116(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more nodes CR1116(1)-1116(N) may correspond to virtual machines (VMs).

[0200] In at least one embodiment, the grouped computing resources 1114 may include individual groups of nodes CR1116 housed within one or more racks (not shown) or within a plurality of racks in data centers (also not shown) located in different geographical locations. The individual groups of nodes CR1116 within the grouped computing resources 1114 may include the group's computing, networking, memory, or storage resources, which may be configured or allocated to support one or more workloads. In at least one embodiment, a plurality of nodes CR1116, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0201] Resource coordinator 1112 may be configured or otherwise control one or more nodes CR1116(1)-1116(N) and / or grouped computing resources 1114. In at least one embodiment, resource coordinator 1112 may include a Software Design Infrastructure (“SDI”) management entity for data center 1100. Resource coordinator 1112 may include hardware, software, or some combination thereof.

[0202] In at least one implementation, such as Figure 11 As shown, framework layer 1120 may include a job scheduler 1128, a configuration manager 1134, a resource manager 1136, and / or a distributed file system 1138. Framework layer 1120 may include a framework for software 1132 supporting software layer 1130 and / or one or more applications 1142 supporting application layer 1140. Software 1132 or application 1142 may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1120 may be, but is not limited to, a type of free and open-source software web application framework that can utilize the distributed file system 1138 for large-scale data processing (e.g., "big data"), such as Apache Spark. TM(“Spark”). In at least one embodiment, job scheduler 1128 can include a Spark driver to facilitate scheduling of workloads supported by layers of data center 1100. Configuration manager 1134 can be capable of configuring different layers, such as software layer 1130 and framework layer 1120 including Spark and distributed file system 1138 for supporting large scale data processing. Resource manager 1136 can be capable of managing clustered or grouped computing resources mapped to or allocated for supporting distributed file system 1138 and job scheduler 1128. In at least one embodiment, clustered or grouped computing resources can include grouped computing resources 1114 at data center infrastructure layer 1110. Resource manager 1136 can coordinate with resource orchestrator 1112 to manage these mapped or allocated computing resources.

[0203] In at least one embodiment, software 1132 included in software layer 1130 can include software used by at least portions of node C.R.s 1116(1)-1116(N), grouped computing resources 1114, and / or distributed file system 1138 of framework layer 1120. One or more types of software can include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0204] In at least one embodiment, application programs 1142 included in application layer 1140 can include one or more types of applications used by at least portions of node C.R.s 1116(1)-1116(N), grouped computing resources 1114, and / or distributed file system 1138 of framework layer 1120. One or more types of applications can include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications (including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments).

[0205] In at least one embodiment, any of configuration manager 1114, resource manager 1136, and resource orchestrator 1112 can implement any number and type of self-modifying actions based at least in part on any number and type of data acquired in any technically feasible manner. Self-modifying actions can free data center operators of data center 1100 from making potentially poor configuration decisions and can avoid underutilization and / or poor performance of portions of data center.

[0206] According to one or more embodiments described herein, the data center 1100 can include tools, services, software, or other resources in order to train one or more machine learning models or use one or more machine learning models to predict or infer information. For example, one or more machine learning models can be trained by computing weight parameters according to a neural network architecture using the software and / or computing resources described above with respect to the data center 1100. In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks can be used to infer or predict information using the resources described above with respect to the data center 1100 by using weight parameters computed through one or more training techniques, such as but not limited to those described herein.

[0207] In at least one embodiment, the data center 1100 can use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or virtual computing resources corresponding thereto) to perform training and / or inference using the resources described above. Moreover, the one or more software and / or hardware resources described above can be configured to allow users to train or perform information inference services, such as image recognition, speech recognition, or other artificial intelligence services.

[0208] Example network environment

[0209] A network environment suitable for implementing embodiments of the present disclosure can include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) can be implemented on one or more instances of the computing device 1000, e.g., each device can include similar components, features, and / or functionality of the computing device 1000. Moreover, where backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices can be included as part of the data center 1100, examples of which are described herein with respect to FIG. 11. Figure 10 in more detail. Figure 11

[0210] Components of the network environment can communicate with each other via one or more networks, which can be wired, wireless, or both. The networks can include multiple networks or networks of networks. As an example, the networks can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet and / or the public switched telephone network (PSTN), and / or one or more private networks. Where the networks include wireless telecommunication networks, components such as base stations, communication towers, or even access points (among other components) can provide wireless connectivity.

[0211] ​A compatible network environment can include one or more peer-to-peer network environments - in which case no servers can be included in the network environment - and one or more client-server network environments - in which case one or more servers can be included in the network environment. In a peer-to-peer network environment, functionality described herein with respect to one or more servers can be implemented on any number of client devices.

[0212] In at least one embodiment, the network environment can include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which can include one or more core network servers and / or edge servers. The framework layer can include a framework that supports a software layer and / or one or more applications of an application layer. The software or applications can include web-based service software or applications, respectively. In embodiments, one or more of the client devices can use the web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer can be, without limitation, a type of free and open-source software web application framework, such as can be used for large-scale data processing (e.g., “big data”) using the distributed file system.

[0213] The cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functionality described herein (or one or more portions thereof). Any of these different functionalities can be distributed across multiple locations from central or core servers (e.g., one or more data centers that can be distributed across states, regions, countries, globally, etc.). The core servers can designate at least a portion of the functionality to edge servers if the connection to the user (e.g., client device) is relatively close to the edge servers. The cloud-based network environment can be private (e.g., limited to a single organization), can be public (e.g., available to many organizations), and / or combinations thereof (e.g., a hybrid cloud environment).

[0214] One or more client devices can include a computing device described herein with respect to Figure 10At least some of the components, features and functionalities of the depicted one or more example computing devices 1000 are described. By way of example and not limitation, a client device can be embodied as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smartwatch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a video camera, a surveillance device or system, a vehicle, a watercraft, a spacecraft, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these depicted devices, or any other suitable device.

[0215] The present disclosure can be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules including routines, programs, objects, components, data structures, and the like, refer to code that performs particular tasks or implements particular abstract data types. The present disclosure can be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general- purpose computers, more specialty computing devices, and the like. The present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0216] As used herein, recitations of “and / or” in reference to two or more elements shall be interpreted to mean one element alone, or a combination of elements. For example, “element A, element B, and / or element C” can include just element A, just element B, just element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Further, “at least one of element A or element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, “at least one of element A and element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0217] The subject matter of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms "step" and / or "block" might be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.

Claims

1. A system comprising: One or more processors, said one or more processors being configured to perform one or more operations, said one or more operations being configured to: Obtain video data including a depth map of the scene; A three-dimensional 3D representation of the scene is reconstructed using at least one or more Gaussian sputtering representations of one or more objects in the scene and the depth map; At least one object in the 3D representation is segmented by the following operations: A two-dimensional 2D segmentation mask for generating a reference view of the video data; Interpolate the 2D segmentation mask across multiple frames of the video data; as well as Map the 2D segmentation mask on the plurality of frames onto at least one corresponding region of the plurality of regions of the 3D representation to segment at least one object in the 3D representation from the scene; update at least one region of the plurality of regions of the 3D representation within a threshold distance of the at least one object in the scene; as well as Generate at least one image depicting at least a portion of the 3D representation of the at least one object for display.

2. The system of claim 1, wherein the one or more operations include at least one operation, the at least one operation being used to: The at least one object is densified by the following operation: Sample multiple points on or around the at least one object in the scene; Generate a voxelized volume of at least one object based at least in part on the plurality of points; and The voxelization is updated at least in part based on the occupancy state of at least one of the plurality of voxels of the voxelization, the occupancy state being at least in part based on at least one rendered depth map.

3. The system of claim 2, wherein the one or more processors are configured to perform the one or more operations, the one or more operations comprising: The interior of the voxelized body is filled at least in part by injecting multiple volume elements into the interior of the at least one object, which includes multiple interior regions; as well as Simulate one or more interactions of the voxelized volume of at least one compacted object to update at least one physical property of the at least one object, the at least one compacted object corresponding to a volume representation.

4. The system of claim 3, wherein, in order to simulate the one or more interactions of the voxelized body, the one or more processors are configured to perform a rigid simulation, the rigid simulation comprising: Using a first physical model, a first plurality of transformations are applied to the at least one compacted object to obtain a plurality of rigid motions of the at least one compacted object, at least in part, based on the following operations: The energy function is determined using at least one of multiple scene parameters or multiple object parameters; Minimize the energy function to determine multiple rigid states of the at least one compacted object; as well as The plurality of rigid states are applied to simulate the plurality of rigid movements of the at least one compacted object over time.

5. The system of claim 3, wherein, in order to simulate the one or more interactions of the voxelized body, the one or more processors are configured to perform a resilient simulation, the resilient simulation comprising: Using a second physical model, a second plurality of transformations are applied to the at least one densified object to obtain a plurality of post-deformation states of the at least one densified object, at least in part, based on the following operations: The energy function is determined using at least one of multiple scene parameters or multiple object parameters; Minimize the energy function to determine multiple updates for multiple control points; as well as One or more deformations of the at least one densified object are calculated, at least in part, based on the multiple updates of the multiple control points and the multiple corresponding skin fields.

6. The system according to claim 1, wherein: The one or more operations used to reconstruct the 3D representation of the scene are further based on at least one of: (i) at least one refined pose of the video source, or (ii) a plurality of two-dimensional 2D frames of the video data; and The one or more operations for reconstructing the 3D representation of the scene also include at least one operation for updating at least one initial pose of the video source to the at least one refined pose, at least in part, based on aligning the 3D representation with the plurality of frames of the video data.

7. The system of claim 6, wherein the one or more operations further comprises at least one operation, the at least one operation being used to: An initial Gaussian sputtering representation is generated, at least in part, based on depth data from the depth map and the at least one initial pose corresponding to the video data; and The 3D reconstruction is generated at least in part based on the initial Gaussian sputtering representation, the at least one refined pose of the video data, and the plurality of frames of the video data.

8. The system of claim 1, wherein the one or more operations for segmenting at least one object in the 3D representation include at least one operation using a segmentation model, and wherein the reference view is at least partially based on user input that selects the at least one object, the reference view corresponding to one of the plurality of frames of the video data.

9. The system of claim 1, wherein the one or more operations for updating at least one region of the plurality of regions represented by the 3D within the threshold distance include at least one operation for: At least one region among the plurality of regions within the threshold distance is filled based at least in part on sampling data from one or more neighboring regions; and Remove one or more elements from at least one of the plurality of regions within the threshold distance, and update at least one of the plurality of regions based at least in part on sampling data of one or more of the plurality of regions in the 3D representation.

10. The system of claim 1, wherein the one or more processors are included in at least one of the following: The system used to execute the game; Systems used for performing content streaming; A system used for creating collaborative content; A system used to perform simulation operations; A system for collaborative content creation of 3D assets; A system for generating synthetic data; A system that includes one or more Visual Language Models (VLMs); Systems that include one or more large language model LLMs; Systems used to perform conversational AI operations; A system for performing optical transmission simulation; A system used to perform deep learning operations; Systems used to perform digital twin operations; Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system containing one or more virtual machines (VMs); Systems implemented using robots; Systems implemented using edge devices; A system that is at least partially implemented in a data center; A system that utilizes cloud computing resources at least in part; A system for generating interactive 3D visualizations; or A system that uses at least part of an augmented reality (AR) or virtual reality (VR) platform.

11. One or more processors, comprising: One or more circuits, said one or more circuits being used for: Obtain video data including a depth map of the scene; The scene is reconstructed into a three-dimensional 3D representation using at least one or more Gaussian sputtering representations and the depth map; At least one object in the 3D representation is segmented at least in part based on mapping a two-dimensional 2D segmentation mask of a reference view of the video data on multiple frames onto at least one corresponding region of a plurality of regions of the 3D representation; Update at least one region of the plurality of regions of the 3D representation near the at least one object in the scene; as well as Display at least one image of the 3D representation, the at least one image depicting at least a portion of the at least one object.

12. One or more processors according to claim 11, wherein the one or more circuits are used for: The at least one object is densified by the following operation: Sample multiple points on or approximately around the at least one object; Generate a voxelized volume of at least one object based at least in part on the plurality of points; and The voxelization is updated at least in part based on the occupancy state of at least one of the plurality of voxels of the voxelization, wherein the occupancy state is at least in part based on at least one rendered depth map.

13. One or more processors according to claim 12, wherein the one or more circuits are used for: The interior of the voxelized body is filled at least in part by injecting multiple volume elements into the interior of the at least one object, which includes multiple interior regions; and Simulate one or more interactions of the voxelized volume of at least one compacted object to update at least one physical property of the at least one object, the at least one compacted object corresponding to a volume representation.

14. The processor of claim 13 or more, wherein simulating the one or more interactions comprises: Perform a rigidity simulation, the rigidity simulation including: Using a first physical model, a first plurality of transformations are applied to the at least one compacted object to obtain a plurality of rigid motions of the at least one compacted object, at least in part, based on the following operations: The energy function is determined using at least one of multiple scene parameters or multiple object parameters; Minimize the energy function to determine multiple rigid states of the at least one densified object; and The plurality of rigid states are applied to simulate the plurality of rigid movements of the at least one compacted object over time.

15. The processor of claim 13 or more, wherein simulating the one or more interactions comprises: Performing an elasticity simulation, the elasticity simulation including: Using a second physical model, a second plurality of transformations are applied to the at least one densified object to obtain a plurality of post-deformation states of the at least one densified object, at least in part, based on the following operations: The energy function is determined using at least one of multiple scene parameters or multiple object parameters; Minimize the energy function to determine multiple updates for multiple control points; and One or more deformations of the at least one densified object are calculated, at least in part, based on the multiple updates of the multiple control points and the multiple corresponding skin fields.

16. One or more processors according to claim 11, wherein: The 3D representation of the scene is reconstructed based on at least one of the following: (i) at least one refined pose of the video source, or (ii) multiple two-dimensional 2D frames of the video data; and The 3D representation of the scene is reconstructed by updating at least one initial pose of the video source to the at least one refined pose, at least in part, based on aligning the 3D representation with the plurality of 2D frames of the video data.

17. A method comprising: Video data, including a depth map of the scene, is obtained by one or more processors; The scene is reconstructed into a three-dimensional 3D representation by the one or more processors using at least one or more Gaussian sputtering representations and the depth map; At least one object in the 3D representation is segmented by the one or more processors; The one or more processors update at least one region of the plurality of regions of the 3D representation within a distance of the at least one object in the scene; The at least one object is densified by one or more processors through the following operations: Generate a voxelized version of the at least one object; and The voxelization is updated at least in part based on the occupancy state of at least one of the plurality of voxels of the voxelization, the occupancy state being at least in part based on at least one rendered depth map; The one or more processors simulate one or more interactions of the voxelized volume of at least one dense object; as well as The one or more processors display at least one image depicting at least a portion of the at least one densified object.

18. The method of claim 17, further comprising: The interior of the voxelized body is filled by the one or more processors, at least in part, based on the injection of multiple volume elements into the interior of the at least one object, which includes multiple interior regions.

19. The method of claim 17, wherein simulating the one or more interactions comprises: Perform a rigidity simulation, the rigidity simulation including: The one or more processors use a first physical model to apply a first plurality of transformations to the at least one densified object to obtain a plurality of rigid motions of the at least one densified object based at least in part on the following operations: The energy function is determined using at least one of multiple scene parameters or multiple object parameters; Minimize the energy function to determine multiple rigid states of the at least one densified object; and The plurality of rigid states are applied to simulate the plurality of rigid movements of the at least one compacted object over time.

20. The method of claim 17, wherein simulating the one or more interactions comprises: Performing an elasticity simulation, the elasticity simulation including: The one or more processors use a second physical model to apply a second plurality of transformations to the at least one densified object to obtain a plurality of post-deformation states of the at least one densified object based at least in part on the following operations: An energy function is determined using at least one of multiple scene parameters or multiple object parameters; the energy function is minimized to determine multiple updates for multiple control points; and One or more deformations of the at least one densified object are calculated, at least in part, based on the multiple updates of the multiple control points and the multiple corresponding skin fields.

Images