Gradient guided object reconstruction

Abstract

Gradient guided object reconstruction includes rendering a predicted image from a current object model of an object in a scenario. Multiple gradients are generated based on a comparison of the predicted image with a real image. The current object model is augmented with the gradients to generate a reconstruction input. A reconstruction network processes the reconstruction input to generate an updated object model as the current object model and the updated object model is stored.

Description

GRADIENT GUIDED OBJECT RECONSTRUCTIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a non-provisional application of U.S. Provisional Patent Application Serial Number 63 / 562,681, filed on March 7, 2024, which is hereby incorporated by reference in its entirety.BACKGROUND

[0002] A virtual world is a computer-simulated environment, which enables a player to interact in a three-dimensional space as if the player were in the real world. In some cases, the virtual world is designed to replicate at least some aspects of the real world. For example, the virtual world may include objects and background reconstructed from the real world. The reconstructing of objects and background from the real world allows the system to replicate aspects of the real world.

[0003] For example, one way to bring realism is by obtaining sensor data from the real world describing a scenario, modifying the scenario to create an augmented scenario, and then allowing the player to interact with the augmented scenario. When the player interacts with the augmented scenario, different objects may be in different relative positions than in the real world. Thus, in order to modify the real world, an accurate set of models should be created and used in the virtual world.

[0004] A problem exists in generating an accurate set of models from sensor data of real-world objects. Neural rendering uses a pre-scene optimization process that recreates the sensor data using differential rendering. This optimization process needs significant computing resources. With large scenes having many objects, the generation of object models is often cost prohibitive in terms of computing resources required. Additionally, when viewpoint changes are large, artifacts may exist in the rendered image.SUMMARYIn general, in one aspect, one or more embodiments relate to a method that includes rendering a predicted image from a current object model of an objectin a scenario. Multiple gradients are generated based on a comparison of the predicted image with a real image. The current object model is augmented with the gradients to generate a reconstruction input. A reconstruction network processes the reconstruction input to generate an updated object model as the current object model and the updated object model is stored.In general, in one aspect, one or more embodiments relate to a system that includes storage and a computer processor connected to storage for executing instructions that cause the computer processor to perform operations. The operations include rendering a predicted image from a current object model of an object in a scenario. Multiple gradients are generated based on a comparison of the predicted image with a real image. The current object model is augmented with the gradients to generate a reconstruction input. A reconstruction network processes the reconstruction input to generate an updated object model as the current object model and the updated object model is stored.In general, in one aspect, one or more embodiments relate to a non-transitory computer readable medium that includes computer readable program code for causing a computing system to perform operations. The operations include rendering a predicted image from a current object model of an object in a scenario. Multiple gradients are generated based on a comparison of the predicted image with a real image. The current object model is augmented with the gradients to generate a reconstruction input. A reconstruction network processes the reconstruction input to generate an updated object model as the current object model and the updated object model is stored.

[0005] Other aspects of one or more embodiments will be apparent from the following description and the appended claims.BRIEF DESCRIPTION OF DRAWINGS

[0006] FIG. 1 shows a diagram of an autonomous training and testing system in accordance with one or more embodiments.

[0007] FIG. 2 shows a flowchart of the autonomous training and testing system in accordance with one or more embodiments.

[0008] FIG. 3 shows a diagram of a model generation system in accordance with one or more embodiments.

[0009] FIG. 4 shows a flowchart for creating an object model in accordance with one or more embodiments.

[0010] FIG. 5 shows an example diagram of gradient guided object reconstruction in accordance with one or more embodiments.

[0011] FIG. 6 shows an example in accordance with one or more embodiments.

[0012] FIG. 7A and FIG. 7B show a computing system in accordance with one or more embodiments.

[0013] Like elements in the various figures are denoted by like reference numerals for consistency.DETAILED DESCRIPTION

[0014] In general, embodiments are directed to generating realistic object models of objects in the real world using sensor data. One or more embodiments reduce the time to generate the realistic object models by performing gradient guided object reconstruction. A current object model is used to generate a predicted image of a scenario from a particular perspective. The predicted image is compared to a real image to generate a set of gradients describing the differences. The current object model and the set of gradients are combined into reconstruction input that is used by a reconstruction network. The reconstruction network is a machine learning model that produces an updated object model from the reconstruction input. By incorporating the gradients with the current object model and using the reconstruction network, iterations to update the object model are reduced. Thus, one or more embodiments create a realistic object model using fewer computing resources.

[0015] The objects that correspond to the object models may be part of generating a simulated environment for the training and testing of autonomoussystems. An autonomous system is a self-driving mode of transportation that does not require a human pilot or human driver to move and react to the real-world environment. Rather, the autonomous system includes a virtual driver that is the decision-making portion of the autonomous system. The virtual driver is an artificial intelligence system that leams how to interact in the real world. The autonomous system may be completely autonomous or semi-autonomous. As a mode of transportation, the autonomous system is contained in a housing configured to move through a real-world environment. Examples of autonomous systems include self-driving vehicles (e.g., self-driving trucks and cars), drones, airplanes, robots, etc. The virtual driver is the software that makes decisions and causes the autonomous system to interact with the real world including moving, signaling, and stopping or maintaining a current state.

[0016] The real-world environment is the portion of the real world through which the autonomous system, when trained, is designed to move. Thus, the real- world environment may include interactions with concrete and land, people, animals, other autonomous systems, human driven systems, construction, and other objects as the autonomous system moves from an origin to a destination. In order to interact with the real-world environment, the autonomous system includes various types of sensors, such as LiDAR sensors, which are used to obtain measurements of the real-world environment, and cameras that capture images from the real-world environment.

[0017] The testing and training of the virtual driver of the autonomous systems in the real-world environment is unsafe because of the accidents that an untrained virtual driver can cause. Thus, the augmented images generated using the object models from claimed techniques may be provided as part of a video to a virtual camera of an autonomous system. The player in the example is the virtual driver of the autonomous system and the perspective of the augmented images in the video is based on the position of the virtual camera in the virtual world.

[0018] The testing and training of a virtual driver may be performed using the simulator (100) described in FIG. 1. As shown in FIG. 1, a simulator (100) isconfigured to train and test a virtual driver (102) of an autonomous system. For example, the simulator (100) may be a unified, modular, mixed reality, closed-loop simulator for autonomous systems. The simulator (100) is a configurable simulation framework that enables not only evaluation of different autonomy components in isolation, but also as a complete system in a closed-loop manner. The simulator reconstructs “digital twins” of real-world scenarios automatically, enabling accurate evaluation of the virtual driver at scale. The simulator (100) may also be configured to perform mixed-reality simulation that combines real-world data and simulated data to create diverse and realistic evaluation variations to provide insight into the virtual driver’s performance. The mixed reality closed-loop simulation allows the simulator (100) to analyze the virtual driver’s (102) action on counterfactual “what-if’ scenarios that did not occur in the real world. The simulator (100) further includes functionality to simulate and train on rare yet safety-critical scenarios with respect to the entire autonomous system and closed-loop training to enable automatic and scalable improvement of autonomy.

[0019] The simulator (100) creates the simulated environment (104) which is a virtual world. The virtual driver (102) is the player in the virtual world. The simulated environment (104) is a simulation of a real -world environment, which may or may not be in actual existence, in which the autonomous system is designed to move. As such, the simulated environment (104) includes a simulation of the objects (z.e., simulated objects or assets) and background in the real world, including the natural objects, construction, buildings and roads, obstacles, as well as other autonomous and non-autonomous objects. The simulated environment (104) simulates the environmental conditions within which the autonomous system may be deployed. Additionally, the simulated environment (104) may be configured to simulate various weather conditions that may affect the inputs to the autonomous systems. The simulated objects may include both stationary and nonstationary objects. Nonstationary objects are actors in the real-world environment.

[0020] The simulator (100) also includes an evaluator (110). The evaluator (110) is configured to train and test the virtual driver (102) by creating various scenarios in the simulated environment (104). Each scenario is a configuration of the simulated environment (104) including, but not limited to, static portions, movement of simulated objects, actions of the simulated objects with each other, and reactions to actions taken by the autonomous system and simulated objects. The evaluator (110) is further configured to evaluate the performance of the virtual driver (102) using a variety of metrics.

[0021] The evaluator (110) assesses the performance of the virtual driver (102) throughout the performance of the scenario. Assessing the performance may include applying rules. For example, the rules may be that the automated system does not collide with any other actor, compliance with safety and comfort standards (e.g., passengers not experiencing more than a certain acceleration force within the vehicle), the automated system not deviating from an executed trajectory, or other rule. Each rule may be associated with the metric information that relates a degree of breaking the rule with a corresponding score. The evaluator (110) may be implemented as a data-driven neural network that learns to distinguish between good and bad driving behavior. The various metrics of the evaluation system may be leveraged to determine whether the automated system satisfies the requirements of the success criterion for a particular scenario. Further, in addition to system level performance, for modular based virtual drivers, the evaluator (110) may also evaluate individual modules such as segmentation or prediction performance for actors in the scenario with respect to the ground truth recorded in the simulator (100).

[0022] The simulator (100) is configured to operate in multiple phases as selected by the phase selector (108) and modes as selected by a mode selector (106). The phase selector (108) and mode selector (106) may be a graphical user interface or application programming interface component that is configured to receive a selection of phase and mode, respectively. The selected phase and mode define the configuration of the simulator (100). Namely, the selected phase andmode define which system components communicate and the operations of the system components.

[0023] The phase may be selected using a phase selector (108). The phase may be a training phase or a testing phase. In the training phase, the evaluator (110) provides metric information to the virtual driver (102), which uses the metric information to update the virtual driver (102). The evaluator (110) may further use the metric information to further train the virtual driver (102) by generating scenarios for the virtual driver (102). In the testing phase, the evaluator (110) does not provide the metric information to the virtual driver (102). In the testing phase, the evaluator (110) uses the metric information to assess the virtual driver (102) and to develop scenarios for the virtual driver (102).

[0024] The mode may be selected by the mode selector (106). The mode defines the degree to which real -world data is used, whether noise is injected into simulated data, the degree of perturbations of real-world data, and whether the scenarios are designed to be adversarial. Example modes include open-loop simulation mode, closed-loop simulation mode, single module closed-loop simulation mode, fuzzy mode, and adversarial mode. In an open-loop simulation mode, the virtual driver is evaluated with real-world data. In a single module closed-loop simulation mode, a single module of the virtual driver is tested. An example of a single module closed-loop simulation mode is a localizer closed- loop simulation mode in which the simulator evaluates how the localizer estimated pose drifts over time as the scenario progresses in simulation. In a training data simulation mode, the simulator is used to generate training data. In a closed-loop evaluation mode, the virtual driver and simulation system are executed together to evaluate system performance. In the adversarial mode, the actors are modified to perform adversarially. In the fuzzy mode, noise is injected into the scenario (e.g., to replicate signal processing noise and other types of noise). Other modes may exist without departing from the scope of the system.

[0025] The simulator (100) includes the controller (112) which includes functionality to configure the various components of the simulator (100)according to the selected mode and phase. Namely, the controller (112) may modify the configuration of each of the components of the simulator (100) based on the configuration parameters of the simulator (100). Such components include the evaluator (110), simulated environment (104), autonomous system model (116), sensor simulation models (114), asset models (117), actor models (118), latency models (120), and a training data generator (122).

[0026] The autonomous system model (116) is a detailed model of the autonomous system in which the virtual driver (102) will execute. The autonomous system model (116) includes model, geometry, physical parameters (e.g., mass distribution, points of significance), engine parameters, sensor locations and type, the firing pattern of the sensors, information about the hardware on which the virtual driver executes (e.g., processor power, amount of memory, and other hardware information), and other information about the autonomous system. The various parameters of the autonomous system model may be configurable by the user or another system.

[0027] For example, if the autonomous system is a motor vehicle, the modeling and dynamics may include the type of vehicle (e.g., car, truck), make and model, geometry, physical parameters such as the mass distribution, axle positions, type and performance of the engine, etc. The vehicle model may also include information about the sensors on the vehicle (e.g., camera, LiDAR, etc.), the sensors’ relative firing synchronization pattern, and the sensors’ calibrated extrinsics (e.g., position and orientation) and intrinsics (e.g., focal length). The vehicle model also defines the onboard computer hardware, sensor drivers, controllers, and the autonomy software release under test.

[0028] The autonomous system model (116) includes an autonomous system dynamic model. The autonomous system dynamic model is used for dynamics simulation that takes the actuation actions of the virtual driver (e.g., steering angle, desired acceleration) and enacts the actuation actions on the autonomous system in the simulated environment (104) to update the simulated environment (104) and the state of the autonomous system. To update the state, a kinematicmotion model may be used, or a dynamics motion model that accounts for the forces applied to the vehicle may be used to determine the state. Within the simulator, with access to real log scenarios with ground truth actuations and vehicle states at each time step, embodiments may also optimize analytical vehicle model parameters or learn parameters of a neural network that infers the new state of the autonomous system given the virtual driver outputs.

[0029] In one or more embodiments, the sensor simulation model (114) models, in the simulated environment, active and passive sensor inputs. Passive sensor inputs capture the visual appearance of the simulated environment (104) including stationary and nonstationary simulated objects from the perspective of one or more cameras based on the simulated position of the camera(s) within the simulated environment (104). Examples of passive sensor inputs include inertial measurement unit (IMU) and thermal. Active sensor inputs are inputs to the virtual driver of the autonomous system from the active sensors, such as LiDAR, RADAR, global positioning system (GPS), ultrasound, etc. Namely, the active sensor inputs include the measurements taken by the sensors, and the measurements being simulated based on the simulated environment, based on the simulated position of the sensor(s), within the simulated environment. By way of an example, the active sensor measurements may be measurements that a LiDAR sensor would make of the simulated environment over time and in relation to the movement of the autonomous system.

[0030] The sensor simulation model (114) is configured to simulate the sensor observations of the surrounding scene in the simulated environment (104) at each time step according to the sensor configuration on the vehicle platform. The scene is the scenario at a particular timestep. Namely, the scene is the static position of objects including orientation and other attributes of the objects in the scenario at the particular timestep. When the simulated environment directly represents the real-world environment, without modification, the sensor output may be directly fed into the virtual driver (102). For light-based sensors, the sensor model simulates light as rays that interact with objects in the scene togenerate the sensor data. Depending on the asset representation (e.g., of stationary and nonstationary objects), embodiments may use graphics-based rendering for assets with textured meshes, neural rendering, or a combination of multiple rendering schemes. Leveraging multiple rendering schemes enables customizable world building with improved realism. Because assets are compositional in 3D and support a standard interface of render commands, different asset representations may be composed in a seamless manner to generate the final sensor data. Additionally, for scenarios that replay what happened in the real world and use the same autonomous system as in the real world, the original sensor observations may be replayed at each time step.

[0031] The sensor simulation model (114) is closely connected to object models. The object models are the models of the objects in the simulated environment that may exist in the real world. Object models created from real- world objects may be created by the model generation system (300) described in FIG. 3. The object models including the object models generated by the model generation system (300) described in FIG. 3 may include asset models (117) and actor models (118) shown in FIG. 1. Further, one or more of the asset models may be a background model, which is a model of the background objects as if the background were a single object.

[0032] In one or more embodiments, the system may be a bifurcated system whereby the operations (e.g., trajectories or positioning) of the assets and actors are defined separately from the appearance, which is generated by the model generation system (300).

[0033] Asset models (117) include multiple models, each model modeling a particular type of individual asset in the real world. The assets may include inanimate objects such as construction barriers or traffic signs, parked cars, and background. The background may include any asset farther than a predefined distance from the player, and may include buildings, vegetation, sky, and other components that do not have independent movement of other assets in the background. Each of the entities in a scenario may correspond to an individualasset. As such, an asset model (117), or instance of a type of asset model (117), may exist for each of the objects or assets in the scenario. The assets can be composed together to form the three-dimensional simulated environment. An asset model (117) provides all the information needed by the simulator to simulate the asset. The asset model (117) provides the information used by the simulator (100) to represent and simulate the asset in the simulated environment (104).

[0034] Closely related to, and possibly considered part of the set of asset models (117) are actor models (118). An actor model (118) represents an actor in a scenario. An actor is a sentient being that has an independent decision-making process. Namely, in the real world, the actor may be an animate being (e.g., a person or animal) that makes a decision based on an environment or may be another autonomous system. The actor makes active movement rather than or in addition to passive movement. An actor model (118), or an instance of an actor model (118) may exist for each actor in a scenario. The actor model (118) is a model of the actor. If the actor is in a mode of transportation, then the actor model (118) includes the model of transportation mode in which the actor is located. For example, actor models may represent pedestrians, children, vehicles being driven by drivers, pets, bicycles, and other types of actors.

[0035] The actor model leverages the scenario specification and assets to control all actors in the scene and the actor’s actions at each time step. The actor’s behavior is modeled in a region of interest centered around the autonomous system. Depending on the scenario specification, the actor simulation controls the actors in the simulation to achieve the desired behavior. Actors can be controlled in various ways. One option is to leverage heuristic actor models, such as an intelligent-driver model (IDM) that tries to maintain a certain relative distance or time-to-collision (TTC) from a lead actor or heuristic-derived lane-change actor model. Another is to directly replay actor trajectories from a real log or to control the actor(s) with a data-driven traffic model. Through the configurable design, embodiments may mix and match different subsets of actors to be controlled by different behavior models. For example, far-away actors thatinitially may not interact with the autonomous system and can follow a real log trajectory, but when near the vicinity of the autonomous system may switch to a data-driven actor model. In another example, actors may be controlled by a heuristic or data-driven actor model that still conforms to the high-level route in a real log. This mixed-reality simulation provides control and realism.

[0036] Further, actor models may be configured to be in a cooperative or adversarial mode. In cooperative mode, the actor model models actors to act rationally in response to the state of the simulated environment. In adversarial mode, the actor model may model actors acting irrationally, such as exhibiting road rage and bad driving.

[0037] The latency model (120) represents timing latency that occurs when the autonomous system is in the real-world environment. Several sources of timing latency may exist. For example, a latency may exist from the time that an event occurs to the sensors detecting the sensor information from the event and sending the sensor information to the virtual driver. Another latency may exist based on the difference between the computing hardware executing the virtual driver in the simulated environment as compared to the computing hardware of the virtual driver. Further, another timing latency may exist between the time that the virtual driver transmits an actuation signal to the autonomous system changing (e.g., direction or speed) based on the actuation signal. The latency model (120) models the various sources of timing latency.

[0038] Stated another way, in the real world, safety-critical decisions in the real world may involve fractions of a second affecting response time. The latency model simulates the exact timings and latency of different components of the onboard system. To enable scalable evaluation without strict requirements on exact hardware, the latencies and timings of the different components of the autonomous system and sensor modules are modeled while running on different computer hardware. The latency model may replay latencies recorded from previously collected real-world data or have a data-driven neural network that infers latencies at each time step to match the hardware in a loop simulation setup.

[0039] The training data generator (122) is configured to generate training data. For example, the training data generator (122) may modify real -world scenarios to create new scenarios. The modification of real -world scenarios is referred to as mixed reality. For example, mixed-reality simulation may involve adding in new actors with novel behaviors, changing the behavior of one or more of the actors from the real world, and modifying the sensor data in that region while keeping the remainder of the sensor data the same as the original log. In some cases, the training data generator (122) converts a benign scenario into a safety-critical scenario.

[0040] The simulator (100) is connected to a data repository (105). The data repository (105) is any type of storage unit or device that is configured to store data. The data repository (105) includes data gathered from the real world. For example, the data gathered from the real world includes real actor trajectories (126), real sensor data (128), real trajectories of the system capturing the real trajectory (130), and real latencies (132). Each of the real actor trajectories (126), real sensor data (128), real trajectory of the system capturing the real trajectory (130), and real latencies (132) is data captured by or calculated directly from one or more sensors from the real world (e.g., in a real -world log). In other words, the data gathered from the real world are actual events that happened in real life. For example, in the case that the autonomous system is a vehicle, the real-world data may be captured by a vehicle driving in the real world with sensor equipment.

[0041] Further, the data repository (105) includes functionality to store one or more scenario specifications (140). A scenario specification (140) specifies a scenario and evaluation setting for testing or training the autonomous system. For example, the scenario specification (140) may describe the initial state of the scenario, such as the current state of the autonomous system (e.g., the full 6D pose, velocity and acceleration), the map information specifying the road layout, and the scene layout specifying the initial state of all the dynamic actors and objects in the scenario. The scenario specification may also include dynamic actor information, describing how the dynamic actors in the scenario shouldevolve over time, which are inputs to the actor models. The dynamic actor information may include route information for the actors, desired behaviors, or aggressiveness. The scenario specification (140) may be specified by a user, programmatically generated using a domain-specification language (DSL), procedurally generated with heuristics from a data-driven algorithm, or adversarial-based generation. The scenario specification (140) can also be conditioned on data collected from a real-world log, such as taking place on a specific real-world map or having a subset of actors defined by their original locations and trajectories.

[0042] The interfaces between the virtual driver (102) and the simulator (100) match the interfaces between the virtual driver (102) and the autonomous system in the real world. For example, the sensor simulation model (114) and the virtual driver (102) matches the virtual driver (102) interacting with the sensors in the real world. The virtual driver (102) is the actual autonomy software that executes on the autonomous system. The simulated sensor data that is output by the sensor simulation model (114) may be in or converted to the exact message format that the virtual driver takes as input as if the virtual driver were in the real world, and the virtual driver can then run as a black box virtual driver with the simulated latencies incorporated for components that run sequentially. The virtual driver (102) then outputs the exact same control representation that it uses to interface with the low-level controller on the real autonomous system. The autonomous system model (116) will then update the state of the autonomous system in the simulated environment (104). Thus, the various simulation models of the simulator (100) run in parallel asynchronously at their own frequencies to match the real -world setting.

[0043] FIG. 2 shows a flow diagram for executing the simulator in a closed- loop mode. In Block 201, a scenario is generated as a simulated environment state. Log data from the real world may be used to generate an initial virtual world. The log data defines which asset and actor models are used in the initial positioning of assets. For example, using convolutional neural networks on thelog data, the various asset types within the real world may be identified. As other examples, offline perception systems and human annotations of log data may be used to identify asset types. In some cases, different asset and actors may be placed in the scenario or assets and actors may be moved in the scenario.Additionally, the goal locations (z.e., the locations in which the actors are directed to moving) may be changed. For example, a vehicle turning left may be changed to stay straight or turn right.

[0044] Accordingly, corresponding asset and actor models may be identified based on the asset types and added to the positions of the real actors and assets in the real world. Thus, the asset and actor models create an initial three dimensional virtual world.

[0045] In Block 203, the sensor simulation model is executed on the simulated environment state to obtain simulated sensor output. The sensor simulation model may use beamforming and other techniques to replicate the view to the sensors of the autonomous system. Each sensor of the autonomous system has a corresponding sensor simulation model and a corresponding system. The sensor simulation model executes based on the position of the sensor within the virtual environment and generates simulated sensor output. The simulated sensor output is in the same form as would be received from a real sensor by the virtual driver. In one or more embodiments, Block 203 may be performed as shown in FIG. 5 and FIG. 6 (described below) to generate camera output for a virtual camera. The processing of FIG. 5 and FIG. 6 may be performed for each of the virtual cameras on the autonomous system. Some of the operations may be performed once and the data generated reused for the different cameras or even for different scenarios. For example, the same generated source light representation may be used for generating augmented images for multiple cameras without regenerating the source light representation and for generating multiple different lighting scenarios. Similarly, the same selection of a target light representation may be used for multiple cameras. The operations of FIG. 5 and FIG. 6 may be performed for each camera and LiDAR sensor on the autonomous system to simulate the outputof the corresponding camera and LiDAR sensor. Location and viewing direction of the sensor with respect to the autonomous vehicle may be used to replicate the originating location of the corresponding virtual sensor on the simulated autonomous system. Thus, the various sensor inputs to the virtual driver match the combination of inputs if the virtual driver were in the real world.

[0046] The simulated sensor output is passed to the virtual driver. In Block 205, the virtual driver executes based on the simulated sensor output to generate actuation actions. The actuation actions define how the virtual driver controls the autonomous system. For example, for a self-driving vehicle, the actuation actions may be the amount of acceleration, movement of the steering, triggering of a turn signal, etc. From the actuation actions, the autonomous system state in the simulated environment is updated in Block 207. The actuation actions are used as input to the autonomous system model to determine the actual actions of the autonomous system. For example, the autonomous system dynamic model may use the actuation actions in addition to road and weather conditions to represent the resulting movement of the autonomous system. For example, in a wet or snowy environment, the same amount of acceleration action as in a dry environment may cause less acceleration than in the dry environment. As another example, the autonomous system model may account for possibly faulty tires (e.g., tire slippage), mechanical based latency, or other possible imperfections in the autonomous system.

[0047] In Block 209, actors’ actions in the simulated environment are modeled based on the simulated environment state. Concurrently with the virtual driver model, the actor models and asset models are executed on the simulated environment state to determine an update for each of the assets and actors in the simulated environment. Here, the actors’ actions may use the previous output of the evaluator to test the virtual driver. For example, if the actor is adversarial, the evaluator may indicate based on the previous action of the virtual driver, the lowest scoring metric of the virtual driver. Using a mapping of metrics on theactions of the actor model, the actor model executes to exploit or test that particular metric.

[0048] Thus, in Block 211, the simulated environment state is updated according to the actors’ actions and the autonomous system state to generate an updated simulated environment state. The updated simulated environment includes the change in position of the actors and the autonomous system. Because the models execute independently of the real world, the update may reflect a deviation from the real world. Thus, the autonomous system is tested with new scenarios. In Block 213, a determination is made whether to continue. If the determination is made to continue, testing of the autonomous system continues using the updated simulated environment state in Block 203. At each iteration, during training, the evaluator provides feedback to the virtual driver. Thus, the parameters of the virtual driver are updated to improve the performance of the virtual driver in a variety of scenarios. During testing, the evaluator is able to test using a variety of scenarios and patterns including edge cases that may be safety critical. Thus, one or more embodiments improve the virtual driver and increase the safety of the virtual driver in the real world.

[0049] As shown, the virtual driver of the autonomous system acts based on the scenario and the current learned parameters of the virtual driver. The simulator obtains the actions of the autonomous system and provides a reaction in the simulated environment to the virtual driver of the autonomous system. The evaluator evaluates the performance of the virtual driver and creates scenarios based on the performance. The process may continue as the autonomous system operates in the simulated environment.

[0050] FIG. 3 shows a diagram of a model generation system (300) to generate object models. As shown in FIG. 3, the model generation system (300) includes a data repository (302) connected to an object model framework (304). Both of these components are described below.

[0051] The data repository (302) is a type of storage unit or device (e.g., a file system, database, data structure, or any other storage mechanism) for storing data.The data repository (302) may include multiple different, potentially heterogeneous, storage units and / or devices. The data repository (302) includes functionality to store sensor data (128) and object models (e.g., object model X (306), object model Y (308), background model (310)).

[0052] The sensor data (128) is real sensor data (128) described in FIG. 1. The sensor data (128) is data gathered from actual or real sensors. For example, the sensor data may be from sensors attached to an actual object moving through the real world. Sensor data (128) may include LiDAR point clouds (328) and actual images (330). LiDAR point clouds are images captured using a LiDAR sensor. Actual images (330) are images captured with a camera, such as a visible light camera.

[0053] The object models (object model X (306), object model Y (308), background model (310)) are models of actors or assets in the real world. Specifically, the object models are the asset models (117) or actor models (118) described in reference to FIG. 1. The object models may also include a background model as described in reference to FIG. 1. Different storage techniques may be used to store each object model. For example, the object model may be stored as a neural hash grid. In a neural hash grid, the object is represented as a grid having a hierarchy of resolutions and each location in the grid has neural features. As another example, the object model may be represented as a set of 3D Gaussians. A 3D Gaussian is defined in 3D space. Each Gaussian represents a particular point on the virtual object and, correspondingly, has a particular location on the object. The Gaussian has a set of attributes. For example, the attributes may define a position in three dimensional (3D) space, scale, orientation, color, and opacity. The Gaussian may be a 3D neural Gaussian. A 3D neural Gaussian may include or be a feature vector describing latent features. A feature vector has encoded information about the scene or object. In some cases, the 3D Gaussian has the attributes described above as well as the latent feature vector describing latent features. The Gaussian may be a 3D color Gaussian. The 3D color Gaussian includes attributes thatexplicitly define the color at the location along with other attributes of the location on the object.

[0054] Continuing with FIG. 3, the data repository (302) is connected to an object model framework (304). The object model framework (304) is software configured to generate an object model from sensor data (128). The object model framework (128) is designed to generate a realistic object model representing objects in the real world even when sensor data does not exist for the different perspectives at the object. For example, the object model may generate a 3D model when sensor data is provided only for a discrete set of locations in the real world looking at one side of the corresponding real -world object.

[0055] The object model framework (304) includes a conversion model (332), an image generator (334), a gradient generator (336), a reconstruction network (338), and a reconstruction network loss function (340). The conversion model (332) is a machine learning model that is configured to generate 3D color Gaussians from 3D neural Gaussians. In one or more embodiments, the conversion model (332) is a multilayer perceptron (MLP) model.

[0056] The image generator (334) is configured to render a predicted image directly or indirectly from the object models (object model X (306), object model Y (308), background model (310)). The predicted image is the image that is generated based on the current object models if the image generation process is accurate. For example, the image generator (334) may be configured to define location and orientation of objects in a scene. The image generator may be a tile rasterizer, configured to project each point in the scene onto an image plane based on scale, color, orientation, and opacity.

[0057] The gradient generator (336) is configured to compare the predicted image to an actual image. The actual image may be a LiDAR image from the LiDAR point clouds (328) in the real sensor data (128). As another example, the actual image may be a camera image from a visible light camera or one of actual images (330). The comparison is a set of gradients. The gradient generator (336) is configured to determine the difference between the rendered image and theactual image and then backpropagate the gradients through the scene, including the points on the various object models in the scene.

[0058] The reconstruction network (338) is a machine learning model that is configured to generate an updated object for an object using reconstruction input that includes the current object model and the gradients. For example, the reconstruction network (338) may be a convolutional neural network. The convolutional neural network may be a sparse convolutional neural network because most of the points in the scene do not have values (e.g., an object is not at the location in the scene).

[0059] The reconstruction network loss function (340) is a configured to identify losses for backpropagating through reconstruction network (338). Specifically, the reconstruction network loss function (340) is configured to determine a loss for training the reconstruction network.

[0060] FIG. 4 shows a flowchart for generating an object model in accordance with one or more embodiments. The operations of FIG. 4 may be implemented using the system of FIG. 1 and FIG. 3 in one or more embodiments. Further, the output of FIG. 4 may be used by the system of FIG. 1 to perform a simulation during a scenario in one or more embodiments.

[0061] Block 400 includes obtaining a current object model. Different techniques may be used to initialize the object model. For example, image recognition may be performed on a real image to identify an object type of a real- world object. Then, a predefined object model may be obtained from a library of object models that matches the type. For example, if the object type is a sedan, a predefined object model may be obtained that matches the sedan type. As another example, if the object is detected as being a particular make and model of a vehicle, then the object model may be initialized from a library for that make and model. Notably, the initialized object model may not account for the customizations or the deficits of the particular object in the real world. In some cases, the current object model is defined as a set of 3D Gaussians defining points of the object. The set of 3D Gaussians may be initially obtained from a library ortranslated from a model in the library. Further, in some cases, the 3D Gaussians may each be augmented with a latent feature vector. The latent feature vector may be randomly initialized.

[0062] Block 402 includes rendering a predicted image from a current object model of an object in a scenario. To render the predicted image, the objects are defined in a scene as defined by a real image. Specifically, the locations of the object models corresponding to the objects and a background model are defined. The location of the virtual sensor in the scene is defined to match the location of the virtual sensor.

[0063] A tile rasterizer or other model may project the scene onto an image plane corresponding to the sensor. In the case that the object models are 3D neural Gaussians, for each point projected, a conversion model may convert the 3D neural Gaussians into 3D color Gaussians. Specifically, the conversion may be performed by a MLP model that includes a skip connection to update 3D Gaussian attributes in the 3D color Gaussians. Further, opacity may be used to blend color values for the points that are projected onto the same part of the plane. The result of Block 402 is a predicted image. The predicted image is the real image if the object models are accurate.

[0064] Block 404 includes generating gradients based on a comparison of the predicted image with a real image. The values of the predicted image are compared with the values of the real image to identify a difference amount for each location in the predicted image. The difference amount is used to generate a loss that is backpropagated onto points of the object models, including the background model, that is projected on that location.

[0065] More specifically, in at least some embodiments, the generation of gradients begins with a pixel-wise comparison between the predicted image and the real image. For each pixel location in the image plane, the system calculates an error value representing the difference between the predicted pixel value and the real image pixel value. These error values collectively form an error map across the entire image.

[0066] The error map is then used to compute a loss function that measures the overall difference between the predicted and real images. Once the loss is computed, the system determines how this loss should affect each point in the current object model through backpropagation. Determining how loss should affect each point involves tracing the contribution of each 3D point in the object model to pixels in the rendered image. The differentiable Tenderer maintains this relationship during the forward rendering pass, enabling the system to attribute error values at specific pixel locations back to the corresponding 3D points that influenced those pixels.

[0067] For each point in the object model, the system computes a gradient vector indicating how the point’s parameters (position, scale, orientation, color, and opacity) should change to reduce the overall loss. Points that contribute to regions with high error values receive larger gradient magnitudes, while points that do not contribute to any pixels in the current view receive gradient values of zero.

[0068] The gradients encode the direction and magnitude of necessary changes to bring the predicted image closer to the real image. As explained below, the system then augments the current object model with these gradients and processes them through the reconstruction network. This enables the network to learn updated strategies that consider relationships between different parts of the scene, producing more coherent updates to the object model than standard gradient descent while requiring fewer iterations to achieve high-quality results.

[0069] Block 406 includes augmenting the current object model with the gradients to generate a reconstruction input. Thus, for the current object model in Block 402, each point in the current object model is augmented with the corresponding gradient. The point may be concatenated with the gradient. In some cases, such as for points on an opposite side of the object model from the image plane, the gradient has a null or zero value. The result of Block 406 is reconstruction input having points (e.g., points of the Gaussian or particularlocations in the case of other types of object models) in the object model each associated with a gradient.

[0070] Block 408 includes a reconstruction network processing the reconstruction input to generate an updated object model. The reconstruction network processes the object model through a series of neural network layers. In one or more embodiments, each point and gradient may be processed individually through the reconstruction network to obtain a new value for the point. The result is an updated object model having different values for each point in the object model.

[0071] Block 410 includes making a determination whether to continue updating the object model. The determination may be based on a predefined completion criterion, such as convergence or amount of loss. If a determination is made to continue updating the object model, then the updated object model is set as the current object model in Block 412. The flow then returns to Block 402 to render a predicted image using the updated object model.

[0072] The processing of Blocks 404-410 may be performed for several iterations until a completion criterion is satisfied. Specifically, a second predicted image is rendered from the current object model. The second predicted image is used to generate updated gradients based on a second comparison of the second predicted image with a second real image. The current object model is then augmented with the updated gradients to generate the second reconstruction input, which is processed by the reconstruction network. The reconstruction network processes the second reconstruction input to generate an updated object model for the current object model. Across the iterations, multiple perspectives may be used. For example, each iteration may have a different perspective or location of the sensor within the scene. In some embodiments, Block 402 is performed multiple times for multiple images, of different perspectives or of different scenes, to determine multiple differences which are combined into a gradient for each point in the object model. Thus, the gradients in Block 404 may be determinedbased on a comparison of multiple predicted images with multiple corresponding real images.

[0073] If a determination is made not to continue in Block 410, the flow proceeds to Block 414. Block 414 includes storing the updated object model. The object model is stored for use in simulations as described in FIG. 2. Simulations may include a virtual world having realistic objects, as defined by object models, generated using FIG. 4 with the player moving through the virtual world.

[0074] The reconstruction network is a machine learning model. As such, the reconstruction network is trained. Training the reconstruction network may be performed on multiple objects having multiple heterogenous visual characteristics. For example, object models having various characteristics are generated using other methodologies that have greater resource usage. The object models may span a variety of scenes and a variety of scenarios. The result is an expected output of the reconstruction network. The expected input of the reconstruction network is as defined in Block 402 of FIG. 4. Through several iterations of processing of FIG. 4, the reconstruction network is trained to produce the expected output.

[0075] FIG. 5 shows an example diagram of an implementation in accordance with one or more embodiments. The following implementation is for example purposes only and not intended to limit the scope of the invention. Other implementations may be used.

[0076] An input to the example implementation is a set of source camera images Isrc=and an approximate geometry scaffold M (e.g., obtained from either LiDAR or points from multi-view stereo) captured in-the-wild by a sensor platform moving through a large dynamic scene. The goal output of the example implementation is to efficiently reconstruct a realistic and editable 3D representation 5 for accurate real-time camera simulation. One or more implementations can create modifiable digital clones of large real-world scenes (> 10,000m2) in two minutes or less, and that renders novel views with highphotorealism at >90 frames per second (FPS). The example implementation may iteratively update a 3D neural Gaussian, initialized from the scaffold , with a single neural network. The network takes the gradient feedback signals by differentiably rendering the representation to reconstruct the source images Isrc.

[0077] 3D Gaussian splatting may use a differentiable rasterization technique that allows real-time rendering of photorealistic scenes learned from posed images and an initial set of points. The example implementation represents the scene, in part, with a set of 3D Gaussians (, points) Cj = {gt} <t<M, where g G IR14consists of position (IR3), scale (IR3), orientation (IR4), color (IR3), and opacity (IR1). These Gaussian points Cj can be rendered to 2D images with camera poses n using a differentiable tile rasterizerrast(( / , n), where each point is projected and splatted to the image plane based on the scale and orientation. Then, the color is blended with other points based on the opacity and depth to camera. The example implementation augments the representation above with a latent feature vector, which is referred to as a 3D neural Gaussian, providing additional capacity for generalizable reconstruction and learning-based optimization. Second, the scene is decomposed into the nearby static scene, dynamic actors, and a distant region to enable modelling of large unbounded dynamic scenes.

[0078] FIG. 5 shows an overview of the example implementation (500). The example implementation may model the generalizable reconstruction as an iterative process, where the 3D neural Gaussians 5^ (502) are iteratively refined with reconstruction network Ge(504). The example implementation may first lift the source 2D images Isrc(506) to 3D space by backpropagating the rendering procedure to get the gradients (508) with respect to the representation V5(t) . Then the reconstruction network Getakes the 3D representationthe gradientand the iteration step t as input and predicts an updated 3D representation(510). To train the network, the example implementation may renderat source and novel views (512) and compute loss. The backward gradient flow for training Geis highlighted with dashed arrows.

[0079] The scene representation 5 is defined as a set of 3D neural Gaussians, 5={ i}i<i<M, where each point is represented by a feature vectorE IRC. The latent representation helps encode information about the scene during the iterative updates in the learning-based optimization. To render, the 3D neural Gaussians are converted to a set of explicit color 3D Gaussians Cj = {gt} <t<M, using a multilayer perceptron (MLP) network gt= fm\phi). To encode geometry and additional physical information about the scene intoand ensure stable optimization, the first 14 channels are designated as the 3D Gaussian attributes and add a skip connection in fmipsuch that it updates these channels to generate gt-

[0080] The example implementation may decompose the dynamic scene and its set of 3D neural Gaussians 5 into a static background 5s, a set of dynamic actors 5^, and a distant region(e.g., far-away buildings and sky). Rigid motionmay be used for dynamic actors, where T is the rigid transformation and are the actor extrinsics. The dynamic points S'Aare moved across different frames using 3D bounding boxes that specify each foreground actor’s size and location. The example implementation may initialize the 3D neural Gaussians for the static background and dynamic actors using the provided approximate geometry scaffolds M (e.g., aggregated LiDAR points or multi -view stereo points). The example implementation may further position a fixed number of points at a large distance to model the distant region (e.g., as a background object).

[0081] To render, the example implementation may use 5 and camera poses n = {Kj, fy], where K(and fy are the camera intrinsics and extrinsics for view i. The example implementation may convert 5 to 3D Gaussians Cj and then leverage the differentiable tile rasterizer [?] to render the images I:

[0082] The example implementation may lift 2D images to 3D space by “rendering and backpropagating” to obtain gradients with respect to the 3D representation. Compared to leveraging networks to process images independently, 3D gradients provide a unified representation that may efficiently aggregate as many images as needed. Moreover, 3D gradients take the rendering procedure into account, naturally handling occlusions. The example implementation may also enable adjustment of the 3D representation, which is not done in traditional depth rendering for view warping. Finally, the 3D gradients are fast to compute with modern differentiable rasterization engines.

[0083] Specifically, given the 3D representation S, the example implementation may first render the scene to source input views Isrc= / renderG^l nsrc) using Eqn. 2. Then, the rendered images are compared with the inputs Isrc, compute the reconstruction loss L, and backpropagate the difference to 3D representation 5 to get accumulated gradients Vs= VsL(S, Isrc; nsrc), with

[0084] The differentiable function render builds a connection between 2D and 3D, and the gradient Vsencodes the 2D images in 3D using 5 as the proxy.

[0085] The example implementation may iteratively refine the scene representation 5 given the source images Isrc. At each step t, the current 3D representation 5^^ is taken as a proxy to compute the gradient V5(t) via differentiable rendering, thereby un-projecting 2D source images Isrcto 3D, and then feed V5(t) into the network Geto predict the updated 3D representation

[0086] y (t) defines the update scale at different step t. Intuitively, similar to gradient descent, the example implementation may use a decaying schedule y(t) and a small T so that the network can predict an initial coarse representation andthen quickly refine it. Further, a cosine scheduler may be used for y(t). The reconstruction network has sparse convolution as Geto process the neural Gaussians 5. The iterative process allows the refinement of the 3D representation to achieve better quality and use a smaller network that is more efficient and easier to learn.

[0087] The reconstruction network may be two generalizable networks with the same architecture for the static background and dynamic scene. The reconstruction network may use an encoder-decoder architecture for sparse scenes. The entire scene may be processed by the reconstruction network. The 3D neural Gaussians and gradients may be concatenated as the input to the reconstruction network. The timestep positional encodings are concatenated with point features output from the last encoder layer and fed to the decoder. For the background reconstruction network, a 2D convolutional neural network (CNN) may be used with two residual blocks, without down sampling or up sampling. For the transformation MLP network fm\pthat converts the 3D neural Gaussians to a set of explicit 3D Gaussians, the example implementations may adopt one linear layer with a tanh activation. The output is combined with a learning rate decay factor y(t) to ensure gradual updates. The input raw gradient values may be normalized for each channel by dividing them by the maximal absolute value in that channel.

[0088] The training process to train the learned optimizer Geand neural decoding MLP fmipis presented below. For each scene, the example implementation may initialize the scene representation 5® from the geometry scaffold J f . One or more implementations may iteratively refine 5 with the network prediction for T steps. To enhance the generalizability of the reconstruction network, the example implementation may render the updated representation to both source views Isrcand novel views Itgtduring training (I = [Isrc, Itgt]), and backpropagate the gradients to the parameters of the reconstruction network Geand themip. Note that in Eqn. 3 only the gradients from source views are used as input to Gefor the next iteration, as the targetviews will not be available at test time. Ggis trained to minimize final rendering loss for every iteration step t. One or more implementations may train the networks across many large outdoor scenes. The total loss £ is: -^mseO' D + ^-Ipips pips Cb I) + TregTreg(^), (6) where I is the rendered images, £mseis the photometric loss, £lpipsis the perceptual loss[?], and £regis the regularization term applied on the shape of the transformed Gaussians Cj to be flat for better alignment with the surface.where dmis the minimal value of the 3-channel scale for each Gaussian gt. One or more implementations may encourage it to be smaller than a threshold e.

[0089] During inference time, the pre-trained reconstruction network Geand neural Gaussian decoder MLP fmipmay be used to reconstruct scenes not seen during training. Specifically, the input images Isrcare used for the novel scene and the 3D neural Gaussian initialization 5® to iteratively compute the gradients Vsand refine the 3D representation. Finally, theare exported to standard 3D Gaussiansfor real-time rerasterization.

[0090] FIG. 6 shows an example time sequence (600) of updates to the object models in accordance with one or more embodiments. The object models in FIG. 6, include the actor models of the particular vehicles that are driving, the asset models of the parked vehicles, and the background models that include the buildings and the street. The object models are iteratively updated as shown in the time sequence in order from tl to t8. Notably, the iterations from tl to t8 are likely not consecutive as many additional iterations exist.

[0091] As shown, one or more embodiments relate to 3D reconstruction using a learnable optimizer, providing a method for efficiently reconstructing realistic three-dimensional object models from sensor data with significantly reduced computational resources. Traditional neural rendering approaches rely on per- scene optimization through thousands of gradient descent iterations, resulting inhigh computational costs and limited robustness to viewpoint changes. One or more embodiments addresses these limitations through a learnable optimizer implemented as a reconstruction network that incorporates gradient feedback signals. This approach enables faster convergence while maintaining or improving quality by rendering a predicted image from a current object model, generating gradients by comparing it with a real image, augmenting the model with these gradients, and processing this input through a reconstruction network to generate an updated object model. The reconstruction network functions as a learned optimizer trained across multiple heterogeneous objects, making more intelligent update decisions than traditional gradient descent while generalizing effectively to new objects and scenes. Thus, embodiments may exhibit computational speedups of at least an order of magnitude, improved robustness to viewpoint changes, generalization across diverse object types, and compatibility with various object representations, including 3D Gaussians with optional latent feature vectors.

[0092] One or more embodiments may be implemented on a computing system specifically designed to achieve an improved technological result. When implemented in a computing system, the features and elements of the disclosure provide a significant technological advancement over computing systems that do not implement the features and elements of the disclosure. Any combination of mobile, desktop, server, router, switch, embedded device, or other types of hardware may be improved by including the features and elements described in the disclosure.

[0093] For example, as shown in FIG. 7A, the computing system (700) may include one or more computer processor(s) (702), non-persistent storage device(s) (704), persistent storage device(s) (706), a communication interface (708) (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and numerous other elements and functionalities that implement the features and elements of the disclosure. The computer processor(s) (702) may be an integrated circuit for processing instructions. The computer processor(s) (702) may be oneor more cores, or micro-cores, of a processor. The computer processor(s) (702) includes one or more processors. The computer processor(s) (702) may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), combinations thereof, etc.

[0094] The input device(s) (710) may include a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device. The input device(s) (710) may receive inputs from a user that are responsive to data and messages presented by the output device(s) (712). The inputs may include text input, audio input, video input, etc., which may be processed and transmitted by the computing system (700) in accordance with one or more embodiments. The communication interface (708) may include an integrated circuit for connecting the computing system (700) to a network (not shown) (e.g, a local area network (PAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) or to another device, such as another computing device, and combinations thereof.

[0095] Further, the output device(s) (712) may include a display device, a printer, external storage, or any other output device. One or more of the output device(s) (712) may be the same or different from the input device(s) (710). The input device(s) (710) and output device(s) (712) may be locally or remotely connected to the computer processor(s) (702). Many different types of computing systems exist, and the aforementioned input device(s) (710) and output device(s) (712) may take other forms. The output device(s) (712) may display data and messages that are transmitted and received by the computing system (700). The data and messages may include text, audio, video, etc., and include the data and messages described above in the other figures of the disclosure.

[0096] Software instructions in the form of computer readable program code to perform embodiments may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer readable medium such as a solid state drive (SSD), compact disk (CD), digital video disk (DVD), storage device, a diskette, a tape, flash memory, physical memory, or any other computer readablestorage medium. Specifically, the software instructions may correspond to computer readable program code that, when executed by the computer processor(s) (702), is configured to perform one or more embodiments, which may include transmitting, receiving, presenting, and displaying data and messages described in the other figures of the disclosure.

[0097] The computing system (700) in FIG. 7 A may be connected to, or be a part of, a network. For example, as shown in FIG. 7B, the network (720) may include multiple nodes (e.g., node X (722) and node Y (724), as well as extant intervening nodes between node X (722) and node Y (724)). Each node may correspond to a computing system, such as the computing system shown in FIG. 7A, or a group of nodes combined may correspond to the computing system shown in FIG. 7A. By way of an example, embodiments may be implemented on a node of a distributed system that is connected to other nodes. By way of another example, embodiments may be implemented on a distributed computing system having multiple nodes, where each portion may be located on a different node within the distributed computing system. Further, one or more elements of the aforementioned computing system (700) may be located at a remote location and connected to the other elements over a network.

[0098] The nodes (e.g., node X (722) and node Y (724)) in the network (720) may be configured to provide services for a client device (726). The services may include receiving requests and transmitting responses to the client device (726). For example, the nodes may be part of a cloud computing system. The client device (726) may be a computing system, such as the computing system shown in FIG. 7A. Further, the client device (726) may include or perform all or a portion of one or more embodiments.

[0099] The computing system of FIG. 7 A may include functionality to present data (including raw data, processed data, and combinations thereof) such as results of comparisons and other processing. For example, presenting data may be accomplished through various presenting methods. Specifically, data may be presented by being displayed in a user interface, transmitted to a differentcomputing system, and stored. The user interface may include a graphical user interface (GUI) that displays information on a display device. The GUI may include various GUI widgets that organize what data is shown, as well as how data is presented to a user. Furthermore, the GUI may present data directly to the user, e.g., data presented as actual data values through text, or rendered by the computing device into a visual representation of the data, such as through visualizing a data model.

[0100] As used herein, the term “connected to” contemplates multiple meanings. A connection may be direct or indirect (e.g., through another component or network). A connection may be wired or wireless. A connection may be a temporary, permanent, or a semi-permanent communication channel between two entities.

[0101] The various descriptions of the figures may be combined and may include, or be included within, the features described in the other figures of the application. The various elements, systems, components, and steps shown in the figures may be omitted, repeated, combined, or altered as shown in the figures. Accordingly, the scope of the present disclosure should not be considered limited to the specific arrangements shown in the figures.

[0102] In the application, ordinal numbers (e.g., first, second, third, etc.) may be used as an adjective for an element (z.e., any noun in the application). The use of ordinal numbers is not to imply or create any particular ordering of the elements, nor to limit any element to being only a single element unless expressly disclosed, such as by the use of the terms “before,” “after,” “single,” and other such terminology. Rather, ordinal numbers distinguish between the elements. By way of an example, a first element is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.

[0103] Further, unless expressly stated otherwise, the conjunction “or” is an inclusive “or” and, as such, automatically includes the conjunction “and,” unless expressly stated otherwise. Further, items joined by the conjunction “or” mayinclude any combination of the items with any number of each item, unless expressly stated otherwise.

[0104] In the above description, numerous specific details are set forth in order to provide a more thorough understanding of the disclosure. However, it will be apparent to one of ordinary skill in the art that the technology may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description. Further, other embodiments not explicitly described above can be devised which do not depart from the scope of the claims as disclosed herein. Accordingly, the scope should be limited only by the attached claims.

Claims

CLAIMSWhat is claimed is:

1. A method comprising : rendering a predicted image from a current object model of an object in a scenario; generating a plurality of gradients based on a comparison of the predicted image with a real image; augmenting the current object model with the plurality of gradients to generate a reconstruction input; processing, by a reconstruction network, the reconstruction input to generate an updated object model as the current object model; and storing the updated object model.

2. The method of claim 1, further comprising: for each of a plurality of iterations, until a completion criterion is satisfied: rendering a second predicted image from the current object model, generating an updated plurality of gradients based on a second comparison of the second predicted image with a second real image, augmenting the current object model with the updated plurality of gradients to generate a second reconstruction input, and processing, by the reconstruction network, the second reconstruction input to generate the updated object model for the current object model.

3. The method of claim 2, wherein generating the second predicted image is with a plurality of perspectives across the plurality of iterations.

4. The method of claim 1, further comprising: generating the current object model as a set of three dimensional (3D) Gaussians defining a plurality of points of the object.

5. The method of claim 4, further comprising: augmenting 3D Gaussian in the set of 3D Gaussians with a latent feature vector.

6. The method of claim 4, further comprising: converting, by a conversion model, the set of 3D Gaussians from a plurality of 3D neural Gaussians into a plurality of 3D color Gaussians with the conversion model.

7. The method of claim 6, wherein the conversion model is a multilayer perceptron model.

8. The method of claim 6, wherein the conversion model comprises a skip connection to update a plurality of 3D Gaussian attributes in the set of 3D Gaussians.

9. The method of claim 1, wherein augmenting the current object model with the plurality of gradients comprises, for each point represented in the current object model, concatenating the point with a corresponding gradient in the plurality of gradients.

10. The method of claim 1, further comprising: training the reconstruction network on a plurality of objects having a plurality of heterogenous visual characteristics.

11. The method of claim 1, further comprising: obtaining a plurality of object models comprising the object model and a background model, wherein rendering the predicted image is performed using the plurality of object models and the background model, and wherein the plurality of gradients is for the plurality of object models and the background model.

12. A system comprising: storage; and a computer processor connected to storage for executing instructions that cause the computer processor to perform operations comprising: rendering a predicted image from a current object model of an object in a scenario; generating a plurality of gradients based on a comparison of the predicted image with a real image; augmenting the current object model with the plurality of gradients to generate a reconstruction input; processing, by a reconstruction network, the reconstruction input to generate an updated object model as the current object model; and storing the updated object model.

13. The system of claim 12, further comprising: for each of a plurality of iterations, until a completion criterion is satisfied: rendering a second predicted image from the current object model, generating an updated plurality of gradients based on a second comparison of the second predicted image with a second real image, augmenting the current object model with the updated plurality of gradients to generate a second reconstruction input, and processing, by the reconstruction network, the second reconstruction input to generate the updated object model for the current object model.

14. The system of claim 13, wherein generating the second predicted image is with a plurality of perspectives across the plurality of iterations.

15. The system of claim 13, wherein the operations further comprise: generating the current object model as a set of three dimensional (3D) Gaussians defining a plurality of points of the object.

16. The system of claim 15, wherein the operations further comprise: augmenting 3D Gaussian in the set of 3D Gaussians with a latent feature vector.

17. The system of claim 13, wherein augmenting the current object model with the plurality of gradients comprises, for each point represented in the current object model, concatenating the point with a corresponding points with a corresponding gradient in the plurality of gradients.

18. The system of claim 13, wherein the operations further comprise: training the reconstruction network on a plurality of objects having a plurality of heterogenous visual characteristics.

19. The system of claim 13, wherein the operations further comprise: obtaining a plurality of object models comprising the object model and a background model, wherein rendering the predicted image is performed using the plurality of object models and the background model, and wherein the plurality of gradients is for the plurality of object models and the background model.

20. A non-transitory computer readable medium comprising computer readable program code for causing a computing system to perform operations comprising: rendering a predicted image from a current object model of an object in a scenario; generating a plurality of gradients based on a comparison of the predicted image with a real image; augmenting the current object model with the plurality of gradients to generate a reconstruction input; processing, by a reconstruction network, the reconstruction input to generate an updated object model as the current object model; and storing the updated object model.

Images