Neural hash grid based multi-sensor simulation

Abstract

The neural hash grid-based sensor simulation includes interpolating hash grid features adjacent to a location in a neural hash grid defined for the target object to obtain a set of location features. A multilayer perceptron (MLP) model processes the set of location features to generate a set of image features for the location. The method further includes completing ray casting to the target object using the set of image features to generate a feature image, generating a rendered image from the feature image, and processing the rendered image.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a non-provisional application of, and therefore claims the benefit of, U.S. Patent Application No. 63 / 424,865, filed November 11, 2022, which is incorporated herein by reference in its entirety. [Background technology]

[0002] background A virtual world is a computer-simulated environment that allows a player to interact in three-dimensional space as if they were in the real world. In some cases, a virtual world is designed to replicate at least some aspects of the real world. For example, a virtual world may include objects and environments reconstructed from the real world. Reconstructing objects and environments from the real world allows the system to replicate aspects of the real world. Summary of the Invention [Problem to be solved by the invention]

[0003] For example, one way to create a sense of realism is to obtain sensor data from the real world that describes a scenario, modify the scenario to create a modified scenario, and then allow a player to interact with the modified scenario. When a player interacts with the modified scenario, different objects may be in different relative positions than in the real world. Therefore, to modify the real world, an accurate set of models should be created and used in the virtual world. [Means for solving the problem]

[0004] overview In general, in one aspect, one or more embodiments relate to a method that includes interpolating hash grid features adjacent to a location in a neural hash grid defined for a target object to obtain a set of location features. A multilayer perceptron (MLP) model processes the set of location features to generate a set of image features for the location. The method further includes completing ray casting to the target object using the set of image features to generate a feature image, generating a rendered image from the feature image, and processing the rendered image.

[0005] In general, in one aspect, one or more embodiments relate to a system comprising a memory and a computer processor including computer-readable program code for performing operations. The operations include interpolating hash grid features adjacent to a location in a neural hash grid defined for a target object to obtain a set of location features. A multilayer perceptron (MLP) model processes the set of location features to generate a set of image features for the location. The operations further include completing ray casting and volume rendering to the target object using the set of image features to generate a feature image, generating a rendered image from the feature image, and processing the rendered image.

[0006] In general, in one aspect, one or more embodiments relate to a non-transitory computer-readable medium including computer-readable program code for performing operations. The operations include interpolating hash grid features adjacent to a location in a neural hash grid defined for a target object to obtain a set of location features. A multilayer perceptron (MLP) model processes the set of location features to generate a set of image features for the location. The operations further include completing ray casting and volume rendering to the target object using the set of image features to generate a feature image, generating a rendered image from the feature image, and processing the rendered image.

[0007] Other aspects of the present invention will become apparent from the following description and appended claims. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram of an autogenic training and testing system according to one or more embodiments. [Figure 2] 1 is a flowchart of an autogenic training and testing system according to one or more embodiments. [Figure 3] FIG. 1 is a diagram of a rendering system according to one or more embodiments. [Figure 4] FIG. 1 is an exemplary architecture diagram of a rendering system according to one or more embodiments. [Figure 5] 1 is a flowchart for neural hash grid training according to one or more embodiments. [Figure 6] 1 is a flowchart for generating a virtual environment according to one or more embodiments. [Figure 7] FIG. 1 illustrates an example of a neural hash grid-based environment in accordance with one or more embodiments. [Figure 8A]FIG. 1 illustrates an exemplary simulation scenario modified from the real world in accordance with one or more embodiments. [Figure 8B] FIG. 1 illustrates an exemplary simulation scenario modified from the real world in accordance with one or more embodiments. [Figure 9A] FIG. 1 illustrates a computing system in accordance with one or more embodiments of the present invention. [Figure 9B] FIG. 1 illustrates a computing system in accordance with one or more embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Like elements in the various figures are designated with like reference numerals for consistency. Detailed Description In general, embodiments are directed to reconstructing the real world in a virtual world while allowing for some variation during reconstruction. For a particular object, a neural hash grid is defined, including hash grid features for the object. In one or more embodiments, both stationary and moving objects may be represented by respective neural hash grids. The neural hash grid features describe a target object. To render the image, ray casting may be performed. When ray casting is performed, a ray may intersect with a location within the target object. To render a portion of the target object at a location, neural hash features neighboring the location are interpolated to generate a location feature. The location features are processed through a multilayer perceptron model (MLP) to generate an object's appearance relative to the location. The ray casting is completed using the object's appearance to generate a feature image. That is, the collection of rays simulates the player's view of the real world such that the player has the same input as if the player were in the real world (i.e., if the virtual world were real). The feature image may be further processed by a convolutional neural network (CNN) to generate a rendered image. For example, the CNN may perform upscaling and correct artifacts, resulting in a more realistic virtual world.

[0010] In one or more embodiments, processing by the system may be used to generate a virtual world that mimics the real world but implements a different scenario. For example, the altered scenario may have dynamic and / or static objects in different locations, a player's viewpoint changed so that the player is in a different location than the player was in the real world, or other aspects of the real world are different.

[0011] Embodiments of the present invention may be used as part of generating simulated environments for training and testing autonomous systems. An autonomous system is an automated driving mode of transportation that does not require a human pilot or human driver to navigate and react to a real-world environment. Rather, the autonomous system includes a virtual driver, which is the decision-making portion of the autonomous system. The virtual driver is an artificial intelligence system that learns how to interact in the real world. The autonomous system can be fully autonomous or semi-autonomous. In the transportation aspect, the autonomous system is housed in a housing configured to navigate a real-world environment. Examples of autonomous systems include autonomous vehicles (e.g., autonomous trucks and cars), drones, airplanes, robots, etc. The virtual driver is software that makes decisions and allows the autonomous system to interact with the real world, including navigating, signaling, and stopping or maintaining a current state.

[0012] The real-world environment is the portion of the real world that the autonomous system is designed to travel through once trained. Thus, the real-world environment may include interactions with concrete and terrain, people, animals, other autonomous systems, human-driven systems, buildings, and other objects as the autonomous system travels from an origin to a destination. To interact with the real-world environment, the autonomous system includes various types of sensors, such as LiDAR sensors, among other types, that are used to obtain measurements of the real-world environment, and cameras that capture images from the real-world environment.

[0013] Testing and training virtual drivers for autonomous systems in real-world environments is unsafe due to potential accidents caused by untrained virtual drivers. Therefore, as shown in FIG. 1, a simulator (100) is configured to train and test virtual drivers (102) for an autonomous system. For example, the simulator may be a unified, modular, mixed reality, closed-loop simulator for an autonomous system. The simulator (100) is a configurable simulation framework that allows different autonomy components to be evaluated not only individually but also as a complete system in a closed-loop manner. The simulator automatically reconstructs a "digital twin" of a real-world scenario, enabling accurate evaluation of virtual drivers at scale. The simulator (100) may also be configured to perform mixed reality simulations that combine real-world data and simulated data to create diverse and realistic evaluation variations, providing insight into the virtual driver's performance. Mixed reality closed-loop simulations enable the simulator (100) to analyze the virtual driver's behavior for counterfactual "what-if" scenarios that did not occur in the real world. The simulator (100) further includes the capability to simulate and train rare but safety-critical scenarios and closed-loop training for the entire autonomous system to enable automatic and scalable improvement of autonomy.

[0014] The simulator (100) creates a simulated environment (104), which is a virtual world. The virtual driver (102) is a player within the virtual world. The simulated environment (104) is a simulation of a real-world environment, which may or may not actually exist, within which the autonomous system is designed to navigate. Thus, the simulated environment (104) includes simulations of objects (i.e., simulated objects or assets) and real-world backgrounds, including natural objects, structures, buildings and roads, obstacles, and other autonomous and non-autonomous objects. The simulated environment simulates environmental conditions in which the autonomous system may be deployed. Additionally, the simulated environment (104) may be configured to simulate various weather conditions that may affect inputs to the autonomous system. The simulated objects may include both stationary and non-stationary objects. Non-stationary objects are actors in the real-world environment.

[0015] The simulator (100) also includes an evaluator (110). The evaluator (110) is configured to train and test the virtual driver (102) by creating various scenarios in the simulated environment. Each scenario is a configuration of the simulated environment that includes, but is not limited to, static portions, motion of simulated objects, actions of the simulated objects with each other, and reactions to actions taken by the autonomous system and the simulated objects. The evaluator (110) is further configured to evaluate the performance of the virtual driver using various metrics.

[0016] The evaluator (110) evaluates the virtual driver's performance throughout the execution of the scenario. Evaluating performance may include applying rules. For example, the rules may require the automated system not to collide with other actors, adherence to safety and comfort criteria (e.g., passengers not experiencing acceleration forces exceeding a certain value in the vehicle), the automated system not to deviate from the executed trajectory, or other rules. Each rule may be associated with metric information that associates the degree to which the rule was violated with a corresponding score. The evaluator (110) may be implemented as a data-driven neural network that learns to distinguish between good and bad driving behaviors. Various metrics of the evaluation system may be used to determine whether the automated system meets the requirements of the success criteria for a particular scenario. Furthermore, in addition to system-level performance, in the case of a module-based virtual driver, the evaluator may also evaluate individual modules, such as the segmentation or prediction performance of actors in a scene, with respect to the ground truth recorded in the simulator.

[0017] The simulator (100) is configured to operate in a plurality of phases selected by a phase selector (108) and a mode selected by a mode selector (106). The phase selector (108) and the mode selector (106) may be graphical user interface or application programming interface components configured to receive the phase and mode selections, respectively. The selected phases and modes define the configuration of the simulator (100). That is, the selected phases and modes define which system components communicate and the operation of the system components.

[0018] The phase may be selected using a phase selector (108). The phase may be a training phase or a test phase. In the training phase, the evaluator (110) provides metric information to the virtual driver (102), which uses the metric information to update the virtual driver (102). The evaluator (110) may further use the metric information to further train the virtual driver (102) by generating scenarios for the virtual driver. In the test phase, the evaluator (110) does not provide metric information to the virtual driver. In the test phase, the evaluator (110) uses the metric information to evaluate the virtual driver and develop scenarios for the virtual driver (102).

[0019] The modes define how much real-world data is used, whether noise is injected into the simulated data, how much variability is added to the real-world data, and whether the scenario is designed to be adversarial. Exemplary modes include open-loop simulation mode, closed-loop simulation mode, single-module closed-loop simulation mode, fuzzy mode, and adversarial mode. In open-loop simulation mode, a virtual driver is evaluated with real-world data. In single-module closed-loop simulation mode, a single module of the virtual driver is tested. An example of a single-module closed-loop simulation mode is a localizer closed-loop simulation mode, in which a simulator evaluates how the localizer estimated pose drifts over time as the scenario progresses within the simulation. In training data simulation mode, a simulator is used to generate training data. In closed-loop evaluation mode, the virtual driver and simulation system are run together to evaluate system performance. In adversarial mode, actors are modified to perform adversarially. In fuzzy mode, noise (e.g., to replicate signal processing noise and other types of noise) is injected into the scenario. Other modes may exist without departing from the scope of the system.

[0020] The simulator (100) includes a controller (112) that includes functionality for configuring various components of the simulator (100) according to a selected mode and phase. That is, the controller (112) may modify the configuration of each component of the simulator based on configuration parameters of the simulator (100). Such components include an evaluator (110), a simulated environment (104), an autonomous system model (116), a sensor simulation model (114), an asset model (117), an actor model (118), a latency model (120), and a training data generator (122).

[0021] The autonomous system model (116) is a detailed model of the autonomous system that the virtual driver will run on. The autonomous system model (116) includes the model, geometry, physical parameters (e.g., mass distribution, significant points), engine parameters, sensor locations and types, sensor firing patterns, information about the hardware on which the virtual driver runs (e.g., processor power, amount of memory, and other hardware information), and other information about the autonomous system. Various parameters of the autonomous system model may be configurable by a user or another system.

[0022] For example, if the autonomous system is a vehicle, the modeling and dynamics may include the vehicle type (e.g., car, truck), make and model, geometry, physical parameters such as mass distribution, axle positions, engine type and performance, etc. The vehicle model may also include information about the sensors (e.g., camera, LiDAR, etc.) on the vehicle, the relative firing synchronization patterns of the sensors, and the calibrated external (e.g., position and orientation) and internal (e.g., focal length) properties of the sensors. The vehicle model also defines the on-board computer hardware, sensor drivers, controllers, and the autonomy software release under test.

[0023] The autonomous system model includes an autonomous system dynamic model. The autonomous system dynamic model is used for dynamics simulation to obtain the actuation actions of a virtual driver (e.g., steering angle, desired acceleration) and execute the actuation actions on the autonomous system in the simulated environment to update the state of the simulated environment and the autonomous system. To update the state, a kinematic motion model may be used, or a dynamic motion model that considers forces applied to the vehicle may be used to determine the state. By accessing actual logged scenarios with ground truth actuation and vehicle states at each time step within the simulator, embodiments may also optimize analytical vehicle model parameters or learn parameters of a neural network that infers a new state of the autonomous system given the virtual driver outputs.

[0024] In one or more embodiments, the sensor simulation model (114) models active and passive sensor inputs in the simulated environment. The passive sensor inputs capture the visual appearance of the simulated environment, including stationary and non-stationary simulated objects, from the perspective of one or more cameras based on the simulated positions of the cameras in the simulated environment. Examples of passive sensor inputs include inertial measurement units (IMUs) and thermal inputs. The active sensor inputs are inputs to the virtual driver of the autonomous system from active sensors, such as LiDAR, RADAR, global positioning system (GPS), and ultrasonic. That is, the active sensor inputs include measurements taken by sensors, which are simulated based on the simulated environment based on the simulated positions of the sensors in the simulated environment. As an example, the active sensor measurements may be measurements a LiDAR sensor makes of the simulated environment over time and in relation to the movement of the autonomous system. In one or more embodiments, all or part of the sensor simulation model (114) may be or include the rendering system (300) shown in Figure 3. In such a scenario, the rendering system of the sensor simulation model (114) may perform the operations of Figures 5 and 6.

[0025] The sensor simulation model (114) is configured to simulate sensor observations of the surrounding scene in the simulated environment (104) at each time step according to the sensor configuration on the vehicle platform. If the simulated environment directly represents the real-world environment without modification, the sensor output may be provided directly to the virtual driver. For light-based sensors, the sensor model simulates light as rays interacting with objects in the scene to generate sensor data. Depending on the asset representation (e.g., of stationary and non-stationary objects), embodiments may use graphics-based rendering for assets with textured meshes, neural rendering, or a combination of multiple rendering methods. Utilizing multiple rendering methods enables customizable world-building with enhanced realism. Because assets are synthetic in 3D and support a standard interface for rendering commands, different asset representations may be seamlessly composed to generate the final sensor data. Furthermore, in scenarios that replay real-world events and use the same autonomous systems as in the real world, the original sensor observations may be replayed at each time step.

[0026] The asset model (117) includes multiple models, each modeling an individual asset of a particular type in the real world. Assets may include inanimate objects such as construction barriers or traffic signs, parked vehicles, and background (e.g., vegetation or the sky). Each entity in a scenario may correspond to an individual asset. Thus, an asset model, or an instance of a type of asset model, may exist for each object or asset in a scenario. Assets can be composited together to form a three-dimensional simulated environment. Asset models provide all the information needed by the simulator to simulate the asset. Asset models provide the information used by the simulator to represent and simulate the asset in the simulated environment.

[0027] Closely related and sometimes considered part of the set of asset models (117) are actor models (118). Actor models represent actors in a scenario. An actor is a sentient entity with an independent decision-making process. That is, in the real world, an actor may be an active entity (e.g., a person or an animal) that makes decisions based on the environment. An actor may perform active movements instead of, or in addition to, passive movements. For each actor in a scenario, there may be an actor model, or an instance of an actor model. An actor model is a model of the actor. If the actor is in a mode of transportation, the actor model includes a model of the transportation in which the actor is located. For example, actor models may represent pedestrians, children, vehicles operated by drivers, pets, bicycles, and other types of actors.

[0028] The actor model leverages a scenario specification and assets to control all actors in a scene and their actions at each time step. Actor behavior is modeled in a region of interest centered around the autonomous system. Depending on the scenario specification, the actor simulation controls the actors in the simulation to achieve desired behavior. Actors can be controlled in a variety of ways. One option is to utilize a heuristic actor model, such as an intelligent-driver model (IDM), that attempts to maintain a specific relative distance or time-to-collision (TTC) from a leading actor or a heuristic-derived lane-change actor model. Another is to directly replay actor trajectories from actual logs or control actors with a data-driven traffic model. Through a configurable design, embodiments may mix and match different subsets of actors to be controlled by different behavioral models. For example, distant actors may not initially interact with the autonomous system and can follow real-world log trajectories, but may switch to a data-driven actor model when in the vicinity of the autonomous system. In another example, actors may be controlled by heuristic or data-driven actor models that still fit into high-level routes in the real world. This mixed reality simulation provides control and a sense of presence.

[0029] Additionally, actor models may be configured to be in cooperative or adversarial modes. In cooperative mode, the actor model models actors that behave rationally in response to the conditions of the simulated environment. In adversarial mode, the actor model may model actors that behave irrationally, such as exhibiting road rage or bad driving.

[0030] In one or more embodiments, the actor model (118), asset model (117), and background may be part of a rendering system (described below with reference to FIG. 3). As another example, the system may be a branching system in which asset and actor behavior (e.g., trajectory or positioning) is defined separately from appearance, which is part of the rendering system.

[0031] The latency model (120) represents the timing latency that occurs when the autonomous system is in a real-world environment. There may be several sources of timing latency. For example, there may be latency from the time an event occurs to the sensor that detects sensor information from an event and sends the sensor information to the virtual driver. Another latency may exist based on differences between the computing hardware running the virtual driver in the simulated environment compared to the computing hardware of the virtual driver. Additionally, there may be another timing latency between the time the virtual driver sends an actuation signal to the autonomous system that changes (e.g., direction or speed) based on the actuation signal. The latency model (120) models various sources of timing latency.

[0032] In other words, in the real world, safety-critical decisions can involve fractions of a second, which impact response times. The latency model simulates the precise timing and latency of different components of the on-board system. To enable scalable evaluation without strict requirements for precise hardware, the latency and timing of different components of the autonomous system and sensor module are modeled while running on different computer hardware. The latency model may replay latencies recorded from previously collected real-world data, or it may have a data-driven neural network that infers latency at each time step to match the hardware in an in-loop simulation setup.

[0033] The training data generator 122 is configured to generate training data. For example, the training data generator 122 may modify a real-world scenario to create a new scenario. Modification of a real-world scenario is called mixed reality. For example, a mixed reality simulation may include adding new actors with new behaviors, changing the behavior of one or more actors from the real world, and modifying sensor data within the area while keeping the rest of the sensor data the same as the original log. In some cases, the training data generator 122 converts a benign scenario into a safety-critical scenario.

[0034] The simulator (100) is connected to a data repository (105). The data repository (105) is any type of storage device or device configured to store data. The data repository (105) includes data collected from the real world. For example, the data collected from the real world includes real actor trajectories (126), real sensor data (128), real trajectories (130) of a system capturing the real world, and real latency (132). Each of the real actor trajectories (126), real sensor data (128), real trajectories (130) of a system capturing the real world, and real latency (132) is data captured by or calculated directly from one or more sensors in the real world (e.g., in a real-world log). In other words, the data collected from the real world is actual events that occurred in the real world. For example, if the autonomous system is a vehicle, the real-world data may be captured by the vehicle traveling through the real world using sensor equipment.

[0035] Additionally, the data repository (105) includes the ability to store one or more scenario specifications (140). The scenario specification (140) specifies a scenario and evaluation setting for testing or training an autonomous system. For example, the scenario specification (140) may describe the current state of the autonomous system (e.g., full 6D attitude, velocity, and acceleration), map information specifying road layout, and the initial state of the scene, such as a scene layout specifying the initial state of all dynamic actors and objects in the scenario. The scenario specification may also include dynamic actor information describing how dynamic actors in the scenario should evolve over time, which is input to the actor model. The dynamic actor information may include actor route information, desired behaviors, or aggressiveness. The scenario specification (140) may be specified by a user, programmatically generated using a domain specification language (DSL), procedurally generated using heuristics from a data-driven algorithm, or adversarially generated. The scenario specification (140) can also be conditional on data collected from real-world logs, such as taking place on a particular real-world map or having a subset of actors defined by their original locations and trajectories.

[0036] The interface between the virtual driver and the simulator matches the interface between the virtual driver and a real-world autonomous system. For example, the sensor simulation model (114) and virtual driver match a virtual driver that interacts with sensors in the real world. The virtual driver is the actual autonomy software running on the autonomous system. The simulated sensor data output by the sensor simulation model (114) can be in or converted to the exact message format that the virtual driver would take as input as if it were in the real world, and the virtual driver can then operate as a black-box virtual driver with simulated latency built into its sequentially operating components. The virtual driver then outputs the exact same control expressions it would use to interface with a lower-level controller on the actual autonomous system. The autonomous system model (116) then updates the state of the autonomous system in the simulated environment. Thus, the various simulation models of the simulator (100) run asynchronously in parallel at their own frequencies to match the real-world setting.

[0037] FIG. 2 shows a flow diagram for running a simulator in closed-loop mode. In block 201, a digital twin of a real-world scenario is generated as a simulated environmental state. Log data from the real world is used to generate an initial virtual world. The log data defines which asset and actor models to use in the initial positioning of assets. For example, a convolutional neural network may be used on the log data to identify various asset types in the real world. As another example, an offline perception system and human annotation of the log data may be used to identify asset types. Corresponding asset and actor models may then be identified based on the asset types and added to the positions of real actors and assets in the real world. The asset and actor models thus create the initial three-dimensional virtual world.

[0038] In block 203, the sensor simulation model is executed against simulated environmental conditions to obtain simulated sensor outputs. The sensor simulation model may use beamforming and other techniques to replicate the view to the sensors of the autonomous system. Each sensor of the autonomous system has a corresponding sensor simulation model and a corresponding system. The sensor simulation model is executed based on the position of the sensor in the virtual environment to generate simulated sensor outputs. The simulated sensor outputs are in the same form as those received from the real sensors by the virtual driver. In one or more embodiments, block 203 may be implemented as shown in FIGS. 5 and 6 to generate camera outputs and lidar sensor outputs for each virtual camera and virtual lidar sensor, respectively. The operations of FIGS. 5 and 6 may be performed for each camera and lidar sensor on the autonomous system to simulate the corresponding camera and lidar sensor outputs. The sensor locations and line of sight directions relative to the autonomous vehicle may be used to replicate the emission locations of the corresponding virtual sensors on the simulated autonomous system. Thus, the various sensor inputs to the virtual driver match the combinations of inputs that the virtual driver would receive in the real world.

[0039] The simulated sensor outputs are passed to a virtual driver. In block 205, the virtual driver acts on the simulated sensor outputs to generate actuation actions. The actuation actions define how the virtual driver controls the autonomous system. For example, for an SDV, the actuation actions may be an amount of acceleration, a steering movement, a turn signal trigger, etc. In block 207, the actuation actions update the autonomous system state in the simulated environment. The actuation actions are used as inputs to an autonomous system model to determine the actual behavior of the autonomous system. For example, an autonomous system dynamic model may use actuation actions in addition to road and weather conditions to represent the resulting behavior of the autonomous system. For example, in a wet or snowy environment, the same amount of acceleration effort as in a dry environment may cause less acceleration than in a dry environment. As another example, the autonomous system model may consider potentially failed tires (e.g., tire slippage), machine-based latency, or other possible defects in the autonomous system.

[0040] At block 209, actor actions within the simulated environment are modeled based on the simulated environment state. Concurrently with the virtual driver model, the actor model and asset model are executed with the simulated environment state to determine updates for each of the assets and actors within the simulated environment. Here, the actor's actions may test the virtual driver using previous outputs of the evaluator. For example, if the actor is hostile, the evaluator may indicate the lowest scoring metric for the virtual driver based on the virtual driver's previous behavior. Using the mapping of metrics to the actor model's actions, the actor model is executed to utilize or test that particular metric.

[0041] Thus, in block 211, the simulated environment state is updated according to the actor's actions and the autonomous system state to generate an updated simulated environment state. The updated simulated environment includes changes in the position of the actor and the autonomous system. Because the model runs independently of the real world, the updates may reflect deviations from the real world. Thus, the autonomous system is tested in a new scenario. In block 213, a decision is made whether to continue. If the decision to continue is made, in block 203, testing of the autonomous system continues using the updated simulated environment state. In each iteration, during training, the evaluator provides feedback to the virtual driver. Thus, the virtual driver's parameters are updated to improve the virtual driver's performance in various scenarios. During testing, the evaluator can be tested using various scenarios and patterns, including edge cases that may be safety-critical. Thus, one or more embodiments improve the virtual driver and increase the virtual driver's safety in the real world.

[0042] As shown, the virtual driver of the autonomous system operates based on the virtual driver's scenario and currently learned parameters. The simulator captures the autonomous system's actions and provides the responses in the simulated environment to the virtual driver of the autonomous system. The evaluator evaluates the virtual driver's performance and creates scenarios based on the performance. The process may continue while the autonomous system operates in the simulated environment.

[0043] 3 illustrates a diagram of a rendering system (300) according to one or more embodiments. The rendering system (300) is a system configured to generate virtual sensor inputs using a neural hash grid for an object. In particular, the rendering system (300) may be configured to render camera images and lidar images. The rendering system (300) includes a data repository (302) connected to a model framework (304). The data repository (302) includes sensor data (128), object models (e.g., object model X (306), object model Y (308)), a target region background model (310), an external region background model (312), and a constraint vector space (314).

[0044] The sensor data (128) is the sensor data previously described with reference to FIG. 1 . The sensor data (128) includes a LiDAR point cloud (328) and actual images (330). The LiDAR point cloud (328) is a point cloud captured by a LiDAR sensor performing a LiDAR sweep of a geographic region. The actual images (330) are images captured by one or more cameras of the geographic region. For example, the detection vehicle may have a camera and a LiDAR sensor that collect sensor data from the geographic region as the detection vehicle moves through the geographic region. In particular, the sensor data (128) is time-series data captured along the trajectory of the detection vehicle. Therefore, the sensor data (128) generally omits some side views of a three-dimensional object. Therefore, challenges exist when performing closed-loop simulations to generate three-dimensional object models of an object (e.g., object model X (306), object model Y (308)) when some of the views of the object are not present. For example, for many objects, certain sides of the object may have no sensor data, while other sides may only have sensor data from one perspective view (e.g., a corner perspective). As a more specific example, consider a sensing vehicle moving along a road. A camera on the sensing vehicle may directly capture the sides of other vehicles parked along the road, as well as a small amount of the front and rear of the unobscured parked vehicles. The camera may also capture an image of another vehicle driving in front of the sensing vehicle. As the other vehicle turns, the camera may capture a different side, but not the front. Thus, the sensor data (128) is incomplete because it does not capture a 360-degree view of the object.

[0045] The object models (e.g., Object Model X (306), Object Model Y (308)) are three-dimensional object models of objects. Each object model (e.g., Object Model X (306), Object Model Y (308)) includes a neural hash grid (e.g., Neural Hash Grid X (320), Neural Hash Grid Y (322)) and a constraint vector (e.g., Constraint Vector X (324), Constraint Vector Y (326)).

[0046] The neural hash grid (e.g., neural hash grid X (320), neural hash grid Y (322)) is a grid of neural network features generated for the corresponding object. Each location has a corresponding location in the neural hash grid, whereby the relative location between two locations on the object matches the relative location of the matching point in the neural hash grid. In other words, the neural hash grid is a scaled model of the object, whereby corresponding points have learned features from the object. In one or more embodiments, the neural hash grid has a hierarchy of resolution. The hierarchy of resolution may be defined by representing the model of the object as a cube containing subcubes. For example, at the lowest resolution, the object is represented by a first set of cubes, each cube having features defined for the entire cube. Each cube in the first set of cubes may be divided into subcubes (e.g., nine subcubes, etc.). A subcube is a cube that is entirely contained within another cube. Each subcube has a set of features for that particular subcube that are features defined for matching locations in the object. Each subcube may be further divided into subcubes with a corresponding set of features defined, and the process can be repeated down to the lowest resolution. Each cube has a corresponding region on the object, regardless of resolution. As an example of a vehicle, at a first resolution, the vehicle may include separate cubes for the front, middle, and rear of the vehicle. The middle region cube may include individual subcubes for the portion of the vehicle having the left front door, left front window, left rear door, and left rear window, without specifically identifying or defining doors, windows, handles, etc. The neural hash grids (e.g., Neural Hash Grid X (320), Neural Hash Grid Y (322)) are feature grids learned from actual sensor data (128) and therefore may not include direct attributes such as color, luminosity, etc., but rather may include encoded features learned by machine learning.

[0047] Constraint vectors (e.g., constraint vector X (324), constraint vector Y (326)) are vectors specific to an object model learned from multiple objects. Constraint vectors serve to constrain the features of the object model. Constraint vectors are defined by a constraint vector space (314). A constraint vector space is a shared representation of an object. That is, a constraint vector space is learned from multiple objects, allowing for the mutual use of information between objects. As an example, a constraint vector space allows missing views from one object to be learned from views of other objects. However, objects may not be identical and therefore may not have identical features. For example, the front of a red sports car cannot be copied onto a blue SUV to be exact. Therefore, missing views are learned from a combination of views of other objects and its own features, rather than by direct copying. Thus, the constraint vector of an object generated by the constraint vector space is the object prior used to generate the object model.

[0048] Continuing with FIG. 3 , the target area background model (310) and the external area background model (312) define different types of background. The target area background has background objects that are within the region of interest of the autonomous system (i.e., the target area). For example, the region of interest may be within 150 meters ahead of the autonomous system, within 40 meters behind the autonomous system, and within 20 meters on each side of the autonomous system. The target area background model (310) may represent the entire target area, as described above with reference to the object model. However, rather than representing individual objects, the target area background model may incorporate the entire target area or specific sub-regions thereof into the same model.

[0049] The external region background model (312) is a background model of something outside the target region. Outside or external refers to a location that is geographically farther away than the current target region. In the above example, the external region is the region 150 meters ahead of the autonomous system, 40 meters behind the autonomous system, and more than 20 meters on each side of the autonomous system. For the external region background model (312), the region may be represented by an inverted sphere. A spherical projection or optimization may be used as the external region background model.

[0050] The model framework (304) is configured to generate an object model and perform a neural hash grid sensor simulation. The model framework (304) includes a hypernetwork (340), a shared multilayer perceptron model (342), a ray casting engine (344), an interpolator (346), a convolutional neural network (CNN) decoder (348), a LiDAR decoder (350), a classifier network (352), a loss function (354), and a trajectory refinement model (356).

[0051] The hypernetwork (340) is an MLP network configured to generate an actor-neural hash grid representation from the constraint vector space (314). In one or more embodiments, the hypernetwork (340) is trained over an object model.

[0052] The shared MLP model (342) is an MLP model configured to generate geometric and appearance features from an object model. Generally, the MLP model is a feedforward artificial neural network with at least three layers of nodes. The layers include an input layer, a hidden layer, and an output layer. Each layer has multiple nodes. Each node includes an activation function with learnable parameters. Through training and backpropagation of losses, the parameters are updated, and the MLP model correspondingly improves in making predictions.

[0053] Although a single shared MLP model (342) is shown, the MLP model may include multiple MLP models. The MLP geometry model maps point locations to signed distances, which then map to volumetric densities. The MLP surface model is trained based on sensor data. The signed distance function of the NeRSDF model maps locations in three-dimensional space to the location's signed distance from the object's surface (i.e., the object surface). The signed distance is a distance that can be positive, 0, or negative from the target object surface, depending on the location's position relative to the object surface. For signed distance, positive values ​​are outside the first surface of the object through which the ray passes, 0 at the object surface, negative within the object, most negative at the center of the object, less negative closer to the second surface of the object, 0 at the second surface of the object through which the ray passes, and then positive outside the second surface through which the object passes. The signed distance may then be mapped by a position function that maps the signed distance to 1 if the location is within the object's interior, and to 0 otherwise.

[0054] The second MLP may be a feature descriptor MLP. The second MLP may take as input a geometry feature vector and a viewpoint encoding and predict a neural feature descriptor. The neural feature descriptor includes neural features for a particular point. In one or more embodiments, the second MLP is a single MLP that takes a location and a view direction as input and directly outputs a neural feature descriptor.

[0055] The ray casting engine (344) is configured to define and cast rays from the sensor to a target object. With respect to the object, the ray has a first endpoint at the virtual sensor and a second endpoint on the opposite side of the target object or where the ray intersects with the target object. The ray may pass through the target object. Thus, the ray passes through at least a near point on the surface of the target object and a far point on the surface of the target object. Because the ray is a line, there are an infinite number of locations along the ray. One or more embodiments use a sampling technique to sample locations along the ray and determine feature descriptors for each location using the remainder of the model framework (304). In one or more embodiments, the ray casting engine (344) is configured to aggregate feature descriptors along the ray's locations. A single ray may pass through multiple objects. Thus, accumulation may be through multiple objects.

[0056] The interpolator (346) is configured to interpolate features from the object model across different resolutions and from different locations in the plurality of resolutions. Specifically, the interpolator (346) is configured to generate a set of interpolated features for particular locations in the object model from the neural hash grid.

[0057] The CNN decoder (348) is a convolutional neural network. The CNN decoder is configured to decode the output of the shared MLP model (342) to generate color values ​​for particular sample locations along a ray. In one or more embodiments, the CNN decoder (348) takes as input neural feature descriptors for particular locations and generates as output color values ​​(e.g., red, green, blue (RGB) values) for the particular locations. The CNN decoder (348) may be configured to upscale the output of the shared MLP model (342). As another example, the CNN decoder may be configured to remove artifacts from the original image.

[0058] The LiDAR decoder (350) is a neural network configured to generate LiDAR points based on the output of the shared MLP model (342). LiDAR points have depth and intensity pairs that are accumulated along LiDAR rays. The depth values ​​of LiDAR rays are calculated as an accumulation of depths. An accumulation depth function may be used to calculate the depth values. The accumulation depth function weights the depth values ​​according to the location and the accumulated transmittance. The decoder MLP model of the LiDAR decoder (350) takes the rendered accumulated volumetric features as input and outputs the intensity.

[0059] In one or more embodiments, the classifier network (352) is a classifier model configured to train a CNN decoder. The classifier network (352) is the classifier portion of a generative adversarial network. The classifier network (352) receives as input simulated images generated using the ray casting engine and the CNN decoder (348) and real images (330). The classifier network (352) attempts to classify the simulated and real images as either simulated or real. If the classifier network is correct in classifying the simulated image (i.e., the classifier network correctly classifies the simulated image as a simulated image), the classification contributes to a loss for updating the CNN decoder (348). If the classifier network is incorrect in classifying the simulated image and the real image (i.e., the classifier network incorrectly classifies the simulated image as a real image, or vice versa), the classification contributes to a loss for updating the classifier network (352). The classifier network (352) is therefore adversarial to the CNN decoder (348), whereby classification leads to the loss of one of the two networks.

[0060] The loss function (354) is a function used to calculate the loss of the system. The loss function (354) uses various outputs of the model framework (304) to calculate losses that are used to update the model framework (304) through backpropagation. During backpropagation, one or more layers may be frozen to calculate losses for other layers.

[0061] The trajectory refinement model (356) is a model configured to refine the object trajectory to ensure that the object trajectory is more accurate. The model may minimize a loss function and optimize the object's six degree of freedom (DoF) pose at each timestamp to more accurately reflect the object location.

[0062] In one or more embodiments, the trajectory refinement model uses prior imagery and LiDAR data to refine the trajectory.

[0063] Referring to Figure 4, Figure 4 illustrates an implementation (400) of the rendering system (300) shown in Figure 3. Specifically, Figure 4 illustrates an example architecture diagram of a rendering system according to one or more embodiments.

[0064] As shown in Figure 4, a scene is divided into three components: a static scene (404), a remote region (e.g., the sky) (402), and dynamic actors (406). Actors are the objects described above. The three components of the scene are modeled using the same architecture, but with different feature grid sizes. For each dynamic actor, its feature grid F is generated by a hypernet from a latent code z. First, for each point sampled along a ray r with location x and view direction d, a feature descriptor f (410) is queried from the neural hash grid. Next, volume rendering (412) is performed to obtain a rendered feature descriptor f (414). A CNN decoder (416) is used to decode the feature descriptor patch into an RGB image (418), and a LiDAR intensity MLP decoder (420) decodes the LiDAR intensity l for the ray r. int The various parts correspond to the similarly named components in Figure 3.

[0065] After the neural hash grid is constructed, one or more embodiments search for features at each sampled point via cubic linear interpolation and apply a small MLP to generate intermediate features. The view direction is then concatenated with the features before sending the concatenated view direction features to the final linear layer. rgb (416) is applied on top of the rendered feature map to generate the final image. One or more embodiments also include an additional decoder for lidar intensity (g int ) (418). Each of the components will be described in more detail below.

[0066] The scene representation is described below. One or more embodiments first define a region of interest using the SDV trajectory. Then, one or more embodiments generate a volumetric occupancy grid and set the voxel size. For static scene models, the multi-resolution feature grid may have 16 levels in the resolution hierarchy. The resolution may increase exponentially. For dynamic actor models, the multi-resolution feature grid may have several levels, with the resolution increasing exponentially. A spatial hash function may be used to map each feature grid to a fixed number of features.

[0067] Next, we describe the dynamic actor model (i.e., the neural hash grid in Figure 3). Each actor model is represented by an independent feature grid generated from a shared hypernet (408). The hypernet fz is a multi-layer MLP. The provided dynamic actor tracklets, even when human-annotated, can be inaccurate, leading to blurred results. Therefore, the actor tracklets can be refined during training. Each dynamic actor A has a trajectory initialized by a set of poses. i For , one or more embodiments jointly optimize the rotational and translational components of the pose at each time step. A symmetry prior along the longitudinal axis may be incorporated for vehicle objects. During training, one or more embodiments randomly flip the input point and view direction when querying the neural feature field.

[0068] The sky model (i.e., the exterior region background model in FIG. 3) is described below. For distant regions outside the volume, one or more embodiments model the distant regions using inverse spherical parameterization. One or more embodiments sample 16 points for the distant sky region during volume rendering.

[0069] The neural feature field NFF may be obtained as follows: bg ) and actor (f A) to obtain the geometry s and the feature descriptor f from the feature grid, bg Both f and fA contain multiple sub-networks. The first sub-network may be an MLP that takes the interpolated features and predicts the geometry (signed distance values) and intermediate geometry features. The second sub-network may be an MLP that takes the intermediate geometry features and the viewpoint encoding as input and predicts the neural feature descriptor f.

[0070] The following describes a CNN decoder (e.g., a camera RGB decoder). As shown in the enlarged section (420) of Figure 4, the camera RGB decoder may have multiple residual blocks. A convolutional layer is first applied to convert input features into a first channel set, and another convolutional layer is applied to predict the final output image. An upsample layer may be between different residual blocks of the CNN.

[0071] To improve realism at unobserved viewpoints, one or more embodiments render images at new camera viewpoints during training. To create "pseudo" camera poses, one or more embodiments randomly jitter the translational components of the training camera poses with standard Gaussian noise. Because ground truth supervision is not available for the new views, one or more embodiments may apply adversarial training to force rendered image patches to look similar to unperturbed pose image patches.

[0072] LiDAR Intensity Decoderg int may be a multi-layer MLP. Training may be performed as follows. One or more embodiments may have multi-stage training to accelerate convergence and for better stability. In the first stage, the CNN decoder is frozen and only the NFF is trained for multiple iterations. In the second stage, one or more embodiments simultaneously train the CNN and NFF for multiple iterations. In the final stage, one or more embodiments add an adversarial loss to jitter pose for multiple iterations. In these stages, one or more embodiments sample multiple LiDAR rays plus camera rays per iteration.

[0073] Figures 5 and 6 illustrate flowcharts according to one or more embodiments. Figure 5 illustrates a flowchart for training a rendering system, and Figure 6 illustrates a flowchart for using a rendering system. While the various steps in these flowcharts are presented and described sequentially, at least some steps may be performed in a different order, combined, or omitted, or at least some steps may be performed in parallel. Furthermore, steps may be performed actively or passively.

[0074] At block 502, a neural hash grid is initialized for multiple objects, including stationary and moving objects. Additionally, the constraint vectors may initially be set to zero. To initialize the neural hash grid, the hypervisor network takes the constraint vector of each moving object and directly predicts the neural hash grid within the volume of the object's bounding box. The background scene (e.g., the target region and the external region) does not have a hypervisor network, and one or more embodiments directly learn the target region background model and the external region background model.

[0075] At block 504, a location is selected. For a particular virtual sensor, a set of rays is defined based on the internal and external properties of the sensor. Because the virtual sensor may replicate a real sensor on a real autonomous system, the internal and external properties of the virtual sensor may be defined by the corresponding real sensor. A ray casting engine casts rays into a scene (e.g., defined by a simulation system). During training, the scene is set as a real-world scene. Thus, real camera and LiDAR images can match the virtual scene being rendered. For each ray, points along the ray are sampled. Each sampled point corresponds to a location, which may be selected at block 504.

[0076] At block 506, hash grid features adjacent to the sampled points in the object's corresponding neural hash grid are interpolated to obtain a set of location features. A cubic linear interpolation may be performed. Specifically, for a particular location, the object at that location is identified and the neural hash grid for the object is obtained. A cube of the neural hash grid on which the location is located at each resolution is determined. Interpolation is applied to the cube to calculate a specific location feature at that particular location in continuous space.

[0077] At block 508, an MLP model is run on the set of location features to obtain a set of image features for the location. In one or more embodiments, the MLP model is a shared MLP model that generates neural features (i.e., image features) from the set of location features. The location features are processed as feature vectors through layers of the MLP model to generate neural features. The neural features may be further processed via volume rendering to generate an image feature map. One or more embodiments volume render the image feature map. The feature map is processed by a CNN decoder to generate the final image. The image features in the feature map are different from the hash grid features. The image features are generated by a shared MLP that takes the hash grid features and view direction as input. Equation (1) below characterizes the generation of image features in one or more embodiments.

[0078] At block 510, a determination is made whether another location exists. If another location exists, flow returns to block 504 to select the next location. For example, the next ray or the next sample along the ray may be determined.

[0079] At block 512, ray casting is performed to generate a feature image from the image features. The ray casting engine combines features from the feature map along the ray to generate a cumulative feature for the ray. This process is repeated for each ray by the ray casting engine.

[0080] At block 514, a CNN decoder model is run on the feature image to generate a rendered image. The CNN has a first layer that takes the feature image as input. Through processing through multiple layers, the CNN upscales the feature image and corrects artifacts. The result is an output image.

[0081] At block 516, a LiDAR decoder is executed on the output of the MLP model. In one or more embodiments, because the LiDAR sensor may be located in a different location than the camera on the autonomous system, the same model may be used for the LiDAR and the camera, but different outputs of the MLP model may be used for the LiDAR and the camera. LiDAR points have a distance value and an intensity value. The distance value may be calculated directly from the sample points along the ray. The LiDAR decoder model may predict the intensity value from the output of the sample points along the ray.

[0082] At block 518, losses are computed using the labeled sensor data. In one or more embodiments, a single loss value is computed as a combination of losses. The single loss is backpropagated through the model of the network. The losses are determined using observations obtained from the real world. For example, a sensing vehicle traveling on a road may have a camera and a lidar sensor to capture various observations of a target object. The losses include RGB pixel loss, LiDAR loss, regularization loss, and adversarial loss.

[0083] RGB loss is the camera image loss accumulated across patches using color values ​​in the rendered image and sensor data for the same viewing direction and angle. For each of at least a subset of pixels, the observed color value of the corresponding pixel in the target image is determined. Specifically, the difference between the observed color value and the simulated color value is calculated. The average of the differences is the camera image loss.

[0084] The camera image loss may be a perceptual loss. The perceptual loss may use a pre-trained network to compute feature maps from the image. The difference between the feature maps generated by the pre-trained network on the actual image and the feature maps generated by the pre-trained network on the rendered image is the perceptual loss.

[0085] The adversarial loss may be calculated based on whether the pre-trained classifier correctly classifies the rendered image as a simulated image or a real image. As described above with reference to Figure 3, the output of the classification may contribute to the camera image loss or to the classifier loss for further training the classifier network.

[0086] The LiDAR loss is the accumulated loss over a subset of LiDAR rays and may be calculated using the LiDAR points determined for the LiDAR ray and sensor data with the same line of sight and angle as the LiDAR ray. For each LiDAR ray, observed LiDAR points are obtained for the target object with the same line of sight and angle as the LiDAR ray. The observed LiDAR points are compared to the simulated LiDAR depth. Specifically, the difference between the depth at the observed LiDAR point value and the simulated depth at the simulated LiDAR point value is calculated as the depth difference. Similarly, the difference between the intensity at the observed LiDAR point value and the simulated intensity at the simulated LiDAR point value is calculated as the intensity difference. The depth difference and intensity difference are combined, such as by a weighted sum, to generate a total difference for the LiDAR points. The average of the total differences over the LiDAR points is the LiDAR loss.

[0087] In at least some embodiments, regularization terms are computed and used as part of the total loss, and may include a term to encourage the signed distance function to satisfy the eikonal equation and a smoothness term to encourage the reconstructed target object to be smooth.

[0088] The total loss may be calculated as a weighted sum of the losses, each weighted by a parameter for weighting the loss, the parameter being configurable.

[0089] The total loss is backpropagated through the model of the rendering system. Different models may be frozen for different periods of training to calculate the total loss. Specifically, the total loss is backpropagated through MLP, LiDAR, CNN, and hypervisor networks. The process may iteratively train the model framework to iteratively improve the rendering system.

[0090] FIG. 6 shows a flowchart for rendering a scene using a trained neural hash grid according to one or more embodiments. At block 602, a scene of objects and an autonomous system is defined. The scene may be defined based on a predefined scenario. For example, a simulation system may define a scenario for testing an autonomous system. As another example, predefined scenarios may exist. As another example, the system may perturb an existing scenario by moving a player or objects within the scene to generate a scenario. Defining a scene specifies the location of three-dimensional virtual objects and autonomous systems, and more generally, the player, in a virtual environment. As another example, the virtual environment may be defined based on previous interactions. Various mechanisms may be used to define a scene.

[0091] At block 604, a location is selected. At block 606, hash grid features adjacent to the sampled point in the object's corresponding neural hash grid are interpolated to obtain a set of location features. At block 608, an MLP model is run on the set of location features to obtain a set of image features for the location. At block 610, a determination is made whether another location exists. If another location exists, flow returns to block 604 to select the next location. At block 612, ray casting is completed to generate a feature image. At block 614, a CNN decoder model is run on the feature image to generate a rendered image. At block 616, a LiDAR decoder is run on the output of the MLP model. The processing of blocks 604-616 may be performed during training as in use, but for different scenarios. The processing may be repeated for each iteration of the simulation. For example, the process may be repeated for each execution of block 203 of FIG. 2. Because a trained rendering system is used, the output is a realistic representation of the simulated environment, but with objects in different positions than in the real world.

[0092] At block 618, the rendered image is processed. Processing the rendered image may include sending the rendered image to a different machine for display, displaying the rendered image, processing the rendered image with a virtual driver (e.g., to determine an action or reaction based on the rendered image), storing the rendered image, or performing other processing.

[0093] Figure 7 shows an example of a 3D scene (700) with a neural hash grid overlaid on a geographic region. As shown in Figure 7, the 3D scene is decomposed into a static background (gray) and a set of dynamic actors (road image). The neural hash grid is queried separately against the static background model and the dynamic actor model. Volume rendering is performed to generate neural feature descriptors. The static scene is modeled with a sparse feature grid. A hypernetwork is used to generate a representation for each actor from a trainable latent layer. Finally, a CNN is used to decode the feature patches into an image.

[0094] One or more embodiments may be used to create real-world sensor inputs for mixed reality. For example, as shown in Figures 8A and 8B, a simulator can create a mixed-reality world in which an actor's actions deviate from the real world. In Figure 8A, the image on the left (802) shows an image of the real world captured via an actual camera on an autonomous system. The image on the right (804) of Figure 8A shows a mixed-reality image that deviates from a real event. That is, the image on the right (804) shows the actor moving into a different lane. As shown, the image captures the actor even when there is a variation in the actor's gaze direction.

[0095] Figure 8B illustrates a different deviation. Similar to Figure 8A, the image on the left (806) shows an image of the real world captured via an actual camera on an autonomous system, while the image on the right (808) shows a mixed reality image that deviates from the actual event. In Figure 8B, a self-driving vehicle (SDV) switches lanes. Thus, the line of sight of all objects changes in the image. The deviation means that portions of objects that were hidden in the original image are now visible in the simulated image. One or more embodiments adapt to the changes via a three-dimensional object model to create a realistic image of the real world. Using embodiments of the present disclosure, the camera image and LiDAR point cloud from the simulated mixed reality simulation are indistinguishable from the actual camera image and LiDAR point cloud. Thus, the embodiments generate more realistic sensor inputs in the virtual sensor.

[0096] To make safe self-driving vehicles (SDVs) a reality, autonomous systems are being rigorously tested. This testing generates safety-critical scenarios beyond those that can be safely collected globally, as many scenarios rarely occur on public roads. To accurately evaluate performance, one or more embodiments require testing the SDV in these scenarios in a closed-loop manner, with the SDV and other actors interacting with each other at each time step. Previously recorded driving logs provide a rich resource for constructing these new scenarios, but for closed-loop evaluation, one or more embodiments require modifying sensor data based on new scene configurations and SDV decisions, as actors may be added or removed and the trajectories of existing actors and SDVs differ from the original logs. One or more embodiments relate to a neural sensor simulator that takes a single recorded log captured by a sensor-equipped vehicle and transforms it into a realistic closed-loop multi-sensor simulation. Neural feature grids are generated to reconstruct both the static background and dynamic actors in the scene, which are then synthesized to simulate LiDAR and camera data in new perspectives, added or removed actors, and new configurations. To better handle extrapolated views, one or more embodiments incorporate learnable priors for dynamic objects and leverage convolutional networks to complete unseen regions, resulting in realistic sensor data with a small domain gap on downstream tasks.

[0097] Given camera images and LiDAR point clouds captured by a moving platform, as well as a log with their relative poses in a reference frame, the goal is to build an editable and controllable digital twin from which one or more embodiments can generate realistic multimodal sensor simulations and counterfactual scenarios of interest. One or more embodiments build a model based on the intuition that the 3D world can be decomposed as a static background and a set of moving actors. By effectively isolating and modeling each component, one or more embodiments can generate new scenarios and manipulate the actors to simulate sensor observations from new perspectives.

[0098]

number

[0099]

number

[0100]

number

[0101]

number

[0102]

number

[0103]

number

[0104]

number

[0105]

number

[0106] After constructing the scene representations of the static and dynamic world, the next step is to render the scene into the data modality of interest. In the following, we will discuss camera images and LiDAR point clouds. However, other decoders may be applied without departing from the scope of the claims.

[0107]

number

[0108]

number

[0109]

number

[0110]

number

[0111]

number

[0112]

number

[0113] Each term in the loss function is explained below.

[0114]

number

[0115]

number

[0116] Because LiDAR observations are noisy, one or more embodiments filter outliers and encourage the model to focus on reliable observations. In practice, one or more embodiments optimize for 95% of the rays in each batch that have the smallest depth error.

[0117]

number

[0118]

number

[0119] Embodiments may be implemented on a computing system specifically designed to achieve improved technical results. When implemented in a computing system, the features and elements of the present disclosure provide a significant technological advance over computing systems that do not implement the features and elements of the present disclosure. Any combination of mobile, desktop, server, router, switch, embedded device, or other type of hardware may be improved by including the features and elements described in this disclosure. For example, as shown in FIG. 9A , a computing system (900) may include one or more computer processors (902), non-persistent storage (904), persistent storage (906), communication interfaces (912) (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and numerous other elements and functions that implement the features and elements of the present disclosure. The computer processor (902) may be an integrated circuit for processing instructions. The computer processor may be one or more cores or micro-cores of a processor. The computer processor (902) includes one or more processors. The one or more processors may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), combinations thereof, and the like.

[0120] The input device (910) may include a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device. The input device (910) may receive input from a user in response to data and messages presented by the output device (908). The input may include text input, audio input, video input, etc., which may be processed and transmitted by the computing system (900) according to the present disclosure. The communication interface (912) may include integrated circuits for connecting the computing system (900) to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or any other type of network) and / or to another device, such as another computing device.

[0121] Additionally, the output device(s) (908) may include a display device, a printer, an external storage device, or any other output device. One or more of the output devices may be the same as or different from the input devices. The input and output devices may be locally or remotely connected to the computer processor (902). Many different types of computing systems exist, and the aforementioned input and output devices may take other forms. The output device (908) may display data and messages sent or received by the computing system (900). The data and messages may include text, audio, video, etc., and may include the data and messages described above in other figures of this disclosure.

[0122] Software instructions in the form of computer-readable program code for implementing the embodiments may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer-readable medium such as a CD, DVD, storage device, diskette, tape, flash memory, physical memory, or any other computer-readable storage medium. In particular, the software instructions may correspond to computer-readable program code configured, when executed by a processor, to implement one or more embodiments, which may include sending, receiving, presenting, and displaying data and messages as described in other figures of this disclosure.

[0123] The computing system (900) of FIG. 9A may be connected to or be part of a network. For example, as shown in FIG. 9B, the network (920) may include multiple nodes (e.g., node X (922), node Y (924)). Each node may correspond to a computing system such as the computing system shown in FIG. 9A, or a group of nodes combined may correspond to the computing system shown in FIG. 9A. As an example, an embodiment may be implemented on a node of a distributed system connected to other nodes. As another example, an embodiment may be implemented on a distributed computing system having multiple nodes, with each portion located on a different node within the distributed computing system. Furthermore, one or more elements of the computing system (900) described above may be located at a remote location and connected to other elements via a network.

[0124] Nodes in network 920 (e.g., node X 922, node Y 924) may be configured to provide services to client devices 926, including receiving requests and sending responses to the client devices 926. For example, the nodes may be part of a cloud computing system. The client devices 926 may be computing systems, such as the computing system illustrated in FIG. 9A. Additionally, the client devices 926 may include and / or implement all or part of one or more embodiments.

[0125] The computing system of FIG. 9A may include functionality for presenting raw data and / or processed data, such as the results of comparisons and other processing. For example, presenting data may be achieved through various presentation methods. Specifically, data may be presented by being displayed on a user interface, transmitted to a different computing system, or stored. The user interface may include a GUI that displays information on a display device. The GUI may include various GUI widgets that organize what data is displayed as well as how the data is presented to the user. Furthermore, the GUI may present data directly to the user, for example, data presented as actual data values ​​via text, or data rendered into a visual representation of the data by the computing device, such as by visualizing a data model.

[0126] As used herein, the term "connected to" contemplates multiple meanings. The connection may be direct or indirect (e.g., through another component or network). The connection may be wired or wireless. The connection may be a temporary, permanent, or semi-permanent communication channel between two entities.

[0127] The various illustrations of the figures may be combined and may include or be included within features described in other figures of this application. Various elements, systems, components, and steps shown in the figures may be omitted, repeated, combined, and / or modified as shown in the figures. Thus, the scope of the disclosure should not be considered limited to the specific configurations shown in the figures.

[0128] Ordinal numbers (e.g., first, second, third, etc.) may be used in this application as adjectives of elements (i.e., any noun in this application). The use of ordinal numbers does not imply or create a particular order of elements, nor does it limit any element to only a single element, unless expressly disclosed, such as by use of "before," "after," "single," and other such terms. Rather, the use of ordinal numbers is to distinguish elements. As an example, a first element is different from a second element, and a first element may encompass two or more elements and follow (or precede) the second element in the order of elements.

[0129] Furthermore, unless otherwise stated, or is an "inclusive or," and thus includes "and." Furthermore, items joined by "or" may include any combination of items with any number of each item, unless otherwise stated.

[0130] In the above description, numerous specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present technology may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description. Furthermore, other embodiments not expressly described above may be devised that do not depart from the scope of the claims disclosed herein. Accordingly, the scope should be limited only by the scope of the appended claims.

Claims

1. interpolating a plurality of hash grid features adjacent to a location within a neural hash grid defined for the target object to obtain a set of location features; processing the set of location features with a multi-layer perceptron (MLP) model to generate a set of image features for the location; completing ray casting to the target object using the set of image features to generate a feature image; generating a rendering image from the feature image; and processing the rendered image.

2. The computer-implemented method of claim 1 , further comprising: executing a convolutional neural network (CNN) decoder on the feature image to generate the rendered image.

3. 3. The computer-implemented method of claim 2, wherein completing the ray casting to generate the feature image generates the feature image at a first resolution, and wherein the CNN decoder upscales the feature image at the first resolution to a second resolution when generating the rendered image.

4. The computer-implemented method of claim 2 , wherein completing the ray casting to generate the feature image corrects rendering artifacts in the feature image.

5. 10. The computer-implemented method of claim 1, further comprising: executing a LiDAR decoder using object appearances generated from the neural hash grid to generate a LiDAR point cloud.

6. determining that the target object intersects with the ray during the ray casting; 2. The computer-implemented method of claim 1, further comprising: selecting the neural hash grid that matches the target object from a plurality of neural hash grids for a plurality of objects based on the target object intersecting the ray.

7. initializing the neural hash grid with the hash grid features set to random values; calculating a loss based on a difference between the neural hash grid and a sensor image; 2. The computer-implemented method of claim 1, further comprising backpropagating the loss through the MLP model and the neural hash grid to update the plurality of hash grid features.

8. 8. The computer-implemented method of claim 7, wherein the plurality of random values ​​are used to initialize non-empty portions of the neural hash grid, and empty portions of the neural hash grid are initialized with predefined values.

9. running a convolutional neural network (CNN) decoder on the feature image to generate the rendered image; after generating the rendered image, comparing the rendered image with a real image to generate an image loss; The computer-implemented method of claim 7 , wherein calculating the loss uses the image loss.

10. Comparing the rendered image to the real image comprises: detecting a plurality of differences between pixels of the rendered image and corresponding pixels of the real image; and calculating the image loss based on the plurality of differences.

11. running a convolutional neural network (CNN) decoder on the feature image to generate the rendered image; After generating the rendered image, dividing the rendered image into a plurality of image patches; Dividing a real image into a plurality of real image patches; running a classifier model on the plurality of image patches and the plurality of real image patches to predict a plurality of classifications; generating a classifier loss based on the inaccurate classifications; generating a CNN loss based on the accurate classifications; and updating the CNN using the CNN loss; and The computer-implemented method of claim 1 , further comprising: updating the classifier model using the classifier loss.

12. modeling each of a plurality of objects with a corresponding latent code to generate a plurality of constraint vectors for the plurality of objects; training a hypervisor network to use the corresponding latent code in a constraint vector of the plurality of constraint vectors corresponding to the object; 2. The computer-implemented method of claim 1, further comprising: jointly optimizing the constraint vectors across the objects to constrain representations of the objects.

13. The computer-implemented method of claim 1 , further comprising simultaneously optimizing rotational and translational components of the pose of the target object at each of a plurality of time steps.

14. Memory and a computer processor including computer readable program code for performing operations, said operations comprising: interpolating a plurality of hash grid features adjacent to a location within a neural hash grid defined for the target object to obtain a set of location features; processing the set of location features with a multi-layer perceptron (MLP) model to generate a set of image features for the location; completing ray casting to the target object using the set of image features to generate a feature image; generating a rendering image from the feature image; and processing the rendered image.

15. The operation is The system of claim 14 , further comprising: executing a convolutional neural network (CNN) decoder on the feature image to generate the rendered image.

16. 16. The system of claim 15, wherein completing the ray casting to generate the feature image generates the feature image at a first resolution, and wherein the CNN decoder upscales the feature image at the first resolution to a second resolution when generating the rendered image.

17. The system of claim 15 , wherein the completing the ray casting to generate the feature image corrects rendering artifacts in the feature image.

18. 15. The system of claim 14, further comprising: running a LiDAR decoder using object appearances generated from the neural hash grid to generate a LiDAR point cloud.

19. A non-transitory computer readable medium containing computer readable program code for performing operations, said operations comprising: interpolating a plurality of hash grid features adjacent to a location within a neural hash grid defined for the target object to obtain a set of location features; processing the set of location features with a multi-layer perceptron (MLP) model to generate a set of image features for the location; completing ray casting to the target object using the set of image features to generate a feature image; generating a rendering image from the feature image; and processing the rendered image.

20. The operation is 20. The non-transitory computer-readable medium of claim 19, further comprising: executing a convolutional neural network (CNN) decoder on the feature image to generate the rendered image.

Images