Real-time image rendering of large-scale scenes

Abstract

Real-time image rendering of a large-scale scene involves performing operations that include identifying the camera position of a camera within a geographical area, rasterizing a UV feature map using the camera position and polygonal mesh, and obtaining a feature buffer. Furthermore, the operation includes processing the feature buffer using the camera's view orientation with a shading machine learning model to generate an image rendering that includes opacity and color values. The operation further includes generating a rendered image from the opacity and color values.

Description

Technical Field

[0001] Cross - reference to related applications This application is a non - provisional application of U.S. Patent Application No. 63 / 450,623, filed on March 7, 2023. Accordingly, the benefit of this U.S. Patent Application is claimed, and this U.S. Patent Application is hereby incorporated by reference in its entirety into this specification.

Background Art

[0002] Background A virtual world is an environment created by computer simulation and interacts in a three - dimensional space similar to the real world. In some cases, the virtual world is designed to replicate at least some aspects of the real world. For example, the virtual world can include objects and backgrounds reconstructed from the real world. By reconstructing objects and backgrounds from the real world, the system can replicate aspects of the real world.

[0003] Interaction with the virtual world is generally done by rendering images of large - scale scenes. The scene is based on the camera placement of a virtual camera in the virtual world. In many applications of the virtual world, speed and realism are important. Specifically, entities interacting with the virtual world should feel as if they are interacting in the real world both in terms of time and space. That is, the scene must appear as a realistic rendering at the frame rate of interaction for an immersive and seamless experience. Achieving both speed and realism in the rendering of large - scale scenes has been a challenge for many years.

Summary of the Invention

[0004] Summary Generally, in one embodiment, one or more embodiments relate to a method that includes identifying the camera position of a camera within a geographical area, rasterizing a UV feature map using the camera position and polygonal mesh, and obtaining a feature buffer. The method further includes processing the feature buffer using the camera's view direction with a shading machine learning model to generate an image rendering including opacity values ​​and color values. The method further includes generating a rendered image from the opacity values ​​and color values.

[0005] In general, in one embodiment, one or more embodiments relate to a system including memory and a computer processor containing computer-readable program code for performing an operation. The operation includes identifying the camera position of a camera within a geographical area, rasterizing a UV feature map using the camera position and polygonal mesh, and obtaining a feature buffer. Furthermore, the operation includes processing the feature buffer using the camera's view direction with a shading machine learning model to generate an image rendering including opacity values ​​and color values. The operation further includes generating a rendered image from the opacity values ​​and color values.

[0006] In general, in one embodiment, one or more embodiments relate to a non-temporary computer-readable medium containing computer-readable program code for performing an operation. The operation includes identifying the camera position of a camera within a geographical area, rasterizing a UV feature map using the camera position and polygonal mesh, and obtaining a feature buffer. Furthermore, the operation includes processing the feature buffer using the camera's view direction with a shading machine learning model to generate an image rendering including opacity and color values. The operation further includes generating a rendered image from the opacity and color values.

[0007] Other aspects of the present invention will become apparent from the following description and the appended claims. [Brief explanation of the drawing]

[0008] [Figure 1] The diagram shows an autonomous training and testing system according to one or more embodiments. [Figure 2] This shows a flowchart of an autonomous training and testing system according to one or more embodiments. [Figure 3] The diagram shows a rendering system according to one or more embodiments. [Figure 4] This shows a flowchart for generating a rendered image using one or more embodiments. [Figure 5] This shows a flowchart for using a UV code map according to one or more embodiments. [Figure 6] A flowchart for processing the background according to one or more embodiments is shown. [Figure 7] This diagram shows a flow chart for training a rendering system using one or more embodiments. [Figure 8] This document presents an example of a rendering system for performing real-time rendering of large-scale scenes using one or more embodiments. [Figure 9] This shows an example of vector quantization for generating a code map using one or more embodiments. [Figure 10A] This document illustrates a computing system according to one or more embodiments of the present invention. [Figure 10B] This document illustrates a computing system according to one or more embodiments of the present invention. [Modes for carrying out the invention]

[0009] Similar elements in various figures are indicated with the same reference number for consistency. Detailed explanation Generally, the embodiments relate to real-time photorealistic rendering of large-scale scenes. In particular, one or more embodiments define a polygonal mesh and a UV feature map for a virtual world. The UV feature map has neural features about locations in the virtual world. Neural features are trained features generated by machine learning. Therefore, the neural features do not have to explicitly represent the appearance of objects in the virtual world. The neural features and polygonal mesh in the UV feature map are used by a rasterization engine that extracts neural features corresponding to the camera's position in the virtual world. A shading machine learning model is configured to convert the neural features into an image rendering with opacity and color values. The shading machine learning model is a lightweight model that uses the neural features output from the rasterization engine along with the camera orientation to generate the image rendering. For example, instead of using raycasting, the shading machine learning model can process a feature vector containing neural features and view orientation through a neural network layer. In one or more embodiments, rendering can be performed in real time by using the lightweight machine learning model.

[0010] A rendering system can be used in conjunction with a virtual reality system in the training or testing of autonomous systems. In a virtual reality system, the user interacts with the virtual world as if they were in the real world. The virtual world may be a computer-generated environment, which may or may not be a virtualized version of the real world. For example, the virtual environment may have objects and backgrounds that do not exist in the real world. The rendering system provides rendered images to other components of the overall system, such as a display, another model, or other components. Furthermore, to simulate multiple cameras, the rendering system can operate in parallel for different cameras.

[0011] In some embodiments, the system's processing can be used to generate a virtual world that mimics the real world but implements different scenarios. For example, the modified scenarios may include dynamic and / or static objects being in different locations, the player's viewpoint being altered because the player is located in a different place than in the real world, or other aspects of the real world being different.

[0012] Embodiments of the present invention can be used as part of generating a simulation environment for training and testing autonomous systems. An autonomous system is an autonomous mode of movement that moves and reacts to the real-world environment without requiring a human pilot or driver. Rather, the autonomous system includes a virtual driver, which is the decision-making part of the autonomous system. The virtual driver is an artificial intelligence system that learns how to interact with the real world. The autonomous system may be fully autonomous or semi-autonomous. As a mode of movement, the autonomous system is housed in a housing configured to move through the real-world environment. Examples of autonomous systems include self-driving vehicles (e.g., self-driving trucks and cars), drones, airplanes, robots, and the like. The virtual driver is software that makes decisions and causes the autonomous system to interact with the real world, such as moving, signaling, stopping, or maintaining its current state.

[0013] The real-world environment is a part of the real world designed for an autonomous system to navigate once it has been trained. Therefore, the real-world environment may include interactions with concrete and land, humans, animals, other autonomous systems, human-driven systems, buildings, and other objects as the autonomous system travels from its starting point to its destination. To interact with the real-world environment, the autonomous system may include various types of sensors used to acquire measurements of the real-world environment, particularly LiDAR sensors, and cameras to capture images from the real-world environment.

[0014] Testing and training virtual drivers of autonomous systems in real-world environments is unsafe due to potential accidents caused by untrained virtual drivers. Therefore, as shown in Figure 1, a simulator (100) is configured to train and test virtual drivers (102) of an autonomous system. For example, the simulator may be a unified, modular, mixed-reality closed-loop simulator for the autonomous system. The simulator (100) is a configurable simulation framework that allows for the evaluation of various autonomous components individually, as well as the evaluation of the entire system in a closed-loop manner. The simulator automatically reconstructs a "digital twin" of real-world scenarios, enabling accurate evaluation of virtual drivers at scale. Furthermore, the simulator (100) may be configured to perform mixed-reality simulations that combine real-world data with simulation data to generate diverse and photorealistic evaluation variations, providing insights into the performance of the virtual drivers. Mixed-reality closed-loop simulations allow the simulator (100) to analyze the behavior of virtual drivers in non-factual "hypothetical" scenarios that do not occur in the real world. The simulator (100) further includes the ability to simulate and train on rare but safety-critical scenarios with respect to the entire autonomous system and closed-loop training, in order to enable automatic and scalable improvements in autonomy.

[0015] The simulator (100) generates a simulation environment (104), which is a virtual world. The virtual driver (102) is the player in the virtual world. The simulation environment (104) is a simulation of a real-world environment that may or may not actually exist, and is designed for the autonomous system to move through. Thus, the simulation environment (104) includes simulations of real-world objects and backgrounds (i.e., simulated objects or assets), such as natural objects, structures, buildings and roads, obstacles, and other autonomous and non-autonomous objects. The simulation environment simulates environmental conditions in which the autonomous system may operate. Furthermore, the simulation environment (104) may be configured to simulate various weather conditions that may affect the input to the autonomous system. The simulated objects may include both stationary and non-stationary objects. Non-stationary objects are actors in the real-world environment.

[0016] The simulator (100) further includes an evaluation unit (110). The evaluation unit (110) is configured to train and test a virtual driver (102) by generating various scenarios in a simulation environment. Each scenario is a simulation environment setting that includes, but is not limited to, stationary parts, the movement of simulated objects, the actions of simulated objects on each other, and responses to actions performed by the autonomous system and the simulated objects. The evaluation unit (110) is further configured to evaluate the performance of the virtual driver using various metrics.

[0017] The evaluation unit (110) evaluates the performance of the virtual driver through the execution of scenarios. Evaluating the performance can include applying rules. For example, the rules can be that the autonomous system does not collide with other actors, follows safety and comfort criteria (e.g., passengers in the vehicle do not experience acceleration forces exceeding a certain acceleration force), does not deviate from the trajectory executed by the autonomous system, or other rules. For each rule, metric information that associates the degree of rule violation with a corresponding score can be combined. The evaluation unit (110) can be implemented as a data-driven neural network that learns to distinguish between good driving behavior and bad driving behavior. By utilizing various metrics of the evaluation system, it can be determined whether the autonomous system meets the requirements of the success criteria for a specific scenario. Furthermore, in addition to system-level performance, in the case of a module-based virtual driver, the evaluation unit can also evaluate individual modules such as the segmentation or prediction performance of actors in the scene regarding the ground truth recorded by the simulator.

[0018] The simulator (100) is configured to operate in a plurality of phases selected by the phase selection unit (108) and a mode selected by the mode selection unit (106). The phase selection unit (108) and the mode selection unit (106) can be graphical user interface or application programming interface components configured to receive the selection of the phase and the mode, respectively. The selected phase and mode define the configuration of the simulator (100). That is, the selected phase and mode define which system components communicate and the operation of the system components.

[0019] The phase can be selected using the phase selection unit (108). The phase may be a training phase or a test phase. In the training phase, the evaluation unit (110) provides metric information to the virtual driver (102), and the virtual driver (102) updates itself using the metric information. Furthermore, the evaluation unit (110) can use the metric information to further train the virtual driver (102) by generating scenarios for the virtual driver. In the test phase, the evaluation unit (110) does not provide metric information to the virtual driver. In the test phase, the evaluation unit (110) evaluates the virtual driver using the metric information and creates scenarios for the virtual driver (102).

[0020] The mode may be selected by a mode selection unit (106). The mode defines the degree of use of real-world data, whether noise is injected into the simulated data, the degree of perturbation of real-world data, and whether the scenario is designed to be adversarial. Exemplary modes include an open-loop simulation mode, a closed-loop simulation mode, a single-module closed-loop simulation mode, a fuzzy mode, and an adversarial mode. In the open-loop simulation mode, the virtual driver is evaluated with real-world data. In the single-module closed-loop simulation mode, a single module of the virtual driver is tested. An example of the single-module closed-loop simulation mode is the localizer closed-loop simulation mode, in which the simulator evaluates how the localizer estimated pose drifts over time as the scenario progresses within the simulation. In the training data simulation mode, the simulator is used to generate training data. In the closed-loop evaluation mode, the virtual driver and the simulation system are executed together to evaluate system performance. In the adversarial mode, the actor is modified to execute adversarially. In the fuzzy mode, noise is injected into the scenario (e.g., to replicate signal processing noise and other types of noise). Without departing from the scope of the system, other modes may exist.

[0021] The simulator (100) includes a controller (112) having a function of setting various components of the simulator (100) according to the selected mode and phase. That is, the controller (112) can change the settings of each component of the simulator based on the set parameters of the simulator (100). Such components include an evaluation unit (110), a simulation environment (104), an autonomous system model (116), a sensor simulation model (114), an asset model (117), an actor model (118), a latency model (120), and a training data generation unit (122).

[0022] The autonomous system model (116) is a detailed model of the autonomous system run by the virtual driver. The autonomous system model (116) includes the model, geometry, physical parameters (e.g., mass distribution, significant points), engine parameters, sensor locations and types, sensor firing patterns, information about the hardware run by the virtual driver (e.g., processor power, memory size, and other hardware information), as well as other information about the autonomous system. Various parameters of the autonomous system model may be configurable by the user or another system.

[0023] For example, if the autonomous system is a car, the modeling and dynamics may include the type of vehicle (e.g., car, truck), manufacturer and model, geometry, physical parameters such as mass distribution, axle position, engine type and performance, etc. Furthermore, the vehicle model may include information about sensors on the vehicle (e.g., cameras, LiDAR, etc.), the relative ignition synchronization patterns of the sensors, and the calibrated external properties (e.g., position and orientation) and internal properties (e.g., focal length) of the sensors. In addition, the vehicle model defines the on-board computer hardware, sensor drivers, controllers, and the autonomous software release under test.

[0024] The autonomous system model includes an autonomous system dynamic model. The autonomous system dynamic model is used in a dynamics simulation that acquires the operation actions of a virtual driver (e.g., steering angle, desired acceleration) and performs operation actions on the autonomous system within the simulation environment to update the state of the simulation environment and the autonomous system. To update the state, a kinematic motion model can be used, or a dynamics motion model that takes into account the forces applied to the vehicle can be used to determine the state. By accessing actual log scenarios with ground truth operations and vehicle states at each time step within the simulator, embodiments can optimize the analysis vehicle model parameters, or learn the parameters of a neural network that infers new states of the autonomous system in light of the virtual driver output.

[0025] In one or more embodiments, the sensor simulation model (114) models active and passive sensor inputs in the simulation environment. Passive sensor inputs capture the visual appearance of the simulation environment, including stationary and non-stationary simulated objects from the viewpoint of one or more cameras, based on the simulated positions of cameras in the simulation environment. Examples of passive sensor inputs include inertial measurement units (IMUs) and thermal. Active sensor inputs are inputs from active sensors, such as LiDAR, RADAR, global positioning systems (GPS), and ultrasonic sensors, to a virtual driver of the autonomous system. That is, active sensor inputs include measurements taken by the sensors, which are simulated based on the simulation environment, based on the simulated positions of the sensors in the simulation environment. As an example, active sensor measurements may be measurements taken by a LiDAR sensor over time in relation to the movement of the autonomous system in the simulation environment. In one or more embodiments, all or part of the sensor simulation model (114) may be the rendering system (300) shown in Figure 3, or may include the rendering system (300) shown in Figure 3. In such scenarios, the rendering system of the sensor simulation model (114) can perform the operations shown in Figures 4 to 6.

[0026] The sensor simulation model (114) is configured to simulate sensor observations of the surrounding scene in the simulation environment (104) at each time step, according to the sensor configuration on the vehicle platform. If the simulation environment directly represents the real-world environment without modification, the sensor output may be supplied directly to the virtual driver. In the case of light-based sensors, the sensor model simulates light as rays interacting with objects in the scene to generate sensor data. Depending on the asset representation (e.g., stationary and non-stationary objects), the embodiment may use graphics-based rendering, neural rendering, or a combination of multiple rendering methods for assets with texture meshes. Utilizing multiple rendering methods enables the construction of customizable worlds with improved realism. Since assets are constructive in 3D and support a standard interface for rendering commands, different asset representations can be seamlessly combined to generate the final sensor data. Furthermore, in scenarios where events that occurred in the real world are reproduced and the same autonomous system as in the real world is used, the original sensor observations can be reproduced at each time step.

[0027] An asset model (117) includes multiple models, each modeling a specific type of individual asset in the real world. Assets can include inanimate objects such as construction obstacles or traffic signs, parked vehicles, and backgrounds (e.g., vegetation or sky). Each entity in a scenario can correspond to an individual asset. Thus, an asset model, or an instance of a certain type of asset model, can exist for each object or asset in a scenario. Assets can be combined with each other to form a three-dimensional simulation environment. The asset model provides all the information the simulator needs to simulate the asset. The asset model provides the information used by the simulator to represent and simulate the asset in the simulation environment.

[0028] The actor model (118) is closely related to the asset model (117) and is likely to be part of the asset model (117). The actor model represents an actor in a scenario. An actor is a sentient living being with an independent decision-making process. That is, in the real world, an actor may be a living being (e.g., a human or an animal) that makes decisions based on its environment. An actor performs active movements, not passive movements, in addition to passive movements. For each actor in a scenario, there may be an actor model, or an instance of the actor model. The actor model is a model of the actor. If the actor is in a mode of transport, the actor model includes a model of the transport in which the actor is located. For example, the actor model may represent a pedestrian, a child, a driver-driven vehicle, a pet, a bicycle, and other types of actors.

[0029] The actor model leverages scenario specifications and assets to control all actors in a scene and their behavior at each time step. Actor behavior is modeled within a domain of interest centered on the autonomous system. Depending on the scenario specifications, the actor simulation controls actors within the simulation to achieve desired behavior. Actors can be controlled in various ways. One option is to leverage heuristic actor models, such as an intelligent-driver model (IDM) or a heuristic-derived lane-changing actor model, which attempts to maintain a specific relative distance or time-to-collision (TTC) from the leading actor. Another option is to directly reconstruct the actor's trajectory from the actual log, or to control the actor with a data-driven traffic model. Through a configurable design, embodiments can mix and harmonize different subsets of actors controlled by different behavior models. For example, a distant actor may initially have no possibility of interacting with the autonomous system and follow the trajectory of the actual log, but as it approaches the autonomous system, it can switch to a data-driven actor model. In another example, actors may be controlled by a heuristic or data-driven actor model that still matches high-level roots in the actual log. This mixed reality simulation provides control and photorealism.

[0030] Furthermore, the actor model can be configured to operate in either a cooperative or adversarial mode. In cooperative mode, the actor model represents actors who act rationally according to the state of the simulation environment. In adversarial mode, the actor model can represent actors who act irrationally, such as exhibiting reckless driving or poor driving habits.

[0031] In one or more embodiments, the actor model (118), asset model (117), and background may be part of a rendering system (described below with reference to Figure 3). As another example, the system may be a binary system in which the behavior of assets and actors (e.g., trajectories or positions) is defined separately from the appearance, which is part of the rendering system.

[0032] The latency model (120) represents the timing latency that occurs when an autonomous system is in a real-world environment. Several causes of timing latency may exist. For example, latency may exist from the time an event occurs until the sensor detects sensor information from the event and sends the sensor information to the virtual driver. Another latency may exist based on the difference between the computing hardware of the virtual driver and the computing hardware running the virtual driver in the simulation environment. Furthermore, another timing latency may exist between the time the virtual driver sends an operation signal to the autonomous system and the time of change (e.g., direction or velocity) based on the operation signal. The latency model (120) models the various causes of timing latency.

[0033] In other words, in the real world, safety-critical decisions can involve fractions of a second or less that affect response time. Latency models simulate the precise timing and latency of various components of an in-vehicle system. To enable scalable evaluation without strict requirements for specific hardware, the latency and timing of various components of an autonomous system and sensor module are modeled while running on different computer hardware. Latency models can reconstruct latency recorded from previously collected real-world data, or they can have a data-driven neural network that infers latency at each time step to match the hardware in a loop simulation setting.

[0034] The training data generation unit (122) is configured to generate training data. For example, the training data generation unit (122) can modify real-world scenarios to generate new scenarios. Modifying real-world scenarios is called mixed reality. For example, a mixed reality simulation may include adding new actors with novel behavior, changing the behavior of one or more actors from the real world, and modifying sensor data within a region while keeping the rest of the sensor data the same as the original log. In some cases, the training data generation unit (122) converts harmless scenarios into safety-critical scenarios.

[0035] The simulator (100) is connected to a data repository (105). The data repository (105) is any type of storage unit or device configured to store data. The data repository (105) contains data collected from the real world. For example, data collected from the real world includes the trajectories of real actors (126), real sensor data (128), the real trajectories of a system capturing the real world (130), and real latency (132). Each of the trajectories of real actors (126), real sensor data (128), the real trajectories of a system capturing the real world (130), and real latency (132) is data captured by one or more sensors from the real world (e.g., in a real-world log) or calculated directly from them. In other words, data collected from the real world is actual events that occurred in real life. For example, if the autonomous system is a vehicle, the real-world data may be captured by a vehicle traveling in the real world equipped with sensor devices.

[0036] Furthermore, the data repository (105) includes the ability to store one or more scenario specifications (140). A scenario specification (140) specifies a scenario and evaluation setting for testing or training an autonomous system. For example, a scenario specification (140) may describe the initial state of a scene, such as the current state of the autonomous system (e.g., full 6D attitude, velocity, and acceleration), map information specifying the road layout, and a scene layout specifying the initial state of all dynamic actors and objects in the scenario. In addition, a scenario specification may include dynamic actor information that describes how dynamic actors in the scenario should unfold over time, which is input to an actor model. Dynamic actor information may include actor route information, desired behavior, or aggressiveness. A scenario specification (140) may be specified by the user, programmatically generated using a domain-specification-language (DSL), procedurally generated using heuristics from a data-driven algorithm, or generated adversarially. Furthermore, the scenario specification (140) can also be conditional on data collected from real-world logs, such as taking place on a specific real-world map or having a subset of actors determined by their original positions and trajectories.

[0037] The interface between the virtual driver and the simulator corresponds to the interface between the virtual driver and the real-world autonomous system. For example, the sensor simulation model (114) and the virtual driver correspond to a virtual driver that interacts with a sensor in the real world. The virtual driver is the actual autonomous software that runs on the autonomous system. The simulated sensor data output by the sensor simulation model (114) may be converted to the exact message format that the virtual driver would take as input, as if the virtual driver were in the real world, and the virtual driver can then operate as a black-box virtual driver with simulated latency incorporated into sequentially executed components. The virtual driver then outputs the exact same control representation that it would use to interface with lower-level controllers on the actual autonomous system. The autonomous system model (116) then updates the state of the autonomous system in the simulation environment. In this way, the various simulation models of the simulator (100) operate asynchronously and in parallel at their respective frequencies to match the real-world settings.

[0038] Figure 2 shows a flowchart for running the simulator in closed-loop mode. In block 201, a digital twin of a real-world scenario is generated as the simulation environment state. Log data from the real world is used to generate the initial virtual world. The log data defines which asset and actor models are used in the initial positioning of assets. For example, a convolutional neural network can be used on the log data to identify different asset types in the real world. As another example, an offline perception system and manual annotations on the log data can be used to identify asset types. Thus, the corresponding asset and actor modes can be identified based on the asset type and added to the real-world actor and asset positions. In this way, the asset and actor models generate the initial three-dimensional virtual world.

[0039] In block 203, a sensor simulation model is executed for the simulated environment state, and simulated sensor outputs are obtained. The sensor simulation model can replicate views to sensors in an autonomous system using beamforming and other techniques. Each sensor in the autonomous system has a corresponding sensor simulation model and a corresponding system. The sensor simulation model is executed based on the position of the sensor in the virtual environment and generates simulated sensor outputs. The simulated sensor outputs are in the same form that the virtual driver would receive from the actual sensors. In one or more embodiments, block 203 can be executed as shown in Figures 4-6 (described later) to generate the respective camera outputs and lidar sensor outputs of virtual cameras and virtual lidar sensors. The operation in Figures 4-6 can be performed for each camera and each lidar sensor on the autonomous system to simulate the corresponding camera and lidar sensor outputs. Using the sensor position and view direction relative to the autonomous vehicle, the originating position of the corresponding virtual sensor on the simulated autonomous system can be replicated. In this way, the various sensor inputs to the virtual driver match the combination of inputs if the virtual driver were in the real world.

[0040] The simulated sensor output is passed to the virtual driver. In block 205, the virtual driver executes based on the simulated sensor output to generate an operational action. The operational action defines how the virtual driver controls the autonomous system. For example, in the case of SDV, the operational action may be the amount of acceleration, steering movement, turn signal trigger, etc. From the operational action, the autonomous system state in the simulated environment is updated in block 207. The operational action is used as input to the autonomous system model to determine the actual behavior of the autonomous system. For example, the autonomous system dynamic model may use the operational action in addition to road and weather conditions to represent the resulting movement of the autonomous system. For example, in a rainy or snowy environment, the amount of acceleration action may be the same as in a dry environment, but the acceleration may be less than in a dry environment. As another example, the autonomous system model may take into account the possibility of tire malfunction (e.g., tire slippage), machine-based latency, or other imperfections that may exist in the autonomous system.

[0041] In block 209, the behavior of actors in the simulation environment is modeled based on the state of the simulation environment. Simultaneously with the virtual driver model, the actor model and asset model are executed for the state of the simulation environment to determine the updates of each asset and actor within the simulation environment. Here, the actor's behavior can be used to test the virtual driver using the previous output of the evaluation unit. For example, if the actor is adversarial, the evaluation unit can indicate the lowest scoring metric for the virtual driver based on its previous behavior. Using the mapping of metrics to the behavior of the actor model, the actor model is executed to utilize or test that particular metric.

[0042] In this way, in block 211, the simulation environment state is updated according to the actor's actions and the autonomous system state, and an updated simulation environment state is generated. The updated simulation environment includes changes in the positions of the actor and the autonomous system. Since the model runs independently of the real world, the updates can reflect deviations from the real world. Thus, the autonomous system is tested in new scenarios. In block 213, a decision is made whether to continue or not. If the decision is to continue, in block 203, the testing of the autonomous system continues using the updated simulation environment state. In each iteration, during training, the evaluation unit provides feedback to the virtual driver. Thus, the parameters of the virtual driver are updated to improve the performance of the virtual driver in various scenarios. During testing, the evaluation unit can perform tests using various scenarios and patterns, including edge cases that may be safety-critical. Thus, one or more embodiments improve the virtual driver and enhance the safety of the virtual driver in the real world.

[0043] As shown in the diagram, the virtual driver of the autonomous system operates based on the scenario and the currently learned parameters of the virtual driver. The simulator acquires the behavior of the autonomous system and provides the response in the simulation environment to the autonomous system's virtual driver. The evaluation unit evaluates the performance of the virtual driver and generates scenarios based on the performance. The process can continue when the autonomous system operates in the simulation environment.

[0044] Figure 3 shows a diagram of a rendering system (300) according to one or more embodiments. The rendering system (300) is configured to generate rendered images based on the position of a virtual camera in a geographical region of a virtual world. In one or more embodiments, the geographical region is a subregion of the virtual world. For example, the subregion may be the region from the virtual camera to a threshold distance in the virtual world. As another example, the geographical region may be the entire virtual world. In one or more embodiments, the rendering system (300) can render images inactively when the viewpoint of the virtual camera changes or when an object moves in the virtual world. The rendering system (300) includes a data repository (302) connected to a model framework (304).

[0045] The data repository (302) may contain one or more of the following: sensor data (128), polygonal mesh (306), background feature map (308), UV feature map (310), and code map (312).

[0046] The sensor data (128) is the sensor data described above with reference to Figure 1. The sensor data (128) includes actual images (330). Actual images (330) are images captured by one or more cameras in a geographical area. For example, when a detection vehicle is moving through a geographical area, the detection vehicle may have cameras that collect sensor data from the geographical area. In particular, the sensor data (128) is time-series data captured along the trajectory of the detection vehicle.

[0047] A polygonal mesh (308) is a mesh structure that maintains the geometric shape of a geographical area. The polygonal mesh (308) as defined herein corresponds to the standard definition used in the art and is sometimes also called a polygon mesh. A mesh has edges that connect vertices. Vertices have specific locations that are mapped to locations within the geographical area. Combinations of edges and vertices define faces. Faces correspond to the surfaces of objects within the geographical area. Faces are polygonal faces such as triangles, quadrilaterals, or other n-sided polygons. The number of vertices, and therefore faces, of a polygonal mesh (308) can be configurable based on resolution and processing speed. For example, a polygonal mesh (308) may have 50,000 vertices or 500,000 vertices.

[0048] A UV feature map (310) is a UV map having neural features in three dimensions. The term UV map corresponds to the standard definition used in the art. A UV map maps the positions of a three-dimensional object onto a two-dimensional plane. Thus, each position on the two-dimensional plane has a corresponding position on the three-dimensional object. Similarly, each position on the three-dimensional object has a corresponding position on the two-dimensional plane. Positions on a UV map can be defined by horizontal and vertical values. The horizontal value can be called the U-axis, and the vertical value can be called the V-axis. Thus, positions u,v, where u ε U and v ε V are mapped to specific positions on the three-dimensional object.

[0049] The UV feature map (310) is a UV map, but it has a third dimension, which is a feature vector. The feature vector learns neural features, which are the features of the appearance of the corresponding object at the positions corresponding to positions u,v. In other words, the third dimension is a feature vector that has neural features about the positions defined by the first two dimensions. The neural features are learned from real sensor data (128) and therefore do not include direct attributes such as color and luminosity, but rather may include encoded features learned by machine learning.

[0050] In one or more embodiments, at least a portion of the virtual world is represented by a single UV feature map (310) and a single polygonal mesh (308). For example, the single UV feature map and the single polygonal mesh may have all or some of the stationary objects within the geographical region of the virtual world. In some embodiments, non-stationary objects within the virtual world are represented by individual corresponding UV feature maps (310) and polygonal meshes (308). Thus, non-stationary objects can move within the virtual world independently of the geographical environment.

[0051] A background feature map (308) is one or more feature maps defined for the background of a geographical area. The background area is the area that is farther than a threshold distance from the virtual camera. Multiple background feature maps can be defined, each corresponding to a different distance range from a location within the geographical area. For example, a first background feature map may be for an object that would be 400 to 600 meters away from the virtual world camera if the virtual world were the real world, a second background feature map may be for an object 600 to 2000 meters away, and a third background feature map may be for an object such as the sky that is more than 2000 meters away. The scope in the embodiments is for illustrative purposes only and is not intended to limit the scope of the claims.

[0052] A background feature map may be a set of neural skyboxes or skydomes that project the background onto a cube. The use of the terms skybox or skydome corresponds to the standard definition used in the field of computer graphics. However, in one or more embodiments, the values ​​at a particular location, rather than color values, are feature vectors. The feature vectors have neural features that may be similar to the feature vectors of a UV feature map. For example, the neural features of a background feature map (308) may be learned from real-world sensor data (128) and therefore may not include direct attributes such as color or luminosity, but rather may include encoded features learned by machine learning. In one or more embodiments, a neural skybox or skydome represents a scene as a set of cuboids, spheres, or other three-dimensional layers that may be defined on a two-dimensional plane. In the case of a cuboid layer, each layer may have six feature maps, each feature map corresponding to a plane of the cuboid. Each layer may be a separate feature map.

[0053] In some embodiments, a code map (312) is used. In such embodiments, the UV feature map (310) or background feature map (308) stores codes that map to feature vectors, rather than directly storing feature vectors. In such embodiments, the third dimension of the UV feature map is the codes that map to neural features. The code map (312) maintains the mapping between codes and feature vectors. Thus, multiple locations in the UV feature map can use the same stored feature vectors, and therefore the size of the UV feature map can be reduced. In one or more embodiments, the code map (312) is learned by a machine learning process. The code map (312) is a quantization of the actual UV feature map. Specifically, if multiple feature vectors are close but not identical, they can use the same code, and a single feature vector can be stored for multiple feature vectors. Thus, the size of the code map can be further reduced.

[0054] In one or more embodiments, separate code maps may exist for background feature maps, either similarly or alternatively. In such scenarios, the code maps may be specifically trained and generated for one or more of the background feature maps. For example, each background feature map may have a separate code map, or a set of background feature maps may share a code map. In other embodiments, the code map for a background feature map may be the same as the code map for a UV feature map. Regardless of the configuration, the code maps, and therefore the background feature maps that use the code maps, operate and can be trained in the same or similar manner as the code maps for UV feature maps.

[0055] Continuing to refer to Figure 3, the model framework (304) includes a rasterization engine (314), a shading machine learning model (316), a code map training engine (318), a compositing unit (320), and a loss function (322). The rasterization engine is software configured to perform rasterization on a geographical area based on a camera. Rasterization is the process of taking a three-dimensional model (e.g., defined by a polygonal mesh) and converting the three-dimensional model into a raster image consisting of pixels. The pixels are based on the camera's position. In one or more embodiments, each pixel has a corresponding feature vector for that pixel. In one or more embodiments, the rasterization engine has several different processes. A first process can perform rasterization, and a second process can sample a UV feature map or a background feature map based on the rasterization.

[0056] A shading machine learning model (316) is a machine learning model configured to convert neural features into color and opacity values. A shading machine learning model (316) may be a lightweight machine learning model. For example, a shading machine learning model (316) may be a multilayer perceptron (MLP) model. Generally, an MLP model is a feedforward artificial neural network having at least three layers of nodes. The layers include an input layer, a hidden layer, and an output layer. Each layer has multiple nodes. Each node contains an activation function with learnable parameters. Through training and backpropagation of losses, the parameters are updated, and therefore the MLP model improves when making predictions.

[0057] A shading model may include multiple machine learning models (e.g., MLPs). Each of these machine learning models may be an independent MLP. For example, a shading machine learning model may include multiple individual machine learning models, each machine learning model for at least one of several background feature maps, UV feature maps, and object feature maps. In this example, a UV feature map may have an independent machine learning model separate from the machine learning models for one or more of the background feature maps. As another example, a background feature map may have an independent machine learning model. As yet another example, each non-static object may have an independent machine learning model for that object. Each machine learning model may be independent of other machine learning models in that the machine learning model may be trained at least partially independently. Furthermore, each machine learning model may be distinct in that at least one or more layers do not overlap with the layers of other models.

[0058] The code map training engine (318) is configured to train the code map (312). Specifically, the code map training engine (318) is configured to train which feature vectors should be mapped to which codes in the code map.

[0059] The compositing unit (320) is configured to composite images based on the output of a shading machine learning model. Image compositing combines different colors into a single image based on the opacity of each part of the image. For example, compositing combines a specified foreground with a background and any non-static objects by processing a UV map.

[0060] The loss function (322) is a function used to calculate the loss of the system. The loss function (322) uses the various outputs of the model framework (304) to calculate the loss used to update the model framework (304) by backpropagation. During backpropagation, one or more layers can be frozen to calculate the losses of the other layers.

[0061] Figures 4 to 7 show flowcharts for one or more embodiments. Figure 5 shows a flowchart for generating a rendered image, and Figure 6 shows a flowchart for using a code map. Figure 6 shows a flowchart for rendering a background and compositing the rendered image. The various steps in these flowcharts are presented and explained in order, but at least some steps may be performed in a different order, combined, or omitted, and at least some steps may be performed in parallel. Furthermore, the steps may be performed actively or passively.

[0062] In block 402, the camera position is identified. In the virtual world, the player can move around. The player, in the case of a human, may be called the ego system, autonomous system, or viewpoint of the display. Therefore, at any given time, the player has a corresponding position in the virtual world. The player has a corresponding position for at least one virtual camera. In the case of virtual reality, the corresponding position of the virtual camera is the position of the player's eyes in the virtual world. In the case of an autonomous system, the corresponding position of the camera is based on the camera's position relative to the autonomous system. Therefore, the camera position in the virtual world can be identified. In some embodiments, there may be multiple virtual cameras. If there are multiple virtual cameras, the process in Figures 4 to 6 is repeated for each camera.

[0063] In block 404, the UV feature map is rasterized using the camera position and polygonal mesh to obtain a feature buffer. The rasterization process obtains the u,v coordinates in the UV feature map for each pixel of the rendered image. The UV feature map is sampled to obtain feature vectors in UV coordinates. If a code map is used, sampling may be performed as described in Figure 5. The process is repeated for each pixel in the rendered image to obtain a feature buffer with feature vectors. Each feature vector is associated with the corresponding pixel in the rendered image. The rasterization engine can also generate an opacity mask from the polygonal mesh. The opacity mask indicates, pixel by pixel, whether the pixel is covered by the polygonal mesh.

[0064] In some embodiments, an opacity mask may be used in the synthesis stage to combine images. In other embodiments, an opacity mask can be used to determine whether to obtain feature vectors from a UV feature map.

[0065] In block 406, the shading machine learning model processes a feature buffer using the camera's view direction to generate a first set of opacity and color values. In one or more embodiments, the shading machine learning model processes a feature vector independently for each pixel. For each feature vector, the view direction at the pixel is determined and combined with the feature vector to generate a combined feature vector. For example, the combination may be concatenated. Other combinations may be used without departing from the claims. In one or more embodiments, the view direction indicates the angle of the camera origin to a particular pixel. The shading machine learning model then processes the combined feature vector. For example, the combined feature vector may be processed through one or more layers of a neural network to generate color and opacity values ​​for the pixel. This process is repeated for each pixel to generate an image rendering.

[0066] In block 408, a rendering image is generated using a first set of opacity and color values. In some embodiments, the image rendering is a rendering image that can be used directly. In such scenarios, the rendering image is output from the rendering system.

[0067] The process shown in Figure 4 may be performed simultaneously for multiple cameras. Each camera may have a corresponding view direction and camera position. When performing the process, the same polygonal mesh and UV feature map can be used for multiple cameras. Therefore, multiple rendered images can be generated and output simultaneously.

[0068] Figure 5 shows a flowchart for using a code map. In block 502, locations within the UV feature map are identified. Locations are obtained as described in block 404. In block 504, codes are obtained from the locations within the UV feature map. Instead of directly obtaining feature vectors from the UV feature map, codes are obtained from the UV feature map. The codes are used as a lookup to the code map. In block 506, a set of features mapped to codes is obtained from the code map. In one or more embodiments, the set of features is a feature vector mapped to the code map. The set of features can be used to perform the processes described above, referring to blocks 404 and 406 in Figure 4. The UV feature map is smaller because the size of the codes is smaller than the size of the feature vectors. The process in Figure 5 is repeated for each location in the UV feature map identified by rasterization.

[0069] Figure 6 shows a flowchart for processing the background. As mentioned above, the background may undergo different processing than the foreground to save processing time. In other embodiments, polygonal meshes and UV feature maps may be used to generate the background, as described in Figure 4.

[0070] In block 602, camera locations within the geographical area are identified. Camera location identification can be performed as described above, with reference to block 402 in Figure 4.

[0071] In block 604, the background feature map is rasterized using the camera position and a background feature buffer is obtained. For each pixel of the image, a camera ray is projected from the camera to a position in the background feature map. The intersection position with each background feature map is determined to identify the position within each background feature map. A feature set for each position within the background feature map is obtained and added to the corresponding feature buffer. Thus, in one or more embodiments, each background feature buffer has a feature set associated with each position.

[0072] In block 606, the shading machine learning model processes a background feature buffer to generate one or more background renderings. The processing is similar to that in block 406 of Figure 4. In one or more embodiments, the processing in block 606 is performed independently for each background pixel or each pixel in the image, for each feature set. In one or more embodiments, the feature set is processed through one or more layers of the neural network model of the shading machine learning model. In some embodiments, the view direction is appended to the feature vector. In other embodiments, the view direction is not used when the shading machine learning model processes the feature set.

[0073] A shading machine learning model can include multiple machine learning models. A separate machine learning model may exist for the foreground, as well as for the background. Furthermore, a separate machine learning model may exist for each background layer. Therefore, processing the background feature buffer can involve processing the feature set of each pixel in the background layer using a trained neural network layer of the corresponding machine learning model to obtain the opacity and color values ​​of the pixels in the layer.

[0074] The process in Figure 6 can be extended to include non-stationary objects in a virtual world. As described above, non-stationary objects may have trained corresponding feature maps and trained polygonal meshes (i.e., object polygonal meshes). Thus, the location of the non-stationary object in the virtual world is identified. Based on the camera's position relative to the non-stationary object and polygonal mesh, the object feature map is rasterized and an object feature buffer is obtained. The shading machine learning model then processes the object feature buffer to obtain color and opacity values ​​for the corresponding locations on the object in the image. This process can be repeated for each non-stationary object in the image.

[0075] In block 608, one or more background renderings are combined with a first set of opacity and color values ​​generated in block 406 of Figure 4 to produce a rendered image. After processing in block 606 and block 406 of Figure 4, each pixel may have one or more pairs of opacity and color values ​​defined for that pixel. If multiple pairs of opacity and color values ​​are defined for the same pixel, the relative distance to the camera based on the layer in which the pixel is located is used to order the pairs from the pair closest to the camera to the pair furthest from the camera. For example, pairs for the foreground are closer than pairs of background pixels ordered according to the layer. The pairs are processed in order. For each pair defined for a pixel, the opacity value is used to determine the amount of color used for the pixel by comparing it to the amount of color from the furthest pair.

[0076] In other words, at the processing stage of block 608, the same pixel in the image may have pairs of foreground color and opacity values, pairs of background color and opacity values, and pairs of color and opacity values ​​for 0, 1, or more objects. The pairs of values ​​for a pixel are ordered according to their distance from the camera. The opacity value of each current pair is then used in order of distance to calculate the amount of color value from more distant pairs compared to the color value of the current pair. For example, an opacity value of 0 for the nearest pair indicates that the color value of the nearest pair will be used. In another example, an opacity value of 50 percent indicates that 50 percent of the color value from the current pair will be combined with 50 percent from a more distant pair. The combination may also be a weighted combination of the current pair and a more distant pair, with the weights determined by the opacity value of the current pair. Furthermore, the combination may be performed independently for the red, green, and blue (RGB: Red, Green, Blue) portions of the color values. Although RGB has been described, other color channels of other color models may be used with the same technique.

[0077] Figure 7 shows a flowchart for training a system according to one or more embodiments. In block 702, a model is used to generate a rendered image. In block 704, a UV feature map, a background feature map, a code map, and a shading model are trained. A stored camera image is compared to the rendered image to generate a loss. Multiple comparisons may be performed, and the loss may be a composite loss. The first comparison may be a direct comparison of the color values ​​of each pixel. Specifically, for each pixel, the difference between the color value of the rendered image and the color value of the real camera image is calculated. The differences are combined across different pixels to obtain the color loss. A second loss may be a perceived loss. A perceived loss is a loss determined from the entire image. In the case of a perceived loss, the trained machine learning model generates a first value from the rendered image and a second value from the real camera image. The difference between the first and second values ​​is the perceived loss. A third loss may be based on vector quantization. That is, the third loss may be based on the amount of quantization of the code map. The vector quantization loss can be calculated as the difference between the original feature and the nearest code in the code map.

[0078] A composite loss can be generated by combining various loss methods, such as performing a weighted average. This composite loss can then be backpropagated through the network. With respect to the UV feature map, the loss updates the neural features; that is, the system learns neural network features that are referenced by the code or directly referenced in the UV feature map.

[0079] In block 706, a decision is made as to whether to continue training. If training is to continue, the flow proceeds to block 702.

[0080] Figures 8 and 9 illustrate examples of one or more embodiments. The description of Figures 8 and 9 is illustrative and not intended to limit the scope of the claims.

[0081] One or more embodiments aim to perform real-time rendering of large scenes. Given a set of presented images and a reconstructed mesh of moderate quality, our method generates a scene mesh with neural texture maps and view-dependent fragment shaders. Using the initial mesh, one or more embodiments first learn the neural textures by generating UV parameterization. Then, one or more embodiments learn together a discrete texture feature codebook and a view-dependent lightweight MLP that can effectively represent the appearance of the scene. Finally, one or more embodiments bake the texture feature codebook and MLP into a set of neural texture maps and fragment shaders that can be executed in real time in an existing graphics pipeline. Here, one or more embodiments first introduce our method for representing large scenes (Section 3.1), then describe how one or more embodiments render and learn the scene (Sections 3.2-3.3), and finally describe how one or more embodiments export our model to a real-time graphics pipeline (Section 3.4).

[0082] Figure 8 shows an example of a rendering system for performing real-time rendering of large-scale scenes in one or more embodiments. Figure 8 shows an example of rendering a large-scale outdoor scene. Hybrid techniques can be used to handle a potentially infinite depth range (e.g., sky, vegetation, mountains, etc.) as well as the nearby area. The entire 3D scene can be divided into two regions: an inner cuboid region (foreground) modeled by a polygonal mesh texturized with neural features, and an outer cuboid region (background) modeled by a neural skybox. Such a hybrid scene representation allows for modeling with fine detail in both near and far areas and enables rendering with significant camera movement.

[0083] For the foreground region, one or more embodiments can leverage an explicit geometric mesh scaffold to learn and render neural textures. Various sources of existing polygonal meshes can be used. For example, one or more embodiments can leverage existing neural reconstruction methods or other methods. Initially, the reconstructed mesh may have tens of millions or more triangular faces that well represent the geometric shape, but may have self-intersecting and overlapping vertices. One or more embodiments can preprocess the resulting polygonal mesh to reduce computational cost and improve UV mapping quality. One or more embodiments may first cluster nearby vertices together, perform secondary mesh decimation to simplify the mesh while maintaining structure, and then perform face culling to remove invisible triangular faces (e.g., source camera views). Finally, the mesh can be unfolded using a UV map generation tool to obtain a UV mapping for each vertex of the polygonal mesh.

[0084]

number

[0085] Regarding the background, in this example, modeling distant background regions with a polygonal mesh may present challenges due to the complexity and scale of those regions. Alternatively, one or more embodiments may use the concepts of multiplanar and multispherical images to represent background regions using a neural skybox. A neural skybox is an example of a background feature map.

[0086]

number

[0087] Specifically, referring to the example in Figure 8, the rendering pipeline is shown. In summary, in Figure 8, the foreground mesh and neural skybox are rasterized with a neural texture map to the desired viewpoint, generating a set of image feature buffers. The feature buffers are then processed by an MLP to generate a set of rendering layers that are synthesized to combine the final RGB image. Next, we will explain the process in Figure 8.

[0088]

number

[0089]

number

[0090]

number

[0091]

number

[0092]

number

[0093]

number

[0094]

number

[0095]

number

[0096]

number

[0097]

number

[0098]

number

[0099]

number

[0100]

number

[0101]

number

[0102]

number

[0103]

number

[0104] Here, each loss term is explained in more detail below.

[0105]

number

[0106]

number

[0107]

number

[0108] Perceptual loss can be calculated as follows. One or more embodiments use additional perceptual loss to improve the quality of the rendered image. Perceptual loss measures "perceptual similarity" that better matches human visual perception.

[0109]

number

[0110]

number

[0111]

number

[0112]

number

[0113]

number

[0114] Figure 9 shows an example of a vector quantization process for generating a code map (902) and a quantized UV feature map (904) according to one or more embodiments. As shown in Figure 9, the original UV feature map (906) having continuous feature values ​​T(u,v) at positions u,v is processed via a codebook (i.e., code map) function to generate a code map (902) and a quantized feature map (904).

[0115]

number

[0116] As illustrated, one or more embodiments can use a scaffold mesh as input, incorporate a neural texture field to model view-dependent effects, and then export this for real-time rendering with a standard rasterization engine. One or more embodiments can render urban driving scenes at over 100 FPS at a resolution of 1920 x 1080 while providing photorealism comparable to existing neural rendering techniques. Thus, one or more embodiments can be used for scalable and immersive experiences for autonomous driving simulations and virtual reality applications.

[0117] Embodiments can be implemented on a computing system specifically designed to achieve improved technical results. When implemented in a computing system, the features and elements of the Disclosure provide a significant technical advancement compared to a computing system that does not implement the features and elements of the Disclosure. Any combination of mobile, desktop, server, router, switch, embedded device, or other types of hardware can be improved by including the features and elements described in the Disclosure. For example, as shown in Figure 10A, a computing system (1000) may include one or more computer processors (1002), non-persistent storage (1004), persistent storage (1006), communication interfaces (1012) (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and a number of other elements and functions that implement the features and elements of the Disclosure. The computer processor (1002) may be an integrated circuit for processing instructions. The computer processor may be one or more cores or microcores of a processor. The computer processor (1002) includes one or more processors. One or more processors can include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), or a combination of these.

[0118] The input device (1010) may include a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device. The input device (1010) may receive input from a user in response to data and messages presented by the output device (1008). The input may include text input, audio input, video input, etc., which may be processed and transmitted by the computing system (1000) according to this disclosure. The communication interface (1012) may include an integrated circuit for connecting the computing system (1000) to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or any other type of network) and / or another device such as another computing device.

[0119] Furthermore, the output device (1008) may include a display device, a printer, an external storage device, or any other output device. One or more of the output devices may be the same as or different from the input devices. The input and output devices may be connected locally to the computer processor (1002) or remotely. Many different types of computing systems exist, and the aforementioned input and output devices may take other forms. The output device (1008) can display data and messages transmitted and received by the computing system (1000). The data and messages may include text, audio, video, etc., and may include the data and messages described in other figures of this disclosure.

[0120] Software instructions in the form of computer-readable program code for executing embodiments may be stored, in whole or in part, temporarily or permanently on a non-temporary computer-readable medium such as a CD, DVD, storage device, diskette, tape, flash memory, physical memory, or any other computer-readable storage medium. Specifically, software instructions may correspond to computer-readable program code configured to execute one or more embodiments that, when executed by a processor, may include sending, receiving, presenting, and displaying data and messages as shown in other figures of this disclosure.

[0121] The computing system (1000) in Figure 10A may be connected to a network or be part of a network. For example, as shown in Figure 10B, the network (1020) may include multiple nodes (e.g., node X (1022), node Y (1024)). Each node may correspond to a computing system such as the computing system shown in Figure 10A, or a group of combined nodes may correspond to the computing system shown in Figure 10A. As an example, the embodiment may be implemented on a node of a distributed system connected to other nodes. As another example, the embodiment may be implemented on a distributed computing system having multiple nodes, where each part may be located on a different node within the distributed computing system. Furthermore, one or more elements of the aforementioned computing system (1000) may be located remotely and connected to other elements via a network.

[0122] Nodes within the network (1020) (e.g., node X (1022), node Y (1024)) may be configured to provide services for the client device (1026), such as receiving requests and sending responses to the client device (1026). For example, the nodes may be part of a cloud computing system. The client device (1026) may be a computing system, such as the computing system shown in Figure 10A. Furthermore, the client device (1026) may include, and / or execute, all or part of one or more embodiments.

[0123] The computing system in Figure 10A may include the ability to present raw and / or processed data, such as the results of comparisons and other processing. For example, data presentation can be achieved by various presentation methods. Specifically, data may be presented by being displayed on a user interface, transmitted to different computing systems, and stored. The user interface may include a GUI that displays information on a display device. The GUI may include various GUI widgets that organize what data is presented and how the data is presented to the user. Furthermore, the GUI may present the data directly to the user, for example, by presenting the data as actual data values ​​via text, or by rendering the data into a visual representation by a computing device, such as through the visualization of a data model.

[0124] As used herein, the term "connected to..." has multiple meanings. The connection may be direct or indirect (e.g., via another component or network). The connection may be wired or wireless. The connection may be a temporary, permanent, or semi-permanent communication channel between two entities.

[0125] The various descriptions in the figures may be combined, include features described in other figures of this application, or be contained within features described in other figures of this application. The various elements, systems, components, and steps shown in the figures may be omitted, repeated, combined, and / or modified as shown in the figures. Therefore, the scope of this disclosure should not be considered to be limited to any particular configuration shown in the figures.

[0126] In this application, ordinal numbers (e.g., first, second, third, etc.) may be used as adjectives for elements (i.e., any noun in this application). The use of ordinal numbers does not imply or create any particular order of elements, nor does it limit any element to only a single element, unless expressly disclosed by the use of terms such as “first,” “after,” “single,” etc., or other such terms. Rather, the use of ordinal numbers is for distinguishing between elements. For example, the first element is distinct from the second element, and the first element may encompass two or more elements, and may follow (or precede) the second element in the order of elements.

[0127] Furthermore, unless otherwise specified, "or" is an "inclusive OR" and therefore includes "and". Moreover, unless otherwise specified, items connected by "or" can include any combination of those items, and the number of each item is arbitrary.

[0128] In the above description, numerous specific details have been included to provide a more complete understanding of this disclosure. However, it will be apparent to those skilled in the art that the art can be carried out without these specific details. In other cases, well-known features have not been described in detail to avoid unnecessarily complicating the description. Furthermore, it is possible to devise other embodiments that are not expressly described above but do not deviate from the claims disclosed herein. Therefore, the scope should be limited only by the appended claims.

Claims

1. Identifying the first camera position of the first camera within a geographical area, Using the first camera position and polygonal mesh, the UV feature map is rasterized and a first feature buffer is obtained. The shading machine learning model processes the first feature buffer using the first view direction of the first camera to generate a first image rendering including a first set of opacity values ​​and color values. To generate a first rendering image from the first set of opacity values ​​and color values, A method performed by a computer, including [this].

2. Using the aforementioned first camera position, the background feature map is rasterized and a background feature buffer is obtained. The shading machine learning model processes the background feature buffer and generates a background rendering that includes a second set of opacity and color values. The first set of opacity values ​​and color values ​​is combined with the second set of opacity values ​​and color values, The method performed by a computer according to claim 1, further comprising:

3. The computer-based method according to claim 2, wherein the shading machine learning model comprises a first machine learning model for generating the image rendering and a second machine learning model for generating the background rendering, separate from the first machine learning model.

4. The process includes rasterizing a plurality of background feature maps and obtaining a plurality of background feature buffers using the first camera position, each of which has a distance from the first camera position. The shading machine learning model processes the multiple background feature buffers and generates multiple background renderings. The aforementioned image rendering is combined with the aforementioned multiple background renderings, The method performed by a computer according to claim 1, further comprising:

5. The computer-based method according to claim 4, wherein the shading machine learning model comprises a plurality of separate machine learning models for specific background features of the plurality of background feature maps and for at least one of the UV feature maps.

6. Using the first camera position and object polygon mesh described above, the object feature map is rasterized and an object feature buffer is obtained. The shading machine learning model processes the object feature buffer and generates an object rendering. The aforementioned image rendering is combined with the aforementioned object rendering, The method performed by a computer according to claim 1, further comprising:

7. The process includes rasterizing a plurality of background feature maps and obtaining a plurality of background feature buffers using the first camera position, each of which has a distance from the first camera position. The shading machine learning model processes the multiple background feature buffers and generates multiple background renderings. Using the first camera position and object polygon mesh, the object feature map of the object is rasterized and an object feature buffer is obtained, wherein the object is moving within the geographical area. The shading machine learning model processes the object feature buffer and generates an object rendering. The image rendering is combined with the object rendering and the multiple background renderings, It further includes, The computer-based method according to claim 1, wherein the shading machine learning model comprises a plurality of separate machine learning models for at least one of the plurality of background feature maps, the UV feature maps, and the object feature maps.

8. Rasterizing the aforementioned UV feature map is Identifying the position within the aforementioned UV feature map, Obtaining a code from the aforementioned location in the UV feature map, Obtaining a set of features mapped to the aforementioned code from the code map, A computer-based method according to claim 1, including the method described in claim 1.

9. The aforementioned UV feature map is trained to learn multiple neural network features from camera images. The method performed by a computer according to claim 1, further comprising:

10. Training the UV feature map, code map, and shading machine learning model using multiple camera images, The method performed by a computer according to claim 1, further comprising:

11. Identifying the second camera position of the second camera within the aforementioned geographical area, Using the second camera position and the polygonal mesh, the UV feature map is rasterized to obtain a second feature buffer, The shading machine learning model processes the second feature buffer using the second view direction of the second camera to generate a second image rendering including a second set of opacity and color values. A second rendering image is generated from the second set of opacity values ​​and color values, Outputting the first rendering image and the second rendering image simultaneously, The method performed by a computer according to claim 1, further comprising:

12. Memory and The system comprises a computer processor containing computer-readable program code for performing an operation, and the operation is: Identifying the camera location of cameras within a geographical area, Using the aforementioned camera position and polygonal mesh, the UV feature map is rasterized to obtain a first feature buffer. The shading machine learning model processes the first feature buffer using the camera's view direction to generate an image rendering that includes a first set of opacity and color values. A rendering image is generated from the first set of opacity and color values, A system that includes this.

13. The aforementioned operation is, Using the aforementioned camera position, the background feature map is rasterized and a background feature buffer is obtained. The shading machine learning model processes the background feature buffer and generates a background rendering that includes a second set of opacity and color values. The first set of opacity values ​​and color values ​​is combined with the second set of opacity values ​​and color values, The system according to claim 12, further comprising:

14. The system according to claim 13, wherein the shading machine learning model comprises a first machine learning model for generating the image rendering and a second machine learning model for generating the background rendering, which is separate from the first machine learning model.

15. The aforementioned operation is, The process includes rasterizing multiple background feature maps and obtaining multiple background feature buffers using the aforementioned camera position, wherein each of the multiple background feature maps has a respective distance from the camera position. The shading machine learning model processes the multiple background feature buffers and generates multiple background renderings. The aforementioned image rendering is combined with the aforementioned multiple background renderings, The system according to claim 12, further comprising:

16. The system according to claim 15, wherein the shading machine learning model comprises a plurality of separate machine learning models for specific background features of the plurality of background feature maps and for at least one of the UV feature maps.

17. The aforementioned operation is, Using the aforementioned camera position and object polygon mesh, the object feature map is rasterized and an object feature buffer is obtained. The shading machine learning model processes the object feature buffer and generates an object rendering. The aforementioned image rendering is combined with the aforementioned object rendering, The system according to claim 12, further comprising:

18. The aforementioned operation is, The process includes rasterizing multiple background feature maps and obtaining multiple background feature buffers using the aforementioned camera position, wherein each of the multiple background feature maps has a respective distance from the camera position. The shading machine learning model processes the multiple background feature buffers and generates multiple background renderings. This includes rasterizing the object feature map of the object and obtaining an object feature buffer using the camera position and object polygon mesh, wherein the object is moving within the geographical area. The shading machine learning model processes the object feature buffer and generates an object rendering. The image rendering is combined with the object rendering and the multiple background renderings, It further includes, The system according to claim 12, wherein the shading machine learning model comprises a plurality of separate machine learning models for at least one of the plurality of background feature maps, the UV feature maps, and the object feature maps.

19. Identifying the camera location of cameras within a geographical area, Using the aforementioned camera position and polygonal mesh, the UV feature map is rasterized to obtain a first feature buffer. The shading machine learning model processes the first feature buffer using the camera's view direction to generate an image rendering that includes a first set of opacity and color values. A rendering image is generated from the first set of opacity and color values, A non-temporary computer-readable medium containing computer-readable program code for performing operations including those described above.

20. The aforementioned operation is, The process includes rasterizing multiple background feature maps and obtaining multiple background feature buffers using the aforementioned camera position, wherein each of the multiple background feature maps has a respective distance from the camera position. The shading machine learning model processes the multiple background feature buffers and generates multiple background renderings. This includes rasterizing the object feature map of the object and obtaining an object feature buffer using the camera position and object polygon mesh, wherein the object is moving within the geographical area. The shading machine learning model processes the object feature buffer and generates an object rendering. The image rendering is combined with the object rendering and the multiple background renderings, It further includes, The non-temporary computer-readable medium according to claim 19, wherein the shading machine learning model comprises a plurality of separate machine learning models for at least one of the plurality of background feature maps, the UV feature maps, and the object feature maps.

Images