Generative AI model for image rendering and reverse rendering

Abstract

The invention relates to a generative AI model for image rendering and reverse rendering. Embodiments of the present disclosure relate to rendering and reverse rendering using one or more generative models. 'Rendering' refers to a process of generating a final visual image or animation from a 2D or 3D model. "Reverse rendering" refers to a process of inferring or estimating properties of a scene (e.g., texture or other properties, such as geometry, illumination, and texture) from observed images or visual data. Essentially, it aims to reverse the traditional rendering process. Aspects of the present disclosure introduce editable light and texture controls into a generative model to support artistic authoring. Various embodiments integrate a generative model as a renderer of a classical rendering pipeline to upgrade the style of retrofit and enhance rendered content.

Description

BACKGROUND

[0001] Generative models represent a frontier of artificial intelligence in image processing and machine learning. For example, video generative models aim to generate realistic and coherent video frames from various inputs, such as static images or other video frames. For example, some video generative models are capable of producing highly realistic animations from input descriptions.

[0002] However, these generative models and other image processing techniques face technical challenges in identity preservation (e.g., maintaining consistent and recognizable features of an object or character across multiple video frames) and providing precise user control over attributes such as lighting, material properties, and scene layout. These limitations and others hinder their ability to fully replicate and leverage useful functionality used in classical graphics rendering workflows. SUMMARY

[0003] Embodiments of the present disclosure relate to rendering and inverse rendering using one or more generative models (e.g., diffusion models (DMs)). “Rendering” refers to the process of generating a final visual image or animation of a scene, which can include considering object geometry and other visual properties. “Inverse rendering” refers to the process of inferring or estimating scene properties (e.g., material maps or other properties such as geometry, lighting, and textures) from observed images. In essence, it aims to reverse the traditional rendering process. Various aspects of the present disclosure introduce editable light and material controls into generative models to support artistic authoring. Various embodiments integrate generative models as renderers to upgrade and enhance the style of graphically rendered content.

[0004] Some embodiments specifically relate to a diffusion-based renderer (e.g., DM) that uses specific inputs (e.g., material maps, noise vectors, lighting maps, and natural language textual descriptions) to render one or more frames and allows for lighting control, relighting, and image enhancement. A material map of a scene defines how one or more properties vary or appear on the surface of one or more objects in that scene. Thus, a material map can define surface properties, including color (albedo), surface detail (normals), reflectivity (metallic), roughness, and / or ambient occlusion. A lighting map represents shading and / or lighting characteristics related to one or more objects. It captures how light interacts with object surfaces, including effects such as shadows, highlights, and / or overall illumination. To generate such maps, some embodiments first receive user input requesting that material properties and / or lighting conditions be incorporated into output frames.

[0005] Subsequently, some embodiments provide, as input to a machine learning model (e.g., a diffusion model), a first noise vector along with a representation (e.g., a pre-processed version) of the texture map and / or the lighting map to generate an output frame that serves as a final rendered frame. The first noise vector corresponds to an initial starting point for a diffusion process performed by the machine learning model.

[0006] Some embodiments additionally or alternatively perform static image and / or video inverse rendering using a machine learning model (e.g., a generative model). In an example instance of inverse rendering, some embodiments first receive an input frame (e.g., a particular video frame). Subsequently, some embodiments provide, as input to a machine learning model, a first noise vector and a representation of the input frame to generate a texture map. For example, in the context of a diffusion model, this noise vector serves as an initial input from which the model iteratively refines its output to generate the texture map. BRIEF DESCRIPTION OF DRAWINGS

[0007] The present system and method for subcutaneous authentication will be described in detail below with reference to the attached drawings, in which:

[0008] Figure 1 is a block diagram of a rendering / inverse rendering system in accordance with some embodiments of the present disclosure;

[0009] Figure 2 shows a pipeline of a rendering process in accordance with some embodiments of the present disclosure, in which a diffusion model (DM) generates a rendered image based on processing a particular input;

[0010] Figure 3 shows a pipeline of an inverse rendering process in accordance with some embodiments of the present disclosure, in which a DM generates one or more texture maps as output based on a particular input;

[0011] Figure 4 shows a pipeline of an inverse rendering process in accordance with some embodiments of the present disclosure, in which a transformer generates an environment map as output based on a particular input;

[0012] Figure 5 shows a pipeline for performing 2D frame relighting by intrinsic decomposition and neural rendering in accordance with some embodiments of the present disclosure;

[0013] Figure 6 shows a pipeline for performing 3D frame relighting by intrinsic decomposition and neural rendering in accordance with some embodiments of the present disclosure;

[0014] Figure 7 shows a pipeline for providing identity-preserving image enhancement (e.g., object insertion, synthesis, or style transfer) in accordance with some embodiments of the present disclosure;

[0015] Figure 8is a screenshot of an example user interface for editing an input image by inverse rendering, in accordance with some embodiments of the present disclosure;

[0016] Figure 9 is a screenshot of an example user interface for generating a video frame by rendering, in accordance with some embodiments of the present disclosure;

[0017] Figure 10 is a flowchart of an example process for training or fine-tuning a machine learning model, in accordance with some embodiments of the present disclosure;

[0018] Figure 11 is a flowchart of an example process for generating an output frame or image, in accordance with some embodiments of the present disclosure;

[0019] Figure 12A is an illustration of an example autonomous vehicle, in accordance with some embodiments of the present disclosure;

[0020] Figure 12B is an example of a camera position and field of view of an example autonomous vehicle, in accordance with some embodiments of the present disclosure; Figure 12A

[0021] Figure 12C is a block diagram of an example system architecture of an example autonomous vehicle, in accordance with some embodiments of the present disclosure; Figure 12A

[0022] Figure 12D is a system diagram for communication between a cloud-based server and an example autonomous vehicle, in accordance with some embodiments of the present disclosure; Figure 12A

[0023] Figure 13 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0024] Figure 14 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0025] ​​​As noted above, the limitations of generative models and image processing techniques generally prevent them from fully replicating the capabilities of classic graphics rendering workflows. For example, with respect to materials and surface details, classic graphics rendering includes detailed material maps (e.g., albedo, normals, roughness, metallic, ambient occlusion) to define how surfaces interact with light, providing precise control over textures and reflectivity. As a result, artists can manually adjust various aspects of these materials to achieve the desired look. However, generative models learn from a dataset and generate new images by generalizing the patterns seen in the data. They currently lack the ability to capture the precise details and variations in materials and surface details needed for photo-realism. Converting detailed material properties to the latent space of a generative model is a challenge that limits the ability to fine-tune textures and surface details and allow for any edits.

[0026] In another example, with respect to scene layout and spatial consistency, classic rendering workflows allow artists to have full control over the position and properties of objects in a scene, ensuring spatial consistency and precise interactions between objects. Classic workflows allow for precise positioning and animation of objects, which is critical for maintaining the intended composition and dynamic effects of a scene. However, for generative models, they often produce artifacts and inconsistencies, especially in complex scenes that contain multiple objects and interactions. For example, there can be rapid and noticeable changes in brightness or color between consecutive frames, resulting in flickering effects. There can also be temporal jitter, which is an unnatural jump or shift in an object’s motion in a video. These anomalies, among others, result in poor frame quality and inaccurate frame predictions for videos. Furthermore, the poor artifacts and anomalies are not limited to video generation, they can occur in any digital media format, such as digital photographs. For digital photographs, artifacts can include elastic deformation, pixel misplacement, or pixel saturation. Additionally, many generative models operate in 2D space, making it difficult to maintain consistent object relationships and perspectives across different parts of an image or between frames in a video.

[0027] In another example, with respect to temporal coherence of animations, classic graphics rendering techniques ensure that each frame is consistent with the previous frame, maintaining temporal coherence. Techniques such as keyframe animation, motion capture, and procedural animation allow for detailed and precise control over motion and interactions. However, generative models struggle to maintain temporal coherence in video generation, resulting in flickering or inconsistent appearance of objects between frames. Generative models generally struggle to handle dynamic scenes where objects move or change properties over time, as the temporal dependencies are complex and difficult to accurately model.

[0028] Regarding user control and customization, classic rendering techniques include interactive tools that give artists full control over various aspects of a scene, from lighting and materials to camera angles and object placement. This allows for highly customized and precise adjustments. However, generative models can be seen as black boxes, where fine-tuning specific properties requires adjusting a high-dimensional latent space, which is not intuitive. Current generative AI tools often lack the interactive and fine-grained control that artists are accustomed to in traditional rendering software.

[0029] Finally, regarding realistic light simulation, some classic rendering techniques simulate the complex interactions of light with materials in a physically accurate manner (e.g., through ray and path tracing), resulting in images with a photo-realistic level of detail. More advanced classic rendering techniques can account for multiple bounces of light, reflections, refractions, and shadows. However, these techniques are computationally intensive and time-consuming, often requiring powerful hardware and long rendering times, especially for high-resolution images and animations. Regarding generative models, they can be, for example, generative adversarial networks (GANs) that approximate the process of light interaction based on training data. While they can produce visually appealing images, they often lack the fine details and precise control over light transport compared to classic rendering techniques. Generative models also struggle to scale to the same level of physical accuracy provided by path and ray tracing without significant computational resources and complex training techniques.

[0030] Various aspects of the present disclosure bridge the gap between generative models and classic computer graphics by: (1) introducing editable light and material controls into generative models to enable more fine-grained and precise control over artistic creation; and (2) integrating generative models as renderers into classic software rendering pipelines to upgrade and enhance the style of rendered content. Some embodiments specifically relate to a diffusion-based renderer that uses specific inputs (e.g., material maps and natural language textual descriptions) to render one or more frames and allows for lighting control, relighting, and image enhancement.

[0031] In operation, some embodiments perform static image and / or video rendering (e.g., rendering a digital twin of a self-machine traversing an environment) using machine learning models (e.g., generative models). As used herein, “rendering” refers to the process of generating a final visual image, video, or animation from a 2D or 3D model using computer software. This process involves several steps and calculations to transform a model containing shapes, textures, lighting, and camera angles into a fully realized image. Some embodiments first receive one or more material maps (e.g., albedo, normals, roughness) and / or one or more light maps. A material map defines the way one or more properties vary or appear on the surface of one or more objects. Thus, a material map defines surface properties, including color (albedo), surface detail (normals), reflectivity (metallic), roughness, and ambient occlusion.

[0032] A light map represents shading and / or lighting characteristics related to one or more objects. Thus, a light map (or lightmap) is a data structure in computer graphics used to store precomputed lighting information for a 3D scene. It captures how light interacts with object surfaces, including effects such as shadows, highlights, and / or overall illumination. By using a light map, a rendering engine can achieve realistic lighting effects without complex real-time calculations, boosting performance, especially in static or semi-static scenes.

[0033] To generate such maps, some embodiments first receive user input requesting, specifying, or otherwise indicating material properties and / or lighting conditions to incorporate into an output frame. Some embodiments generate material maps and / or light maps based at least in part on the user input. For example, a user can first input a noisy image and specify “shiny wood floor” in natural language. In response, various embodiments generate multiple material maps that define specific aspects of the surface properties of the wood floor. For example, for a wood floor, an albedo map would contain the basic appearance of the wood without any applied lighting or shading (e.g., texture and color), showing the wood plank’s texture pattern and natural color variations. A normal map would depict the fine details of the wood grain, subtle imperfections, and slight bumps and dips on the wooden surface. It enhances the depth and texture of the wood floor. For a shiny wood floor, the roughness value of the roughness map is low, indicating that its surface is smooth and reflective. The map can still have slight variations to reflect subtle surface imperfections or differences in the wood grain. Since wood is a non-metallic material, the metallic map would appear completely black, indicating that the wood floor does not have any metallic properties.

[0034] In another example where the user provides lighting information, the user can indicate in natural language that “bright afternoon sunlight is shining in from the large window on the left side of the room.” Various embodiments can first parse the user input (e.g., through natural language processing, such as named entity recognition) to identify key elements, such as time of day: afternoon (implying warm, intense light), light source: sunlight, direction: from the left, intensity: bright, modifier: shining in through the window (implying some soft shadows). Then, some embodiments create a lighting environment through various algorithms. For example, in spherical harmonics (SH), embodiments use SH coefficients to approximate ambient lighting. Some embodiments use an environment map to create or select an environment map that matches the description of bright afternoon with sunlight. Some embodiments use light source properties to define the properties of the main light source (sunlight). In response, various embodiments subsequently generate a lighting map as follows. The directional light source models the sunlight, which involves setting the direction, intensity, and color temperature (warm afternoon light). There can also be soft shadows. Since the light is shining through the window, some embodiments add soft shadows in the lighting map to reflect the diffusion of the light through the window glass. Some embodiments add ambient lighting to model the overall brightness of the room, ensuring that areas not directly illuminated by the sunlight still get some illumination.

[0035] Then, some embodiments take the noise vector, as well as the representation of the material map and / or the lighting map (e.g., a vector), as input to a machine learning model (e.g., a diffusion model) to generate an output frame that serves as the final rendered frame. The noise vector corresponds to the initial starting point of the diffusion process performed by the machine learning model. In some embodiments, the machine learning model is a diffusion model. A diffusion model is a class of probabilistic models that map an easy-to-sample distribution (e.g., pixel white noise) to a target distribution that is difficult to sample (e.g., a clean image or video frame without noise or artifacts). The noise distribution for frame prediction can be a standard normal distribution for each pixel and RGBA channel in the predicted frame. The diffusion model is trained to progressively convert samples in the noise distribution (represented by the noise vector) to samples in the training distribution (e.g., video frames in a sequence of video frames). In one example example, the diffusion model can be trained to convert standard normal pixel noise to multiple video frames in a sequence of video frames.

[0036] A diffusion model typically performs the diffusion process by progressively converting the noise distribution or noise vector to the target distribution or frame through multiple steps, where the state of all previous steps is encoded in a representation of the same dimensionality as the noise and image. The diffusion model can use one or more steps (e.g., 5 or more steps) in this diffusion process. Each step of the diffusion process progressively converts the “noisy” representation of the image (initially, the input is only noise) to a slightly “less noisy” representation, so that at the last step of the process, we get a sample of the clean image.

[0037] In practice, the diffusion model can be conditioned by changing the“prompt” of the target distribution of the noise-to-image process. The diffusion model generates frames (e.g., images) by iteratively denoising a noisy input frame, gradually refining it to generate a clean frame. The conditioning mechanism changes this denoising process to guide the model to generate frames that meet the specific criteria provided by the prompt. In some embodiments, such specific criteria or conditioning information includes material maps, lighting maps, and / or user input. Cross-attention layers are integrated into the model in order to incorporate conditioning information (e.g., material maps, lighting maps, user prompts) at multiple stages of the denoising process. By using cross-attention mechanisms, the diffusion model can effectively integrate and condition on material maps, lighting maps, and user input, as described in more detail below. This enables the model to generate high-quality frames that conform to specific user-defined criteria, combining the strengths of neural networks with the properties of traditional computer graphics to achieve photorealistic rendering.

[0038] The noise vector is combined with representations of the material map and the lighting map. This combination can be achieved through concatenation or other mathematical operations that integrate the noise with the scene properties. In some embodiments, the combined input (two or more of the noise vector, material map, and lighting map, etc.) is fed into the diffusion model. This generative model then iteratively refines the noisy input to generate a final output frame. To do this, in some embodiments, the diffusion model starts with an initial noise vector. This noisy input is gradually refined through several iterations. At one or more (e.g., each) iteration, the diffusion model performs the following: it receives the current noisy representation, which includes integrated information from the material map and the lighting map. Then, the diffusion model applies a denoising step using a neural network that is trained to reduce noise and enhance details based on material properties and lighting properties. It then generates an intermediate output that is less noisy and more accurate than the previous iteration. This iterative process continues for a set number of steps, with each iteration improving the quality and accuracy of the output. After the last iteration, the diffusion model will produce a high-quality output frame that serves as the final rendered frame. This output frame incorporates material properties and lighting conditions, creating a realistic and detailed image.

[0039] In some embodiments, the diffusion model is trained on a dataset of rendered frames and their corresponding material maps and lighting maps. During training, the model learns to predict a final rendered frame by iteratively refining a noisy input using scene properties. During actual use (inference), the model receives a noise vector along with representations of the material map and the lighting map, processes it through iterative denoising steps, and outputs a final rendered frame.

[0040] In some embodiments, these final rendered frames (and / or other objects, such as material maps) are editable. Thus, particular embodiments will edit or otherwise modify one or more features based on performing user requests. User editing is more convenient and closer to classic computer graphics workflows, thanks in large part to the use of material maps and / or the structured, modular rendering approach. By using these material maps, users can adjust different properties of a scene independently without affecting others. Each material map represents a particular aspect of surface appearance, making it easier for users to understand and edit the properties they want to change. For example, if a user wants to lower the reflectivity of a surface, they can directly edit the specular map without having to change the color or texture. Furthermore, the iterative process of the diffusion model allows users to see incremental improvements and changes in real-time or near real-time. This feedback loop helps make fine adjustments and efficiently achieve the desired visual result.

[0041] Some embodiments additionally or alternatively perform inverse rendering of static images and / or videos using machine learning models (e.g., generative models). As used herein, “inverse rendering” refers to the process of inferring or estimating scene properties (e.g., material maps or other properties such as geometry, lighting, and textures) from observed images or visual data. In essence, it aims to reverse the traditional rendering process. While traditional rendering generates images from 3D models and scene descriptions, inverse rendering is used to deconstruct scene images into representations of scene properties.

[0042] In one illustrative example of inverse rendering, some embodiments first receive an input frame (e.g., a particular video frame). Then, some embodiments provide a first noise vector and a representation of the input frame as inputs to a machine learning model to generate a material map. For example, in the context of a diffusion model, this noise vector is used as an initial input, and the model will iteratively refine its output from this initial input to generate the material map. In one illustrative example, the input frame is first passed through a feature extractor, such as a convolutional neural network (CNN). The feature extractor identifies important features from the image, such as edges, textures, and color distributions.

[0043] The first noise vector is combined with the representation (e.g., features) extracted from the input frame. This combination can be done by concatenation or other methods, such as adding or multiplying the noise vector with the image features. The combined input (noise vector + image representation) is fed into the diffusion model. At each step, the model uses the combined input to gradually reduce the noise and refine its estimate of the material map.

[0044] The diffusion model starts with an initial noise vector. This vector is a random, noisy representation that will be refined over several iterations. In one or more (e.g., each) iteration, the diffusion model: receives the current noisy representation and applies a denoising step using features extracted from the input frame, which involves using a neural network trained to reduce noise and bring the representation closer to a true material map. The diffusion model generates an intermediate output that is slightly less noisy and more accurate than the previous iteration. This process repeats for a predetermined number of iterations, with each step bringing the output closer to a final high-quality material map. After the last iteration, the output of the diffusion model is a set of material maps that describe the surface properties of the scene.

[0045] In some embodiments, initially, the diffusion model is trained on a large dataset where each input frame is paired with a corresponding material map. The model learns to predict material maps by iteratively refining noisy inputs to match the training data. In actual use (inference), the model receives an input frame and a noise vector, processes it through iterative denoising steps, and outputs a material map. These material maps can then be used for various applications, such as rendering scenes under different lighting conditions, integrating virtual objects, or creating augmented reality experiences.

[0046] With various embodiments of the present disclosure, various technical effects and improvements can be achieved. For example, the accuracy and fidelity of the output (e.g., rendered frames) is improved. This is because, unlike existing generative models, various embodiments incorporate the technical solutions of material maps, lighting maps, and / or noise vectors. Each of these solutions ensures a high level of fidelity and quality, e.g., ensuring that a requested material “shiny” surface actually has a shine.

[0047] Another technical effect is that human-computer interaction is improved. Existing generative models do not support material maps or other powerful control or editing features. Therefore, various embodiments allow non-expert users to specify lighting and material properties through simple text commands or user interface selections. For example, a user can type “make the floor wood-like with a shiny finish,” and the model will adjust the material map accordingly. Users can intuitively modify images and videos by describing the changes they want in natural language, without needing in-depth 3D graphics or material science technical knowledge. This level of control allows for precise modifications, resulting in highly accurate and customized visual output.

[0048] Another technical effect is reduced computational resource consumption, e.g., reduced latency and I / O. For example, some embodiments conditionally adjust the model (e.g., via cross-attention) according to certain parameters (e.g., material maps, lighting, user input). By conditioning the diffusion model on certain graphical parameters such as material and ambient lighting, the present invention ensures that only relevant data is processed. This reduces the amount of data that needs to be loaded and processed, saving memory and reducing I / O operations. For example, by focusing on key material properties and necessary lighting conditions, the need to process irrelevant data is avoided, resulting in more efficient resource utilization. In another example, the model in some embodiments utilizes parallel processing capabilities to perform multiple computations simultaneously. This can include parallelizing the denoising step and concurrently processing multiple portions of the image or input (e.g., material maps, lighting maps, and / or user input). Parallel processing significantly reduces latency by distributing the computational load across multiple processors or cores.

[0049] The systems and methods described herein can be used by, but are not limited to, ego machines such as non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driving assistance systems (ADAS)), manned and unmanned robotic or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, watercraft, space shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, airplanes, construction vehicles, trains, underwater vehicles, remote-controlled vehicles (e.g., drones), and / or other vehicle types. Further, the systems and methods described herein can be used for a variety of uses such as, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security monitoring, simulation and digital twin, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twin, data center processing, conversational artificial intelligence, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, generative artificial intelligence, and / or any other suitable application. For example, one or more output frames described herein can represent a simulation of a digital twin ego machine as the ego machine moves through an environment.

[0050] The disclosed embodiments can be included in various different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, marine systems, smart regional monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems that include one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in data centers, systems for performing conversational AI operations, systems that implement one or more language models (e.g., one or more large language models (LLMs)), systems for performing optical transport simulation, systems for performing collaborative content creation for 3D assets, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0051] Reference is made to Figure 1 , Figure 1 is a block diagram of a rendering / inverse rendering system 100 (referred to as “system 100”) in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements, for example, machines, interfaces, techniques, orderings, components, etc., can be used without departing from the scope of the present disclosure. It should also be understood that, depending on the example, certain elements can or can not be used, that certain elements can be used multiple times in Figures 12A-12D , and that certain elements can be grouped in a common component, in different components, or multiple components. In this context, and for brevity, certain steps or components can be described as being “operable to do” something in order to more Figure 13 conveniently convey the implementation. Still further, functions described herein as being performed by one component can be performed by other components. Figure 14 In some embodiments, the systems, methods, and flows described herein can be implemented using components, features, and / or functions similar to those of the example autonomous vehicle 1200 in Figure 1 , the example computing device 1300 in Figure 13 , and / or the example data center 1400 in Figure 14 . In the illustrated embodiment of Figure 1 , the system 100 includes a material map generator 102, a light map generator 104, a noise generator 108, a generative model 112, and a storage 105, each communicatively coupled over a network 110 (e.g., a wide area network (WAN), a local area network (LAN), an interconnect, an internal bus structure on which all components are hosted on the same device, etc.). In some embodiments, the material map generator 102, the light map generator 104, and / or the noise generator 108 are contained within the generative model 112, rather than being separate components as illustrated in Figure 1 .

[0052] In one or more implementations, the material map generator 102 is generally responsible for generating one or more material maps. In some embodiments, the material maps are generated based on a generative model 112 that generates material maps. For example, the generative model 112 takes as input an input image, a noise vector (generated by the noise generator 108), and / or one or more lighting maps (generated by the lighting map generator 104) to generate one or more material maps, which will be described in more detail below (see, e.g., Figure 3 ).

[0053] In some embodiments, material maps can alternatively or additionally be generated from the baseline input image. Creating material maps from input images involves a series of computational steps to extract surface properties such as color, texture, reflectance, and surface normals. For example, for a reflectance map, various embodiments extract the base color of the surface of the input image without the lighting effects such as shadows and highlights. Some embodiments first remove lighting effects from the input image to isolate the inherent color of the material. Then, some embodiments separate the image into its inherent components: lighting and reflection (reflectance). In some embodiments, the extraction of this reflectance map is achieved by the following equation:

[0054] I(x,y) = R(x,y) · L(x,y)

[0055] where I(x,y) is the input image, R(x,y) is the reflection (reflectance) map, and L(x,y) is the lighting map. Some embodiments estimate (x,y) by techniques such as Retinex theory or optimization algorithms that can minimize the variation of R under varying L.

[0056] With respect to normal maps, some embodiments derive or extract surface normals from the input image to represent fine details and textures. In some embodiments, this is achieved using photometric stereo imaging, e.g., taking multiple images under different lighting conditions to estimate surface normals. Some embodiments use the following equation to estimate surface normals from shading variations in a single image: I = N · S, where I is the intensity vector of the image under different light sources, N is the normal vector, and S is the light source direction vector. Some embodiments additionally or alternatively perform shape-from-shading algorithms, e.g., represented as:

[0057] I(x,y) = p(x,y) · (N(x,y) · L)

[0058] where I(x, y) is the intensity at pixel (x, y), p(x, y) is the albedo at pixel (x, y), N(x, y) is the normal at pixel (x, y), and L is the light direction. Particular embodiments solve for N using an optimization method that minimizes the difference between the observed intensity and the predicted intensity.

[0059] Some embodiments estimate a roughness map of an input image by estimating the surface roughness, which indicates how smooth or rough the surface is. This estimation can include analyzing the size and distribution of specular highlights to estimate the roughness. Particular embodiments analyze the frequency content of an image to distinguish smooth and rough regions by the following equation:

[0060] Roughness = σ21 i,j ∑(I specular (i,j)-μ specular )2

[0061] where Ispecular(i,j) is the intensity of the specular reflection at pixel (i,j), μspecular is the average intensity of the specular reflection, and σ2 is the variance of the specular reflection intensity. Thus, various embodiments compute the variance of the specular highlight intensity to estimate the roughness.

[0062] With respect to the metallic map, particular embodiments determine whether each pixel represents a metal or a dielectric material. Metallic surfaces typically have unique reflective properties and lack diffuse reflected color. Some embodiments compute the ratio of specular to diffuse reflection to classify as metal and non-metal. Thus, various embodiments compute the variance of the specular highlight intensity to estimate the roughness.

[0063] With respect to the ambient occlusion map, some embodiments estimate the degree of occlusion of ambient light, indicating how much light is blocked by surrounding geometry. To do so, some embodiments analyze the 3D geometry to determine areas that are occluded by ambient light. Some embodiments use depth information to estimate occlusion, for example, by the following equation:

[0064] AO(x, y) = 1 - π1∫hemisphereV(ω, x, y)(N(x, y) · ω)dω

[0065] where V(ω, x, y) is the visibility function at direction ω and pixel (x, y), and N(x, y) is the normal at pixel (x, y). Thus, various embodiments all utilize the integral of the visibility function over a hemisphere to compute the ambient occlusion.

[0066] In some embodiments, a material map can additionally or alternatively be generated based on receiving user input. Generating a material map based on user input describing surface properties involves converting qualitative descriptions or user interface selections into quantitative parameters that define the appearance of a surface. This process utilizes a pre-trained model or pre-defined rules to convert user descriptions into specific material map values.

[0067] In one example instance, the user input can be to generate a "wooden floor with a glossy finish" in the output frame. Various embodiments then identify keywords that describe the material type and surface properties, such as through named entity recognition (NER) or other NLP-based techniques. For example, various embodiments generate the following labels (denoted with <>) : "wooden" <material type>, "glossy" <surface finish>. Various embodiments then map the description to label or other identifier for identifying a material map or property, such as to the pair "wooden" <material type>. For example, some embodiments assign basic color and texture properties according to the material type. As shown in the above figure, "wooden" maps to a particular color and wood grain texture. Various embodiments then map these material maps to their corresponding equations (as shown in the above figure) to derive the material map. For example, a lookup data structure or other hash map can be used, where the key is represented by the material map identifier or label (e.g. "albedo"), and the value in the lookup structure represents the equation that is actually accessed and used to generate the corresponding material map.

[0068] The lighting map generator 104 is generally responsible for generating one or more maps that describe lighting and / or shading / coloring. As with the material map generator 102, in some embodiments, the lighting map generator 104 is generated based on extracting information (e.g., pixel-by-pixel information) from the input image. For example, to generate a lighting map from an input image, some embodiments use spherical harmonics, environment maps, and / or latent variables by decomposing the lighting information in the input image and representing it in a way that models (e.g., diffusion models) can effectively utilize.

[0069] For example, some embodiments extract lighting information from an input image through spherical harmonics (SH). SHs are a set of orthogonal basis functions defined on the surface of a sphere. SHs are particularly useful in computer graphics for efficiently representing and manipulating functions defined on the domain of a sphere, such as lighting environments. They provide a compact representation of light distribution on a sphere, which is particularly useful for environment lighting. Thus, various embodiments project the environment captured in the input image onto SHs - they project the ambient lighting captured in the input image onto the spherical harmonics basis functions.

[0070] Some embodiments first project the ambient lighting captured in the input image onto the spherical harmonics basis functions:

[0071] L(0, f) ~ l=0 Lm=-l l clm Ylm(0, f)

[0072] where L(0, f) is the lighting function at spherical coordinates (0, f), Ylm(0, f) is the spherical harmonic basis function, and clm is the SH coefficient.

[0073] Environment maps capture the entire lighting environment around a scene. They can be represented as a high dynamic range (HDR) image, providing detailed information about the light sources and their intensities. Some embodiments convert an input image into an environment map by using an image-based lighting technique for converting an input image into an environment map, such as a cubemap generation. For example, this conversion can be done by the following equation:

[0074] E(x, y) = HDR(x, y)

[0075] where E(x, y) is the environment map value at pixel (x, y), and HDR(x, y) is the high dynamic range value at pixel (x, y).

[0076] Latent variables are abstract representations learned by a neural network to capture complex features of the lighting environment. In these embodiments, some embodiments extract latent features by encoding lighting information into latent variables using a neural network (e.g., a variational autoencoder), such as by the following equation: z = fencoder(I), where z is the latent vector, fencoder is the encoding function of the neural network, and I is the input image.

[0077] Then, some embodiments concatenate or combine the SH coefficients, the environment map, and the latent variables to create a comprehensive lighting map, such as by the following equation:

[0078] Lfinal(x, y) = aLSH(0, f) + bLenv(x, y) + gLlatent

[0079] where Lfinal(x, y) is the final lighting map value at pixel (x, y), a, b, g are weights balancing the contributions of the SH, the environment map, and the latent variables. In other words, various embodiments compute a lighting map by capturing ambient light using spherical harmonics, convert an input image into an environment map to obtain detailed lighting, and / or encode lighting features into latent variables using a neural network. Then, various embodiments concatenate or combine these features by computing lighting contributions from the SH, the environment map, and the latent variables. Some embodiments combine these components using a weighted sum to create a final lighting map. By following these operations and using the corresponding equations, a lighting map can be generated from an input image containing ambient lighting, detailed ambient lighting, and abstract features captured by latent variables.

[0080] As described above with respect to the material map generator 102, in some embodiments, the lighting map generator 104 can also or instead generate a lighting map based on receiving and processing user input, based on user input describing a particular lighting condition, the lighting map generator 104 converts the natural language description to actionable parameters that can be used to create a lighting map. This process can utilize natural language processing (NLP) techniques (e.g., named entity recognition (NER)) to extract key elements from the user input and map them to corresponding lighting concepts. For example, a user can specify a lighting condition such as “bright afternoon sunlight shining in from the large window on the left side of the room.” NER can then responsively extract entities from this user input such as “time of day” (afternoon), “light source” (sunlight), “intensity” (bright), “direction” (left side), and “modifier” (shining in through the window). An example of NER output is as follows: time of day: “afternoon,” light source: “sunlight,” intensity: “bright,” direction: “left side,” and modifier: “shining in through the window.”

[0081] Some embodiments then map the extracted information to lighting parameters by converting the extracted entities to parameters for creating a lighting map. For example, in a lookup structure, the keys can represent parameters and the values can represent the extracted entities. For example, lighting parameters can first be defined such as time of day: “afternoon” indicates warm light, high light, color temperature around 5000-6000 K (Kelvin). Light source: “sunlight” indicates directional light. Intensity: “bright” indicates high intensity. Direction: “left side” indicates light direction vector. Modifier: “shining in through the window” indicates soft shadows.

[0082] Then, some embodiments can calculate a directional light source (e.g., based on user input stating light source position information), such as by the following formula: L = I · DL = I · D, where L is the lighting vector, I is the light intensity, and D is the direction vector. Some embodiments can also map time of day to color temperature, create an environment map reflecting the user-specified lighting condition, and apply soft shadows to simulate a window effect. By utilizing natural language processing (NLP) and mathematical models, a user description can be effectively converted to a detailed lighting map suitable for realistic rendering in a 3D scene. This approach allows intuitive and flexible lighting control based on natural language input.

[0083] The noise generator 108 is generally responsible for generating one or more noise vectors to generate frames or other intermediate objects, such as a texture map. A noise vector is a random vector drawn from a particular distribution, such as a Gaussian distribution. It represents a highly noisy version of the final frame or other object (e.g., a texture map or lighting map). In the context of a diffusion model, the noise vector is gradually transformed into a clean frame or object through a series of denoising steps. In the context of a diffusion model, alpha (a) represents the level of noise in the image. The initial alpha value (a0) is typically set close to 1, representing a high level of noise, such as pure noise.

[0084] Some embodiments first sample from a Gaussian distribution and then apply an initial noise level. For example, with respect to sampling, a noise vector z is sampled from a Gaussian distribution (0, I), where I is the identity matrix - z ~ N(0, I). With respect to applying an initial noise level, the noise vector at the initial alpha value can be the sampled Gaussian noise, as a0~ 1.

[0085] In one or more implementations, the noise vector is gradually denoised through multiple iterations, guided by parameters learned by the generative model 112. In some embodiments, a forward diffusion process (adding noise) is first performed, starting from a clean frame (e.g., an input image) x0x0, and then adding noise at a time step t. At each time step t, the image xtis represented as:

[0086]

[0087] where a is the noise level at time step t, and z ~ N(0, I) is Gaussian noise. In some embodiments, the noise generator 108 subsequently performs a reverse diffusion process (denoising), in which the generative model 112 (e.g., a diffusion model) model learns to reverse the noise addition process, starting from xT(a highly noisy version) and gradually denoising it to generate the final output frame. At each reverse step t, the denoising process involves predicting the original image x0from xt:

[0088] xt-1= xt- e0(xt, t)

[0089] where e0(xt, t) is the model’s 112 prediction of the noise component at step t. As the iterations proceed, a is decreased, thereby reducing the noise in the image: a = a -1- a. The model 112 gradually refines the image, progressively removing noise according to the learned noise patterns.

[0090] In one or more implementations, the generative model 112 takes as input one or more of: an input image, a noise vector (generated by the noise generator 108), one or more material maps (generated by the material map generator 102), and / or one or more lighting maps (generated by the lighting map generator 104). The generative model 112 generates an output frame and / or other objects, such as material maps (generated by the material map generator 102), using the input information, which will be described in more detail below.

[0091] The system 100 also includes a storage 105. The storage 105 represents any suitable data storage area (e.g., a database or other data structure) or storage device (e.g., a storage area network (SAN), RAM, RAID, disk) for storing any suitable data, such as frames, models, routines, models, etc. For example, the storage 105 can include input frames uploaded by users and output frames generated by the generative model 112, as well as any other objects, such as material maps generated by the material map generator 102, lighting maps generated by the lighting map generator 104, and noise vectors generated by the noise generator 108.

[0092] Figure 2 is a pipeline 200 illustrating a rendering process according to some embodiments, in which a diffusion model (DM) generates a rendered image 210 (output frame) based on processing particular inputs. In some embodiments, the material map 202 represents a material map generated by the material map generator 102 of Figure 1 , the lighting 202 represents a lighting map generated by the lighting map generator 104 of Figure 1 , the noise vector 206 represents a noise vector generated by the noise generator 108 of Figure 1 , and the DM 208 represents the generative model 112 of

[0093] Figure 2 illustrates a workflow for neural rendering, in which it is shown how a diffusion model (DM) 208 can be used to augment or replace classic physics-based rendering techniques (e.g., path tracing). The DM 208 takes as input a material map 202, a noise vector 206, and lighting 204 to generate a final rendered image 210 that reflects the specified lighting and material properties. The DM 208 uses the provided information / input (i.e., 202, 204, 206, 212, and 214) to condition a denoising process that progressively refines a noisy input into a high-quality rendered image 210.

[0094] In some embodiments, DM 208 represents a U-NET machine learning model. U-Net (“U-shaped Network”) is a convolutional neural network (CNN) architecture that can be used for image segmentation tasks. However, it can also be adapted for various other tasks, including generating frames or images. A characteristic of the U-Net architecture is the U-shaped structure, where a contracting path captures context and an expansive path enables precise localization.

[0095] In some embodiments, the U-Net takes input data that provides context for frame generation. This input can include one or more frames, or additional information that describes the desired content or conditions (i.e., noise vector 206, material map 202, and lighting 204). The input data can then pass through a series of convolutional layers, each followed by an activation function (e.g., ReLU) and / or a normalization layer (e.g., batch normalization). A down-sampling operation can then occur, such as max-pooling or stride convolution, which reduces the spatial resolution while capturing hierarchical features. A bottleneck function can then occur—the contracting path leads to a bottleneck layer, in which the most abstract and compressed representation of the input data is obtained. An expansive path function or decoder function is then executed. The expansive path involves an up-sampling operation (e.g., transposed convolution) to recover the spatial resolution step-by-step. Skip connections can then concatenate feature maps from the contracting path at corresponding levels, which helps to recover fine details and prevent information loss. The final layer of the U-Net deterministically generates an output frame based on the processed information. For deterministic frame generation, a linear activation function can be used if the pixel values are directly predicted. Alternatively, other activation functions suitable for the particular task can be employed. During training, a deterministic loss function can be used to measure the difference between the predicted frame and a ground truth frame. For example, the loss function for deterministic frame generation includes mean squared error (MSE) or other regression-based loss functions, depending on the nature of the output.

[0096] The cross-attention mechanism in DM 208 enables DM 208 to effectively integrate the conditioning information (albedo, normal map, lighting) in the denoising process. This involves aligning and focusing on relevant parts of the conditioning input while refining the noisy image. Various embodiments first generate input embeddings that contain noise vector 206 and the conditioning inputs (i.e., material map 202 and lighting 204). These embodiments encode the albedo, normal map (and / or other material maps), and lighting information using one or more neural networks to obtain feature embeddings.

[0097] To perform encoding, one or more embodiments use an encoder path (downsampling) by passing the noisy image (i.e., the noise vector 206) through a series of convolutional layers to extract hierarchical features at different scales. At each layer / , some embodiments compute a feature map F1. At each layer / of the encoder, various embodiments apply a cross-attention mechanism that integrates the conditional features with the feature map F1. In some embodiments, the computation of the cross-attention mechanism is as follows:

[0098] Attention(Q, K, V) = softmax(QKT / √d k )V

[0099] where Q (query) = feature map F1from the noisy image, K (key) and V (value) represent the conditional features from albedo, normal map, and / or lighting. This produces a refined feature map that contains conditional information. The lowest resolution layers in the U-Net capture the most abstract features. Then, various embodiments integrate global context from the latent variables at this stage to provide additional scene information.

[0100] Then, some embodiments engage a decoder path (upsampling) by passing the features through a series of transposed convolutional layers to reconstruct the image 210 step-by-step. Various embodiments use skip connections to combine high-resolution features from the encoder with the upsampled features to preserve spatial details. The last layer of the decoder produces a denoised image that integrates the denoising process with the conditional information to generate the final rendered image 210. Thus, the lowest resolution layers in the U-Net capture the most abstract features. Various embodiments also integrate global context from the latent variables at this stage to provide additional scene information. Then, various embodiments pass these features through a series of transposed convolutional layers to reconstruct the image step-by-step. Various embodiments use skip connections to combine high-resolution features from the encoder with the upsampled features to preserve spatial details. The last layer of the decoder produces a denoised image that integrates the denoising process with the conditional information to generate the final rendered image.

[0101] With continued reference to Figure 2 , the numeral 212 denotes that a user-initiated video control 212 can be incorporated, which is contained in additional parameters 216. The video control 212 indicates that the DM 208 can manipulate the video input, thereby ensuring temporal consistency across frames. The video control 212 allows the user to specify (e.g., through natural language or user interface selection) material properties and lighting conditions. In response, various embodiments generate the corresponding material map 202 and lighting 204, e.g., as described with respect to Figure 1The material map generator 102 and the lighting map generator 104 described above. For example, the user can specify in natural language that an object in the rendered image 210 should be a particular color throughout the video sequence.

[0102] Similarly, the edit / refine controls 214 allow the user to edit and refine the rendered output through text controls and material parameter adjustments. By providing intuitive scene modification controls, non-expert users can easily get up to speed with the rendering process. For example, the DM 208 can scale the shininess according to the user instruction "increase the shininess of the wood floor by 20%". In another example, a particular embodiment can change the albedo map according to the user input indicating "change the color of the sofa from red to blue". The user can additionally or alternatively replace or modify textures on surfaces, such as changing the pattern on a rug or the fabric on a chair. For example, "apply a new floral texture on the sofa fabric". The user can additionally or alternatively adjust the reflectance and specular highlights on surfaces. For example, the user can indicate "increase the reflectance of the marble countertop" to change the material map. The user can also adjust the intensity of the light sources in the scene (e.g., by the natural language statement "dim the main light source by 30%"). The user can also change the direction of the light to simulate different times of day or lighting setups, such as by the statement "change the sunlight direction to shine from the east". The user can control the softness of the shadows, making them harder or softer, such as by the statement "soften the shadows of the window light projection".

[0103] To integrate the video controls 212 and editing / refinement controls 214 into the architecture of the diffusion model (DM) 208, in some embodiments, the model 208 needs to be augmented with additional modules and mechanisms to handle temporal consistency, dynamic user input, and real-time editing functionality. For example, the DM can include a spatio-temporal encoder-decoder (not shown in the figure) to manipulate the video input and maintain temporal consistency across frames. The spatio-temporal encoder will incorporate 3D convolutional layers to process spatial and temporal information. Various embodiments will then use recurrent layers (e.g., LSTM, GRU, etc.) or temporal convolutional layers to capture temporal dependencies. The spatio-temporal decoder will then use transposed 3D convolutions to upsample in the spatial and temporal dimensions, incorporating skip connections to preserve spatial details and maintain consistency. Various embodiments will then integrate a temporal attention mechanism to focus on relevant features from previous frames. Some embodiments use NLP techniques to parse user input and map the user input to corresponding material properties and lighting changes. Some embodiments dynamically integrate user input in the denoising process using conditional embeddings and cross-attention mechanisms. Some embodiments create embeddings for user-specified changes (e.g., shininess, color). Some embodiments then apply a cross-attention mechanism to integrate these embeddings into intermediate features of the model. By integrating spatio-temporal mechanisms for video control and cross-attention mechanisms for dynamic user input manipulation, the diffusion model 208 can achieve temporal consistency and interactive refinement capabilities. This enhanced architecture allows users to control material properties and lighting in real-time, ensuring high-quality, consistent video output.

[0104] Figure 3 An inverse rendering pipeline 300 is shown, in accordance with some embodiments, in which a diffusion model 308 generates one or more material maps 302 as output based on certain inputs. In some embodiments, the noise vector 306 represents a noise vector generated by the noise vector generator 108 in FIG. 1, the DM 308 represents the generative model 112, and the material maps 302 represent material maps generated by the material map generator 102 in FIG. 1. Figure 1 Figure 1

[0105] Figure 3 ​​An intrinsic decomposition inverse rendering process is shown, in which the DM 308 generates material maps 302 as output based on processing of an input image 303 (e.g., a user-uploaded image) and a noise vector 306 as input. The user requests the extraction of intrinsic properties, such as albedo, normals, and other material maps, from the input image 303. The noise vector 306 is added to the input image, and the diffusion model 308 will process this noise vector to generate the material maps 302. Additional parameters 316 represent additional parameters or conditions that guide the diffusion model. These parameters can include learned weights, prior knowledge, or task-specific constraints. Intrinsic decomposition refers to the process of separating the input image 303 into basic components that describe different intrinsic properties of the scene. In this context, intrinsic properties include the material maps 302.

[0106] To generate the material maps 302 at the output, various functions are used according to some embodiments. The input image I is combined with the noise z (i.e., the noise vector 306): X0= I + z. The noisy image X0is then passed to an encoder, which contains several convolutional layers that progressively downsample the image 303 and extract hierarchical features. At each layer of the encoder, a cross-attention mechanism is applied to integrate conditional information (e.g., albedo, normal maps, and lighting). The cross-attention mechanism allows the model to focus on relevant parts of the conditional input while processing the noisy image. Abstract features and global context are captured at the lowest resolution layers. Regarding the decoder path, the image 303 is reconstructed through the decoder and uses skip connections to preserve spatial details. Regarding the output layer, separate convolutional layers produce the material maps 302, such as albedo maps, normal maps, metallic maps, and roughness maps. At the last layer of the decoder, the model outputs different material maps. Each map corresponds to a specific intrinsic property of the input image. By leveraging the cross-attention mechanism in the U-Net architecture, the diffusion model 308 can effectively process noisy inputs and generate detailed material maps while maintaining high fidelity and consistency with the input image. In some embodiments, the DM 316 is configured to produce the material maps 302 through the video controls 212 and / or the edit / refine controls 214, as described in connection with FIG. 2. Figure 1

[0107] Figure 4 An inverse rendering pipeline is shown, in which the transformer 408 generates an environment map 410 as output based on a specific input according to some embodiments. In some embodiments, the transformer 408 represents the generative model 112 in Figure 1

[0108] Figure 4 ​​A process is shown that involves eigen decomposition and lighting estimation / modeling for generating an environment map 410 from an input image 402 using a spherical coordinate transformer 408. A “spherical coordinate transformer” 408 is a transformer model that operates on data represented in spherical coordinates. Spherical coordinates are used to describe points in 3D space (e.g., with three values: radial distance, polar angle, and azimuthal angle). In the context of machine learning or neural networks, a “spherical coordinate transformer” 408 is a specialized model designed to manipulate data that naturally fits a spherical coordinate system. This is especially useful in tasks involving 3D spatial data, as traditional Cartesian coordinates (x, y, z) can be less efficient or less intuitive.

[0109] The input image 402 (e.g., a 2D image) is first processed to extract relevant features. For example, a particular embodiment uses a convolutional neural network (CNN) to detect and encode features such as edges, textures, and color information. The features extracted from the input image 402 are then mapped to spherical coordinates. This involves transforming the 2D image features into a 3D representation. For example, each pixel in the image can be projected onto a unit sphere, converting Cartesian coordinates to spherical coordinates (radius r, polar angle θ, and azimuthal angle φ). The transformation from Cartesian coordinates to spherical coordinates is as follows:

[0110] θ = arccos(z / √x 2 +y 2 +z 2 )

[0111] φ = arctan2(y, x)

[0112] where (x, y, z) are the Cartesian coordinates and (θ, φ) are the spherical coordinates.

[0113] The spherical coordinate transformer 408 processes the spherical coordinate data. This transformer model 408 contains layers that use attention mechanisms specifically designed for spherical data. These layers can effectively capture dependencies and relationships between different parts of the spherical data, which is crucial for tasks such as lighting estimation. With the spherical features processed by the transformer, the model 408 generates an environment map 410. The environment map represents how light interacts with the environment and is used for rendering to simulate realistic lighting conditions. The output map includes information about the light sources and their intensity, direction, and color. Specific loss functions (e.g., mean squared error (MSE) and structural similarity index loss (SSIM)) are used to ensure that the output environment map accurately represents the lighting conditions during fine-tuning or training. These loss functions aim to measure the difference between the predicted environment map and the ground truth lighting conditions. This process leverages the strengths of CNNs in feature extraction and transformers in handling complex dependencies in spherical data, resulting in accurate and realistic environment maps for lighting estimation and modeling.

[0114] Figure 5 A pipeline 500 for performing 2D frame relighting by eigen-decomposition and neural rendering is shown, in accordance with some embodiments. In some embodiments, the noise vectors 506 and 511 are noise vectors generated by the noise generator 108 in Figure 1 In some embodiments, the diffusion models 508 and 512 represent the generative model 112 in Figure 1 In some embodiments, the user-specified lighting 504 represents user input and / or lighting maps generated by the lighting map generator 104 in Figure 1 In some embodiments, the user-specified lighting 504 represents user input and / or lighting maps generated by the lighting map generator 104 in Figure 1 In some embodiments, the user-specified lighting 504 represents user input and / or lighting maps generated by the lighting map generator 104 in

[0115] Figure 5 An input 2D image 503 is shown being fed into the pipeline for video relighting, as indicated in the relit image 510. The pipeline 500 includes taking the input image 503 and a noise vector 506, processing them through a diffusion model (DM) 508 to extract a material map 503, and then generating a relit image 510 using user-specified lighting 504.

[0116] In particular, a real-world input image 503 for which the user requests relighting (change in lighting parameters) is first provided as input to the DM 508. For example, an image can be provided as input to the DM 508 along with a request to change the position of a light source from coordinate A to coordinate B. A noise vector 506 is also added to the input image 503 to create an initial noisy input to the diffusion model. As described above, for example, a noise vector z is added to an input image I to create an initial noisy input X0 (X0 = I + az) through a forward and / or backward diffusion process.

[0117] The diffusion model 508 processes the input image 503 and the noise 506 to decompose the image into its eigen-properties, including one or more material maps 502. The one or more material maps 502 can include: an albedo map (base color of surfaces in the scene, without any shading or lighting effects), a normal map (encodes the orientation of surfaces in the scene), and optional maps containing other eigen-properties such as depth, metallic, and roughness, which can be extracted if needed. The eigen-decomposition step involves a neural network that analyzes the input image and decomposes it into these essential components that are critical for realistic relighting.

[0118] As described herein, the DM 508 is trained, updated, or fine-tuned to generate the material maps 502. For example, for training purposes, ground truth albedo maps, normal maps, and optionally other maps (depth maps, metallic maps, and roughness maps) are obtained in a dataset. In some embodiments, training involves iteratively optimizing model parameters to minimize the difference between predicted intrinsic material maps and ground truth material maps. In a forward pass of training, some embodiments first generate a noise vector z and add it to the input image I: X0= I + az. The noisy input X0is passed through an encoder to extract features. A decoder processes these features to predict, for example, albedo maps and normal maps.

[0119] Some embodiments define suitable loss functions to measure the reconstruction accuracy of each intrinsic property. For example, for albedo loss, the L2 loss (mean squared error) between the predicted albedo map and the ground truth albedo map is computed by the following equation:

[0120] L albedo = ||A pred- A true ||2.

[0121] By following this procedure, the diffusion model 508 learns to decompose an input image into its intrinsic properties, effectively handling noise and producing accurate albedo maps and normal maps, and / or other material maps.

[0122] After the material maps 502 have been generated, these maps, along with the user-specified lighting 504 and a second noise vector 511, are provided as input to a second DM 512 to produce the final relit image 510. The user provides lighting conditions 504, which are encoded as parameters that the second DM 512 can understand and use. These parameters include the direction, color, and / or intensity of the light sources.

[0123] After generating the material maps 502, some embodiments generate a noise vector 511 (z) and add it to the material maps 502 to create an initial noisy input. Some embodiments also convert the user-specified lighting conditions 504 into a format that the model can understand, such as lighting vectors or lighting maps (e.g., lighting vectors or lighting maps generated by the lighting map generator 104 in FIG. 1). Figure 1 For example, the user can specify: lighting direction: from the top right corner; intensity: high intensity (e.g., 1.5 times the ambient light); color: warm light (e.g., color temperature 3000K); type: point light source.

[0124] The various embodiments then use the DM 512, which contains the material map 502, the noise 511, and the user-specified lighting 504, to generate a relit image 510. For example, some embodiments pass the noisy material map through an encoder to extract features. In some embodiments, the encoder contains multiple convolutional layers, each followed by a non-linear activation function (e.g., ReLU) and / or other operations such as batch normalization. The goal of the encoder is to progressively downsample the input while extracting increasingly abstract and higher-level features. The input to the encoder is the concatenation of the albedo map, the normal map (and / or any other material maps), and the noise vector 511. If the albedo map and the normal map 502 are both 3 x H x W tensors (where H and W are the height and width of the image, respectively), and the noise vector 511 is also 3 x H x W, then the combined input will be a 9 x H x W tensor.

[0125] Each convolutional layer applies a set of learnable filters (corresponding to 511, 502, and 504, respectively) to the input. These filters are small matrices (e.g., 3 x 3 x 3 or 5 x 5 x 5) that slide over the input and perform a dot product at each spatial position. For an input tensor X and a filter W, the convolution at position (i,j) can be written as:

[0126]

[0127] where Y i,j,k is the output feature map at position (i,j) in channel k, C is the number of input channels, Fh and Fw are the height and width of the filter, and W k,m,n,o is the weight of the filter in channel k. Pooling layers (e.g., max-pooling) are used to reduce the spatial dimensions of the feature maps while retaining the most important information. This helps to reduce the computational load and introduce spatial invariance.

[0128] The various embodiments then use a cross-attention mechanism or other conditioning mechanism to integrate the user-specified lighting 504 with the extracted features. The lighting vector influences the way the features are processed, ensuring that the new lighting conditions are applied correctly. First, the user-specified lighting conditions 504 are encoded into a form that the model can use. For example, this encoding involves converting the lighting parameters (e.g., direction, intensity, color) into a feature vector. For example, the lighting direction can be encoded as a vector indicating the direction of the light. The intensity can be encoded as a scalar value indicating the intensity of the light. The color is encoded as an RGB vector indicating the color of the light.

[0129] The cross-attention mechanism allows the model 512 to integrate the user-specified lighting 504 into the feature extraction process. This is achieved by attending to the lighting features while processing the image features. For example, the keys (K) and values (V) of the cross-attention mechanism indicate the encoded lighting information. The cross-attention mechanism computes a weighted sum of the values (V), where the weights are determined by the similarity between the queries (Q) and the keys (K), as described above. The feature maps from the encoder represent the image features. The encoded lighting vector is reshaped to match the spatial dimensions of the feature maps. The feature maps extracted by the encoder will have a shape of (B, C, H, W), where B is the batch size, C is the number of channels, H is the height, and W is the width. A fully connected (FC) layer is used to encode the lighting vector L into a higher-dimensional feature vector. For example, if the lighting vector L has a shape of (B, 7), where 7 represents the components of the lighting vector (direction x, y, z, intensity, color r, g, b). A neural network (e.g., a series of fully connected layers) is used to transform the encoded lighting vector into a feature vector. The encoded lighting vector is reshaped to match the spatial dimensions of the feature maps. In some embodiments, this involves adding two singleton dimensions at the end of the encoded vector, making it compatible for broadcasting. Broadcasting ensures that the lighting vector is replicated across the spatial dimensions of the feature maps.

[0130] For image reconstruction at the relit image 510, the second diffusion model 512 takes the material map 502 and the user-specified lighting 504, as well as another noise vector 511, as input. The model 512 uses these inputs to render the final relit image 510 and apply the new lighting conditions to the scene. The cross-attention mechanism ensures that the user-specified lighting is accurately applied to the material map to generate a realistic relit image 510. The final output is the relit image 510, which looks as if it were illuminated by the new lighting specified by the user at 504.

[0131] Figure 6 A pipeline 600 for performing 3D frame relighting by eigen-decomposition and neural rendering is shown, in accordance with some embodiments. In some embodiments, the noise vectors 606 and 611 are noise vectors generated by the noise generator 108 of Figure 1 In some embodiments, the diffusion models 608 and 612 represent the generative model 112 of Figure 1 In some embodiments, the material maps 602 and / or 610 represent material maps generated by the material map generator 102 of Figure 1 In some embodiments, the specified lighting 604 represents user input and / or a lighting map generated by the lighting map generator 104 of Figure 1

[0132] ​At a first time, the input images 603 (e.g., driving video frames, YOUTUBE video frames, SORA video frames) and the noise vector 606 are provided as inputs to the DM 608 to generate the output albedo material map 602 (e.g., as described in connection with Figure 5 the DM 508 in Figure 3 the DM 316 in ). At a second time, various embodiments generate a 3D proxy scene 609 based on the albedo map 602 (and / or other material maps) as input. The 2D information (e.g., albedo 602) is used to create the 3D proxy scene 609. This scene represents a 3D reconstruction of the environment depicted in the input images 603.

[0133] In an illustrative example, a proxy neural radiance field and / or a 3D geometry scan (3DGS) is used to generate the 3D proxy scene 609. A neural radiance field (“NeRF”) is a technique for representing a scene in 3D space as a continuous, parameterized function that can be used to output color and density given a 3D position and 2D viewing direction to render a 2D image of any position in the scene. It learns this function by optimizing parameters to fit a set of 2D images of the scene indicated in the input images 603. The inputs are the albedo map 603 (providing base colors of surfaces) and / or other material maps (e.g., normals, depth, roughness, etc. to provide additional geometry and surface information).

[0134] NeRF uses positional encoding to convert 3D coordinates to a higher dimensional space. This helps the network capture high frequency details. Techniques to create a NeRF typically use a multi-layer perceptron (MLP) to model the scene. Positional encoding transforms the input 3D coordinates to a higher dimensional space to capture high frequency variations. This encoding is crucial because standard MLPs struggle to learn high frequency functions using low dimensional inputs.

[0135] The positional encoding function y maps a 3D coordinate x = (x,,) to a higher dimensional vector:

[0136] y(x) = (sin(2 0 πx), cos(2 0 πx), sin(2 1 πx), cos(2 1 πx), …, sin(2 L-1 πx), cos(2 L-1 πx)) where L is the number of frequencies used for encoding.

[0137] In one or more embodiments, the MLP in NeRF models a scene by learning to map position- encoded coordinates and viewing directions to color and density values. In some embodiments, the MLP contains several fully connected layers with ReLU activation functions. The network receives position- encoded coordinates and viewing directions and outputs density and color. The MLP takes as input the encoded 3D coordinates and viewing directions and outputs color and density. Volume rendering can be used to generate 2D images from NeRF. Rays are cast through the scene from a target viewpoint in the scene, and the network predicts color and density information along these rays. The color and density are then integrated to produce pixel colors. The network is trained using gradient descent to minimize the difference between the rendered images and ground truth images. Position encoding converts 3D coordinates into a high-dimensional space and encodes viewing directions in a similar fashion, enabling the model to consider the direction along which a viewpoint is taken and providing the MLP with a richer feature set, facilitating learning of complex high-frequency patterns in the scene. For a ray r(t) = o + td, where o is the origin and d is the direction, its color C(r) C(r) is given by:

[0138] C(r) = ∫t n tf T(t) σ(r(t)) c(r(t)) dt

[0139] where σ(r(t)) is the density at point r(t) and c(r(t)) is the color at point r(t).

[0140] 3D geometry scanning (3DGS) involves reconstructing the 3D geometry of a scene from 2D images (e.g., 603) using techniques such as photogrammetry or structured light scanning. The input is an albedo map 602 (and / or other material maps such as a normal map and a depth map (providing the distance from the camera to the surface)).

[0141] If multiple views of a scene (input images 603) are available, then multi-view stereo (MVS) techniques can be used to reconstruct the 3D geometry by finding correspondences between images and triangulating 3D points. Structure from Motion (SfM) can be used to estimate camera parameters and the 3D structure of a scene from a sequence of 2D images. If only single-view images are available, then a depth estimation network can predict a depth map from the images. The 3D points obtained from MVS or SfM are used to construct a mesh representing the 3D geometry of the scene. The albedo map and other material maps are projected onto the mesh to create a textured 3D model.

[0142] In some embodiments, various training stages are performed so that the model can learn to produce a 3D proxy scene given a material map as input. A photometric loss function is used, which measures the difference between the rendered image and the ground truth image. NeRF can be used with volume rendering techniques to integrate the predicted density and color along a ray to produce pixel colors. Various embodiments use gradient descent to minimize the photometric loss on the training dataset. After training, the model can generate a 3D proxy scene from albedo maps and other material maps. During inference, the trained model takes albedo maps, normal maps, and / or other material maps as input to generate a 3D proxy scene.

[0143] Continuing Figure 6 The 3D proxy scene 606 is used as input to produce material maps 602 (albedo, normal, and / or any other material maps) that are extracted from the 3D proxy scene 609. The 3D proxy scene 609 is a simplified or intermediate representation of the scene that captures the geometric and material properties of the scene. As described above, this scene can be generated using techniques such as Neural Radiance Fields (NeRF) or 3D Geometry Scanning (3DGS).

[0144] In some embodiments, a neural network is used to process the rendered information from the 3D proxy scene 609 to generate the desired material maps at 610. In some embodiments, this network includes an encoder to extract features and a decoder to generate a particular material map. The encoder extracts features from the 3D proxy scene 609 and captures a high-level representation of the scene geometry and material properties in the 3D proxy scene 609. In some embodiments, the input to the encoder is a multi-channel image or tensor representing the 3D proxy scene. This can include albedo maps, normal maps, depth maps, and other material properties. The encoder processes the input using a series of convolutional layers. Each convolutional layer applies a set of learnable filters to the input, producing a feature map that captures local patterns and structures. Each layer applies a convolution operation (as described above). The feature maps output by each convolutional layer capture information at different levels of abstraction, such as low-level features (e.g., edges, textures, and simple patterns), mid-level features (e.g., shapes, contours, and structures), and high-level features (e.g., object parts, semantics, and material properties). The deeper layers of the encoder capture high-level representations that combine geometric and material information. These high-level features are useful for tasks such as relighting and material map generation.

[0145] In some embodiments, separate decoders are used to generate different material maps 610, such as albedo maps and normal maps. Each decoder is designed to generate one type of material map. In some embodiments, the decoders include a series of transpose convolutional layers (also referred to as deconvolutional layers) for upsampling the feature maps to the original input resolution. For example, the albedo map decoder takes high-level features as input and outputs a 3-channel image representing an albedo map. Likewise, the normal map decoder uses a similar architecture but outputs a 3-channel image representing a normal map.

[0146] Continuing Figure 6 , the material maps 610, the noise vector, and the specified lighting conditions 604 (e.g., user-specified changes to light source positions) are provided as input to a second DM 612, which then generates a final rendered relit image 610 representing a 3D relit version of the input image 603. In some embodiments, the DM 612 generates the relit image 610 in the same manner as described, for example, with respect to the DM 208 in Figure 2 , which also takes material maps, a noise vector, and lighting as input to generate a rendered image 210.

[0147] Figure 7 A pipeline 700 for providing identity-preserving image enhancement (e.g., object insertion, synthesis, or style transfer) is shown, in accordance with some embodiments. In some embodiments, the noise vector 706 is generated by the noise generator 108 in Figure 1 . In some embodiments, the diffusion model 708 represents the generative model 112 in Figure 1 . In some embodiments, the material maps 702 and / or 610 represent the material maps generated by the material map generator 102 in Figure 1 . In some embodiments, the specified lighting 704 represents user input and / or the lighting map generated by the lighting map generator 104 in Figure 1 .

[0148] The input image 703 (e.g., an image provided by a user to request image enhancement or repair), an albedo map (e.g., a map representing the base color of surfaces in the image), a normal map (e.g., a map representing surface orientation), random noise 706, and lighting 706 are provided as inputs to a diffusion model (DM) 708. Random noise (denoted as noise vector 706) is added to the input image 703, which the diffusion model 708 uses in a denoising process. In some embodiments, the lighting 704 includes spherical harmonics (SH), an environment map (Env. Map), and / or latent representations, as described herein. This lighting information 704 is used to inform the model of the lighting conditions in the scene indicated in the input image 703. Other parameters of the DM 708 include parameters for adjusting the style or enhancing specific aspects of the image (e.g., through image inpainting).

[0149] The input image 703 is first combined with the noise vector 706 to create a noisy version of the input image 703 that the diffusion model 708 will use to enhance the image (and / or learn how to enhance the image when training). The albedo and normal maps 702 (and / or any other suitable material maps) provide additional information about the scene’s material properties and geometry, which helps preserve the identity of the final output and boost the realism of the final output at 710.

[0150] There are various use cases for generating fixed images 710 from input images 703. For example, in some embodiments, a simulation-to-reality (Sim2Real) technique is used to convert synthetic video frames (e.g., from CARLA) into photo-realistic video frames. The model 708 can take a photo-realism-poor image at 703 (source style) and use the albedo and normal maps 702 to generate a more realistic image at 710.

[0151] In one or more implementations, the input can include one or more of a source image, an albedo map, a normal map, etc. The source image (Is) (represented by input image 703) can contain a photo-realism-poor image. The albedo map (A) (in 702) represents the base color of surfaces without lighting or shading. The normal map (N) (in 702) encodes the surface orientation (normal). Noise (η) (i.e., noise vector 706) is added to the input image 703 to create an initial noisy input. The encoder of the DM 708 processes the input image 703, the albedo map 702, the normal map 702, and the noise 706 to extract meaningful features.

[0152] In one or more implementations, feature extraction can be achieved by using a function F of the encoder that takes the concatenated inputs and extracts features:

[0153] Fenc = F(I s ,A,N,η)

[0154] where F enc represents the extracted features.

[0155] Illumination information (L) (denoted by 704), such as spherical harmonics, an environment map, and / or latent vectors, is integrated into the feature extraction process using a cross-attention mechanism or other conditioning mechanism, as described above. Let L enc be the encoded illumination information. The integrated features F int are computed as: F int = Attention(F enc ,L enc ,L enc ). The decoder of the DM 708 takes the integrated features and reconstructs the image 703 with enhanced photo-realism: I r = G(F int ), where G is a function of the decoder and Ir is the generated more photo-realistic image (denoted by 710).

[0156] To ensure that the output image Ir has photo-realism and preserves the identity of the source image in 703, in some embodiments, the model 708 is trained using multiple loss functions. For example, a reconstruction loss can be used. The reconstruction loss measures the difference between the predicted image (e.g., in 710) and the ground truth photo-realistic image (I gt ):

[0157] L rec = ||I r -I gt ||2.

[0158] Some embodiments additionally or alternatively use a perceptual loss. With this loss, a pre-trained network (e.g., the DM) can be used to measure the perceptual similarity between I r and I gt :

[0159] L perc = i∑||φ i (I r )-φ i (I gt )||2

[0160] where φi represents the feature maps of the i-th layer of the pre-trained network. Additional or alternative losses can be used, such as an illumination consistency loss and / or a style loss. The total loss is a weighted sum of the above losses:

[0161] L total = λ rec Lrec + λ perc L perc + λ style L style + λ light L light

[0162] where λ rec , λ perc , λ style , and λ light are weights for each loss component. In some embodiments, the model parameters are optimized using gradient descent (e.g., Adam) to minimize the total loss.

[0163] Some embodiments additionally or alternatively change the style of the input image 703 while preserving the underlying geometry and material (e.g., material maps). For example, various embodiments apply a different artistic style to the image 703. Let the source image (I s ) (e.g., the input image 703) represent the image whose style is to be changed. Let the image style (I style ) represent an image that provides the desired style (e.g., the fixed image 710).

[0164] In these style transfer embodiments, the encoder of the DM 708 extracts features from the style image to capture the style elements: F style = F(I style ). To transfer the style from I style to I s , various embodiments transform the features F enc of the source image (encoder features) to match the style features F style . Some embodiments utilize adaptive instance normalization (AdaIN), which adjusts the mean and variance of the content features to match the mean and variance of the style features.

[0165] AdaIN(F enc , F style ) = σ(F style )(F enc - μ(F enc ) / σ((F enc ))+ μ(F style )

[0166] where μ and σ represent the mean and standard deviation, respectively. The decoder takes the transformed features and reconstructs an image with the desired style while preserving the underlying geometry and material properties. Thus, the transformed features F trans are given by the equation: Ir= G(F trans ), where G is a function of the decoder, and I r is the generated stylized image.

[0167] To ensure that the output image Irhas the desired style and preserves the underlying geometry and material of the source image, in some embodiments, the model 708 is trained using multiple loss functions. For example, a content loss (measuring the difference between the features of the source image 703 and the reconstructed image 710, ensuring that the geometry and material properties are preserved), a style loss, a material consistency loss (ensuring that the material properties (e.g., albedo and normal map) of the source image and the stylized image represented by 710 remain consistent). This can be achieved by comparing the material map reconstructed from the stylized image and the original material map. The total loss is a weighted sum of the above losses. The model parameters are optimized using a gradient descent method (e.g., Adam) to minimize this total loss.

[0168] Some embodiments additionally or alternatively perform synthesis and / or object insertion. Consider the following notation: input image 703 (I s ): original image of the scene; albedo map (A s ); normal map (N s ); object image (I o ): image representing an object (e.g., a sofa) to be inserted into the scene; object albedo map (A0): encodes the base color of the surface of the object (to be inserted); object normal map (N o ): encodes the surface orientation of the object. Object mask (M0): binary mask indicating the presence of the object in the synthesized image. Lighting information (L): includes spherical harmonics, environment map, and / or latent representation. The goal is to integrate I o into I s to create a synthesized image I c (e.g., fixed image 710) that looks photo-realistic and consistent with the lighting and material properties of the scene.

[0169] The encoder processes the scene and object images, as well as their respective albedo and normal maps, to extract meaningful features. The lighting information (L) is encoded to inform the model of the lighting conditions in the scene. A cross-attention mechanism is used to integrate the features of the object with the scene features while taking into account the lighting conditions. This ensures that the appearance of the object is adjusted to match the lighting of the scene. This enables the object features to be conditioned on both the lighting and the scene environment. Then, at 710, the integrated features F int are used to generate a synthesized image. The object mask M o is used to blend the object into the scene. The synthesis process can be represented as:

[0170] F comp = F s ⊙ (1 - M o ) + F int ⊙ M o

[0171] where denotes element-wise multiplication.

[0172] The decoder processes the synthesized features F comp to generate a final synthesized image I c: I c = G(F comp ). To ensure that the synthesized image has photo-realistic and seamless blending, one or more loss functions are employed during the training process. For example, as described above, the loss functions can include a reconstruction loss, a perceptual loss, a style loss, an illumination consistency loss (ensuring that the illumination conditions in the synthesized image are consistent with the specified illumination). In some embodiments, the total loss is a weighted sum of the above losses. The model parameters are optimized using gradient descent (e.g., Adam) to minimize this total loss.

[0173] Figure 8 is a screenshot 800 of an example user interface for editing an input image by inverse rendering according to some embodiments. The screenshot 800 can correspond to an application that allows a user to upload and edit a photo to see the effect of different materials (e.g., flooring, walls), products (e.g., furniture or decor), or other items in a home. The application needs to accurately understand the lighting and material properties of the user’s room in order to seamlessly integrate new objects or other properties. First, various embodiments receive an upload of an input image 303 in an uploaded image 802 (e.g., Figure 3 that represents the current appearance of the room.

[0174] The user can wish to change: the floor in the uploaded image 802 from a tile surface to a wooden texture, the walls in the uploaded image 802 from green to white, and the location of the light source from the left side of the room to the right side of the room. Accordingly, various embodiments receive corresponding natural language characters in a text field 804 and perform natural language processing (NLP) to generate entities or other inputs that a model (e.g., DM 308) processes to generate a material map (e.g., Figure 3 ), an illumination map, and / or an output rendered image (e.g., Figure 4 in the environment map 410). In some embodiments, such natural language characters or other user inputs (e.g., button selections) represent user inputs that are processed by the illumination map generator 104, the material map generator 102, and / or the generative model 112 as described in connection with Figure 1The DM 308 can generate, for example, a plurality of material maps as the result 806, representing certain properties of the wooden floor, such as an albedo map (representing the base color of the wooden floor, including natural variations of the wood grain and any stains or finishes. This map does not contain lighting information, only the inherent color of the material), a normal map (encoding the surface details and small-scale geometry of the wooden floor, such as the texture of the wood grain and any imperfections or patterns. This map affects the way light interacts with the surface, creating the illusion of depth and texture), a roughness map (indicating the micro-surface details of the wood, specifying the roughness or smoothness of different areas. For a glossy finish, the roughness map will show low roughness values, meaning the surface is smooth and shiny. For a more natural, matte finish, higher roughness values will be used), a specular map (defining the intensity of the wood surface’s specular reflection). This map helps simulate the shiny and reflective properties of the wood finish. For a wooden floor with a glossy finish, the specular map will have higher values where the wood reflectivity is highest), a displacement map (height map; represents the actual height variations of the wood surface. This map can be used to create physical depth during the rendering process, especially for rendering techniques that support displacement mapping. It highlights the grooves, knots, and other height variations in the wood), and an ambient occlusion map (encodes the occlusion (shadowing) effects that occur in the crevices and corners of the wooden floor. This map enhances the perception of depth and detail by darkening areas where light is naturally blocked.

[0175] Figure 9 is a screenshot 900 of an example user interface for generating a video frame by rendering according to some embodiments. The screenshot 900 can correspond to an application that allows a user to generate a video according to their natural language request. First, various embodiments receive the user input indicated in the field 904— requesting to generate a video of a room that shows the lighting conditions of the room from sunrise to sunset over a time range of 30 seconds. In some embodiments, such user input represents Figure 2 the video control 212, while the lighting from sunrise to sunset represents Figure 2 the lighting condition 204. In some embodiments, “over a time range of 30 seconds” represents an additional condition that the DM 208 receives as an additional parameter 216. Thus, upon receiving such natural language input in a particular embodiment, some embodiments generate the corresponding material map 202, lighting 204, and noise vector 206, and then the DM 208 takes all of these inputs (including the video control 212) to generate a rendered image 212 or output frame 906 that represents the user input in 904— namely, a video of the room lighting from sunrise to sunset over a time range of 30 seconds. In other words, some embodiments not only generate a single frame, but also generate a series of frames that represent a video sequence that constitutes a video component.

[0176] Figure 10 is an example process 1000 for training or fine-tuning a machine learning model according to some embodiments. In some embodiments, the method 1000 represents how to fine-tune a DM or any other model described herein. Each block of the methods 1000 (and / or 11000) described herein contains one or more processes that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be implemented by a processor executing instructions stored in memory. These methods can also be embodied as computer-usable instructions stored on computer storage media. These methods can be provided by a standalone application, a service, or a plug-in to another application (standalone or combined with other plug-ins), among others. Further, the methods 1000 (and / or 1100) are presented with reference to functional Figure 1 generative model 112 system of FIG. 11. However, these methods can additionally or alternatively be performed by any one system or any combination of systems, including but not limited to the systems described herein.

[0177] According to block 1002, some embodiments first receive pairs of renderings and / or inverse renderings containing ground truth. In the context of training a rendering / inverse rendering model, pairs in the training data represent a combination of an input and its corresponding output (representing ground truth). These pairs are useful for supervised learning, where a model learns to map an input to the correct output by minimizing the difference between the predicted output and the ground truth output. “Render pairs” include input-output (ground truth) pairs, such as a texture map-rendered image (ground truth) pair, a lighting map-rendered image pair, a user input-rendered image pair, and / or a noise-rendered image pair, where at least the rendered image represents ground truth. In certain embodiments, such pairs can be positive pairs and / or negative pairs, where a positive pair represents a particular texture map, noise vector, lighting, and / or user input that correctly belongs or matches a given ground truth rendered image. A negative pair represents a particular texture map, noise vector, lighting, and / or user input that does not correctly belong or match a given ground truth rendered image. Negative pairs can be used to ensure that the model does not overfit to incorrect attributes. The model learns to distinguish when an attribute does not match a noisy image, thereby predicting a higher error. This helps the model to be more robust and accurate when adjusting based on correct attributes. Thus, using positive and negative pairs can provide robustness by using both matching pairs (positive pairs) and non-matching pairs (negative pairs) during training.

[0178] In some embodiments, the“inverse rendering pairs” include example input-output (ground truth) pairs, such as input image-rendered image pairs (ground truth), input image-texture map pairs (ground truth), noise-texture map pairs (ground truth), and / or input image- environment map pairs (ground truth), where at least the texture map, rendered image, and / or environment map (and / or lighting map) represent ground truth.

[0179] According to block 1003, some embodiments then pre-process the pairs received at block 1002. For example, some embodiments normalize the texture maps, images, and other inputs of the pairs. Additionally or alternatively, some embodiments convert user input text into suitable embeddings using NLP models, such as WORD2VEC. For example, BERT (Bidirectional Encoder Representations from Transformers) generates contextually aware embeddings for sentences and phrases. Additionally or alternatively, particular embodiments tokenize user input (e.g., break words or other character fragments into their constituent parts, before converting into embeddings). Additionally or alternatively, images themselves can be converted from pixel form into numerical representations for further processing, such as feature vectors and / or matrices representing particular pixel values.

[0180] According to block 1004, some embodiments then extract one or more features from the pre-processed pairs. For example, as described herein, some embodiments apply a convolutional neural network (CNN) to extract feature representations from the inputs of the rendering / inverse rendering pairs. For example, an encoder of the CNN extracts features from a combined input of albedo, normal map, and noise by progressively down-sampling the input while extracting high-level features. The output of the encoder is a feature map that captures the essential information needed for the prediction at block 1006.

[0181] According to block 1006, some embodiments then generate predicted rendered images and / or maps (e.g., texture maps or lighting maps). Some embodiments employ all inputs from the rendering and / or inverse rendering pairs (and / or new inputs not in the pairs) to predict new outputs. For example, the model predicts noise (or a clean image). In the denoising task, the goal is to predict the noise added to the original clean image, or to directly predict the clean image from the noisy one. The model is trained to minimize the difference between its prediction and the ground truth.

[0182] According to block 1008, some embodiments then compute one or more losses by measuring the difference between the predicted rendered image / map and ground truth. This ensures that the predicted rendered output image (or other output, e.g., material map) conforms to the graphical properties (material map), textual description, and / or input image conditions. For example, as described herein, various embodiments compute a reconstruction loss, a conditional loss, a temporal consistency loss, and / or a combined loss (a concatenation or sum of all other losses). For example, some embodiments compute a temporal consistency loss as follows:

[0183]

[0184] The temporal consistency loss ensures that the frames generated at consecutive time steps are consistent. The model is trained to minimize the loss function containing the temporal consistency error, reconstruction error, and / or conditional constraints, thereby ensuring that the generated images (or other outputs, e.g., material maps) are realistic and conform to the specified conditions, e.g., textual and material map conditions.

[0185] According to block 1010, some embodiments then proceed with backpropagation, as described herein. Backpropagation refers to the use of an optimizer to compute gradients and update model parameters (e.g., training the model through multiple additional epochs). For example, some embodiments compute the gradient of the total loss with respect to the model parameters by updating the model parameters (e.g., weights and biases) using an optimizer (e.g., Adam) to minimize the total loss. In other words, each layer of the neural network (or node of the neural network) applies a linear transformation (weights and biases) to the input data, followed by an (e.g., non-linear) activation function. This process yields a set of outputs, which are then used to compute the loss. The difference between the predicted output and the ground truth is computed using a loss function (e.g., mean squared error, cross-entropy).

[0186] The loss value quantifies how well the model’s prediction matches the actual value. The loss is backpropagated through the network to compute the gradients. The gradients represent the partial derivatives of the loss with respect to each parameter (weights and biases) in the network. If the neuron’s input (after applying the weights and adding the bias) yields a high value, then the neuron is activated (e.g., through an activation function such as ReLU, Sigmoid, and / or Tanh). If the input yields a low or negative value (depending on the activation function), then the neuron is inhibited (i.e., its output is zero or close to zero). The optimizer adjusts the weights based on the gradients computed during the backpropagation process.

[0187] Figure 11is a flowchart of an example process 1100 for generating an output frame or map according to some embodiments. According to block 1103, some embodiments receive at least one of the following: a first set of one or more material maps, a first set of one or more lighting maps, and / or an input frame. The one or more material maps define how one or more properties vary across the surface of one or more objects. The one or more lighting maps represent at least one of one or more shading or lighting characteristics associated with the one or more objects. In some embodiments, any of the material maps described herein include at least one of the following: an albedo map, a normal map, a roughness map, a metallic map, an ambient occlusion map, a displacement map, a specular map, an emissive map, an opacity map, a cavity map, or a subsurface scattering map.

[0188] In one or more embodiments, the albedo map represents the base color of a surface without any shading or lighting effects. It captures the inherent color of the material. For example, for a wooden surface, it exhibits the natural wood grain and color. It does not contain any reflections, shadows, or highlights. The normal map encodes the direction of surface normals, which are vectors that are perpendicular to the surface. This map is used to create the illusion of complex surface details without adding extra geometry. It influences the interaction of light with the surface, creating intricate textures such as bumps, grooves, and wrinkles. The RGB values in the normal map represent the XYZ components of the normal vector. The roughness map indicates the micro-surface texture of the material, determining how rough or smooth the surface appears. This map controls the reflectivity of the surface. Lower roughness values represent smooth, shiny surfaces (e.g., polished metal), while higher roughness values represent rough, matte surfaces (e.g., untreated wood). The metallic map specifies whether the surface is metallic or non-metallic (dielectric). This metallic map uses binary or grayscale values to distinguish between metallic and non-metallic areas. Metallic surfaces reflect light differently than non-metallic surfaces. Pure metals have high reflectivity and are electrically conductive, while dielectrics do not have high reflectivity and are not electrically conductive.

[0189] The ambient occlusion map encodes the shading or shadowing effects in small crevices and corners of the surface. This map simulates the soft shadows that appear in areas where ambient light is occluded or blocked. It enhances the perception of depth and detail by darkening these occluded areas, thereby increasing the depth and realism of the scene. The displacement (height) map represents the actual height variations of the surface. Unlike the normal map, which only affects lighting, the displacement map modifies the geometry of the surface itself. This map is used to create actual geometric details, such as deep crevices or raised surfaces, by displacing vertices along the normal direction based on the height values.

[0190] A specular map defines the intensity and color of the specular reflection. It determines the brightness and color of the specular highlights (the reflection of the light source on the surface). An emissive map specifies the areas of the surface that emit light. This map is used to make certain parts of the material glow, independent of external lighting. The emissive map defines the color and intensity of the emitted light and can be used to create effects such as glowing screens, lights, or other emissive surfaces.

[0191] An opacity map (Alpha map) defines the transparency of the surface. This map uses grayscale values to indicate the transparency or opacity of the surface. White areas represent full opacity, black areas represent full transparency, and gray shades represent partial transparency. This is useful for materials such as glass, curtains, or leaves. A cavity map highlights small cracks and cavities on the surface. Similar to ambient occlusion, but usually for smaller details. It enhances fine details by darkening the cavities and increasing the visual depth of the surface. A subsurface scattering map controls the way light penetrates and scatters within a translucent material. This map is used for materials such as skin, wax, or marble, where light enters the surface, scatters underneath, and exits from different points. It helps to simulate the soft diffused appearance characteristic of such materials.

[0192] Some embodiments receive a natural language user input requesting to incorporate at least one of a material property or a lighting condition into an output frame. For example, the fields 804 and 904 in Figure 8 and Figure 9 or the user-specified lighting 504 in Figure 5 describe examples of such scenarios. And, based at least in part on the natural language user input, some embodiments generate at least one of one or more material maps or one or more lighting maps, where the output frame is generated based at least in part on the natural language user input, e.g., as described in Figure 5 .

[0193] According to block 1105, some embodiments provide a representation (e.g., tokenized user input, vector, or matrix) of at least one of a noise vector, a first material map, a lighting map, or an input frame to a first machine learning model to generate an output, where the output includes at least one of an output frame, a second material map, or a second lighting map. Examples of block 1105 are directed to Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 and Figure 7The pipelines described in the Background are described. For example, some embodiments provide a first noise vector (e.g., 206) and a representation of at least one of one or more texture maps (e.g., 202) or one or more lighting maps (e.g., 204) as input to one or more first machine learning models (e.g., DM 208) to generate an output frame (e.g., rendered image 210). A “representation” of any payload (e.g., input frame, texture map, or noise vector) described herein represents the payload itself (e.g., all pixel values of an input image or map) or some other encoded or numerical representation of the payload, such as a vector, matrix, etc.

[0194] The first noise vector corresponds to an initial starting point for a diffusion process performed by the one or more first machine learning models. Diffusion models are a type of generative model that learns to generate data (e.g., images) by reversing a diffusion process. This involves starting with a noise vector and progressively denoising it to produce a final coherent image. The process can be understood as follows: a forward diffusion adds noise to data (e.g., texture map 202 and / or input image 303) in multiple steps, gradually transforming the data into pure noise. A reverse diffusion trains a model to reverse the forward process, starting with pure noise and iteratively refining it to recover the original data. The noise vector is the initial input to the reverse diffusion process. It is a randomly generated tensor that serves as a starting point for the model to generate a final image (e.g., rendered image 210).

[0195] A “frame” as described herein refers to a single video frame, digital image (e.g., digital photograph), or other unit / format of digital media. A video frame is a single still image in a sequence of consecutive images that, when played back quickly in succession, creates the appearance of motion or video.

[0196] As described herein, various embodiments perform various combinations of calculations. For example, some embodiments receive an input frame (e.g., 303) and a second noise vector (e.g., 306). Some embodiments then provide a representation of the second noise vector and the input frame and provide it as input to one or more second machine learning models (e.g., DM 308) to generate one or more first texture maps (e.g., texture maps 502), e.g., as described in the Background. Figure 3

[0197] Some embodiments provide an input frame (e.g., 703) as input to one or more first machine learning models (e.g., DM 708), and wherein the first noise vector represents a noisy version of the input frame. Some embodiments then receive a request to enhance the input frame (e.g., as described in the Background). Figure 8 ​(as shown in field 804) (e.g., providing image style transfer, image inpainting, object insertion, Sim2Real, etc.). The output frame (e.g., 710) is generated at least in part based on the request and by taking the input frame as input to one or more first machine learning models. The output frame includes one or more features enhanced relative to the input frame, for example, as referenced. Figure 7 and Figure 8 As stated above.

[0198] Some embodiments provide a two-dimensional input frame (e.g., 503) and a second noise vector (e.g., 506) as input to one or more second machine learning models (e.g., DM 508). And, based at least on the two-dimensional input frame and the second noise vector as input to one or more second machine learning models, some embodiments generate one or more material maps (e.g., material map 502). Thus, the output frame (e.g., 510) represents the two-dimensional input frame, except that the lighting properties in the output frame are modified relative to the input frame, for example, as... Figure 5 As described in [the text].

[0199] Some embodiments provide a two-dimensional input frame (e.g., 603) and a second noise vector (e.g., noise vector 606) as input to one or more second machine learning models (e.g., DM 608). Based at least in part on using the input frame and the second noise vector as input to one or more second machine learning models, some embodiments generate one or more second material maps (e.g., albedo map 602), for example, as per [reference to...]. Figure 6 As described in 603, 606, 608 and 602.

[0200] Some embodiments generate multidimensional frames based on the generation of one or more second material maps (e.g., albedo map 602). Figure 6 The 3D proxy scene 609). A multidimensional frame represents a two-dimensional input frame (e.g., input image 603), the difference being that the multidimensional frame has at least one more dimension than the two-dimensional input frame. Some embodiments then generate one or more first material maps (e.g., based at least in part on the generation of the multidimensional frame). Figure 6 The material map 610 in the image is then used. The output frame (e.g., the relit image 610) is then generated, at least in part, based on the generation of the multi-dimensional frame. This type of functionality will be combined with... Figure 6 Describe it.

[0201] Example autonomous vehicles

[0202] Figure 12Ais a diagram of an example autonomous vehicle 1200 according to some embodiments of the present disclosure. Autonomous vehicle 1200 (alternatively referred to herein as “vehicle 1200”) can include, but is not limited to, a passenger vehicle such as a car, truck, bus, first responder vehicle, shuttle, electric or motorized bicycle, motorcycle, fire vehicle, police vehicle, ambulance, boat, construction vehicle, underwater vessel, robotic vehicle, drone, airplane, vehicle coupled to a trailer (e.g., a semi-tractor-trailer truck for hauling cargo), and / or another type of vehicle (e.g., a self-driving and / or passenger-holding vehicle). Autonomous vehicles are generally described in terms of levels of automation as defined by a department of the United States Department of Transportation, the National Highway Traffic Safety Administration (NHTSA), and the Society of Automotive Engineers (SAE) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806 published June 15, 2018, Standard No. J3016-201609 published September 30, 2016, and prior and future versions of this standard). Vehicle 1200 can be capable of functionality that meets one or more of Levels 3-5 of autonomous driving. For example, depending on the embodiment, vehicle 1200 can be capable of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). The term “autonomous” as used here can include any and / or all types of autonomy of vehicle 1200 or other machines, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, providing assistance autonomous, semi-autonomous, primarily autonomous, or other designations.

[0203] Vehicle 1200 can include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. Vehicle 1200 can include a propulsion system 1250 such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. Propulsion system 1250 can be connected to a drivetrain of vehicle 1200 that can include a transmission to effect propulsion of vehicle 1200. Propulsion system 1250 can be controlled in response to receiving a signal from a throttle / accelerator 1252.

[0204] A steering system 1254, which can include a steering wheel, can be used to steer the vehicle 1200 (e.g., along a desired path or route) while the propulsion system 1250 is operating (e.g., while the vehicle is in motion). The steering system 1254 can receive signals from a steering actuator 1256. For full automation (Level 5) functionality, the steering wheel can be optional.

[0205] A brake sensor system 1246 can be used to operate the vehicle brakes in response to receiving signals from a brake actuator 1248 and / or a brake sensor.

[0206] One or more controllers 1236, which can include one or more system on a chip (SoC) 1204 Figure 12C ) and / or one or more GPUs, can provide signals (e.g., representing commands) to one or more components and / or systems of the vehicle 1200. For example, the one or more controllers can send signals to operate the vehicle brakes via one or more brake actuators 1248, to operate the steering system 1254 via one or more steering actuators 1156, to operate the propulsion system 1250 via one or more throttle / accelerator 1252. The one or more controllers 1236 can include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in driving the vehicle 1200. The one or more controllers 1236 can include a first controller 1236 for autonomous driving functionality, a second controller 1236 for functional safety functionality, a third controller 1236 for artificial intelligence functionality (e.g., computer vision), a fourth controller 1236 for infotainment functionality, a fifth controller 1236 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1236 can handle two or more of the above functionalities, two or more controllers 1236 can handle a single functionality, and / or any combination thereof.

[0207] One or more controllers 1236 can provide signals for controlling one or more components and / or systems of vehicle 1200 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data can be received from, for example and without limitation, a global navigation satellite system (“GNSS”) sensor 1258 (e.g., a global positioning system sensor), a RADAR sensor 1260, an ultrasonic sensor 1262, a LiDAR sensor 1264, an inertial measurement unit (IMU) sensor 1266 (e.g., an accelerometer, a gyroscope, a magnetic compass, a magnetometer, etc.), a microphone 1296, a stereo camera 1268, a wide-angle camera 1270 (e.g., a fisheye camera), an infrared camera 1272, a surround camera 1274 (e.g., a 360-degree camera), a long and / or medium range camera 1298, a speed sensor 1244 (e.g., to measure the speed of vehicle 1200), a vibration sensor 1242, a steering sensor 1240, a brake sensor (e.g., as part of brake sensor system 1246), one or more occupant monitoring system (OMS) sensors 1201 (e.g., one or more interior sensors), and / or other sensor types.

[0208] One or more of controllers 1236 can receive input from instrument cluster 1232 of vehicle 1200 (e.g., represented by input data) and provide output (e.g., represented by output data, display data, etc.) via human-machine interface (HMI) display 1234, audible annunciators, speakers, and / or via other components of vehicle 1200. These outputs can include information such as vehicle speed, velocity, time, map data (e.g., a high-definition (“HD”) map 1222 of Figure 12C

[0209] ​The vehicle 1200 also includes a network interface 1224 that can communicate over one or more networks using one or more wireless antennas 1226 and / or modems. For example, the network interface 1224 can be capable of communicating over Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multi-Carrier (“CDMA2000”), and / or the like. The one or more wireless antennas 1226 can also enable communication between objects (e.g., vehicles, mobile devices, and / or the like) in the implementation environment using one or more local area networks such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, and / or the like and / or one or more low power wide area networks (“LPWANs”) such as LoRaWAN, SigFox, and / or the like.

[0210] Figure 12B For example autonomous vehicle 1200 for Figure 12A An example of camera locations and fields of view for the example autonomous vehicle 1200 according to some embodiments of the present disclosure. The cameras and respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras can be included and / or these cameras can be located at different locations on the vehicle 1200.

[0211] Camera types for the cameras can include, but are not limited to, digital cameras that can be suitable for use with components and / or systems of the vehicle 1200. The cameras can operate at Automotive Safety Integrity Level (ASIL) B and / or at another ASIL. The camera types can have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, and / or the like, depending on the embodiment. The cameras can be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, a color filter array can include a Red-White-White-White (RCCC) color filter array, a Red-White-White-Blue (RCCB) color filter array, a Red-Blue-Green-White (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras such as cameras with a

[0212] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more of the cameras (e.g., all of the cameras) can simultaneously record and provide image data (e.g., video).

[0213] One or more of the cameras can be mounted in mounting assemblies such as custom designed (three-dimensional ("3D") printed) assemblies to cut off stray light and reflections from within the car (e.g., reflections from the dashboard reflected in the windshield mirror) that can interfere with the image data capture capabilities of the cameras. With respect to wing mirror mounting assemblies, the wing mirror assemblies can be custom 3D printed such that the camera mounting plates match the shape of the wing mirrors. In some examples, one or more cameras can be integrated into the wing mirrors. For side view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0214] Cameras with fields of view that include the portion of the environment in front of the vehicle 1200 (e.g., front-facing cameras) can be used for surround view to help identify the forward path and obstacles, and to assist in providing information critical to generating an occupancy grid and / or determining a preferred vehicle path with the help of one or more controllers 1236 and / or control SoCs. Front-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. Front-facing cameras can also be used for ADAS functions and systems including lane departure warning ("LDW"), adaptive cruise control ("ACC"), and / or other functions such as traffic sign recognition.

[0215] A wide variety of cameras can be used in the front-facing configuration, including, for example, monocular camera platforms including complementary metal-oxide-semiconductor ("CMOS") color imagers. Another example can be a wide-angle camera 1270, which can be used to perceive objects (e.g., pedestrians, intersection traffic, or bicycles) entering the field of view from the periphery. Although Figure 12B Although only one wide-angle camera is illustrated in FIG. 12, there can be any number (including zero) of wide-angle cameras 1270 on the vehicle 1200. In addition, long-range cameras 1298 (e.g., pairs of long-view stereo cameras) can be used for depth-based object detection, especially for objects for which a neural network has not been trained. Long-range cameras 1298 can also be used for object detection and classification and basic object tracking.

[0216] Any number of stereo cameras 1268 can also be included in the front-facing configuration. In at least one embodiment, one or more stereo cameras 1268 can include an integrated control unit that includes a scalable processing unit that can provide a multi-core microprocessor with integrated Controller Area Network (“CAN”) or Ethernet interfaces and a field programmable gate array (“FPGA”) on a single chip. Such a unit can be used to generate a 3D map of the vehicle’s environment, including distance estimates for all points in the image. Alternative stereo cameras 1268 can include a compact stereo vision sensor that can include two camera lenses (one on the left and one on the right) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1268 can be used in addition to or instead of those described herein.

[0217] Cameras with fields of view that include portions of the environment to the side of the vehicle 1200 (e.g., side-view cameras) can be used for surround view, providing information used to create and update the occupancy grid and to generate side-crash collision warnings. For example, surround cameras 1274 (e.g., four surround cameras 1274 as shown in FIG. 15B) can be placed on the vehicle 1200. The surround cameras 1274 can include wide-view cameras 1270, fisheye cameras, 360-degree cameras, and / or the like. In one example, four fisheye cameras can be placed on the front, back, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 1274 (e.g., left, right, and back), and can utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround view camera. Figure 12B

[0218] Cameras with fields of view that include portions of the environment to the rear of the vehicle 1200 (e.g., rear-view cameras) can be used to assist with parking, surround view, rear collision warnings, and to create and update the occupancy grid. A wide variety of cameras can be used, including but not limited to cameras that are also suitable as front-facing cameras (e.g., long- and / or mid-range cameras 1298, stereo cameras 1268, infrared cameras 1272, etc.) as described herein.

[0219] ​Cameras with fields of view that include portions of the interior environment within the cabin of vehicle 1200 (e.g., one or more OMS sensors 1201) can be used as part of an occupant monitoring system (OMS), such as but not limited to a driver monitoring system (DMS). For example, OMS sensors (e.g., OMS sensors 1201) can be used to track (e.g., by controller 1236) the gaze direction, head pose, and / or blinks of an occupant and / or driver. This gaze information can be used to determine the level of attention of the occupant or driver (e.g., detect drowsiness, fatigue, and / or distraction), and / or take responsive action to prevent injury to the occupant or operator. In some embodiments, data from OMS sensors can be used to implement gaze-controlled operations triggered by the driver and / or non-driver occupants, such as but not limited to adjusting cabin temperature and / or airflow, opening and closing windows, controlling cabin lighting, controlling an entertainment system, adjusting a rearview mirror, adjusting a seat position, and / or other operations. In some embodiments, OMS can be used for applications such as determining when objects and / or occupants are left behind in the cabin (e.g., by detecting the presence of an occupant after the driver has exited the vehicle).

[0220] Figure 12C An example system architecture for an example autonomous vehicle 1200 in accordance with some embodiments of the present disclosure. Figure 12A A block diagram of an example system architecture for an example autonomous vehicle 1200 in accordance with some embodiments of the present disclosure. It should be understood that this arrangement and other arrangements described herein are set forth merely for illustration. Arrangements other than those shown, and elements of one arrangement can be incorporated into other arrangements, as readily understood by one of ordinary skill in the art. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combinations and locations. Various functions described herein as being performed by an entity can be implemented in hardware, firmware, and / or software. For instance, various functions can be implemented by a processor executing instructions stored in memory.

[0221] Figure 12CEach of the components, features, and systems of the vehicle 1200 are illustrated as being connected via a bus 1202. The bus 1202 can include a controller area network (CAN) data interface (alternatively referred to herein as a "CAN bus"). The CAN can be a network within the vehicle 1200 that is used to assist in controlling various features and functions of the vehicle 1200, such as the driving of brakes, acceleration, braking, steering, windshield wipers, and so forth. The CAN bus can be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, revolutions per minute (RPM) of the engine, button positions, and / or other vehicle status indicators. The CAN bus can be ASIL B compliant.

[0222] Although the bus 1202 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet can be used in addition to or alternatively from the CAN bus. Further, although the bus 1202 is represented with a single line, this is not intended to be limiting. For example, there can be any number of buses 1202, which can include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses that use different protocols. In some examples, two or more buses 1202 can be used to perform different functions, and / or can be used for redundancy. For example, a first bus 1202 can be used for collision avoidance functions, and a second bus 1202 can be used for drive control. In any example, each bus 1202 can communicate with any component of the vehicle 1200, and two or more buses 1202 can communicate with the same components. In some examples, each SoC 1204, each controller 1236, and / or each computer within the vehicle can have access to the same input data (e.g., inputs from sensors of the vehicle 1200), and can be connected to a common bus, such as a CAN bus.

[0223] The vehicle 1200 can include one or more controllers 1236, such as those described herein with respect to Figure 12A The controllers 1236 can be used for a wide variety of functions. The controllers 1236 can be coupled to any of the other various components and systems of the vehicle 1200, and can be used for control of the vehicle 1200, artificial intelligence of the vehicle 1200, infotainment for the vehicle 1200, and / or the like.

[0224] The vehicle 1200 can include one or more system on a chip (SoC) 1204. The SoC 1204 can include a CPU 1206, a GPU 1208, a processor 1210, a cache 1212, an accelerator 1214, a data store 1216, and / or other components and features not illustrated. The SoC 1204 can be used to control the vehicle 1200 in a wide variety of platforms and systems. For example, one or more SoC 1204 can be used in a system (e.g., a system of the vehicle 1200) in conjunction with an HD map 1222 that can obtain map refreshes and / or updates from one or more servers (e.g., the one or more servers 1278) via a network interface 1224. Figure 12D

[0225] The CPU 1206 can include a CPU cluster or CPU complex (alternatively referred to herein as a “CCPLEX”). The CPU 1206 can include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 1206 can include eight cores in a coherent multi-processor configuration. In some embodiments, the CPU 1206 can include four dual-core clusters with each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). The CPU 1206 (e.g., the CCPLEX) can be configured to support simultaneous cluster operation such that any combination of clusters of the CPU 1206 can be active at any given time.

[0226] The CPU 1206 can implement power management capabilities including one or more of the following features: individual hardware blocks can be automatically clock-gated when idle to save dynamic power; each core clock can be gated when the core is not actively executing instructions due to execution of WFI / WFE instructions; each core can be independently power-gated; each core cluster can be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster can be independently power-gated when all cores are power-gated. The CPU 1206 can further implement enhanced algorithms for managing power states in which the allowed power states and desired wake-up times are specified and the hardware / microcode determines the optimal power state for the cores, clusters, and CCPLEX to enter. The processing cores can support a simplified power state entry sequence in software, with the work being offloaded to microcode.

[0227] ​GPU 1208 can include an integrated GPU (alternatively referred to herein as an “iGPU”). GPU 1208 can be programmable and efficient for parallel workloads. In some examples, GPU 1208 can use an enhanced tensor instruction set. GPU 1208 can include one or more streaming microprocessors, where each streaming microprocessor can include an LI cache (e.g., an LI cache having at least 96 KB of storage capacity), and two or more of the streaming microprocessors can share an L2 cache (e.g., an L2 cache having 512 KB of storage capacity). In some embodiments, GPU 1208 can include at least eight streaming microprocessors. GPU 1208 can use a compute application programming interface (API). In addition, GPU 1208 can use one or more parallel computing platforms and / or programming models (e.g., NVIDIA’s CUDA).

[0228] In the case of automotive and embedded uses, GPU 1208 can be power-optimized for best performance. For example, GPU 1208 can be fabricated on a fin field effect transistor (FinFET). However, this is not intended to be limiting, and GPU 1208 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a thread warp scheduler, a dispatch unit, and / or a 64 KB register file. In addition, the streaming microprocessor can include independent parallel integer and floating point data paths to exploit the mixed provision of computation and address computation for efficient execution of workloads. The streaming microprocessor can include independent thread scheduling capabilities to allow for more fine-grained synchronization and cooperation between parallel threads. The streaming microprocessor can include a combined LI data cache and shared memory unit to improve performance while simplifying programming.

[0229] GPU 1208 can include a high bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem that provides approximately 900 GB / s of peak memory bandwidth in some examples. In some examples, in addition to or alternatively from HBM memory, synchronous graphics random access memory (SGRAM) can be used, such as fifth generation graphics double data rate synchronous random access memory (GDDR5).

[0230] GPU 1208 can include a unified memory technology that includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, improving efficiency of memory ranges shared between processors. In some examples, address translation services (ATS) support can be used to allow GPU 1208 to access CPU 1206 page tables directly. In such examples, when a GPU 1208 memory management unit (MMU) experiences a miss, an address translation request can be transmitted to CPU 1206. In response, CPU 1206 can look up a virtual-to-physical mapping for the address in its page tables and transmit the translation back to GPU 1208. In this way, the unified memory technology can allow a single unified virtual address space for memory of both CPU 1206 and GPU 1208, simplifying GPU 1208 programming and porting applications to GPU 1208.

[0231] Further, GPU 1208 can include access counters that can track how frequently GPU 1208 accesses other processors’ memory. The access counters can help ensure that memory pages are migrated to the physical memory of the processor that accesses these pages most frequently.

[0232] SoC 1204 can include any number of caches 1212, including those described herein. For example, caches 1212 can include an L3 cache available to both CPU 1206 and GPU 1208 (e.g., connected to both CPU 1206 and GPU 1208). Caches 1212 can include a write-back cache that can track the state of a line, for example, by using a cache coherency protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache can include 4 MB or more, although smaller cache sizes can also be used.

[0233] SoC 1204 can include an arithmetic logic unit (ALU) that can be utilized in processing to perform any of a variety of tasks or operations with respect to vehicle 1200, such as processing a DNN. Further, SoC 1204 can include a floating point unit (FPU) (or other mathematical co-processor or digital co-processor type) for performing mathematical operations within the system. For example, SoC 104 can include one or more FPUs integrated as execution units within CPU 1206 and / or GPU 1208.

[0234] The SoC 1204 can include one or more accelerators 1214 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC 1204 can include a hardware accelerator cluster that can include optimized hardware accelerators and / or a large on-chip memory. The large on-chip memory (e.g., 4MB SRAM) can enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster can be used to supplement the GPU 1208 and offload some of the tasks of the GPU 1208 (e.g., freeing up more cycles of the GPU 1208 for performing other tasks). As one example, the accelerators 1214 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to be easily controlled for acceleration. As used herein, the term “CNN” can include all types of CNNs, including region-based or region with convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0235] The accelerators 1214 (e.g., hardware accelerator cluster) can include a deep learning accelerator (DLA). The DLA can include one or more tensor processing units (TPUs) that can be configured to provide an additional 100 billion operations per second for deep learning applications and inferencing. The TPUs can be accelerators that are configured to perform and optimized for performing image processing functions (e.g., for CNNs, RCNNs, etc.). The DLA can be further optimized for a specific set of neural network types and floating point operations and inferencing. The design of the DLA can provide higher performance per mm than general purpose GPUs and far exceeds the performance of CPUs. The TPUs can perform several functions including single instance convolution functions, support INT8, INT16, and FP16 data types for both features and weights, for example, and post-processor functions.

[0236] The DLA can perform neural networks, especially CNNs, on processed or unprocessed data for any of a wide variety of functions quickly and efficiently, such as and not limited to: CNNs for object recognition and detection using data from a camera sensor; CNNs for distance estimation using data from a camera sensor; CNNs for emergency vehicle detection and identification and detection using data from a microphone; CNNs for face recognition and vehicle owner identification using data from a camera sensor; and / or CNNs for safety and / or safety related events.

[0237] The DLA can perform any of the functions of the GPU 1208, and by using an inference accelerator, for example, a designer can target the DLA or GPU 1208 for any function. For example, a designer can focus the processing of CNNs and floating point operations on the DLA, and leave other functions to the GPU 1208 and / or other accelerators 1214.

[0238] The accelerators 1214 (e.g., a cluster of hardware accelerators) can include a programmable vision accelerator (PVA), which can be alternatively referred to herein as a computer vision accelerator. The PVA can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA can include, for example and without limitation, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0239] The RISC cores can interact with image sensors (e.g., image sensors of any of the cameras described herein), image signal processors, and / or the like. Each of these RISC cores can include any number of memories. Depending on the embodiment, the RISC cores can use any of several protocols. In some examples, the RISC cores can execute a real-time operating system (RTOS). The RISC cores can be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores can include an instruction cache and / or a tightly coupled RAM.

[0240] The DMA can enable components of the PVA to access system memory independently of the CPU 1206. The DMA can support any number of features to provide optimizations to the PVA, including, but not limited to, supporting multi-dimensional addressing and / or circular addressing. In some examples, the DMA can support addressing up to six or more dimensions, which can include block width, block height, block depth, horizontal block stride, vertical block stride, and / or depth stride.

[0241] The vector processors can be programmable processors that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, a PVA can include a PVA core and two vector processing subsystem partitions. The PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystems can operate as the main processing engines of the PVA and can include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core can include a digital signal processor such as, for example, a single instruction multiple data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and rate.

[0242] Each of the vector processors can include an instruction cache and can be coupled to a dedicated memory. As a result, in some examples, each of the vector processors can be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA can be configured to employ data parallelization. For example, in some embodiments, multiple vector processors included in a single PVA can execute the same computer vision algorithm but on different regions of an image. In other examples, the vector processors included in a particular PVA can execute different computer vision algorithms on the same image simultaneously, or even different algorithms on a sequence of images or portions of an image. Among other things, any number of PVAs can be included in a hardware accelerator cluster, and any number of vector processors can be included in each of those PVAs. Furthermore, the PVAs can include additional error-correcting code (ECC) memory to enhance overall system security.

[0243] The accelerator 1214 (e.g., hardware accelerator cluster) can include an on-chip computer vision network and SRAM to provide high bandwidth, low latency SRAM for the accelerator 1214. In some examples, the on-chip memory can include at least 4 MB of SRAM composed of, for example and without limitation, eight field programmable memory blocks, which can be accessed by both the PVA and the DLA. Each pair of memory blocks can include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and the DLA can access the memory via a backbone that provides high-speed memory access to the PVA and the DLA. The backbone can include an on-chip computer vision network that interconnects the PVA and the DLA to the memory (e.g., using APB).

[0244] The on-chip computer vision network can include an interface that determines that both the PVA and the DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst communications for continuous data transmission. This type of interface can comply with ISO 26262 or IEC 61508 standards, although other standards and protocols can be used.

[0245] In some examples, the SoC 1204 can include a real-time ray tracing hardware accelerator, such as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine locations and extents of objects (e.g., within a world model) in order to generate real-time visualizations simulations for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general wave propagation simulation, for comparison with LiDAR data for purposes of localization and / or other functions, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) can be used to perform one or more ray tracing related operations.

[0246] The accelerator 1214 (e.g., hardware accelerator cluster) has a wide range of autonomous driving uses. The PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithm domains that require predictable processing, low power, and low latency. Thus, the PVA performs well on semi-dense or dense regular computations, and even on small data sets that require predictable runtimes with low latency and low power. Thus, in the context of a platform for autonomous vehicles, the PVA is designed to run classical computer vision algorithms as they are effective at object detection and integer math operations.

[0247] For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. In some examples, a semi-global matching based algorithm can be used, although this is not intended to be limiting. Many applications for level 3-5 autonomous driving require instant motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions on inputs from two monocular cameras.

[0248] In some examples, the PVA can be used to perform dense optical flow. According to a process, raw RADAR data is processed (e.g., using a 4D fast Fourier transform) to provide processed RADAR. In other examples, the PVA is used for time-of-flight depth processing, such as by processing raw time-of-flight data to provide processed time-of-flight data.

[0249] The DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence measure for each object detection. Such a confidence value can be interpreted as a probability, or as providing a relative "weight" for each detection compared to other detections. The confidence value enables the system to make further decisions about which detections should be considered true positive detections and not false positive detections. For example, the system can set a threshold for confidence, and only consider detections above the threshold to be true positive detections. In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform an emergency brake, which is obviously undesirable. Thus, only the most confident detections should be considered a trigger for AEB. The DLA can run a neural network for regression of a confidence value. The neural network can take as its input at least some subset of parameters, such as a bounding box dimension, a ground plane estimate obtained (e.g., from another subsystem), inertial measurement unit (IMU) sensor 1266 outputs related to vehicle 1200 orientation, distance, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 1264 or RADAR sensor 1260), etc.

[0250] SoC 1204 can include one or more data stores 1216 (e.g., memory). Data stores 1216 can be on-chip memory of SoC 1204, which can store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and safety, data stores 1216 can be large enough in capacity to store multiple instances of a neural network. Data stores 1212 can include L2 or L3 cache 1212. References to data stores 1216 can include references to memory associated with PVAs, DLAs, and / or other accelerators 1214 as described herein.

[0251] SoC 1204 can include one or more processors 1210 (e.g., embedded processors). The processors 1210 can include a boot and power management processor, which can be a specialized processor and subsystem for handling boot power and management functions and related security implementations. The boot and power management processor can be part of the SoC 1204 boot sequence and can provide run-time power management services. The boot power and management processor can provide clock and voltage programming, auxiliary system low power state transitions, SoC 1204 thermal and temperature sensor management, and / or SoC 1204 power state management. Each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC 1204 can use the ring oscillator to detect the temperature of the CPU 1206, GPU 1208, and / or accelerator 1214. If it is determined that the temperature exceeds a threshold, the boot and power management processor can enter a temperature fault routine and place the SoC 1204 in a lower power state and / or place the vehicle 1200 in a driver safe park mode (e.g., safely park the vehicle 1200).

[0252] The processors 1210 can also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio over multiple interfaces and a range of widely flexible audio I / O interfaces. In some examples, the audio processing engine is a specialized processor core with a digital signal processor with dedicated RAM.

[0253] The processors 1210 can also include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. The always-on processor engine can include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0254] The processors 1210 can also include a security cluster engine that includes a specialized processor subsystem that handles security management for automotive applications. The security cluster engine can include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In a secure mode, the two or more cores can operate in a lockstep mode and act as a single core with comparison logic that detects any differences between their operations.

[0255] The processors 1210 can also include a real-time camera engine that can include a specialized processor subsystem for handling real-time camera management.

[0256] The processor 1210 can further include a high dynamic range signal processor, which can include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0257] The processor 1210 can include a video image compositor, which can be a processing block (e.g., implemented on a microprocessor), that implements video post-processing functions needed by the video playback application to produce the final image for the player window. The video image compositor can perform lens distortion correction on the wide-angle camera 1270, surround camera 1274, and / or on the cab-in monitor camera sensors. The cab-in monitor camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to recognize cab-in events and respond accordingly. The cab-in system can perform lip reading to activate mobile phone services and place a call, dictate an email, change the vehicle destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode, and are disabled otherwise.

[0258] The video image compositor can include enhanced temporal noise reduction for spatial and temporal noise reduction. For example, where motion is present in the video, the noise reduction appropriately weights the spatial information, reducing the weight of information provided by neighboring frames. Where the image or portions of the image do not include motion, the temporal noise reduction performed by the video image compositor can use information from previous images to reduce noise in the current image.

[0259] The video image compositor can also be configured to perform stereo correction on input stereo lens frames. The video image compositor can further be used for user interface composition when the operating system desktop is in use and the GPU 1208 does not need to continuously render new surfaces. Even when the GPU 1208 is powered on and active, doing 3D rendering, the video image compositor can be used to offload the GPU 1208 to improve performance and responsiveness.

[0260] The SoC 1204 can further include a Mobile Industry Processor Interface (MIPI) camera serial interface for receiving video and input from cameras, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions. The SoC 1204 can further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a particular role.

[0261] The SoC 1204 can also include a wide range of peripheral device interfaces to enable communication with peripherals, audio codecs, power management, and / or other devices. The SoC 1204 can be used to process data from cameras (connected over Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LiDAR sensor 1264, RADAR sensor 1260, etc. that can be connected over Ethernet), data from the bus 1202 (e.g., speed of the vehicle 1200, steering wheel position, etc.), data from GNSS sensor 1258 (connected over Ethernet or CAN bus). The SoC 1204 can also include dedicated high-performance mass storage controllers, which can include their own DMA engines, and which can be used to free up the CPU 1206 from routine data management tasks.

[0262] The SoC 1204 can be an end-to-end platform with a flexible architecture that spans automation levels 3-5, providing an integrated functional safety architecture for a platform that leverages and efficiently uses computer vision and ADAS technology to achieve diversity and redundancy, along with deep learning tools. The SoC 1204 can be faster, more reliable, and even more energy and space efficient than conventional systems. For example, the accelerators 1214, when combined with the CPU 1206, GPU 1208, and data storage 1216, can provide a fast and efficient platform for level 3-5 autonomous vehicles.

[0263] The technology thus provides capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using high-level programming languages such as the C programming language to perform a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to, for example, execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for on-board ADAS applications and for practical level 3-5 autonomous vehicles.

[0264] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a cluster of hardware accelerators, the technology described herein allows multiple neural networks to be executed simultaneously and / or sequentially, and the results to be combined together to achieve level 3-5 autonomous driving functionality. For example, a CNN executed on a DLA or dGPU (e.g., GPU 1220) can include text and word recognition, allowing a supercomputer to read and understand traffic signs, including signs for which a neural network has not been specifically trained. The DLA can also include a neural network that is able to recognize, interpret, and provide a semantic understanding of the sign, and pass that semantic understanding to a path planning module running on the CPU complex.

[0265] As another example, multiple neural networks can be run simultaneously as required for level 3, 4, or 5 driving. For example, a warning sign consisting of the words "Caution: flashing lights indicate icy conditions" along with electric lights can be interpreted by several neural networks independently or collectively. The sign itself can be recognized by a first deployed neural network (e.g., a trained neural network) as a traffic sign, the text "flashing lights indicate icy conditions" can be interpreted by a second deployed neural network that informs the vehicle's path planning software (preferably executing on the CPU complex) that icy conditions exist when flashing lights are detected. The flashing lights can be recognized by operating a third deployed neural network over multiple frames that informs the vehicle's path planning software of the presence (or absence) of flashing lights. All three neural networks can be run simultaneously, for example, within the DLA and / or on the GPU 1208.

[0266] In some examples, a CNN for face recognition and owner recognition can use data from the camera sensor to recognize the presence of an authorized driver and / or owner of the vehicle 1200. A processing engine always on the sensor can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a safe mode, disable the vehicle when the owner leaves the vehicle. In this way, the SoC 1204 provides security against theft and / or carjacking.

[0267] In another example, a CNN for emergency vehicle detection and recognition can use data from the microphones 1296 to detect and recognize emergency vehicle sirens. In contrast to conventional systems that use a general classifier to detect sirens and manually extract features, the SoC 1204 uses a CNN to classify ambient and urban sounds as well as to classify visual data. In a preferred embodiment, a CNN running on the DLA is trained to recognize the relative closing speed of an emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the local area in which the vehicle is operating as recognized by the GNSS sensor 1258. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when in the United States, the CNN will seek to recognize sirens that are only North American. Once an emergency vehicle is detected, a control program can be used to execute an emergency vehicle safety routine to slow the vehicle, pull over to the side of the road, stop the vehicle, and / or idle the vehicle until the emergency vehicle passes, with the assistance of the ultrasonic sensors 1262.

[0268] The vehicle can include a CPU 1218 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 1204 via a high-speed interconnect (e.g., PCIe). The CPU 1218 can include, for example, an X86 processor. The CPU 1218 can be used to perform any of a wide variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC 1204, and / or monitoring the status and health of the controller 1236 and / or infotainment SoC 1230.

[0269] The vehicle 1200 can include a GPU 1220 (e.g., a discrete GPU or dGPU) that can be coupled to the SoC 1204 via a high-speed interconnect (e.g., NVIDIA’s NVLINK). The GPU 1220 can provide additional artificial intelligence functionality, for example, by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based at least in part on input (e.g., sensor data) from sensors of the vehicle 1200.

[0270] The vehicle 1200 can also include a network interface 1224 that can include one or more wireless antennas 1226 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 1224 can be used to enable wireless connections over the Internet with the cloud (e.g., with the server 1278 and / or other network devices), with other vehicles, and / or with computing devices (e.g., client devices of passengers). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across a network and through the Internet). The direct link can be provided using a car-to-car communication link. The car-to-car communication link can provide the vehicle 1200 with information about vehicles that are approaching the vehicle 1200 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 1200). This functionality can be part of a cooperative adaptive cruise control functionality of the vehicle 1200.

[0271] The network interface 1224 can include a SoC that provides modulation and demodulation functionality and enables the controller 1236 to communicate over a wireless network. The network interface 1224 can include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. The frequency conversion can be performed through well-known processes and / or can be performed using a super-heterodyne process. In some examples, the radio frequency front end functionality can be provided by a separate chip. The network interface can include wireless functionality for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0272] The vehicle 1200 can also include a data store 1228, which can include off-chip (e.g., off-SoC 1204) storage. The data store 1228 can include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disks, and / or other components and / or devices that can store data for at least one bit.

[0273] The vehicle 1200 can also include a GNSS sensor 1258. The GNSS sensor 1258 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist in mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 1258 can be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0274] The vehicle 1200 can also include a RADAR sensor 1260. The RADAR sensor 1260 can be used by the vehicle 1200 for long-range vehicle detection, even in darkness and / or adverse weather conditions. The RADAR functional safety level can be ASIL B. The RADAR sensor 1260 can use the CAN and / or the bus 1202 (e.g., to transmit data generated using the RADAR sensor 1260) for control as well as access to object tracking data, in some examples, Ethernet for access to raw data. A wide variety of RADAR sensor types can be used. For example and without limitation, the RADAR sensor 1260 can be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

[0275] The RADAR sensor 1260 can include different configurations, such as long-range with narrow field of view, short-range with wide field of view, short-range side coverage, and so on. In some examples, long-range RADAR can be used for adaptive cruise control functionality. Long-range RADAR systems can provide a wide field of view (e.g., 250 m range) achieved through two or more independent scans. The RADAR sensor 1260 can help distinguish between static and moving objects, and can be used by the ADAS system for emergency brake assist and forward collision warning. The long-range RADAR sensor can include a single-station multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas, as well as a high-speed CAN and FlexRay interface. In examples with six antennas, the central four antennas can create focused beam patterns designed to record the surroundings of the vehicle 1200 at higher speed with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, making it possible to quickly detect vehicles entering or leaving the lane of the vehicle 1200.

[0276] As one example, a mid-range RADAR system can include a range of up to 1260 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 1250 degrees (rear). A short-range RADAR system can include, but is not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor the rear and the blind spot next to the vehicle.

[0277] A short-range RADAR system can be used in an ADAS system for blind spot detection and / or lane change assist.

[0278] The vehicle 1200 can also include ultrasonic sensors 1262. Ultrasonic sensors 1262 that can be placed on the front, rear, and / or sides of the vehicle 1200 can be used for parking assist and / or to create and update an occupancy grid. A wide variety of ultrasonic sensors 1262 can be used, and different ultrasonic sensors 1262 can be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensors 1262 can operate at an ASIL B functional safety level.

[0279] The vehicle 1200 can include LiDAR sensors 1264. The LiDAR sensors 1264 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensors 1264 can be at an ASIL B functional safety level. In some examples, the vehicle 1200 can include multiple LiDAR sensors 1264 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0280] In some examples, the LiDAR sensors 1264 can be capable of providing a list of objects and their distances for a 360-degree field of view. A commercially available LiDAR sensor 1264 can have, for example, an advertised range of approximately 1200 m, a precision of 2 cm - 3 cm, and support for a 1200 Mbps Ethernet connection. In some examples, one or more flush LiDAR sensors 1264 can be used. In such examples, the LiDAR sensors 1264 can be implemented as small devices that can be embedded to the front, rear, sides, and / or corners of the vehicle 1200. In such examples, the LiDAR sensors 1264 can provide a field of view of up to 120 degrees horizontal and 35 degrees vertical with a range of 200 m, even for low reflectivity objects. Front-mounted LiDAR sensors 1264 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0281] In some examples, LiDAR technology such as 3D Flash LiDAR can also be used. 3D Flash LiDAR uses a flash of laser light as a source of emission to illuminate the vehicle’s surroundings up to about 200 m. The Flash LiDAR unit includes a receptor that records the laser pulse transmission time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LiDAR can allow for the generation of highly accurate and distortion-free images of the surroundings with each laser flash. In some examples, four Flash LiDAR sensors can be deployed, one on each side of the vehicle 1200. Available 3D Flash LiDAR systems include solid-state 3D staring array LiDAR cameras (e.g., non-scanning LiDAR devices) that have no moving parts other than a fan. The Flash LiDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture the reflected laser light in the form of 3D range point clouds and co-registered intensity data. By using Flash LiDAR, and because Flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 1264 can be less susceptible to motion blur, vibration, and / or jostling.

[0282] The vehicle can also include an IMU sensor 1266. In some examples, the IMU sensor 1266 can be located at the center of the rear axle of the vehicle 1200. The IMU sensor 1266 can include, for example and without limitation, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in six-axis applications, the IMU sensor 1266 can include an accelerometer and a gyroscope, while in nine-axis applications, the IMU sensor 1266 can include an accelerometer, a gyroscope, and a magnetometer.

[0283] In some embodiments, the IMU sensor 1266 can be implemented as a microelectromechanical systems (MEMS) based inertial navigation system (INS) that combines microelectromechanical systems (MEMS) inertial sensors, high-sensitivity GPS receiver, and advanced Kalman filtering algorithms to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 1266 can enable the vehicle 1200 to estimate heading without input from a magnetic sensor by directly observing the change in velocity from GPS to the IMU sensor 1266 and correlating it. In some examples, the IMU sensor 1266 and the GNSS sensor 1258 can be combined into a single integrated unit.

[0284] The vehicle can include a microphone 1296 placed in and / or around the vehicle 1200. The microphone 1296 can be used for emergency vehicle detection and identification, among other things.

[0285] The vehicle can also include any number of camera types, including stereo cameras 1268, wide-view cameras 1270, infrared cameras 1272, surround-view cameras 1274, long and / or mid-range cameras 1298, and / or other camera types. These cameras can be used to capture image data around the entire periphery of the vehicle 1200. The types of cameras used depend on the embodiment and requirements of the vehicle 1200, and any combination of camera types can be used to provide the necessary coverage around the vehicle 1200. Further, the number of cameras can vary depending on the embodiment. For example, the vehicle can include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As one example and not by way of limitation, the cameras can support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described in more detail herein with respect to Figure 12A and Figure 12B are described in more detail.

[0286] The vehicle 1200 can also include vibration sensors 1242. The vibration sensors 1242 can measure vibrations of components of the vehicle, such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 1242 are used, differences between the vibrations can be used to determine the friction or slip of the road surface (e.g., when there is a difference in vibration between a power driven axle and a free spinning axle).

[0287] The vehicle 1200 can include an ADAS system 1238. In some examples, the ADAS system 1238 can include a SoC. The ADAS system 1238 can include adaptive / automatic / autonomous cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functionality.

[0288] The ACC system can use RADAR sensors 1260, LiDAR sensors 1264, and / or cameras. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately ahead of the vehicle 1200 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Lateral ACC performs distance keeping and, if necessary, suggests a lane change for the vehicle 1200. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0289] CACC uses information from other vehicles, which can be received from other vehicles indirectly via a wireless link or through a network connection (e.g., through the Internet) via the network interface 1224 and / or the wireless antenna 1226. Direct links can be provided by a vehicle-to-vehicle (V2V) communication link, while indirect links can be an infrastructure-to-vehicle (I2V) communication link. Generally, the V2V communication concept provides information about the immediately preceding vehicles (e.g., vehicles immediately ahead of and in the same lane as the vehicle 1200), while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both of I2V and V2V information sources. Given information about vehicles ahead of the vehicle 1200, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0290] FCW systems are designed to alert the driver to a hazard so that the driver can take corrective action. FCW systems use a front-facing camera and / or RADAR sensor 1260 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as displays, speakers, and / or vibrating components. FCW systems can provide warnings in the form of, for example, sound, visual warnings, vibrations, and / or quick brake pulses.

[0291] AEB systems detect an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. AEB systems can use a front-facing camera and / or RADAR sensor 1260 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When an AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the effects of a predicted collision. AEB systems can include technologies such as dynamic brake support and / or crash imminent braking.

[0292] LDW systems provide visual, audible, and / or tactile warnings such as steering wheel or seat vibrations to alert the driver when the vehicle 1200 is crossing lane markers. The LDW system is not activated when the driver indicates an intentional lane departure by activating a turn signal. LDW systems can use a front-side facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as displays, speakers, and / or vibrating components.

[0293] An LKA system is a variation of the LDW system. If the vehicle 1200 begins to leave the lane, the LKA system provides a steering input or brake to correct the vehicle 1200.

[0294] A BSW system detects and warns the driver of vehicles in the car's blind spot. The BSW system can provide visual, audible, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses a turn signal. The BSW system can use rear side-facing cameras and / or RADAR sensors 1260 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0295] A RCTW system can provide visual, audible, and / or tactile notifications when objects are detected outside the range of the rear-facing camera while the vehicle 1200 is backing up. Some RCTW systems include AEB to ensure that vehicle brakes are applied to avoid a collision. The RCTW system can use one or more rear-facing RADAR sensors 1260 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0296] Conventional ADAS systems can be prone to false positive results, which can annoy and distract the driver, but typically are not catastrophic because the ADAS system alerts the driver and allows the driver to decide whether the safety condition is truly present and act accordingly. However, in an autonomous vehicle 1200, in the case of conflicting results, the vehicle 1200 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 1236 or the second controller 1236). For example, in some embodiments, the ADAS system 1238 can be a secondary and / or auxiliary computer for providing perception information to a backup computer plausibility module. The backup computer plausibility monitor can run redundant diverse software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 1238 can be provided to a supervisory MCU. If the outputs from the primary and secondary computers conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0297] In some examples, the host computer can be configured to provide a confidence score to the supervisory MCU indicating the host computer's confidence in the selected result. If the confidence score exceeds a threshold, then the supervisory MCU can follow the host computer's direction, regardless of whether the secondary computer provides conflicting or inconsistent results. In the event that the confidence score does not satisfy the threshold and in the event that the host computer and the secondary computer indicate different results (e.g., a conflict), the supervisory MCU can arbitrate between the computers to determine the appropriate result.

[0298] The supervisory MCU can be configured to run a neural network that is trained and configured to determine conditions under which the secondary computer provides false alarms based at least in part on the output from the host computer and the secondary computer. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying metal objects that are not in fact dangerous, such as drain grates or manhole covers that trigger false alarms. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to disregard the LDW when a cyclist or pedestrian is present and lane departure is in fact the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU can include at least one of a DLA or a GPU suitable for running a neural network with associated memory. In preferred embodiments, the supervisory MCU can include and / or be included as a component of the SoC 1204.

[0299] In other examples, the ADAS system 1238 can include a secondary computer that performs ADAS functions using traditional computer vision rules. As such, the secondary computer can use classic computer vision rules (if-then) and the presence of a neural network in the supervisory MCU can improve reliability, safety, and performance. For example, the diverse implementation and intentional non-identity make the overall system more fault-tolerant, especially with respect to faults caused by software (or software-hardware interface) functions. For example, if there is a software bug or error in the software running on the host computer and the non-identical software code running on the secondary computer provides the same overall result, then the supervisory MCU can be more confident that the overall result is correct and that the bug in the software or hardware on the host computer did not cause a substantial error.

[0300] In some examples, the output of the ADAS system 1238 can be fed to a perception block of the host computer and / or a dynamic driving task block of the host computer. For example, if the ADAS system 1238 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information in identifying the object. In other examples, the secondary computer can have its own neural network that is trained and thus reduces the risk of false positives as described herein.

[0301] The vehicle 1200 can also include an infotainment SoC 1230 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as a SoC, the infotainment system can not be a SoC and can include two or more discrete components. The infotainment SoC 1230 can include a combination of hardware and software that can be used to provide audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming media, etc.), telephony (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.), and / or information services (e.g., a navigation system, a park assist, a radio data system, vehicle-related information such as a fuel level, a total distance covered, a brake fuel level, an oil level, a door open / close, air filter information, etc.) to the vehicle 1200. For example, the infotainment SoC 1230 can include a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, an in-car computer, in-car entertainment, WiFi, steering wheel audio controls, hands-free voice controls, a heads-up display (HUD), an HMI display 1234, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 1230 can further be used to provide information (e.g., visual and / or audible) to a user of the vehicle, such as information from the ADAS system 1238, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0302] The infotainment SoC 1230 can include GPU functionality. The infotainment SoC 1230 can communicate with other devices, systems, and / or components of the vehicle 1200 over the bus 1202 (e.g., a CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1230 can be coupled to a supervisory MCU such that, in the event of a failure of the host controller 1236 (e.g., a primary and / or backup computer of the vehicle 1200), the GPU of the infotainment system can perform some autonomous driving functions. In such examples, the infotainment SoC 1230 can place the vehicle 1200 in a driver safe park mode as described herein.

[0303] The vehicle 1200 can also include an instrument cluster 1232 (e.g., a digital dashboard, electronic instrument cluster, digital instrument panel, etc.). The instrument cluster 1232 can include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 1232 can include a set of instruments, such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seat belt warning light, parking brake warning light, engine malfunction light, supplemental restraint system (SRS) system information, lighting controls, safety system controls, navigation information, and so forth. In some examples, information can be displayed and / or shared between the infotainment SoC 1230 and the instrument cluster 1232. Thus, the instrument cluster 1232 can be included as part of the infotainment SoC 1230, or vice versa.

[0304] Figure 12D FIG. 12 illustrates a system diagram of communication between a cloud-based server and an example autonomous vehicle 1200 in accordance with some embodiments of the present disclosure. Figure 12A FIG. 12 illustrates a system diagram of communication between a cloud-based server and an example autonomous vehicle 1200 in accordance with some embodiments of the present disclosure. The system 1276 can include servers 1278, a network 1290, and vehicles including the vehicle 1200. The servers 1278 can include a plurality of GPUs 1284(A)-1284(H) (collectively referred to herein as GPUs 1284), PCIe switches 1282(A)-1282(D) (collectively referred to herein as PCIe switches 1282), and / or CPUs 1280(A)-1280(B) (collectively referred to herein as CPUs 1280). The GPUs 1284, CPUs 1280, and PCIe switches can be interconnected with high-speed interconnects such as, for example and without limitation, NVLink interfaces 1288 developed by NVIDIA and / or PCIe connections 1286. In some examples, the GPUs 1284 are connected via NVLink and / or NVSwitch SoC connections, and the GPUs 1284 and PCIe switches 1282 are connected via PCIe interconnects. Although eight GPUs 1284, two CPUs 1280, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 1278 can include any number of GPUs 1284, CPUs 1280, and / or PCIe switches. For example, each of the servers 1278 can include eight, sixteen, thirty-two, and / or more GPUs 1284.

[0305] The server 1278 can receive image data from vehicles over the network 1290 and representing images showing unexpected or changing road conditions such as a road work that has recently started. The server 1278 can transmit neural networks 1292, updated neural networks 1292, and / or map information 1294, including information about traffic and road conditions, to vehicles over the network 1290. Updates to the map information 1294 can include updates to the HD map 1222, e.g., information about construction sites, potholes, curves, flooding, or other obstacles. In some examples, the neural networks 1292, updated neural networks 1292, and / or map information 1294 can have been produced from experience using training performed at a data center (e.g., using the server 1278 and / or other servers) and / or from data received from any number of vehicles in the environment.

[0306] The server 1278 can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated using vehicles and / or can be generated in simulations (e.g., using game engines). In some examples, the training data is labeled (e.g., in cases where the neural network benefits from supervised learning) and / or undergoes other pre-processing, while in other examples, the training data is not labeled and / or pre-processed (e.g., in cases where the neural network does not require supervised learning). The training can be performed according to any one or more categories of machine learning techniques, including but not limited to categories such as: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including spare dictionary learning), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning models are trained, the machine learning models can be used by vehicles (e.g., transmitted to vehicles over the network 1290) and / or the machine learning models can be used by the server 1278 to remotely monitor vehicles.

[0307] In some examples, the server 1278 can receive data from vehicles and apply the data to the latest real-time neural networks for real-time intelligent inference. The server 1278 can include deep learning supercomputers and / or specialized AI computers powered by GPUs 1284, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, the server 1278 can include deep learning infrastructure of a data center that is powered using CPUs only.

[0308] The deep learning infrastructure of the server 1278 can be capable of fast real-time inference, and can use this capability to assess and validate the health of the processors, software, and / or associated hardware in the vehicle 1200. For example, the deep learning infrastructure can receive periodic updates from the vehicle 1200, such as a sequence of images and / or objects located in the sequence of images that the vehicle 1200 has located (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify the objects and compare them to the objects identified by the vehicle 1200, and if the results do not match and the infrastructure concludes that the AI in the vehicle 1200 is malfunctioning, the server 1278 can transmit a signal to the vehicle 1200 instructing the fail-safe computer of the vehicle 1200 to take control, notify the passengers, and complete a safe parking operation.

[0309] For inference, the server 1278 can include GPUs 1284 and one or more programmable inference accelerators (such as NVIDIA’s TensorRT). The combination of GPU-powered servers and inference-accelerated can make real-time response possible. In other examples, such as where performance is less important, CPU-, FPGA-, and other processor-powered servers can be used for inference.

[0310] Example Computing Device

[0311] Figure 13 is a block diagram of an example computing device 1300 suitable for implementing some embodiments of the present disclosure. The computing device 1300 can include an interconnect system 1302 that directly or indirectly couples the following devices: memory 1304, one or more central processing units (CPUs) 1306, one or more graphics processing units (GPUs) 1308, a communication interface 1310, input / output (I / O) ports 1312, I / O components 1314, a power supply 1316, one or more presentation components 1318 (e.g., display(s)), and one or more logic units 1320. In at least one embodiment, the computing device(s) 1300 can include one or more virtual machines (VMs), and / or any component thereof can include a virtual component (e.g., a virtual hardware component). For non-limiting examples, one or more of the GPUs 1308 can include one or more vGPUs, one or more of the CPUs 1306 can include one or more vCPUs, and / or one or more of the logic units 1320 can include one or more virtual logic units. As such, the computing device(s) 1300 can include discrete components (e.g., a full GPU dedicated to the computing device 1300), virtual components (e.g., a portion of a GPU dedicated to the computing device 1300), or a combination thereof.

[0312] Although Figure 13 various blocks of the computing device 1300 are shown as being connected by the interconnect system 1302, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 1318 such as a display device can be considered an I / O component 1314 (e.g., if the display is a touch screen). As another example, the CPU 1306 and / or GPU 1308 can include memory (e.g., the memory 1304 can represent a storage device in addition to the memory of the GPU 1308, CPU 1306, and / or other components). Thus, Figure 13 the computing device of FIG. 13 is illustrative. No distinction is made between a “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types in such categories because all are considered within the scope of the computing device of FIG. 13. Figure 13

[0313] The interconnect system 1302 can represent one or more links or buses, such as an address bus, data bus, control bus, or a combination thereof. The interconnect system 1302 can include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 1306 can be directly connected to the memory 1304. Further, the CPU 1306 can be directly connected to the GPU 1308. Where there are direct or point-to-point connections between components, the interconnect system 1302 can include a PCIe link to perform the connection. In these examples, a PCI bus need not be included in the computing device 1300.

[0314] The memory 1304 can include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 1300. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media.

[0315] ​Computer storage media can include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and / or other data types. For example, memory 1304 can store computer readable instructions (e.g., representing program(s) and / or program element(s), such as an operating system). Computer storage media can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computing device 1300. As used herein, computer storage media does not include signals per se.

[0316] Computer storage media can embody computer readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term “modulated data signal” can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.

[0317] The CPUs 1306 can be configured to execute at least some of the computer readable instructions to control one or more components of the computing device 1300 to perform one or more of the methods and / or processes described herein. The CPUs 1306 can each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling multiple software threads concurrently. The CPUs 1306 can include any type of processors and can include different types of processors depending on the type of computing device 1300 being implemented (e.g., a mobile device having fewer cores for a processor and a server having more cores for a processor). For example, depending on the type of computing device 1300, the processor can be an Advanced RISC Machines (ARM) processor implemented using a Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 1300 can include one or more CPUs 1306 in addition to, or as an alternative to, one or more microprocessors or co-processors such as mathematical co-processors.

[0318] In addition to or in place of CPU(s) 1306, GPU(s) 1308 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1300 to perform one or more of the methods and / or processes described herein. One or more of GPUs 1308 can be integrated GPUs (e.g., with one or more of CPUs 1306) and / or one or more of GPUs 1308 can be discrete GPUs. In embodiments, one or more of GPUs 1308 can be a co-processor of one or more of CPUs 1306. GPUs 1308 can be used by computing device 1300 to render graphics (e.g., 3D graphics) or to perform general purpose computing. For example, GPUs 1308 can be used for general purpose computing on GPUs (GPGPU). GPUs 1308 can include hundreds or thousands of cores capable of handling hundreds or thousands of software threads concurrently. GPUs 1308 can generate pixel data for output images in response to rendering commands (e.g., received from CPUs 1306 via a host interface). GPUs 1308 can include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory can be included as part of memory 1304. GPUs 1308 can include two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or can connect the GPUs through a switch (e.g., using NVSwitch). When combined together, each GPU 1308 can generate pixel data or GPGPU data for a different portion of an output or for a different output (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can include its own memory or can share memory with other GPUs.

[0319] In addition to or in lieu of CPU 1306 and / or GPU 1308, logic unit(s) 1320 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1300 to perform one or more of the methods and / or processes described herein. In embodiments, CPU(s) 1306, GPU(s) 1308, and / or logic unit(s) 1320 can perform any combination of the methods, processes, and / or portions thereof, discretely or jointly. One or more of logic unit(s) 1320 can be part of and / or integrated with one or more of CPU 1306 and / or GPU 1308 and / or one or more of logic unit(s) 1320 can be a discrete component or otherwise external to CPU 1306 and / or GPU 1308. In embodiments, one or more of logic unit(s) 1320 can be a co-processor of one or more of CPU 1306 and / or GPU 1308.

[0320] Examples of logic unit(s) 1320 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel visual core (PVC), a visual processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multi-processor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating-point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) element, etc.

[0321] The communication interface 1310 can include one or more receivers, transmitters and / or transceivers that enable the computing device 1300 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. The communication interface 1310 can include components and functionality to enable communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., through Ethernet or InfiniBand), low power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 1320 and / or the communication interface 1310 can include one or more data processing units (DPUs) for transferring data received over a network and / or through the interconnection system 1302 directly to (e.g., memory of) one or more GPUs 1308.

[0322] The I / O ports 1312 can enable the computing device 1300 to be logically coupled to other devices including the I / O components 1314, the presentation component(s) 1318, and / or other components, some of which can be built into (e.g., integrated with) the computing device 1300. Illustrative I / O components 1314 include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. The I / O components 1314 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs can be transmitted to appropriate network elements for further processing. A NUI can implement any combination of speech recognition, pen / mouse recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with a display of the computing device 1300. The computing device 1300 can include a depth camera, such as a stereoscopic camera system, infrared camera system, RGB camera system, touchscreen technology, and combinations of these, for gesture detection and recognition. In addition, the computing device 1300 can include an accelerometer or a gyroscope (e.g., as part of an inertial measurement unit (IMU)) to detect motion. In some examples, the computing device 1300 can use the output of the accelerometer or gyroscope to render an immersive augmented reality or virtual reality.

[0323] The power supply 1316 can include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply 1316 can provide power to the computing device 1300 to enable the components of the computing device 1300 to operate.

[0324] The presentation component 1318 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 1318 may receive data from other components (e.g., GPU 1308, CPU 1306, DPU, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0325] Example Data Center

[0326] Figure 14 An example data center 1400 that may be used in at least one embodiment of this disclosure is shown. The data center 1400 may include a data center infrastructure layer 1410, a framework layer 1420, a software layer 1430, and / or an application layer 1440.

[0327] like Figure 14 As shown, the data center infrastructure layer 1410 may include a resource coordinator 1412, grouped computing resources 1414, and node computing resources (“nodes CRs”) 1416(1)-1416(N), where “N” represents any complete positive integer. In at least one embodiment, nodes CRs 1416(1)-1416(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules and / or cooling modules, etc. In some embodiments, one or more node CRs from nodes CRs 1416(1)-1416(N) may correspond to servers having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CRs1416(1)-14161(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more nodes CRs1416(1)-1416(N) may correspond to virtual machines (VMs).

[0328] In at least one embodiment, the grouped computing resources 1414 may include individual groups of node CRs 1416 housed within one or more racks (not shown), or a plurality of racks housed within a data center in different geographical locations (also not shown). Individual groups of node CRs 1416 within the grouped computing resources 1414 may include grouped computing, networking, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, a plurality of node CRs 1416, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0329] Resource coordinator 1412 may be configured or otherwise control one or more nodes CRs 1416(1)-1416(N) and / or grouped computing resources 1414. In at least one embodiment, resource coordinator 1412 may include a Software Design Infrastructure (SDI) management entity for data center 1400. Resource coordinator 1412 may include hardware, software, or some combination thereof.

[0330] In at least one embodiment, such as Figure 14 As shown, framework layer 1420 may include a job scheduler 1433, a configuration manager 1434, a resource manager 1436, and / or a distributed file system 1438. Framework layer 1420 may include a framework for software 1432 supporting software layer 1430 and / or one or more applications 1442 of application layer 1440. Software 1432 or application 1442 may respectively contain web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1420 may be, but is not limited to, free and open-source software web application frameworks (such as Apache Spark) that can utilize distributed file system 1438 for large-scale data processing (e.g., "big data"). TM(“Spark” hereinafter). In at least one embodiment, job scheduler 1433 can include a Spark driver to facilitate scheduling workloads supported by different tiers of data center 1400. Configuration manager 1434 can be capable of configuring different tiers, such as software tier 1430 and framework tier 1420, which includes Spark and distributed file system 1438 for supporting large scale data processing. Resource manager 1436 can be capable of managing clustered or grouped computing resources mapped to or allocated for supporting distributed file system 1438 and job scheduler 1433. In at least one embodiment, clustered or grouped computing resources can include grouped computing resources 1414 of data center infrastructure layer 1410. Resource manager 1436 can coordinate with resource orchestrator 1412 to manage these mapped or allocated computing resources.

[0331] In at least one embodiment, software 1432 included in software tier 1430 can include software used by at least portions of node C.R.s 1416(1)-1416(N), grouped computing resources 1414, and / or distributed file system 1438 of framework tier 1420. One or more types of software can include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0332] In at least one embodiment, applications 1442 included in application tier 1440 can include one or more types of applications used by at least portions of node C.R.s 1416(1)-1416(N), grouped computing resources 1414, and / or distributed file system 1438 of framework tier 1420. One or more types of applications can include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0333] In at least one embodiment, any of configuration manager 1434, resource manager 1436, and resource orchestrator 1412 can implement any number and type of self-modifying actions based on any quantity and type of data acquired in any technically feasible manner. Self-modifying actions can free data center operators of data center 1400 from making possibly poor configuration decisions and possibly avoid underutilization and / or poor performance portions of a data center.

[0334] According to one or more embodiments described herein, data center 1400 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information. For example, one or more machine learning models can be trained by using the software and / or computing resources described above with respect to data center 1400 to compute weight parameters according to a neural network architecture. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks can be used to infer or predict information using the resources described above with respect to data center 1400 by using weight parameters computed through one or more training techniques (such as, but not limited to, those described herein).

[0335] In at least one embodiment, the data center 1400 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform services that infer information, such as image recognition, speech recognition, or other artificial intelligence services.

[0336] Example network environment

[0337] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 13 This is implemented on one or more instances of computing device 1300—for example, each device may include similar components, features, and / or functions of computing device 1300. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of data center 1400, examples of which are described herein. Figure 14 To describe in more detail.

[0338] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0339] Compatible network environments can include one or more peer-to-peer network environments (in which case servers can not be included in the network environment) and one or more client-server network environments (in which case one or more servers can be included in the network environment). In a peer-to-peer network environment, functionality described herein for servers can be implemented on any number of client devices.

[0340] In at least one embodiment, the network environment can include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which can include one or more core network servers and / or edge servers. The framework layer can include a framework that supports one or more applications of a software layer and / or an application layer. The software or applications can include network-based service software or applications, respectively. In embodiments, one or more client devices can use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer can be, without limitation, a free and open-source software web application framework as can be used for large-scale data processing (e.g., “big data”) using a distributed file system.

[0341] The cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functionality described herein (or one or more portions thereof). Any of these different functionalities can be distributed across multiple locations from central or core servers (e.g., one or more data centers that can be distributed across a state, a region, a country, globally, and the like). The core servers can designate at least a portion of the functionality to edge servers if the connection to the user (e.g., client device) is relatively close to the edge servers. The cloud-based network environment can be private (e.g., limited to a single organization), can be public (e.g., available to many organizations), and / or combinations thereof (e.g., a hybrid cloud environment).

[0342] The client device(s) can include the client devices described herein with respect to Figure 13At least some of the components, features and functionality of the described example computing device 1300 are described. By way of example, and without limitation, a client device can be implemented as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smartwatch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a camera, a surveillance device or system, a vehicle, a boat, a spaceship, a virtual machine, a drone, a robot, a hand-held communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these depicted devices, or any other suitable device.

[0343] The present disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The present disclosure can be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general- purpose computers, more specialty computing devices, etc. The present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0344] As used herein, the term “and / or” with respect to two or more elements should be interpreted as only one element or a combination of elements. For example, “element A, element B, and / or element C” can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. In addition, “at least one of element A or element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, “at least one of element A and element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0345] The subject matter of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms “step” and / or “block” might be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.

[0346] Example text support

[0347] In example embodiments, the one or more processors include one or more processing units to: receive at least one of one or more material maps or one or more lighting maps, the one or more material maps defining one or more properties of surfaces of one or more objects in a scene, the one or more lighting maps representing at least one of one or more shading or lighting characteristics associated with the one or more objects; and provide, as input to one or more first machine learning models, a first noise vector and a representation of at least one of the one or more material maps or the one or more lighting maps to generate an output frame of the scene, the first noise vector corresponding to an initial starting point of a diffusion process performed by the one or more first machine learning models.

[0348] In some embodiments, the one or more material maps include at least one of: an albedo map, a normal map, a roughness map, a metallic map, an ambient occlusion map, a displacement map, a specular map, an emissive map, an opacity map, a cavity map, or a subsurface scattering map.

[0349] In some embodiments, the one or more processing units are further to: receive a user input requesting that at least one of a material property or a lighting condition be incorporated into the output frame; and based at least in part on the user input, generate at least one of the one or more material maps or the one or more lighting maps, and wherein the output frame is generated based at least in part on the user input.

[0350] In some embodiments, the one or more processing units are further to: receive an input frame and a second noise vector; and provide, as input to one or more second machine learning models, a representation of the second noise vector and the input frame to generate the one or more first material maps.

[0351] In some embodiments, the one or more processing units are further to: provide, as input, an input frame to the one or more first machine-learned models, and wherein the first noise vector represents a noisy version of the input frame; and receive a request to enhance the input frame, wherein the output frame is generated based at least in part on the request and the one or more processing units are further to provide, as input, the input frame to the one or more first machine-learned models, and wherein the output frame includes one or more features that have been enhanced relative to the input frame.

[0352] In some embodiments, the one or more processing units are further to: provide, as input, a two-dimensional input frame and a second noise vector to one or more second machine-learned models; and generate the one or more material maps based at least on providing, as input, the two-dimensional input frame and the second noise vector to the one or more second machine-learned models, wherein the output frame represents the two-dimensional input frame including modified lighting properties in the output frame relative to the input frame.

[0353] In some embodiments, the one or more processing units are further to: provide, as input, a two-dimensional input frame and a second noise vector to one or more second machine-learned models; and generate one or more second material maps based at least in part on providing, as input, the input frame and the second noise vector to the one or more second machine-learned models.

[0354] In some embodiments, the one or more processing units are further to: generate, based on the one or more second material maps, a multi-dimensional frame representing a two-dimensional input frame including at least a multi-dimension relative to the two-dimensional input frame; and generate the one or more first material maps based at least in part on generating the multi-dimensional frame, wherein the output frame is generated based at least in part on the multi-dimensional frame.

[0355] In some embodiments, the one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transport simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using edge devices; a system implemented using robots; a system for performing conversational AI operations; a system for generating synthetic data; a system for generating synthetic data using AI; a system for performing one or more operations using one or more large language models (LLMs); a system for performing one or more operations using one or more visual language models (VLMs); a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0356] In embodiments, the system includes one or more processing units to: receive an input frame; and provide, as input to one or more first machine learning models, a first noise vector and a representation of the input frame to generate one or more first material maps, the first noise vector corresponding to an initial starting point for a diffusion process performed by the one or more first machine learning models, the one or more first material maps defining one or more properties of a surface of one or more objects in a scene.

[0357] In some embodiments, the one or more material maps include at least one of: a reflectance map, a normal map, a roughness map, a metallic map, or an ambient occlusion map, a displacement map, a specular map, an emissive map, an opacity map, a cavity map, or a subsurface scattering map.

[0358] In some embodiments, the one or more processing units are further to: receive a user input requesting that at least one of a material property or a lighting condition be incorporated into an output frame; and generate, based at least in part on the user input, at least one of the one or more first material maps or the output frame.

[0359] In some embodiments, the one or more processing units are further to: receive one or more lighting maps, the one or more lighting maps representing at least one of one or more shading or lighting characteristics associated with the one or more objects; and provide, as input to the one or more first machine learning models, a representation of the one or more lighting maps to generate an output frame based at least in part on the first noise vector, the one or more material maps, and the one or more lighting maps.

[0360] In some embodiments, the first noise vector represents a noisy version of the input frame, and wherein the one or more processing units are further to: receive a request to enhance the input frame, wherein the output frame is generated using the one or more first machine learning models based at least in part on the request and input frame, and wherein the output frame includes one or more features that have been enhanced relative to the input frame.

[0361] In some embodiments, the input image represents a two-dimensional input frame, and wherein the one or more processing units are further to: provide, as input to one or more second machine learning models, the one or more first material maps, a user-specified lighting condition, and a second noise vector; and generate an output frame based at least in part on the one or more first material maps, the user-specified lighting condition, and the second noise vector as input to the one or more second machine learning models, wherein the output frame represents the two-dimensional input frame including lighting properties that have been modified in the output frame relative to the input frame.

[0362] In some embodiments, the input frame represents a two-dimensional input frame, and wherein the one or more processing units are further to: generate, based on the one or more first material maps, a multi-dimensional frame representing a two-dimensional input frame including at least a plurality of dimensions relative to the two-dimensional input frame; and generate one or more second material maps based at least on the multi-dimensional frame.

[0363] In some embodiments, the one or more processing units are further to: provide, as input to one or more second machine learning models, a second noise vector and the one or more second material maps to generate an output frame, and wherein the output frame is generated based at least in part on the generation of the multi-dimensional frame.

[0364] In some embodiments, the system is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transport simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using edge devices; a system implemented using robots; a system for performing conversational AI operations; a system for generating synthetic data; a system for generating synthetic data using AI; a system for performing one or more operations using one or more large language models (LLMs); a system for performing one or more operations using one or more visual language models (VLMs); a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0365] In embodiments, a method includes: receiving at least one of: one or more first material maps, one or more first light maps, or an input frame; providing, as input, a representation of at least one of: a noise vector, the one or more first material maps, the one or more first light maps, or the input frame, to one or more first machine learning models; and generating, based on at least one of: the noise vector, the one or more first material maps, the one or more first light maps, or the input image, an output comprising at least one of: an output frame, one or more second material maps, or one or more second light maps.

[0366] In some embodiments, the method is performed by at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transport simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using edge devices; a system implemented using robots; a system for performing conversational AI operations; a system for generating synthetic data; a system for generating synthetic data using AI; a system for performing one or more operations using one or more large language models (LLMs); a system for performing one or more operations using one or more visual language models (VLMs); a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Claims

1. One or more processors, comprising: One or more processing units are used for: Receive at least one of one or more material maps or one or more light maps, wherein the one or more material maps define one or more properties of the surface of one or more objects in the scene, and the one or more light maps represent at least one of one or more shading or lighting properties associated with the one or more objects; as well as The first noise vector and a representation of at least one of the one or more material maps or light maps are provided as input to one or more first machine learning models to generate an output frame of the scene, wherein the first noise vector corresponds to the initial starting point of the diffusion process performed by the one or more first machine learning models.

2. The processors according to claim 1, wherein, The one or more material maps include at least one of the following: albedo map, normal map, roughness map, metal map, ambient occlusion map, displacement map, specular map, emissive map, opacity map, cavity map, or subsurface scattering map.

3. The processors according to claim 1, wherein, The one or more processing units are further configured to: Receive user input requesting that at least one of the material properties or lighting conditions be incorporated into the output frame; and At least one of the one or more material maps or the one or more light maps is generated based at least in part on the user input, and wherein the output frame is generated based at least in part on the user input.

4. The processors according to claim 1, wherein, The one or more processing units are further configured to: Receive the input frame and the second noise vector; and Provide the second noise vector and the representation of the input frame, and provide the second noise vector and the representation of the input frame as input to one or more second machine learning models to generate the one or more first material maps.

5. The processors according to claim 1, wherein, The one or more processing units are further configured to: The input frame is provided as input to one or more first machine learning models, wherein the first noise vector represents a noisy version of the input frame; and The processing unit receives a request to enhance the input frame, wherein the output frame is generated at least in part based on the request, and the one or more processing units are further configured to provide the input frame as input to the one or more first machine learning models, wherein the output frame includes one or more features that have been enhanced relative to the input frame.

6. The processors according to claim 1, wherein, The one or more processing units are further configured to: The two-dimensional input frame and the second noise vector are used as input to one or more second machine learning models; as well as The one or more material maps are generated based at least on providing the two-dimensional input frame and the second noise vector as input to the one or more second machine learning models, wherein the output frame represents the two-dimensional input frame including lighting properties modified in the output frame relative to the input frame.

7. The processors according to claim 1, wherein, The one or more processing units are further configured to: The two-dimensional input frame and the second noise vector are provided as input to one or more second machine learning models; and At least in part, one or more second material maps are generated based on providing the input frame and the second noise vector as input to the one or more second machine learning models.

8. The processors according to claim 7, wherein, The one or more processing units are further configured to: A multidimensional frame is generated based on the one or more second material maps, wherein the multidimensional frame represents a two-dimensional input frame that includes at least one more dimension than the two-dimensional input frame; and The one or more first material maps are generated at least in part based on the generation of the multidimensional frame, wherein the output frame is generated at least in part based on the multidimensional frame.

9. The processors according to claim 1, wherein, The one or more processors are included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system used to perform deep learning operations; Systems used for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; Systems implemented using edge devices; Systems implemented using robots; Systems used to perform conversational AI operations; A system for generating synthetic data; Systems for generating synthetic data using AI; A system for performing one or more operations using one or more large language model LLMs; A system for performing one or more operations using one or more Visual Language Models (VLMs); A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

10. A system comprising one or more processing units for: Receive input frames; and The first noise vector and the representation of the input frame are provided as input to one or more first machine learning models to generate one or more first material maps, the first noise vector corresponding to the initial starting point of a diffusion process performed by the one or more first machine learning models, and the one or more first material maps defining one or more properties of the surface of one or more objects in the scene.

11. The system according to claim 10, wherein, The one or more material maps include at least one of the following: albedo map, normal map, roughness map, metal map, or ambient occlusion map, displacement map, specular map, emissive map, opacity map, cavity map, or subsurface scattering map.

12. The system according to claim 10, wherein, The one or more processing units are further configured to: The user input requesting the inclusion of at least one of the material properties or lighting conditions into the output frame; and At least one of the one or more first material maps or the output frame is generated, based at least in part on the user input.

13. The system according to claim 10, wherein, The one or more processing units are further configured to: Receive one or more lightmaps, the one or more lightmaps representing at least one of one or more shading or lighting properties associated with the one or more objects; as well as The representations of the one or more lightmaps are provided as input to one or more first machine learning models to generate output frames based at least in part on the first noise vector, the one or more material maps, and the one or more lightmaps.

14. The system according to claim 10, wherein, The first noise vector represents a noisy version of the input frame, and wherein the one or more processing units are further configured to: A request to enhance the input frame is received, wherein an output frame is generated using one or more first machine learning models based at least in part on the request and the input frame, and wherein the output frame includes one or more features that have been enhanced relative to the input frame.

15. The system according to claim 10, wherein, The input image represents a two-dimensional input frame, and wherein the one or more processing units are further configured to: The one or more first material maps, user-specified lighting conditions, and a second noise vector are provided as input to one or more second machine learning models; and An output frame is generated based at least in part on the one or more first material maps, the user-specified lighting conditions, and the second noise vector as input to the one or more second machine learning models, wherein the output frame represents the two-dimensional input frame, the two-dimensional input frame including lighting properties modified in the output frame relative to the input frame.

16. The system according to claim 10, wherein, The input frame represents a two-dimensional input frame, and wherein the one or more processing units are further configured to: A multidimensional frame is generated based on the one or more first material maps, wherein the multidimensional frame represents a two-dimensional input frame that includes at least one more dimension than the two-dimensional input frame; and At least based on the multidimensional frame, generate one or more second material maps.

17. The system according to claim 16, wherein, The one or more processing units are further configured to: The second noise vector and the one or more second material maps are provided as input to one or more second machine learning models to generate an output frame, wherein the output frame is generated at least in part based on the generation of the multidimensional frame.

18. The system according to claim 10, wherein, The system is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system used to perform deep learning operations; Systems used for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; Systems implemented using edge devices; Systems implemented using robots; A system for performing conversational AI operations; A system for generating synthetic data; Systems for generating synthetic data using AI; A system for performing one or more operations using one or more large language model LLMs; A system for performing one or more operations using one or more Visual Language Models (VLMs); A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

19. A method comprising: Receive at least one of the following: one or more first material maps, one or more first lightmaps, or an input frame; The representation of at least one of the following is provided as input to one or more first machine learning models: a noise vector, the one or more first material maps, the one or more first light maps, or the input frame; as well as An output is generated based on at least one of the noise vector, the one or more first material maps, the one or more first light maps, or the input image. The output includes an output frame, at least one of the one or more second material maps or the one or more second light maps.

20. The method according to claim 19, wherein, This method is performed by at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system used to perform deep learning operations; Systems used for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; Systems implemented using edge devices; Systems implemented using robots; Systems used to perform conversational AI operations; A system for generating synthetic data; Systems for generating synthetic data using AI; A system for performing one or more operations using one or more large language model LLMs; A system for performing one or more operations using one or more Visual Language Models (VLMs); A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

Images