LEARNING FROM COMPOSITE REPRESENTATIONS OF TIME-CHANGING SCENES FOR AUTONOMOUS AGENTS

Abstract

Scene reconstruction is an image processing technique that creates a model of a scene from a given input, typically involving the creation of a three-dimensional (3D) scene model from one or more input two-dimensional (2D) images of the scene. High-quality scene reconstruction and rendering are useful for various applications, such as autonomous agent applications and scene editing applications. Existing scene reconstruction methods reach their limits with dynamic scenes where moving objects are inconsistent across different views at different times. These methods generally lack the motion cues essential for effective object-environment decomposition, which, moreover, leads to incomplete environment reconstruction of new views due to persistent occlusion of environmental structures.The present disclosure integrates spatial memories or a spatial memory from previous iterations of a scene when learning a representation of a temporally changing scene, thereby providing observations of obscured areas and contextual information for road users to enable more comprehensive and efficient learning of the scene representation.

Description

TECHNICAL AREA

[0001] The present disclosure relates to computer methods for scene reconstruction. background

[0002] Scene reconstruction is an image processing technique that creates a model of a scene from a given input, typically involving the creation of a three-dimensional (3D) scene model from one or more input two-dimensional (2D) images of the scene. High-quality scene reconstruction and rendering is beneficial for autonomous agent applications, such as autonomous vehicles and robotics. For example, reconstructed scenes can be used by autonomous agents to understand and navigate complex and constantly changing environments.

[0003] Recent breakthroughs in radiation fields, such as Neural Radiance Field (NeRF) and 3D Gaussian Splatting (3DGS), have revolutionized scene reconstruction and rendering through differentiated 3D scene representations. Despite superior performance with static scenes, they reach their limits with dynamic urban scenes, as moving objects are inconsistent across different views at different times, leading to rendering artifacts. Previous efforts address dynamic objects through 3D bounding box annotations. However, the reliance on costly 3D labeling limits their efficiency and scalability.

[0004] To eliminate the need for 3D labeling, recent work uses LiDAR and camera sensor data as input and leverages motion as a self-monitoring signal to separate dynamic components from static backgrounds. However, transient objects such as parked cars are static—just like the environment—and therefore lack the motion cues essential for effective object-environment decomposition. Furthermore, their persistent occlusion of surrounding structures leads to an incomplete reconstruction of the environment from new viewpoints. Therefore, learning to construct a complete and separate composite representation of complex urban scenes through self-monitoring remains a significant challenge.

[0005] Therefore, there is a need to address these and / or other problems associated with the state of the art. For example, there is a need to integrate spatial memories from previous runs of a scene when learning a representation of a temporally changing scene, which can provide additional cues such as observations of obscured areas and contextual information for road users, enabling more comprehensive and efficient learning of the scene representation. SUMMARY

[0006] A method, a computer-readable medium, and a system for generating a 3D representation of a real-world scene are disclosed. The method accesses at least one first image capturing a real-world scene at a first time point. It also accesses a representation of time-invariant components of the real-world scene, wherein the representation of the time-invariant components is generated from one or more second images capturing the real-world scene at one or more second times points prior to the first time point. Finally, a 3D representation of the real-world scene at the first time point is generated using the at least one first image and the representation of the time-invariant components. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 illustrates a method for generating a 3D representation of a real scene according to one embodiment. Fig. Figure 2 illustrates a system pipeline for supervised learning of a 3D representation of a real-world scene according to one embodiment. Fig. Figure 3 illustrates an implementation of the system pipeline of Fig. 2 according to one embodiment. Fig. Figure 4 illustrates a method for using a 3D representation of a real scene for scene editing according to one embodiment. Fig. Figure 5 illustrates exemplary inputs and outputs of the procedure of Fig. 4 according to one embodiment. Fig. 6A illustrates an inference and / or training logic according to at least one embodiment; Fig. 6B illustrates an inference and / or training logic according to at least one embodiment; Fig. Figure 7 illustrates the training and deployment of a neural network according to at least one embodiment; Fig. Figure 8 illustrates an exemplary data center system according to at least one embodiment. DETAILED DESCRIPTION

[0007] Fig. Figure 1 illustrates a method 100 for generating a 3D representation of a real-world scene according to one embodiment. In one embodiment, the method 100 can be performed by a device that may include a processing unit, a program, a custom circuit, or a combination thereof. In another embodiment, a system comprising non-volatile memory containing instructions and one or more processors communicating with the memory can execute the instructions to perform the method 100. In yet another embodiment, a non-volatile, computer-readable medium can store computer instructions which, when executed by one or more processors of a device, cause the device to perform the method 100.

[0008] In step 102, at least one first image is accessed, capturing a real-world scene at a first point in time. The real-world scene refers to a time-changing environment as found in real life. In one embodiment, each first image can capture the real-world scene from a different viewpoint at the same or nearly the same first point in time.

[0009] In one embodiment, the at least one first image can include at least one frame of a video that captures the real scene at the first time. In another embodiment, the at least one first image can be generated by a camera. In another embodiment, the camera can be installed on an autonomously driving vehicle that traverses the real scene. In yet another embodiment, the camera can be installed on a drone that traverses the real scene.

[0010] In step 104, a representation of time-invariant components of the real scene is accessed. This representation is generated from one or more second images that capture the real scene at one or more second time points prior to the first. The time-invariant components of the real scene refer to components of the real scene that remain unchanged across two or more time points. In one embodiment, the time-invariant components of the real scene can include (or be limited to) static (e.g., stationary) objects in the real scene, such as buildings in an urban environment.

[0011] The representation of the time-invariant components refers to a representation generated from the one or more second images that capture the real scene at one or more time points prior to the first time point when the at least one first image captured the real scene. In one embodiment, the representation of the time-invariant components of the real scene can exclude transient components of the real scene. Transient components refer to components of the real scene that change or shift over two or more time points. In one embodiment, the transient components of the real scene can include (or be limited to) dynamic (e.g., moving) objects in the real scene, such as moving people, moving vehicles, etc.

[0012] In one embodiment, a spatial memory or spatial storage can contain a representation of the time-invariant components of the real scene. In another embodiment, the representation of the time-invariant components of the real scene can contain 3D Gaussian distributions of the time-invariant components of the real scene. In another embodiment, the 3D Gaussian distributions can be learned using 3D Gaussian splatting (3DGS).

[0013] In one embodiment, the one or more second images can include one or more second images from one or more videos that capture the real scene at the one or more second time points prior to the first time point. In one embodiment, the one or more second images can be generated by the same camera that captured the at least one first image. In another embodiment, the one or more second images can be generated by a camera that differs from the camera that captured the at least one first image. In other embodiments, the one or more second images can capture the same or a different viewpoint of the real scene as at least one first image.

[0014] In step 106, a 3D representation of the real scene at the first time point is generated using the at least one first image and the representation of the time-invariant components of the real scene. The 3D representation of the real scene refers to a representation of the real scene that defines at least the time-invariant components of the real scene in 3D. The 3D representation of the real scene can define the time-invariant components of the real scene based on their representation in both the at least one first image and on the representation of the time-invariant components of the real scene generated from one or more second images.

[0015] In one embodiment, the 3D representation of the real scene can also define the transient components of the real scene. The 3D representation of the real scene can define the transient components of the real scene based on their representation in the at least one first image. In one embodiment, the transient components of the real scene at the first time point can be detected from the at least one first image, and a representation of the transient components of the real scene at that first time point can be generated. The transient components of the real scene at the first time point can be detected based on a depth difference between the at least one first image and the representation of the time-invariant components of the real scene, and a color difference between the at least one first image and the representation of the time-invariant components of the real scene.The representation of the transient components of the real scene can, for example, show 4D Gaussian distributions of the transient components of the real scene.

[0016] In addition to this embodiment, the 3D representation of the real scene at the first time can be generated based on the representation of the transient components of the real scene and the representation of the time-invariant components of the real scene. Thus, for example, the 3D representation of the real scene at the first time can include both the time-invariant components and the transient components of the real scene. In one embodiment, the representation of the transient components and the representation of the time-invariant components of the real scene (i.e., for use in generating the 3D representation of the real scene) can be optimized by minimizing at least one loss function. Accordingly, the 3D representation of the real scene can consist of the optimized representation of the transient components and the optimized representation of the time-invariant components of the real scene.

[0017] In one embodiment, the at least one loss function can be a regularization loss function that promotes the separation of the rendering of the transient components of the real scene from the rendering of the time-invariant components of the real scene. In another embodiment, the at least one loss function can be a conservation loss function that monitors hidden pixels in the rendering of the time-invariant components of the real scene in order to preserve the structure of one or more of the time-invariant components that are hidden by one or more of the transient components. In another embodiment, the at least one loss function can be a rendering loss function. In yet another embodiment, the at least one loss function can be a depth-smoothing loss function.

[0018] To this end, when performed by a device, Method 100 generates a 3D representation of a real scene as it exists at a first time point, from both one or more images of the scene captured at that first time point and one or more images of the scene captured at one or more earlier times point in time. The use of the images of the scene captured at the one or more earlier times point in time enables the 3D representation to define features of the scene that might otherwise be obscured or unobservable in the one or more images of the scene captured at the first time point. When used for scene reconstruction, Method 100 particularly improves the 3D representation result for scenes that change over time.

[0019] In one embodiment, the method 100 further comprises outputting the 3D representation of the real scene to a downstream task. In one embodiment, the downstream task may comprise an image synthesis or image processing application. For example, the downstream task may use the 3D representation of the real scene to perform image synthesis or image processing, which includes removing at least one object (a time-invariant or a transient component) from the at least one first image of the real scene, relocating at least one object (a time-invariant or a transient component) within the at least one first image of the real scene, and / or transferring at least one object (a time-invariant or a transient component) from one or more second images into the at least one first image of the real scene.

[0020] In one embodiment, the downstream task may involve an autonomous agent application, such as an autonomous vehicle application or a robotics application. For example, the downstream task may use the 3D representation of the real-world scene to control an autonomous agent. By way of example only, the downstream task may process the 3D representation of the real-world scene to make decisions regarding whether the autonomous agent should traverse the real-world scene. In various embodiments, the method 100 may be performed by hardware and / or software of the autonomous agent or by hardware and / or software located remotely from the autonomous agent (such as hardware and / or software located in the cloud).

[0021] In an exemplary implementation of Method 100, a 3D representation of a real scene is generated from at least one first image capturing the real scene at a first time point, and a representation of time-invariant components of the real scene, generated from one or more second images capturing the real scene at one or more second times points prior to the first time point. The at least one first image is then processed using the 3D representation of the real scene to generate at least one processed image, wherein the processing includes at least one of: removing at least one object in the at least one first image of the real scene, relocating at least one object in the at least one first image of the real scene, or transferring at least one object from the one or more second images into the at least one first image of the real scene.Furthermore, at least one edited image will be output.

[0022] Further embodiments are now provided in the description of the following figures. It should be noted that the embodiments described herein refer to Method 100 of Fig. The embodiments disclosed in 1 apply to any of the embodiments of the remaining figures below and / or can be used in combination with them.

[0023] Fig. Figure 2 illustrates a system pipeline 200 for supervised learning of a 3D representation of a real-world scene according to one embodiment. The system pipeline 200 can be implemented to perform the method 100 of Fig. 1 in one embodiment. Therefore, the definitions and descriptions provided above can be applied equally to the present embodiment.

[0024] Learning to create a spatiotemporal representation of a constantly changing (e.g., urban) scene, for example with an autonomous agent, is not trivial due to sparse observations, complex compositions, unrestricted dynamics, and varying lighting. In contrast to previous efforts requiring human annotation or LiDARs, the System Pipeline 200 provides a self-monitored and exclusively camera-based paradigm. The learned representations aim for (1) completeness by holistically capturing all elements of the scene—including dynamic objects, static obstacles, and permanent street infrastructure; and (2) granularity, enabling the precise tracking of each instance over time.Therefore, the System Pipeline 200 enables holistic reconstruction, composite rendering and editable simulation, while reducing the need for costly annotations and high-end sensors.

[0025] System Pipeline 200 divides the scene into environment and object fields. The environment field is represented by 3D Gaussian splats, capturing permanent structures such as buildings and roads, while the object field uses 4D Gaussian splats, incorporating an additional dimension to model the likely motion dynamics of transient objects. Specifically, System Pipeline 200 first performs a warm-up phase to create a spatial memory represented by 3D Gaussian splats from multiple past traversals, primarily capturing permanent scene structures such as road infrastructure. It then uses this memory to initialize the current traversal, which differs from the original 3D Gaussian splat, initialized by structure-from-motion (SfM) points.Subsequently, system pipeline 200 adjusts the environment field to account for the changes in appearance, while an object field is created to model previously unseen instances, which can be either dynamic or static. Such adjustment of the environment and object detection is achieved simply by minimizing red-green-blue (RGB) rendering loss plus regularization loss between two fields and is therefore entirely self-monitoring.

[0026] Fig. Figure 3 illustrates an implementation of the system pipeline of Fig. 2 according to one embodiment.

[0027] In the present implementation, an autonomous agent, such as a self-driving vehicle, repeatedly traverses a route and captures images of the same route at different times. The collected images are temporally continuous within a single traverse but temporally separated between traverses. The time interval between successive traverses depends on how frequently the agent revisits the location and varies randomly from several hours to several days. Through repeated navigation, the agent can filter out the time-varying signals across traverses and build a spatial memory that captures the time-invariant components. The agent can then use this spatial memory as a basis for learning composite scene representations in subsequent traverses. Composite Gaussian splatting

[0028] The scene is divided into two fields or areas: the environment field, which corresponds to the time-invariant component (permanent structures), and the object field, which represents the time-varying components (transient entities). The object field contains both dynamic and static, yet transient or temporary objects.

[0029] Surrounding field. The one with G env The designated environment field is a set of 3D Gaussian distributions with attributes (µ, S, R, o, c), where µ ∈ R 3 the mean of the Gaussian distribution is S ∈ R 3 the scaling coefficient is, R ∈ R 4 the rotation quaternion, o ∈ R is the opacity, and c ∈ R dThe d-dimensional feature is defined. The mean µ is linked to the position of the Gaussian point in the 3D world, while S and R influence the shape of the Gaussian distribution. The opacity o determines the visibility of the Gaussian distribution, and c contains spherical harmonic coefficients and possibly self-monitored feature descriptors.

[0030] Object field. Transient entities during a single traverse or pass can be either dynamic or static. This is related to G. opj The designated object field additionally assigns dynamic properties to each Gaussian distribution, including (t0, τ, v, a), where t0 ∈ R and τ ∈ R represent the time of the maximum and the duration, respectively. The opacity of a Gaussian distribution is defined as o(t)=o⋅exp(−(t−t0)22τ2). The vectors v and a denote velocity and acceleration respectively, such that at time t the position of the corresponding Gaussian distribution is given by x(t)=μ+v0(t−t0)+12a(t−t0)2. By combining different segments of a simple uniformly accelerated linear motion, this parameterization can approximate a complex non-uniform motion. initialization

[0031] Environment field initialization. Using RGB images from several previous passes, a universal SfM pipeline (e.g., COLMAP) is used to jointly reconstruct sparse points and calculate camera positions. A differential rendering pipeline, such as 3DGS, is then used to learn the positions, rotations, scales, opacities, and features of the 3D Gaussian distributions. Since transient objects typically appear in only one pass or a few frames and occupy a small portion of the scene, they are automatically skipped to some extent in the 3D rendering. To further clean up the 3D environment rendering, self-supervised mapping techniques are used to remove these transient objects during optimization. Subsequently, the learned Gaussian distributions from previous passes form the G memo designated spatial memory, which is used to G envto initialize.

[0032] Object field initialization. Using RGB images in a new pass, the SfM pipeline is used to perform incremental registration, integrating new images into the previously reconstructed scene to obtain camera positions for the newly acquired data. Next, significant changes in the scene—indicating newly appearing objects in the latest pass—are identified by exploiting two types of inconsistencies. The first is a geometric inconsistency: A pre-trained depth model (e.g., DepthAnything) is used to estimate a depth map for each image in the current pass. The depth map is then used to construct a geometric inconsistency map using spatial memory. memoThe image is rendered, and the depth difference between the current depth and the depth from memory is calculated to represent geometric inconsistencies. Additionally, the RGB difference is calculated to quantify photometric inconsistencies. The difference calculations are simple subtractions, with both the depth map and the RGB image patched using a 4 × 4 patch grid to improve robustness. The dot product of the depth and RGB differences is calculated patch-wise, and patches with a difference value higher than a predefined threshold are selected.

[0033] It should be noted that RGB is sensitive to small changes, while depth estimation is inconsistent across different views, so combining these metrics yields more accurate results than using them separately. For each selected patch, it is not projected using its depth in world coordinates, and then the unprojected 3D point is used as the mean µ of the initialized Gaussian distribution. The feature c is derived from the average feature value of the patch, while the initial velocity v and acceleration a are set to zero. t0 is assigned based on the corresponding image, and the temporal decay factor τ is set as the time interval between two adjacent images. optimization

[0034] After initialization, the goal is to adapt the environment field and optimize the object field based on the latest observations. In some embodiments, however, the object field may capture parts of the environment and vice versa. To promote the disentanglement of the two fields, a regularization loss function is introduced. Lreg=oobj / sg[f(cobj,cenv)], where f is a specific difference measure, such as the L1 loss, and sg removes this term from the gradient calculation. This loss penalizes object opacity if the object and environment fields have similar RGB values, ensuring that the object field only generates Gaussian distributions when necessary. Additionally, a conservation loss is introduced. Lmaint, which uses spatial memory to monitor hidden pixels in the surrounding field, ensuring that areas obscured by newly appearing objects retain their structures. Ultimately, G env and G obj optimized by minimizing the following loss function of equation 1. L=Lreg+Lmaint+Lrender+Ldepth where Lrender denotes the RGB and (optionally) feature rendering loss, and where Depth This refers to the loss of depth smoothing. It should be noted that this entire training goal is purely self-monitored.

[0035] The System Pipeline 200 can be used for several different downstream tasks. In one embodiment, the System Pipeline 200 can be used for 2D object segmentation to effectively hide transient objects, such as vehicles and pedestrians. In another embodiment, the System Pipeline 200 can be used for photorealistic scene processing at the instance level, which in turn can be used for automated simulation technology in self-driving and robotics. In yet another embodiment, the System Pipeline 200 can be used for composite rendering, which can be used in autonomous driving scenarios.

[0036] Fig. Figure 4 illustrates a method 400 for using a 3D representation of a real scene for scene editing according to one embodiment. With respect to the present method 400, the 3D representation of the real scene is obtained via method 100 from Fig. 1 and / or the system pipeline 200 from Fig. 2 generated. In one embodiment, the method 400 can be carried out by an application located in a processing pipeline downstream of the system pipeline 200.

[0037] In step 402, a 3D representation of a real-world scene is received, generated from one or more images of the real-world scene. In step 404, the one or more images of the real-world scene are processed using the 3D representation. In one embodiment, one or more instructions for processing the one or more images can be entered via an image processing application or tool. In one embodiment, the instruction(s) can be entered by a user. In another embodiment, the instruction(s) can be entered as a text command, a mask, or other indicator superimposed on the one or more images, or a combination thereof.

[0038] Thus, one or more images of the real scene can be processed according to the instruction(s). In one embodiment, one or more images of the real scene can be processed to remove at least one object. In another embodiment, one or more images of the real scene can be processed to move at least one object within them. In yet another embodiment, one or more images of the real scene can be processed to transfer at least one object from one or more other images of the real scene into one or more images of the real scene.

[0039] In step 406, the one or more processed images of the real scene (resulting from step 404) are output. In one embodiment, the one or more processed images can be output to a display device. In another embodiment, the one or more processed images can be output to a further downstream application, such as a simulation application used to train an autonomous agent.

[0040] Fig. Figure 5 illustrates exemplary inputs and outputs of Procedure 400. Fig. 4 according to one embodiment. For example, Method 400 processes several input images (top row) to remove a selected object from them (bottom row). For object relocation, Method 400 processes several input images (top row) to relocate or shift a selected object within them (bottom row). For example, for chronoportation (i.e., a transfer between images), Method 400 transfers a selected object from several input source images (top row) to several input target images (middle row) such that the several source images contain the object (bottom row). Machine learning

[0041] Deep neural networks (DNNs), including deep learning models developed on processors, are used for a variety of applications, from self-driving cars to faster drug development, from automatic image labeling in online image databases to intelligent real-time speech translation in video chat applications. Deep learning is a technique that models the neural learning process of the human brain, continuously learning, becoming increasingly intelligent, and delivering faster and more accurate results over time. A child is initially taught by an adult to correctly identify and classify different shapes, eventually becoming able to identify shapes without guidance.Similarly, a deep learning or neural learning system must be trained in object recognition and classification to become more intelligent and efficient at identifying basic objects, hidden objects, etc., while also assigning context to objects.

[0042] At the simplest level, neurons in the human brain consider various incoming inputs, assign a weight to each input, and pass the output on to other neurons that respond accordingly. An artificial neuron, or perceptron, is the most basic model of a neural network. For example, a perceptron might receive one or more inputs representing different features of an object that the perceptron is to recognize and classify, and each of these features is assigned a weight based on its importance in defining the object's shape.

[0043] A deep neural network (DNN) model comprises multiple layers with many connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.) that can be trained on enormous amounts of input data to quickly solve complex problems with high accuracy. In one example, a first layer of the DNN model segments an input image of a car into different sections and looks for basic patterns such as lines and angles. The second layer orders the lines to search for higher-order patterns such as wheels, windshields, and mirrors. The next layer identifies the vehicle type, and the final few layers generate a label for the input image that identifies the model of a specific car brand.

[0044] Once the DNN is trained, it can be deployed and used to identify and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include identifying handwritten numbers on checks deposited at ATMs, identifying pictures of friends in photos, providing movie recommendations to over fifty million users, identifying and classifying different types of automobiles, pedestrians, and road hazards in self-driving cars, or translating human language in real time.

[0045] During training, data flows through the DNN in a forward propagation phase until a prediction is generated that displays a label matching the input. If the neural network does not correctly label the input, the errors between the correct label and the predicted label are analyzed, and the weights for each feature are adjusted during a backward propagation phase until the DNN correctly labels the input and other inputs in a training dataset. Training complex neural networks requires massive amounts of parallel computing power, including floating-point multiplication and addition.Inferencing is less computationally intensive than training because it is a latency-sensitive process in which a trained neural network is applied to new inputs it has not seen before to classify images, translate speech, and generally derive new information. INFERENCE AND TRAINING LOGIC

[0046] As noted above, a deep learning system or a neural learning system must be trained to generate inferences from input data. Details regarding inference and / or training logic 615 for a deep learning system or a neural learning system are given below in conjunction with Fig. 6A and / or Fig. 6B provided.

[0047] In at least one embodiment, the inference and / or training logic 615 may, without limitation, include a data store 601 for storing forward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network that is trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the data store 601 stores weight parameters and / or input / output data of each layer of a neural network that is trained or used in conjunction with one or more embodiments during the forward propagation of input / output data, and / or weight parameters during training and / or inference using aspects of one or more embodiments.In at least one embodiment, any section of the data memory 601 may be located in another on-chip or off-chip data memory, including the cache memory or system memory of a processor L1, L2 or L3.

[0048] In at least one embodiment, any section of the data memory 601 can be internal or external with respect to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the data memory 601 can be a cache memory, a dynamic directly addressable memory (“DRAM”), a static directly addressable memory (“SRAM”), a non-volatile memory (e.g., a flash memory), or another type of memory.In at least one embodiment, the choice of whether the data storage 601 is, for example, internal or external with respect to a processor, or consists of a DRAM, SRAM, Flash or another type of memory, may depend on available on-chip memory versus off-chip memory, latency requirements of training and / or inference functions being performed, batch size of data used in the inference and / or training of a neural network, or a combination of these factors.

[0049] In at least one embodiment, the inference and / or training logic 615 may, without limitation, include a data store 605 for storing backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network that is trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the data store 605 stores weight parameters and / or input / output data of each layer of a neural network that is trained or used in conjunction with one or more embodiments during backward propagation of input / output data, and / or weight parameters during training and / or inference using aspects of one or more embodiments.In at least one embodiment, any section of the data memory 605 can be located in another on-chip or off-chip data memory, including the cache memory or system memory of a processor L1, L2, or L3. In at least one embodiment, any section of the data memory 605 can be internal or external with respect to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the data memory 605 can be a cache memory, a DRAM, an SRAM, a non-volatile memory (e.g., a flash memory), or another type of memory.In at least one embodiment, the choice of whether the data storage 605 is, for example, internal or external with respect to a processor, or consists of a DRAM, SRAM, Flash or another type of memory, may depend on available on-chip memory versus off-chip memory, latency requirements of training and / or inference functions being performed, batch size of data used in the inference and / or training of a neural network, or a combination of these factors.

[0050] In at least one embodiment, the data memory 601 and the data memory 605 can be separate memory structures. In at least one embodiment, the data memory 601 and the data memory 605 can be the same memory structure. In at least one embodiment, the data memory 601 and the data memory 605 can be partly the same memory structure and partly separate memory structures. In at least one embodiment, any portion of the data memory 601 and the data memory 605 can be located in another on-chip or off-chip data memory, including the cache memory or system memory of a processor L1, L2, or L3.

[0051] In at least one embodiment, the inference and / or training logic 615 may, without limitation, include one or more arithmetic logic unit(s) (“ALU(s)”) 610 for performing logical and / or mathematical operations that are at least partially based on or specified by a training and / or inference code, the result of which may lead to activations (e.g., output values ​​of layers or neurons within a neural network) stored in an activation memory 620 and which are functions of input / output and / or weight parameter data stored in the data memory 601 and / or the data memory 605.In at least one embodiment, activations stored in the activation memory 620 are generated according to linear algebraic and / or matrix-based mathematics performed by the ALU(s) 610 in response to the execution of instructions or other code, wherein weight values ​​stored in the data memory 605 and / or the data 601 are used as operands together with other values, such as deviation values, gradient information, pulse values, or other parameters or hyperparameters, any or all of which may be stored in the data memory 605 or the data memory 601 or in another memory on or off the chip.In at least one embodiment, the ALU(s) 610 are contained within one or more processors or other hardware logic devices or circuits, whereas in another embodiment, the ALU(s) 610 may be external to a processor or other hardware logic device or circuit that uses them (e.g., a coprocessor). In at least one embodiment, the ALU(s) 610 may be contained within the execution units of a processor or otherwise in a bank of ALUs that can be accessed by the execution units of a processor, either within the same processor or distributed among different processors of different types (e.g., central processing units, graphics processing units, fixed-function units, etc.).In at least one embodiment, the data memory 601, the data memory 605, and the activation memory 620 can be located on the same processor or other hardware logic device or circuit, whereas in another embodiment, they can be located on different processors or other hardware logic devices or circuits, or a combination of the same and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of the activation memory 620 can be contained in another on-chip or off-chip data memory, including the cache memory or system memory of a processor L1, L2, or L3.Furthermore, inference and / or training code can be stored with other code that can be accessed by a processor or other hardware logic or circuitry and retrieved and / or processed using the retrieval, decoding, scheduling, execution, retirement, and / or other logical circuitry of a processor.

[0052] In at least one embodiment, the activation memory 620 can be a cache memory, a DRAM, an SRAM, a non-volatile memory (e.g., flash memory), or another type of memory. In at least one embodiment, the activation memory 620 can be located wholly or partially inside or outside one or more processors or other logic circuits. In at least one embodiment, the choice of whether the activation memory 620 is, for example, internal or external to a processor, or whether it consists of a DRAM, SRAM, flash memory, or another type of memory, can depend on available on-chip memory versus off-chip memory, the latency requirements of training and / or inference functions being performed, the batch size of data used in the inference and / or training of a neural network, or a combination of these factors.In at least one embodiment, the inference and / or training logic 615, which is described in . Fig. Figure 6A illustrates the use of the inference and / or training logic 615 in conjunction with an application-specific integrated circuit (“ASIC”), such as a Google Tensorflow® processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., “Lake Crest”). In at least one embodiment, the inference and / or training logic 615, which is illustrated in Figure 6A, can be used ... Fig. 6A illustrates how they can be used in conjunction with hardware of a central processing unit (“CPU”), hardware of a graphics processing unit (“GPU”) or other hardware, such as field-programmable gate arrays (“FPGAs”).

[0053] Fig. Figure 6B illustrates an inference and / or training logic 615 according to at least one embodiment. In at least one embodiment, the inference and / or training logic 615 can, without limitation, include hardware logic in which computing resources are dedicated or otherwise used exclusively in connection with weight values ​​or other information that correspond to one or more layers of neurons within a neural network. In at least one embodiment, the inference and / or training logic 615, which is described in Figure 6B, can include hardware logic in which computing resources are dedicated or otherwise used exclusively in connection with weight values ​​or other information that correspond to one or more layers of neurons within a neural network. Fig. Figure 6B illustrates this, in conjunction with an application-specific integrated circuit (ASIC), such as a Google Tensorflow® processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., “Lake Crest”). In at least one embodiment, the inference and / or training logic 615, which is illustrated in Figure 6B, can be used ... Fig. Figure 6B illustrates the inference and / or training logic 615, which can be used in conjunction with hardware of a central processing unit (CPU), a graphics processing unit (GPU), or other hardware, such as field-programmable gate arrays (FPGAs). In at least one embodiment, the inference and / or training logic 615 includes, without limitation, a data memory 601 and a data memory 605, which can be used to store weight values ​​and / or other information, including deviation values, gradient information, pulse values, and / or other parameter or hyperparameter information. In at least one embodiment, which is illustrated in Figure 6B, the inference and / or training logic 615 includes, without limitation, a data memory 601 and a data memory 605, which can be used to store weight values ​​and / or other information, including deviation values, gradient information, pulse values, and / or other parameter or hyperparameter information. Fig. As illustrated in Figure 6B, each of the data storage 601 and the data storage 605 is associated with a dedicated computing resource, such as computing hardware 602 and computing hardware 606, respectively. In at least one embodiment, each of the computing hardware 606 comprises one or more ALUs that perform mathematical functions, such as linear algebraic functions, using only information stored in the data storage 601 and the data storage 605, respectively, and whose result is stored in the activation memory 620.

[0054] In at least one embodiment, each of the data storage devices 601 and 605 and the corresponding computing hardware 602 and 606, respectively, correspond to different layers of a neural network, such that the resulting activation of one "memory / computing pair 601 / 602" of the data storage device 601 and the computing hardware 602 is provided as an input for the next "memory / computing pair 605 / 606" of the data storage device 605 and the computing hardware 606, in order to mirror the conceptual organization of a neural network. In at least one embodiment, each of the memory / computing pairs 601 / 602 and 605 / 606 can correspond to more than one layer of a neural network. In at least one embodiment, additional memory / computing pairs (not shown) may be included following or in parallel to the memory / computing pairs 601 / 602 and 605 / 606 in the inference and / or training logic 615. TRAINING AND USE OF A NEURAL NETWORK

[0055] Fig. Figure 7 illustrates another embodiment for training and deploying a deep neural network. In at least one embodiment, an untrained neural network 706 is trained using a training dataset 702. In at least one embodiment, the training framework 704 is a PyTorch framework, whereas in other embodiments, the training framework 704 is a TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training framework. In at least one embodiment, the training framework 704 trains an untrained neural network 706 and enables it to be trained using processing resources described herein to generate a trained neural network 708. In at least one embodiment, weights can be randomly generated or pre-trained using a deep belief network or a similar training framework.A deep probabilistic network can be selected. In at least one embodiment, the training can be performed in a supervised, partially supervised, or unsupervised manner.

[0056] In at least one embodiment, an untrained neural network 706 is trained using supervised learning, wherein the training dataset 702 contains an input paired with a desired output for an input, or wherein the training dataset 702 contains an input paired with a known output, and the output of the neural network is manually evaluated. In at least one embodiment, an untrained neural network 706 is trained in a supervised manner, processes inputs from the training dataset 702, and compares the resulting outputs with a set of expected or desired outputs. In at least one embodiment, errors are then propagated back by the untrained neural network 706. In at least one embodiment, the training framework 704 adjusts weights that control the untrained neural network 706.In at least one embodiment, the training framework 704 includes tools to monitor how well the untrained neural network 706 converges to a model, e.g., the trained neural network 708, which is suitable for generating correct responses, such as result 714, based on known input data, such as the new data 712. In at least one embodiment, the training framework 704 repeatedly trains the untrained neural network 706 while adjusting weights to refine an output of the untrained neural network 706 using a loss function and an adjustment algorithm, such as stochastic gradient descent. In at least one embodiment, the training framework 704 trains the untrained neural network 706 until the untrained neural network 706 achieves a desired accuracy.In at least one embodiment, the trained neural network 708 can then be used to perform any number of machine learning operations.

[0057] In at least one embodiment, the untrained neural network 706 is trained using unsupervised learning, wherein the untrained neural network 706 attempts to train itself using unlabeled data. In at least one embodiment, the training dataset 702 for unsupervised learning includes input data without associated output data or "ground truth" data. In at least one embodiment, the untrained neural network 706 can learn groupings within the training dataset 702 and determine how individual inputs relate to the untrained dataset 702. In at least one embodiment, the unsupervised training can be used to generate a self-organizing map, which is a type of trained neural network 708 capable of performing operations useful in reducing the dimensionality of the new data 712.In at least one embodiment, the unsupervised training can also be used to perform anomaly detection, enabling the identification of data points in a new data set 712 that deviate from normal patterns of the new data set 712.

[0058] In at least one embodiment, semi-supervised learning can be used, which is a technique in which the training dataset 702 contains a mixture of labeled and unlabeled data. In at least one embodiment, the training framework 704 can be used to perform incremental learning, such as through transferred learning techniques. In at least one embodiment, the incremental learning enables the trained neural network 708 to adapt to the new data 712 without forgetting the knowledge input into the neural network during the initial training. DATA CENTER

[0059] Fig. Figure 8 illustrates an exemplary data center 800, in which at least one embodiment can be used. In at least one embodiment, the data center 800 includes a data center infrastructure layer 810, a framework layer 820, a software layer 830, and an application layer 840.

[0060] In at least one embodiment, as in Fig. As shown in Figure 8, the data center infrastructure layer 810 can include a resource orchestrator 812, clustered compute resources 814, and node compute resources (“node RR”) 816(1) to 816(N), where “N” is any positive integer. In at least one embodiment, the node RR 816(1)-816(N) can include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including accelerators, field-programmable gate arrays (“FPGAs”), graphics processing units, etc.), memory devices (e.g., dynamic read-only memory), data storage devices (e.g., solid-state or disk drives), network input / output devices (“NW I / O” devices), network switches, virtual machines (“VMs”), power modules, and cooling modules, etc.In at least one embodiment, one or more Node-RRs of Node-RR 816(1)-816(N) can be a server comprising one or more of the aforementioned computing resources.

[0061] In at least one embodiment, the grouped compute resources 814 can include separate groupings of node RRs housed in one or more racks (not shown), or many racks housed in data centers at different geographic locations (also not shown). Separate groupings of node RRs within grouped compute resources 814 can include grouped compute, network, storage, or memory resources that can be configured or allocated to support one or more workloads. In at least one embodiment, multiple node RRs, including CPUs or processors, can be grouped in one or more racks to provide compute resources to support one or more workloads.In at least one embodiment, one or more racks can also include any number of power modules, cooling modules and network switches in any combination.

[0062] In at least one embodiment, the resource orchestrator 822 can configure or otherwise control one or more node RR 816(1) to 816(N) and / or grouped computing resources 814. In at least one embodiment, the resource orchestrator 822 can include a management unit of a software design infrastructure (“SDI”) for the data center 800. In at least one embodiment, the resource orchestrator can include hardware, software, or a combination thereof.

[0063] In at least one embodiment, as in Fig. As shown in Figure 8, the framework layer 820 includes a task scheduler 832, a configuration manager 834, a resource manager 836, and a distributed file system 838. In at least one embodiment, the framework layer 820 can include a framework to support software 832 of software layer 830 and / or one or more applications 842 of application layer 840. In at least one embodiment, the software 832 or the application(s) 842 can each be web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 820 can, without restriction, be a type of web application framework for free and open-source software, such as Apache Spark™ (hereinafter “Spark”), which can use the distributed file system 838 for large-scale data processing (e.g., “Big Data”).In at least one embodiment, the task scheduler 832 can include a Spark driver to facilitate the scheduling of workloads supported by various layers of the data center 800. In at least one embodiment, the configuration manager 834 can configure various layers, such as the software layer 830 and the framework layer 820, including Spark and the distributed file system 838, to support large-scale data processing. In at least one embodiment, the resource manager 836 can manage clustered or grouped compute resources allocated or assigned to support the distributed file system 838 and the task scheduler 832. In at least one embodiment, the clustered or grouped compute resources can include a grouped compute resource 814 on the data center infrastructure layer 810.In at least one embodiment, the resource manager 836 can coordinate with the resource orchestrator 812 to manage these allocated or assigned computing resources.

[0064] In at least one embodiment, the software 832, which is included in software layer 830, may include software that is used by at least parts of node RR 816(1) to 816(N), grouped computing resources 814, and / or the distributed file system 838 of framework layer 820. One or more types of software may include, but are not limited to, web browsing software, email scanning software, database software, and streaming video content software.

[0065] In at least one embodiment, the application(s) 842 included in the application layer 840 may include one or more types of applications used by at least parts of node RR 816(1) to 816(N), grouped compute resources 814, and / or the distributed file system 838 of the framework layer 820. One or more types of applications may include any number of a genomics application, a cognitive compute application, and a machine learning application, including, but not limited to, training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0066] In at least one embodiment, the configuration manager 834, the resource manager 836, and the resource orchestrator 812 can implement any number and type of self-modifying actions based on any amount and type of data acquired in any technically feasible manner. In at least one embodiment, self-modifying actions can relieve a data center operator of the data center 800 from potentially making poor configuration decisions and potentially avoiding underutilized and / or underperforming sections of a data center.

[0067] In at least one embodiment, the Data Center 800 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or deriving information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weighting parameters according to a neural network architecture using software and computing resources previously described with reference to the Data Center 800.In at least one embodiment, trained machine learning models corresponding to one or more neural networks can be used to infer or predict information using resources previously described in relation to the Computing Center 800 and weighting parameters calculated using one or more training techniques described in this document.

[0068] In at least one embodiment, the data center can use CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above can be configured as a service to allow users to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0069] The inference and / or training logic 615 is used to perform inference and / or training operations that are associated with one or more embodiments. In at least one embodiment, the inference and / or training logic 615 can be implemented in the system consisting of Fig. 8 for inference or prediction operations, at least partially based on weighting parameters calculated using neural network training operations, neural network functions and / or neural network architectures or neural network use cases described in this document.

[0070] As it is here with regard to the Fig. As described in Figures 1-5, a method, a computer-readable medium, and a system for using a diffusion model to generate a 2D image of a scene from a scene representation comprising 3D blobs are disclosed. The diffusion model can be stored (partially or completely) in one or both data stores 601 and 605 in the inference and / or training logic 615, as described in the Fig. 6A and Fig. Figure 6B illustrates the training and deployment of the diffusion model. Fig. 7 is shown and can be carried out as described here. The distribution of the diffusion model can be carried out using one or more servers in a data center 800, as shown in Fig. 8 shown and as described here.

Claims

[1] Method comprising: on a device: Creating a three-dimensional, 3D, representation of a real scene of: at least one initial image capturing the real scene at an initial point in time, and a representation of time-invariant components of the real scene, generated by one or more second images that capture the real scene at one or more second points in time prior to the first point in time; Processing the at least one initial image using the 3D representation of the real scene to produce at least one processed image, wherein the processing includes at least one of: Removing at least one object from at least one initial image of the real scene, Relocating at least one object within the first image of the real scene, or Transferring at least one object from the one or more second images into the at least one first image of the real scene; and Output of at least one edited image. [2] Method according to claim 1, wherein the representation of the time-invariant components of the real scene comprises 3D Gaussian distributions of the time-invariant components of the real scene. [3] Method according to claim 1 or 2, further comprising, on the device: Recognizing, in at least one initial image, transient components of the real scene at that initial time; and Generating a representation of the transient components of the real scene at the first time point; where the 3D representation of the real scene at the first time point is furthermore generated based on the representation of the transient components of the real scene. [4] The method of claim 3, further comprising, on the device: Optimizing the representation of the transient components of the real scene and the representation of the time-invariant components of the real scene by minimizing at least one loss function. [5] The method of claim 4, wherein the at least one loss function is at least one of: a regularization loss function that promotes a separation of the representation of the transient components of the real scene and the representation of the time-invariant components of the real scene, a conservation loss function that monitors hidden pixels in the representation of the time-invariant components of the real scene to preserve the structure of one or more of the time-invariant components that are hidden by one or more of the transient components, a rendering loss function, or a depth smoothing loss function. [6] Method according to claim 4 or 5, wherein the 3D representation of the real scene consists of the optimized representation of the transient components of the real scene and the optimized representation of the time-invariant components of the real scene. [7] Method comprising: on a device: Accessing at least one initial image that captures a real scene at an initial point in time; Accessing a representation of time-invariant components of the real scene, generated by one or more second images that capture the real scene at one or more second time points prior to the first time point; Generating a three-dimensional, 3D, representation of the real scene at the first time point using at least one first image and the representation of the time-invariant components of the real scene. [8] Method according to claim 7, wherein the at least one first image includes at least one image of a video capturing the real scene at the first time. [9] Method according to claim 7 or 8, wherein the at least one first image is generated by a camera installed on an autonomously driving vehicle traversing the real scene. [10] Method according to any one of claims 7 to 9, wherein the one or more second images comprise one or more second images of one or more videos capturing the real scene at the one or more second time points prior to the first time point. [11] Method according to any one of claims 7 to 10, wherein the time-invariant components of the real scene include static objects in the real scene. [12] Method according to any one of claims 7 to 11, wherein a spatial memory includes the representation of the time-invariant components of the real scene. [13] Method according to any one of claims 7 to 12, wherein the representation of the time-invariant components of the real scene includes 3D Gaussian distributions of the time-invariant components of the real scene. [14] Method according to claim 13, wherein the 3D Gaussian distributions are learned using 3D Gaussian splatting (3DGS). [15] Method according to any one of claims 7 to 14, wherein the representation of the time-invariant components of the real scene excludes transient components of the real scene. [16] Method according to any one of claims 7 to 15, further comprising, on the device: Recognizing, in at least one initial image, transient components of the real scene at that initial time; and Generating a representation of the transient components of the real scene at the first time point; where the 3D representation of the real scene at the first time point is furthermore generated based on the representation of the transient components of the real scene. [17] Method according to claim 16, wherein the transient components of the real scene are detected at the first time based on: a depth difference between the at least one first image and the representation of the time-invariant components of the real scene, and a color difference between the at least one first image and the representation of the time-invariant components of the real scene. [18] Method according to claim 16 or 17, wherein the representation of the transient components of the real scene includes 4D Gaussian distributions of the transient components of the real scene. [19] Method according to any one of claims 16 to 18, further comprising, on the device: Optimizing the representation of the transient components of the real scene and the representation of the time-invariant components of the real scene by minimizing at least one loss function. [20] Method according to claim 19, wherein the at least one loss function is a regularization loss function which promotes a separation of the representation of the transient components of the real scene and the representation of the time-invariant components of the real scene. [21] Method according to claim 19 or 20, wherein the at least one loss function is a conservation loss function that monitors hidden pixels in the representation of the time-invariant components of the real scene to preserve the structure of one or more of the time-invariant components that are hidden by one or more of the transient components. [22] Method according to any one of claims 19 to 21, wherein the at least one loss function is a rendering loss function. [23] Method according to any one of claims 19 to 22, wherein the at least one loss function is a depth smoothing loss function. [24] Method according to any one of claims 19 to 23, wherein the 3D representation of the real scene consists of the optimized representation of the transient components of the real scene and the optimized representation of the time-invariant components of the real scene. [25] Method according to any one of claims 7 to 24, further comprising, on the device: Outputting the 3D representation of the real scene to a downstream task. [26] Method according to claim 25, wherein the downstream task uses the 3D representation of the real scene to perform an image synthesis comprising at least one of: Removing at least one object from at least one initial image of the real scene, Relocating at least one object within the first image of the real scene, or Transferring at least one object from one or more second images into at least one first image of the real scene. [27] System encompassing: a non-volatile memory containing instructions; and one or more processors in communication with memory, wherein the one or more processors execute the instructions to: to access at least one initial image that captures a real scene at an initial point in time; to access a representation of time-invariant components of the real scene, generated by one or more second images that capture the real scene at one or more second time points prior to the first time point; and to generate a three-dimensional (3D) representation of the real scene at the first time point using at least one first image and the representation of the time-invariant components of the real scene. [28] Non-volatile, computer-readable medium that stores computer instructions which, when executed by one or more processors of a device, cause the device to: to access at least one initial image that captures a real scene at an initial point in time; to access a representation of time-invariant components of the real scene, generated by one or more second images that capture the real scene at one or more second time points prior to the first time point; and to generate a three-dimensional, 3D, representation of the real scene at the first time point using at least one first image and the representation of the time-invariant components of the real scene.

Images