Occupancy grid for neural radiation fields

Abstract

An example apparatus disclosed herein queries a neural network for optical densities at sample points along training rays associated with training the neural network to provide a neural representation of the video frame. The disclosed example apparatus also generates the occupancy grid for the video frame based on the optical density at sample points along the training ray, the occupancy grid including voxels for indicating whether a respective portion of a three-dimensional (3D) volume associated with the neural representation is occupied by a geometry.

Description

[0001] Related Applications

[0002] This patent claims the benefit of U.S. Provisional Patent Application No. 63 / 632,366, filed April 10, 2024. U.S. Provisional Patent Application No. 63 / 632,366 is hereby incorporated by reference in its entirety. Priority is hereby claimed to U.S. Provisional Patent Application No. 63 / 632,366. BACKGROUND

[0003] Machine learning models, such as neural networks, multilayer perceptrons, and the like, can be configured to implement a neural representation of a three-dimensional (3D) scene based on a set of two-dimensional (2D) images of the scene associated with a set of reference viewpoints. The neural representation, also referred to as a neural radiance field, is trained to encode structural and color information that can be used to render 2D images of the scene from viewpoints that can be different from the reference viewpoints. A sequence of neural representations or neural radiance fields can also be trained to render respective video frames of a video of the 3D scene. Thus, the sequence of neural representations or neural radiance fields form a neural video of the scene that can be rendered from different viewpoints and thus provide an immersive video experience. BRIEF DESCRIPTION OF DRAWINGS

[0004] FIGS. 1A-1B Details of an example neural network implementing an example neural representation of a scene are shown.

[0005] FIG. 2 An example implementation of a neural network corresponding to an example multilayer perceptron (MLP) is shown. FIGS. 1A-1B

[0006] FIG. 3 An example neural representation training circuit for training a neural representation of a scene based on an occupancy grid in accordance with the teachings of this disclosure is shown. FIG. 2 An example neural representation training circuit for training a neural representation of a scene based on an occupancy grid in accordance with the teachings of this disclosure is shown.

[0007] FIGS. 4-6 An example operation of the neural representation training circuit for constructing an occupancy grid for an example neural network in accordance with the teachings of this disclosure is shown. FIG. 3 An example operation of the neural representation training circuit for constructing an occupancy grid for an example neural network in accordance with the teachings of this disclosure is shown. FIG. 2

[0008] FIG. 7 FIGS. 8A-8B are flowcharts representing example machine-readable instructions and / or example operations that can be performed, instantiated, and / or implemented by example programmable circuitry to implement the neural representation training circuit of FIG. 3

[0009] FIG. 9 ​​​​is a block diagram of an example processing platform that includes programmable circuitry that is structured to execute, instantiate, and / or perform example machine-readable instructions and / or implement FIG. 7 and FIGS. 8A-8B example operations of FIG. 3 to implement neural representation training circuitry of

[0010] FIG. 10 is a block diagram of an example implementation of programmable circuitry of FIG. 9

[0011] FIG. 11 is a block diagram of another example implementation of programmable circuitry of FIG. 9

[0012] FIG. 12 is a block diagram of an example software / firmware / instruction distribution platform (e.g., one or more servers) for distributing software, instructions, and / or firmware (e.g., corresponding to example machine-readable instructions of FIG. 7 and FIGS. 8A-8B to client devices associated with end users and / or consumers (e.g., for licensing, sale, and / or use), retailers (e.g., for sale, resale, licensing, and / or sub-licensing), and / or original equipment manufacturers (OEMs) (e.g., for inclusion in products to be distributed to, e.g., retailers and / or other end users (e.g., direct purchase customers)).

[0013] Generally, like reference numbers will be used to refer to like or similar components throughout the attached drawings and accompanying written description. DETAILED DESCRIPTION

[0014] ​​Images, audio, video, 3D objects, and most media are typically captured and represented by discrete samples. For example, an image is typically represented as a grid of discrete samples (e.g., an array of pixels), where each pixel has a number representing the intensity of red, green, and blue colors at that image location. In contrast, a neural representation is a neural network that stores content (e.g., an image) in the weights and structure of a neural network. Neural representations are becoming ubiquitous in visual computing because they offer advantages in compactness and continuous representation. In particular, they enable a new way of representing and rendering a realistic depiction of a real scene given several photos of that scene by enabling joint reconstruction and encoding of complex geometry and radiance distribution of a three-dimensional (3D) scene from a set of two-dimensional (2D) color images taken from different viewpoints. For example, such a neural network can receive (x, y, z) coordinates as input and output (R, G, B) pixel values representing color distribution in 3D space.

[0015] Neural Radiance Fields (NeRF) is a particular neural representation that trains a neural network (e.g., a multi-layer perceptron (MLP)) to represent the appearance of a 3D scene. NeRF takes several images of a static scene as input, where the cameras are at known positions, and uses deep learning techniques to train a neural network to achieve a neural representation of the scene. Rendering an image from a NeRF is done by volume rendering, which involves repeated querying of the neural network to determine color and opacity at many points along a ray that passes through an input pixel location, such as an input (x, y) coordinate.

[0016] The original NeRF algorithm uses a large multi-level perceptron (MLP) as its main neural network, and the original NeRF algorithm is slow due to the large number of MLP queries that are produced by sampling along many rays. Some techniques to improve NeRF performance include employing sample pruning to eliminate queries for samples along rays that hit empty or occluded regions. Being able to eliminate queries early by sample pruning is an important technique to achieve good NeRF rendering performance.

[0017] Some techniques for pruning spatial queries in empty space or occluded regions are based on storing occupancy information in a spatial data structure. Such pruning techniques involve voxelizing the volume of interest (e.g., corresponding to a 3D scene to be represented by NeRF) using a regular grid and storing an explicit occupancy (e.g., density) value in each voxel. Pruning techniques use this occupancy grid to prune samples located in voxels with low occupancy. Existing pruning techniques update the occupancy information stored in the occupancy grid by querying the NeRF neural network during training to sample density in random locations and storing the maximum optical density value in the corresponding voxel of the occupancy grid. In order to gradually introduce empty space learned by the neural network, the stored density values ​​are subjected to exponential moving average (EMA) decay, which marks a voxel as empty if a density above a certain threshold has not been observed through random sampling for a period of time.

[0018] However, this random occupancy sampling may miss fine details and cause artifacts, as described in further detail. Moreover, in the case of incremental training for video / dynamic data, called Incremental Neural Video (INV), this problem has a cumulative effect. INV is a NeRF-based technique that supports the creation and streaming of 3D video scenes based on an incoming multi-view video stream. INV is based on incrementally training NeRF by training a neural network (e.g., MLP) for new frames by reusing the MLP weights from previous frames. However, in the case of a previous occupancy grid based on random sampling of the volume of interest, the space of voxels is excluded from gradient propagation once they are marked as empty in the occupancy grid and may never be updated. As a result, details gradually disappear from frame to frame and no combination of hyperparameters can fix it.

[0019] The example occupancy grid generation techniques disclosed herein provide a robust way to update the occupancy grid of a neural representation (e.g., NeRF), thereby enabling artifact-free, fast training and rendering of static scenes and incremental neural videos. Instead of relying on random sampling as in the prior art, the examples disclosed herein mark voxels of the occupancy grid as occupied using density data from training ray marches, as the training rays will hit the geometry (if the geometry exists). Moreover, instead of storing the maximum detected density in the occupied grid voxel, at least some of the disclosed example occupancy grid generation techniques store a fixed predefined number (when the light density is detected at the voxel location), thereby avoiding the need for atomic operations when performing parallel updates. In order to enable the neural network to adapt to geometric changes in 3D video sequences, at least some of the disclosed examples delay the occupancy grid test and EMA decay operations at the beginning of training for a new frame.

[0020] The examples disclosed herein fix deficiencies of at least some existing sampling-based pruning techniques for neural radiance fields and enable fast rendering and training of artifact-free neural representations. In addition, the disclosed examples are applicable to neural videos and are efficiently supported on hardware (e.g., discrete GPUs, integrated GPUs, next-generation NPUs, etc.).

[0021] FIGS. 1A-1B Details of an example NeRF neural network are shown. FIG. 1A A conceptual diagram of training a NeRF is shown. FIG. 1B Input and output details of a NeRF neural network are shown. As FIG. 1A shown, several images 100 of a three-dimensional scene can be taken from known positions. A training operation 105 can be performed to construct a neural representation of the three-dimensional (3D) scene from those images. The trained neural representation is a NeRF that represents the scene via the weights of a neural network model. The NeRF can be used to render the scene 110 from viewpoints other than the viewpoints of the input images. As FIG. 1B shown, the trained NeRF can receive a five-dimensional (5D) input 115 that includes a 3D position (x, y, z) and a 2D direction (Θ and Φ). The NeRF can be a neural network 120 (e.g., an MLP) that is trained to respond to the 5D input 115 to produce output 125 color values (R, G, B) and a density σ. Volume rendering techniques can repeatedly access the neural network 120 to render the 3D scene from arbitrary viewpoints to accumulate color and density into 2D images. θ

[0022] FIG. 2 ​An example neural network 200 is shown that implements an MLP that can be trained to provide a neural representation (e.g., NeRF) of a static 3D scene, 3D video (e.g., INV), etc. The neural network 200 is an MLP, which is a fully connected neural network used for NeRF implementations. The neural network 200 of the example shown includes multiple fully connected layers 211, each having 256 channels and an example rectified linear unit (ReLU) activation function 212. An example position input 210A (y(x), e.g., 3D position (x, y, z)) is provided at a first fully connected layer 211 and propagates through eight fully connected layers. An additional instance of the position input 210B is provided to an example fifth fully connected layer 213 via a skip connection. The position input 210B is combined with the fifth fully connected layer 213 via vector concatenation using an example combination function 221. An example eighth fully connected layer 214 outputs a first example output layer 233 that outputs an example volume density 230 of a 256-dimensional feature vector and a position y(x). The 256-dimensional feature vector is concatenated with an example viewing direction 232 (y(d), e.g., 2D direction (0, F)) via an example combination function 231 and processed via a second example output layer 235 to generate an example RGB value 240 for the position y(x) when viewed from the direction y(d). For example, instead of directly using the position x and direction d inputs, the neural network 200 applies an encoding function y to convert the 3 position coordinates x into a multi-channel input with 60 channels for the position and the 2 direction coordinates d into 24 channels for the direction. The use of the encoding function can improve the ability of the neural network representation to capture high frequencies. Examples of the encoding function y include frequency encoding, Fourier features, multi-resolution hash tables, etc.

[0023] The neural network 200 enables the synthesis of new images of a 3D scene as seen from a desired viewpoint at a particular time, even if that viewpoint is not directly captured by a video camera. The neural network 200 also supports INV. As described above, the INV method takes as input multiple video streams with camera parameters (e.g., intrinsic and / or extrinsic parameters) for each stream, where each stream captures the same scene from a different viewpoint. The INV maintains a NeRF-like MLP (e.g., the neural network 200) that is up to date with the current timestamp of the video streams. The NeRF-like MLP 200 is incrementally updated for each incoming time frame (corresponding to multiple views in space). The incremental training uses the MLP weights from the previous frame to train the MLP 200 for the next frame. Thus, the MLP 200 can automatically reuse learning from previous frames and adjust the MPL layer weights for new frames. In this frame-to-frame incremental training mode, the MLP weights exhibit the following behavior: the front layers change from frame to frame, while the back layers change slowly or remain unchanged. This behavior occurs spontaneously in the INV NeRF MPL 200. As described above, experiments have shown that the front layers 250 of the MPL 200 are primarily responsible for encoding motion / deformation, while the back layers 260 primarily encode the colors of the 3D scene patches.

[0024] FIG. 3 is a neural network 200 for training FIG. 3 occupancy grid for the neural network 200. FIG. 3 The neural representation training circuit 300 of the neural network 200 can be instantiated (e.g., created as an instance, brought into existence for any length of time, materialized, realized, etc.) by programmable circuitry (e.g., a central processing unit (CPU)) executing first instructions. Additionally or alternatively, FIG. 3 The neural representation training circuit 300 of the neural network 200 can be instantiated (e.g., created as an instance, brought into existence for any length of time, materialized, realized, etc.) by (i) an application-specific integrated circuit (ASIC) and / or (ii) a field-programmable gate array (FPGA), where the application-specific integrated circuit (ASIC) and / or the field-programmable gate array (FPGA) are structured and / or configured to implement operations corresponding to the first instructions in response to execution of second instructions. Thus, it should be understood that FIG. 3 Some or all of the circuitry in the neural network 200 can be instantiated at the same or different times. For example, FIG. 3 Some or all of the circuitry in the neural network 200 can be executed concurrently on hardware and / or instantiated in one or more threads executed serially on hardware. Moreover, in some examples, FIG. 3Some or all of the circuitry in may be implemented by microprocessor circuitry executing instructions and / or FPGA circuitry performing operations to implement one or more virtual machines and / or containers.

[0025] The example neural representation training circuitry 300 includes an example neural network training circuitry 305, an example optical density query circuitry 310, an example occupancy grid generation circuitry 315, and an example occupancy grid storage 320. The neural network training circuitry 305 implements any suitable neural network training technique or combination of techniques to train the layer weights of the neural network 200 to implement a neural representation of the 3D scene based on training data stored in an example training data storage 325. In the illustrated example, the training data storage 325 can be implemented by any number and / or type of storage devices, memories, etc., and stores training data including a plurality of images captured from different viewpoints of the 3D scene (also referred to herein as multi-view images), a plurality of video streams from different viewpoints of the 3D scene (also referred to herein as multi-view video streams), etc. For example, the neural network training circuitry 305 can implement a forward propagation technique, a backward propagation technique, etc., or any combination thereof, to incrementally train the neural network 200 based on the multi-view video streams stored in the training data storage 325 to implement INV of a stream of output NeRFs that enables rendering of video frames from viewpoints other than the viewpoint of the input multi-view video stream. Additional example training techniques that may be implemented by the neural network training circuitry 305 are described in U.S. Patent Publication No. 2024 / 0135483, entitled “INCREMENTAL NEURALREPRESENTATION FOR FAST GENERATION OF DYNAMIC FREE-VIEWPOINT VIDEOS,” published on April 25, 2024.

[0026] exist FIG. 3 In the illustrated example, neural network 200 implements NeRF, which is a technique for representing the geometry and directional lighting information of a 3D scene. It is based on a positional neural representation (e.g., a combination of a grid, a hash grid, and a neural network, e.g., an MLP) that takes as input the 3D position and direction vector of a point and produces the light density (e.g., opacity) at that point and the radiance leaving the point in a given direction. The resulting representation is compact and expressive, enabling highly detailed and realistic depictions of 3D scenes. NeRF can also be generalized to other types of geometric and non-geometric data, e.g., signed distance functions, electromagnetic fields, etc.

[0027] Once trained by the neural network training circuitry 305, the neural network 200 can render images of a 3D scene from arbitrary viewpoints using volume rendering via ray marching. Ray marching involves projecting a ray for each image pixel, sampling the neural network at points along the ray, and accumulating transparency and weighted radiance to produce the final pixel color.

[0028] The neural network training circuitry 305 trains the neural network 200 to implement a neural representation of a NeRF that fits 3D scene data from images taken from different viewpoints with reference (e.g., known) camera parameters. The neural network training circuitry 305 uses such training views to generate training rays. During training, the neural network training circuitry 305 invokes the neural network 200 to perform volume rendering using the training rays to generate pixel colors that are compared to true pixel colors in the training data. The neural network training circuitry 305 adjusts the weights of the neural network 200 using color differences in a loss function.

[0029] Volume rendering involves querying the underlying neural network 200 at multiple points along a ray. Millions of rays can be used during training and rendering, resulting in hundreds of millions of network queries. To address computational complexity, some volume rendering techniques focus on making each query cheaper (e.g., by simplifying the underlying MLP) and / or reducing the number of queries by skipping sampling points that hit empty space.

[0030] Techniques that reduce the number of queries by skipping sampling points that hit empty space have proven effective, in some cases enabling 75-80% of samples to be removed early in the pipeline. Such sample pruning relies on a method to distinguish, without performing a full network query, whether space at a given 3D location is empty or occupied. Some such pruning techniques use a regular 3D grid, referred to as an occupancy grid, as a helper structure that stores floating-point values indicating whether a grid voxel is occupied. FIG. 4 An example occupancy grid 400 is shown in FIG. 4.

[0031] FIG. 4The example occupancy grid 400 includes voxels (e.g., example voxel 405 and voxel 410) that represent respective 3D portions (e.g., regions) of a 3D volume corresponding to a 3D scene represented by a neural representation (e.g., a NeRF) to be implemented by the neural network 200. Existing occupancy grid techniques store an approximate maximum density of geometry in each voxel. During ray marching, samples along a ray are tested against the grid to determine whether they hit voxels with a density below a threshold, which indicates empty voxels, and if so, those samples are skipped accordingly. In some examples, the voxels of the occupancy grid 400 are also associated with respective occupancy flags, which can be 1-bit values that are asserted or de-asserted to indicate whether the corresponding voxel is occupied by geometry. In FIG. 4 In examples, the shadow voxels (e.g., voxel 405) of the occupancy grid 400 have had the occupancy flag asserted (e.g., set to a logical -1) to indicate that those voxels are associated with space occupied by geometry, while the non-shadow voxels (e.g., voxel 410) of the occupancy grid 400 have had the occupancy flag de-asserted (e.g., set to a logical 0) to indicate that those voxels are associated with space not occupied by geometry.

[0032] At the start of training, the neural network has no information about the 3D scene. Existing pruning techniques attempt to gradually fill the occupancy grid 400 with the densities learned by the network over time by periodically sampling the density at a random location within each voxel and updating the voxel occupancy value with the maximum between the current voxel value in the grid and the density detected at the sampled location in the voxel. Also, in some such maximum density occupancy grid techniques, the density value of a given voxel is reduced by an exponentially moving average (EMA) decay at each update iteration, such that the occupancy grid value of a voxel that has not detected a sample with a high density for a long time will fall below the threshold indicating an empty voxel. In some maximum density occupancy grid techniques, the maximum density occupancy grid 400 is converted to a 1-bit voxel grid that encodes whether a given voxel is empty before performing ray marching.

[0033] Despite being fast and capable of pruning up to 80% of samples, such existing occupancy grid techniques result in artifacts. FIG. 5 The root cause of this deficiency is shown, namely, the use of random sampling to test the density inside a voxel. FIG. 5 An example occupancy grid 500 is shown in which the grid voxels are updated using random sampling. FIG. 5points in the volume (e.g., example point 505) represent sample points selected based on random sampling. For samples corresponding to geometry (e.g., sample 505), the neural network 200 will return a relatively high optical density, while for samples corresponding to empty space (e.g., sample 510), the neural network 200 will return a relatively low optical density. Voxels with sample points having a high optical density (e.g., exceeding a threshold) will be considered occupied, while voxels with sample points having a low optical density (e.g., not exceeding a threshold) will be considered empty. As FIG. 5 As shown by the example of FIG. 5, random sampling can miss small geometric details and falsely declare voxels empty (e.g., example voxel 515). Moreover, once a voxel is declared empty, all samples hitting that voxel are excluded from optimization during training, so the network will never have a chance to recover those details.

[0034] The disclosed example occupancy grid technique implemented by the neural representation training circuit 300 uses density values computed during training in a density probing ray marching pre-pass to fill the occupancy grid, rather than relying entirely on random sampling of the 3D space. FIG. 6 Example operations of the disclosed occupancy grid technique implemented by the neural representation training circuit 300 are shown in FIG. 6.

[0035] FIG. 6 An example occupancy grid 600 generated by the neural representation training circuit 300 is shown, and in particular, an example occupancy grid 600 generated by the optical density query circuitry 310, occupancy grid generation circuitry 315, and occupancy grid storage 320 of the neural representation training circuit 300. FIG. 6 Example training rays 605 used by the neural representation training circuit 300 to train the neural network 200 are also shown. The occupancy grid 600 is generated by the neural representation training circuit 300 based on the following considerations. First, if a geometry (e.g., example geometry 610) is visible in a training image used by the neural network training circuitry 305 to train the neural network 200, the training rays 605 will hit the geometry, and the optical density query circuitry 310 and occupancy grid generation circuitry 315 can reliably detect the presence of such geometry and update voxels (e.g., voxel 615) of the occupancy grid 600 based on the optical density returned by the neural network for sample points (e.g., sample point 620) along the training rays. Second, if a lightweight density query pre-pass to obtain early stopping distances and prune empty space missed by the occupancy grid is implemented by the neural network training circuitry 305, the optical density query circuitry 310 and occupancy grid generation circuitry 315 can reuse density samples from that pass to update voxels of the occupancy grid 600.

[0036] The following description provides additional implementation details of the example occupancy grid techniques disclosed herein, and extends the occupancy grid for use with neural videos.

[0037] Occupancy initialization. During training, at the beginning of each video frame, the occupancy grid generation circuitry 315 resets each voxel of the occupancy grid 600 to a predefined value (e.g., denoted by the variable Dhigh in this document), which initializes the voxel as occupied. The occupancy grid generation circuitry 315 stores the reset occupancy grid 600 for the current frame of training in the occupancy grid storage 320. In some examples, the occupancy grid generation circuitry 315 also maintains respective occupancy flags for the voxels of the occupancy grid, which can be stored as a separate 1 -bit empty space indicator grid in the occupancy grid storage 320. The occupancy grid storage 320 can be implemented by any number and / or type of memory, storage device, etc.

[0038] Occupancy usage and EMA decay delay for training incremental neural videos (INV). For each new frame of the INV, the neural network training circuitry 305 begins training the neural network for the current video frame by reusing the weights of the neural network 200 trained for the previous frame. It is possible that, in the new frame, some empty spaces will become occupied by moving objects, and some occupied spaces will be vacated. In at least some examples, to ensure that the neural network 200 learns these updates, the light density query circuitry 310 and the occupancy grid generation circuitry 315 can delay the occupancy EMA decay and any sample pruning for a portion (e.g., ¼ or some other portion) of the first training epoch of the current frame.

[0039] Light ray sampling. To guarantee maximum coverage of the training data at the beginning of each epoch, in at least some disclosed examples, the neural network training circuitry 305 generates a random permutation of all training light ray indices, and obtains a batch of light rays from the permutation.

[0040] Occupancy EMA decay with optional random occupancy sampling update. Every nth training iteration, the occupancy grid generation circuitry 315 uses an EMA decay step to carve out empty spaces. Optionally, the light density query circuitry 310 and the occupancy grid generation circuitry 315 can also perform random occupancy sampling that limits the maximum density value to Dhigh, thereby avoiding decay getting stuck at very high density values. In some examples, in every nth training iteration, the occupancy grid generation circuitry 315 updates the occupancy flags for the voxels of the occupancy grid 600, and can store the updated flags as a 1 -bit empty space indicator grid in the occupancy grid storage 320.

[0041] Occupancy update. In at least some examples, at each training iteration, the optical density query circuitry 310 performs a density-only ray marching pass operation that separates the current batch of rays into sample points and filters the sample points using the current occupancy grid 600 (e.g., by discarding sample points associated with voxels that are empty as indicated by their occupancy flags). The optical density query circuitry 310 then uses the remaining pruned sample points to query the neural network 200 for optical density at those sample points. The occupancy grid generation circuitry 315 evaluates the returned optical densities and removes low-density samples, and then occluded samples are removed. The occupancy grid generation circuitry 315 uses the resulting set of high-density sample points to update corresponding voxels of the occupancy grid 600. For example, for each high-density sample point, the occupancy grid generation circuitry 315 writes a value Dhigh to the voxel at which the sample point is located and discards the previous voxel value. In this way, each voxel containing contributing geometry will be equally marked as occupied in the occupancy grid 600 regardless of the size of the contribution, resulting in a robust update. Additionally, since the same Dhigh value is written to occupied voxels and replaces previously stored values, this update method does not rely on any ordering of atomic operations or memory writes.

[0042] In light of the foregoing, in some examples, during training of the neural network 200 by the neural network training circuitry 305, the optical density query circuitry 310 of the neural representation training circuit 300 queries the neural network for optical density at the sample points 620 along the training rays 605. As described above, the neural network training circuitry 305 trains the neural network 200 using the training rays 605 to provide a neural representation of a video frame. In some such examples, the occupancy grid generation circuitry 315 of the neural representation training circuit 300 generates an occupancy grid 600 for the video frame based on the optical density at the sample points 620 along the training rays 605. As described above, the occupancy grid 600 includes voxels 615 to indicate whether respective portions of a 3D volume associated with the neural representation are occupied by geometry. In some examples, the neural network training circuitry 305 trains the neural network 200 based on the occupancy grid 600, the training rays 605, and training images corresponding to different camera views respectively associated with the video frame.

[0043] In some examples, to generate the occupancy grid, the occupancy grid generation circuitry 315 updates a first one of the voxels 615 based on the optical density at the sample points 620 along the training rays 605, where the first one of the voxels 615 corresponds to a portion of the 3D volume that includes the sample points 620. For example, the occupancy grid generation circuitry 315 can update the first one of the voxels 615 based on a comparison of the optical density at the sample points 620 along the training rays 605 to a threshold value.

[0044] In some examples, training ray 605 is a first training ray, sample point 620 is a first sample point, and optical density query circuitry 310 queries neural network 200 for optical density at sample point 620 along a plurality of training rays 605 (e.g., including the first sample point of the first training ray), where training rays 605 are associated with a training iteration of neural network 200 (and training rays 605) performed by neural network training circuitry 305. In some such examples, occupancy grid generation circuitry 315 identifies voxels in voxels 615 of occupancy grid 600 that are associated with optical densities that satisfy a threshold value and sets the values ​​of those voxels to the same value (e.g., Dhigh).

[0045] In some examples, voxels 615 of occupancy grid 600 include a value and a flag, wherein the value of the voxel is based on the optical density associated with the corresponding training ray sample point 620, the training ray sample point 620 being included in the portion of the 3D volume respectively associated with the voxel, and the flag is set to indicate whether the portion of the 3D volume respectively associated with the voxel 615 is occupied by a geometric shape. In some such examples, occupancy grid generation circuitry 315 sets the flag of voxel 615 based on a comparison of the value of voxel 615 with a threshold value. In some such examples, occupancy grid generation circuitry 315 updates the value of voxel 615 of occupancy grid 600 in successive training iterations associated with the video frame based on the updated optical density and attenuation values ​​returned by neural network 200 for the updated sample point 620 of the updated training ray 605 associated with the successive training iterations. In some such examples, occupancy grid generation circuitry 315 updates the flag of voxel 615 of occupancy grid 600 at intervals corresponding to the number of training iterations (e.g., at every nth training iteration).

[0046] In some examples, after the training iterations performed by the neural network training circuitry 305 for training the neural network 200 for a given frame are completed, the occupancy grid generation circuitry 315 stores the final occupancy grid 600 for the given frame in the occupancy grid storage 320. In some examples, the final occupancy grid 600 is output by the neural representation training circuitry 300 with the trained parameters of the neural network 200 for the given frame to provide a neural representation (e.g., NeRF) for the given frame. For example, the neural representation training circuitry 300 can store and / or transmit the neural representation (NeRF), including the occupancy grid 600, for use by a computing device (e.g., a media device, a computer, a smartphone, a tablet, etc.) in rendering the frame.

[0047] In some examples, neural representation training circuitry 300 includes means for training a neural network to provide a neural representation. For example, the means for training a neural network can be implemented by neural network training circuitry 305. In some examples, neural network training circuitry 305 can be implemented by programmable circuitry (e.g., FIG. 9 For example, the neural network training circuit system 305 may be instantiated by executing machine executable instructions (e.g., by at least FIG. 7 and / or FIGS. 8A-8B The machine executable instructions implemented by the block FIG. 10 In some examples, the neural network training circuit system 105 can be instantiated by hardware logic circuit system, which can be an ASIC, XPU or other processor configured and / or constructed to perform operations corresponding to machine-readable instructions. FIG. 11 FPGA circuitry 1100. Additionally or alternatively, neural network training circuitry 305 may be instantiated by any other combination of hardware, software, and / or firmware. For example, neural network training circuitry 305 may be implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and / or integrated analog and / or digital circuitry, FPGA, ASIC, XPU, comparator, operational amplifier (op-amp), logic circuit, etc.) that are configured and / or constructed to execute some or all machine-readable instructions and / or perform some or all operations corresponding to the machine-readable instructions without executing software or firmware, although other structures are equally applicable.

[0048] In some examples, neural representation training circuit 300 includes means for querying the neural network for optical density values. For example, the means for querying the neural network can be implemented by optical density query circuitry 310. In some examples, optical density query circuitry 310 can be implemented by programmable circuitry (e.g., FIG. 9 For example, the optical density query circuit system 310 may be implemented by executing machine executable instructions (e.g., by at least FIG. 7 and / or FIGS. 8A-8B The machine executable instructions implemented by the block FIG. 10 In some examples, the optical density query circuit system 310 can be instantiated by hardware logic circuit system, which can be an ASIC, XPU or other processor configured and / or constructed to perform operations corresponding to machine-readable instructions. FIG. 11The optical density query circuitry 310 is implemented by the FPGA circuitry 1100. Additionally or alternatively, the optical density query circuitry 310 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the optical density query circuitry 310 may be implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and / or integrated analog and / or digital circuitry, FPGA, ASIC, XPU, comparator, operational amplifier (op-amp), logic circuit, etc.) that are configured and / or constructed to execute some or all machine-readable instructions and / or perform some or all operations corresponding to the machine-readable instructions without executing software or firmware, although other structures are equally applicable.

[0049] In some examples, the neural representation training circuit 300 includes a unit for generating and / or updating an occupancy grid. For example, the unit for generating and / or updating an occupancy grid can be implemented by an occupancy grid generation circuit system 315. In some examples, the occupancy grid generation circuit system 315 can be implemented by a programmable circuit system (e.g., FIG. 9 For example, the occupancy grid generation circuitry 315 may be instantiated by executing machine executable instructions (e.g., by at least FIG. 7 and / or FIGS. 8A-8B Those implemented by the block) FIG. 10 In some examples, the occupancy grid generation circuitry 315 may be instantiated by hardware logic circuitry, which may be implemented by an ASIC, XPU, or other processor configured and / or constructed to perform operations corresponding to machine-readable instructions. FIG. 11 FPGA circuitry 1100. Additionally or alternatively, occupancy grid generation circuitry 315 may be instantiated by any other combination of hardware, software, and / or firmware. For example, occupancy grid generation circuitry 315 may be implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and / or integrated analog and / or digital circuitry, FPGA, ASIC, XPU, comparator, operational amplifier (op-amp), logic circuit, etc.) that are configured and / or constructed to execute some or all machine-readable instructions and / or to execute some or all operations corresponding to the machine-readable instructions without executing software or firmware, although other structures are equally applicable.

[0050] Although FIG. 3 An example way to implement neural representation training circuit 300 is shown, but FIG. 3One or more elements, processes, and / or devices illustrated in FIG. 1 can be combined, divided, re-arranged, omitted, eliminated, and / or implemented in any other way. Further, the example neural network 200, the example neural network training circuitry 305, the example optical density query circuitry 310, the example occupancy grid generation circuitry 315, the example occupancy grid storage 320, and / or more generally the example neural representation training circuit 300 of FIG. 1 can be implemented by hardware alone, or can be implemented by combinations of hardware and software and / or firmware. FIG. 3 The example neural representation training circuit 300 of FIG. 1 can be implemented by hardware alone or can be implemented by combinations of hardware and software and / or firmware. Thus, for example, any of the example neural network 200, the example neural network training circuitry 305, the example optical density query circuitry 310, the example occupancy grid generation circuitry 315, the example occupancy grid storage 320, and / or more generally the example neural representation training circuit 300 can be implemented by programmable circuitry in combination with machine-readable instructions (e.g., firmware or software), processor circuitry, analog circuitry, digital circuitry, logic circuitry, programmable processor, programmable microcontroller, graphics processing unit (GPU), digital signal processor (DSP), ASIC, programmable logic device (PLD), and / or field programmable logic device (FPLD) (e.g., FPGA). Further, the example neural representation training circuit 300 can include one or more elements, processes, and / or devices in addition to, or instead of, the elements, processes, and / or devices illustrated in FIG. 1, and / or can include any or all of the illustrated elements, processes, and devices. FIG. 3 The example neural representation training circuit 300 of FIG. 1 can be implemented by hardware alone or can be implemented by combinations of hardware and software and / or firmware. Thus, for example, any of the example neural network 200, the example neural network training circuitry 305, the example optical density query circuitry 310, the example occupancy grid generation circuitry 315, the example occupancy grid storage 320, and / or more generally the example neural representation training circuit 300 can be implemented by programmable circuitry in combination with machine-readable instructions (e.g., firmware or software), processor circuitry, analog circuitry, digital circuitry, logic circuitry, programmable processor, programmable microcontroller, graphics processing unit (GPU), digital signal processor (DSP), ASIC, programmable logic device (PLD), and / or field programmable logic device (FPLD) (e.g., FPGA). Further, the example neural representation training circuit 300 can include one or more elements, processes, and / or devices in addition to, or instead of, the elements, processes, and / or devices illustrated in FIG. 1, and / or can include any or all of the illustrated elements, processes, and devices.

[0051] FIG. 7 and FIGS. 8A-8B Flowcharts representative of example machine-readable instructions that can be executed to implement and / or realize the neural representation training circuit 300 of FIG. 1 and / or example operations of the neural representation training circuit 300 are shown in FIGS. 2-4. The machine-readable instructions can be one or more executable programs or portions of programs, which are used to cause programmable circuitry to perform operations, tasks, etc. as discussed below in conjunction with, for example, the programmable circuitry 912 shown in the example processor platform 900 discussed below in conjunction with FIG. 9, and / or one or more functions or portions of functions to be performed by example programmable circuitry (e.g., FPGA). In some examples, the machine-readable instructions cause operations, tasks, etc. to be performed in an automated fashion in the real world. FIG. 3 Flowcharts representative of example machine-readable instructions that can be executed to implement and / or realize the neural representation training circuit 300 of FIG. 1 and / or example operations of the neural representation training circuit 300 are shown in FIGS. 2-4. The machine-readable instructions can be one or more executable programs or portions of programs, which are used to cause programmable circuitry to perform operations, tasks, etc. as discussed below in conjunction with, for example, the programmable circuitry 912 shown in the example processor platform 900 discussed below in conjunction with FIG. 9, and / or one or more functions or portions of functions to be performed by example programmable circuitry (e.g., FPGA). In some examples, the machine-readable instructions cause operations, tasks, etc. to be performed in an automated fashion in the real world. FIG. 3 Flowcharts representative of example machine-readable instructions that can be executed to implement and / or realize the neural representation training circuit 300 of FIG. 1 and / or example operations of the neural representation training circuit 300 are shown in FIGS. 2-4. The machine-readable instructions can be one or more executable programs or portions of programs, which are used to cause programmable circuitry to perform operations, tasks, etc. as discussed below in conjunction with, for example, the programmable circuitry 912 shown in the example processor platform 900 discussed below in conjunction with FIG. 9, and / or one or more functions or portions of functions to be performed by example programmable circuitry (e.g., FPGA). In some examples, the machine-readable instructions cause operations, tasks, etc. to be performed in an automated fashion in the real world. FIG. 9 Flowcharts representative of example machine-readable instructions that can be executed to implement and / or realize the neural representation training circuit 300 of FIG. 1 and / or example operations of the neural representation training circuit 300 are shown in FIGS. 2-4. The machine-readable instructions can be one or more executable programs or portions of programs, which are used to cause programmable circuitry to perform operations, tasks, etc. as discussed below in conjunction with, for example, the programmable circuitry 912 shown in the example processor platform 900 discussed below in conjunction with FIG. 9, and / or one or more functions or portions of functions to be performed by example programmable circuitry (e.g., FPGA). In some examples, the machine-readable instructions cause operations, tasks, etc. to be performed in an automated fashion in the real world. FIG. 10 Flowcharts representative of example machine-readable instructions that can be executed to implement and / or realize the neural representation training circuit 300 of FIG. 1 and / or example operations of the neural representation training circuit 300 are shown in FIGS. 2-4. The machine-readable instructions can be one or more executable programs or portions of programs, which are used to cause programmable circuitry to perform operations, tasks, etc. as discussed below in conjunction with, for example, the programmable circuitry 912 shown in the example processor platform 900 discussed below in conjunction with FIG. 9, and / or one or more functions or portions of functions to be performed by example programmable circuitry (e.g., FPGA). In some examples, the machine-readable instructions cause operations, tasks, etc. to be performed in an automated fashion in the real world. FIG. 11 Flowcharts representative of example machine-readable instructions that can be executed to implement and / or realize the neural representation training circuit 300 of FIG. 1 and / or example operations of the neural representation training circuit 300 are shown in FIGS. 2-4. The machine-readable instructions can be one or more executable programs or portions of programs, which are used to cause programmable circuitry to perform operations, tasks, etc. as discussed below in conjunction with, for example, the programmable circuitry 912 shown in the example processor platform 900 discussed below in conjunction with FIG. 9, and / or one or more functions or portions of functions to be performed by example programmable circuitry (e.g., FPGA). In some examples, the machine-readable instructions cause operations, tasks, etc. to be performed in an automated fashion in the real world.

[0052] The program can be embodied in instructions (e.g., software and / or firmware) stored on one or more non-transitory computer-readable and / or machine-readable storage media, such as cache, a magnetic storage device or disk (e.g., a floppy diskette, a hard drive, etc.), an optical storage device or disk (e.g., a Blu-ray disk, a compact disk, a digital versatile disk, etc.), a RAID, registers, ROM, SSD, SSD memory, non-transitory memory (e.g., electrically erasable programmable read only memory (EEPROM), flash memory, etc.), volatile memory (e.g., any type of random access memory (RAM), etc.), and / or any other storage device or storage disk. The instructions of the non-transitory computer-readable and / or machine-readable media can be programmed and / or executed by programmable circuitry located in one or more hardware devices, although the entire program and / or portions of it can alternatively be executed and / or instantiated by one or more hardware devices other than programmable circuitry and / or embodied in dedicated hardware. The machine-readable instructions can be distributed across multiple hardware devices and / or executed by two or more hardware devices (e.g., server and client hardware devices). For example, a client hardware device can be implemented by an endpoint client hardware device (e.g., a hardware device associated with a human and / or machine user) or an intermediary client hardware device gateway (e.g., a radio access network (RAN)), which can facilitate communications between a server and an endpoint client hardware device. Similarly, the non-transitory computer-readable storage media can comprise one or more media. Moreover, while the computer-readable and machine-readable media is referred to as non-transitory, the computer-readable and machine-readable media can change over time; for example, a ROM can be reprogrammed to hold different instructions. Given the non-transitory nature of the computer-readable and machine-readable media in this context, however, the computer-readable and machine-readable media are not intended to represent interfaces to transitory media such as signals, waves, and the like. FIG. 7 and FIGS. 8A-8BThe flowcharts illustrated in the middle describe example procedures, but many other methods of implementing example neural representation training circuit 300 can be used instead. For example, the order of execution of the blocks of the flowcharts can be changed, and / or some of the blocks described can be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks of the flowcharts can be implemented by one or more hardware circuits (e.g., processor circuitry, discrete and / or integrated analog and / or digital circuitry, FPGAs, ASICs, comparators, op-amps, logic circuits, etc.) structured to perform the corresponding operations without executing software or firmware. Programmable circuitry can be distributed across different network locations and / or local to one or more hardware devices (e.g., single-core processors (e.g., single-core CPUs), multi-core processors (e.g., multi-core CPUs, XPU, etc.)). For example, programmable circuitry can be a CPU and / or FPGA located in the same package (e.g., the same integrated circuit (IC) package or in two or more separate housings), one or more processors in a single machine, multiple processors distributed across multiple servers of a server rack, multiple processors distributed across multiple servers across one or more server racks, etc., and / or any combination thereof.

[0053] The machine-readable instructions described herein can be stored in one or more of a compressed format, an encrypted format, a segmented format, a compiled format, an executable format, a packaged format, etc. The machine-readable instructions described herein can be stored as data (e.g., computer-readable data, machine-readable data, one or more bits (e.g., one or more computer-readable bits, one or more machine-readable bits, etc.), a bitstream (e.g., a computer-readable bitstream, a machine-readable bitstream, etc.), etc.) or a data structure (e.g., as part of an instruction, code, a representation of code, etc.) that can be used to create, manufacture, and / or produce machine-executable instructions. For example, machine-readable instructions can be segmented and stored on one or more storage devices, disks, and / or computing devices (e.g., servers) located in the same or different locations of a network or collection of networks (e.g., in the cloud, in edge devices, etc.). Machine-readable instructions can require one or more of installation, modification, adjustment, updating, combining, supplementing, configuring, decryption, decompression, unpacking, distribution, reassignment, compilation, etc. in order to make them directly readable, interpretable, and / or executable by a computing device and / or other machine. For example, machine-readable instructions can be stored in multiple parts that are individually compressed, encrypted, and / or stored on separate computing devices, where when decrypted, decompressed, and / or combined, the parts form a set of computer-executable and / or machine-executable instructions that implement one or more functions and / or operations that together can form a procedure, such as those described herein.

[0054] In another example, machine-readable instructions can be stored in a state in which the machine-readable instructions can be read by programmable circuitry, but require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc. in order to execute the machine-readable instructions on a particular computing device or other device. In another example, machine-readable instructions can need to be configured (e.g., storage settings, input data, recording network addresses, etc.) before the machine-readable instructions and / or corresponding program can execute in whole or in part. Thus, machine-readable, computer-readable, and / or machine-readable media used herein can include instructions and / or programs irrespective of the particular format or state of the machine-readable instructions and / or programs.

[0055] Machine-readable instructions described herein can be represented by any past, present, or future instruction language, scripting language, programming language, etc. For example, machine-readable instructions can be represented using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, HyperText Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0056] As described above, FIG. 7 and FIGS. 8A-8BExample operations of the neural representation training circuit 300 can be implemented using executable instructions (e.g., computer-readable and / or machine-readable instructions) stored on one or more non-transitory computer-readable and / or machine-readable media. As used herein, the terms non-transitory computer-readable medium, non-transitory computer-readable storage medium, non-transitory machine-readable medium, and / or non-transitory machine-readable storage medium are defined as including any type of computer- readable storage device and / or storage disk, and exclude propagating signals and transitory media. Examples of such non-transitory computer-readable media, non-transitory computer-readable storage media, non-transitory machine-readable media, and / or non-transitory machine-readable storage media include optical storage devices, magnetic storage devices, HDDs, flash memories, ROMs, CDs, DVDs, caches, any type of RAM, registers, and / or any other storage devices or storage disks that store information in any duration (e.g., long-term, permanent, short-term instance, temporary buffer, and / or caching of information). As used herein, the terms “non-transitory computer-readable storage device” and “non-transitory machine-readable storage device” are defined as including any physical (mechanical, magnetic, and / or electrical) hardware that retains information in duration, but exclude propagating signals and transitory media. Examples of non-transitory computer-readable storage devices and / or non-transitory machine-readable storage devices include any type of random access memory, any type of read-only memory, solid state memory, flash memory, optical disk, magnetic disk, disk drive, and / or redundant array of independent disks (RAID) system. The term “device” as used herein refers to a physical structure, e.g., mechanical and / or electrical device, hardware, and / or circuitry, which can or can not be configured by, and / or manufactured to execute, computer-readable instructions, machine-readable instructions, etc.

[0057] FIG. 7 are representative of example machine-readable instructions and / or example operations 700 that can be executed, instantiated, and / or implemented by programmable circuitry to implement FIG. 3 the neural representation training circuit 300. FIG. 7 are representative of example machine-readable instructions and / or example operations 700 that can be executed, instantiated, and / or implemented by programmable circuitry to implement FIG. 3 the neural representation training circuit 300. FIG. 7 are representative of example machine-readable instructions and / or example operations 700 that can be executed, instantiated, and / or implemented by programmable circuitry to implement FIG. 3 the neural representation training circuit 300.

[0058] At block 715, occupancy grid generation circuitry 315 of neural representation training circuitry 300 initializes the values of the respective voxels 615 of the occupancy grid 600 of the current frame to the same initial value (e.g., Dhigh). At block 720, occupancy grid generation circuitry 315 initializes the occupancy flags of the respective voxels 615 of the occupancy grid 600 of the current frame to an asserted value (e.g., logical -1) to indicate that the voxels are occupied.

[0059] At block 725, neural network training circuitry 305 obtains a current batch of training rays 605 of a current batch of training data selected from the training data loaded at block 710. At block 730, optical density query circuitry 310 of neural representation training circuitry 300 queries neural network 200 for optical density values at the position sample points 620 along the current batch of training rays 605, as described above. At block 735, occupancy grid generation circuitry 315 updates the values of the voxels 615 of the occupancy grid 600 based on the optical density values at the position sample points 620 along the current batch of training rays 605, as described above. For example, occupancy grid generation circuitry 315 can compare the optical density values to a density threshold and identify those position sample points 620 having optical density values that satisfy (e.g., meet or exceed) the density threshold. Occupancy grid generation circuitry 315 can set the values of the voxels 615 corresponding to those position sample points 620 to the same constant value (e.g., Dhigh). In some examples, occupancy grid generation circuitry 315 also applies a decay factor to the voxels 615 of the occupancy grid 600 to cause the values of those voxels that are not occupied by a geometry to decrease over time to a low value (e.g., close to 0 or less than the occupancy threshold value).

[0060] At block 740, the neural network training circuitry 305 trains the neural network 200 based on the current batch of training rays, the current batch of training data, and the current occupancy grid 600. At block 745, the occupancy grid generation circuitry 315 determines whether it is time to update the occupancy flags of the voxels 615 of the occupancy grid 600 (e.g., by determining whether the number of training iterations corresponding to a grid update period has elapsed). If so, at block 750, the occupancy grid generation circuitry 315 updates the occupancy flags of the voxels 615 of the occupancy grid 600 based on the current values of the voxels 615, as described above. For example, the occupancy grid generation circuitry 315 can compare the current values of the voxels 615 of the occupancy grid 600 to an occupancy threshold. In some examples, the occupancy grid generation circuitry 315 sets the occupancy flag of a given voxel 615 of the occupancy grid 600 to an asserted value (e.g., a logical 1) if the value of the voxel satisfies (e.g., meets or exceeds) the occupancy threshold, and sets the occupancy flag of the given voxel 615 to a de-asserted value (e.g., a logical 0) if the value of the given voxel 615 does not satisfy (e.g., is less than) the occupancy threshold.

[0061] At block 755, the neural network training circuitry 305 determines whether there are additional batches of training data to be used to train the neural network 200 for the current video frame. If so, processing returns to block 725 and its subsequent blocks to allow the neural representation training circuit 300 to perform another training iteration for the current video frame. However, if there are no additional batches of training data for the current video frame, at block 760, the neural network training circuitry 305 determines whether there are additional video frames (for which to train the neural network 200). If so, processing returns to block 705 and its subsequent blocks to allow the neural representation training circuit 300 to train the neural network 200 to generate a neural representation (e.g., a NeRF) of the next video frame. Otherwise, the example machine-readable instructions and / or example operation 700 ends.

[0062] FIGS. 8A-8B Commonly represented are second example machine-readable instructions and / or example operation 800 that can be executed, instantiated, and / or implemented by programmable circuitry to implement a neural representation training circuit 300 that is configured to FIG. 3 train a neural network 200 to generate a neural representation (e.g., a NeRF) of a frame of a neural video (e.g., an INV). FIGS. 8A-8B The example machine-readable instructions and / or example operation 800 begin at block 805 of FIG. 8A the neural network training circuitry 305 of the neural representation training circuit 300 initializes parameters to be used to train the neural network 200 to generate a neural representation (e.g., a NeRF) of a frame of a neural video (e.g., an INV). For example, the neural network training circuitry 305 sets initial occupancy values (e.g., D HIGH ), occupancy weights (e.g., WOCC ) and an occupancy threshold (e.g., T OCC ), which will be used to generate and update the occupancy grid 600 for the given video frame. The neural network training circuitry 305 also initializes a rendering step size (e.g., δ) to be used to generate the sample points 620 along the training rays 605, which will be used to query the neural network 200 for light density values for generating and updating the occupancy grid 600 for the given video frame.

[0063] At block 810, the neural network training circuitry 305 loads training data (e.g., training images and camera parameters) from the training data storage 325, which will be used to train the neural network 200 for the current video frame (e.g., f). The neural network training circuitry 305 also sets the number of training iterations (e.g., I TRAIN ) to be performed for the current video frame, the number of training rays (e.g., N R ) in a given training batch, and the period (e.g., P OCC ) for occupancy grid updates. At block 810, the occupancy grid generation circuitry 315 of the neural network training circuitry 300 initializes the values of the respective voxels 615 of the occupancy grid 600 for the current frame to the same initial value (e.g., D HIGH ), which corresponds to initializing all of the voxels 615 of the occupancy grid 600 to indicate that they are occupied. The occupancy grid generation circuitry 315 also sets the occupancy flags of the voxels 615 of the occupancy grid 600 to indicate that they are occupied (e.g., by asserting the occupancy flags to a logical -1 value).

[0064] At block 815, the occupancy grid generation circuitry 315 determines whether it is time to update the values and occupancy flags of the voxels 615 of the occupancy grid 600. If it is not time to update the occupancy grid 600, then at block 820, the neural network training circuitry 305 obtains a new batch of training rays 605 (e.g., RAYS) and training data (e.g., PIXELS), which will be used to train the neural network 200 for the current video frame.

[0065] At FIG. 8BAt block 825, the optical density query circuitry 310 of the neural representation training circuit 300 generates the position sample points 620 (e.g., ALL SAMPLES) along the current batch of training rays 605 (e.g., by sampling along the rays based on the rendering step size delta based on queries to the neural network 200). The optical density query circuitry 310 then identifies those position sample points 620 (e.g., SAMPLES) that are associated with occupied (e.g., have an asserted occupancy flag) voxels 615 of the occupancy grid 600. The optical density query circuitry 310 then queries the neural network 200 for the optical density (e.g., R δ ) at those identified position sample points 620.

[0066] At block 825, the occupancy grid generation circuitry 315 further identifies those position sample points 620 that are associated with an optical density that satisfies (e.g., meets or exceeds) an occupancy threshold (e.g., T OCC ). HIGH OCC At block 830, the occupancy grid generation circuitry 315 sets the values of the voxels 615 of the occupancy grid 600 corresponding to those identified position sample points 620 to the same constant value (e.g., D HIGH *W OCC ).

[0067] At block 835, the neural network training circuitry 305 trains the neural network 200 based on the culling position sample points 620 of the training rays 605 that are both associated with occupied voxels 615 of the occupancy grid 600 and have an optical density that satisfies the occupancy threshold. At block 840, the neural network training circuitry 305 determines whether there are additional batches of training data to be used to train the neural network 200 for the current video frame. If so, processing returns to block 815 and its subsequent blocks to allow the neural representation training circuit 300 to perform another training iteration for the current video frame. However, if there are no additional batches of training data for the current video frame, the neural network training circuitry 305 outputs the trained neural network parameters (e.g., trained weights of the neural network 200) and the final occupancy grid 600 as the neural representation (e.g., NeRF) for the current video frame.

[0068] At block 850, the neural network training circuitry 305 determines whether there are additional video frames (for which to train the neural network 200). If so, processing returns to block 810 and its subsequent blocks to allow the neural representation training circuit 300 to train the neural network 200 to generate a neural representation (e.g., NeRF) for the next video frame. Otherwise, the example machine-readable instructions and / or the example operation 800 ends.

[0069] Returning to FIG. 8AIf the occupancy grid generation circuitry 315 determines that it is time to update the value and occupancy flag of a voxel 615 of the occupancy grid 600 at block 815, the process proceeds to block 855. At block 855, the occupancy grid generation circuitry 315 updates the value of the voxel 615 of the occupancy grid based on the EMA decay factor. At block 860, the occupancy grid generation circuitry 315 updates the occupancy flag of the voxel 615 of the occupancy grid 600 based on the current value of the voxel 615. For example, the occupancy grid generation circuitry 315 compares the current value of the voxel 615 of the occupancy grid 600 to an occupancy threshold (e.g., T OCC ) in the illustrated example. If the value of a given voxel 615 satisfies (e.g., meets or exceeds) the occupancy threshold, the occupancy grid generation circuitry 315 sets the occupancy flag of the given voxel 615 of the occupancy grid 600 to an asserted value (e.g., a logic 1), and if the value of the given voxel 615 does not satisfy (e.g., is less than) the occupancy threshold, sets the occupancy flag of the given voxel 615 to a de-asserted value (e.g., a logic 0).

[0070] FIG. 9 is a block diagram of an example programmable circuitry platform 900 that is structured to execute and / or instantiate example machine-readable instructions and / or example operations of FIG. 7 and FIGS. 8A-8B to implement the neural representation training circuitry 300 of FIG. 3 . For example, the programmable circuitry platform 900 can be a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cell phone, a smart phone, a tablet computer (e.g., an iPad TM ), a personal digital assistant (PDA), an internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, a headset (e.g., an augmented reality (AR) headset, a virtual reality (VR) headset, etc.), or other wearable device, or any other type of computing and / or electronic device.

[0071] The programmable circuitry platform 900 of the illustrated example includes programmable circuitry 912. The programmable circuitry 912 of the illustrated example is hardware. For example, the programmable circuitry 912 can be implemented by one or more integrated circuits from any desired family or manufacturer, logic circuits, FPGAs, microprocessors, CPUs, GPUs, DSPs, and / or microcontrollers. The programmable circuitry 912 can be implemented by one or more semiconductor-based devices, e.g., semiconductor-based devices that include circuitry implemented using one or more semiconductor materials, such as silicon. In this example, the programmable circuitry 912 implements the neural network training circuitry 305, the optical density query circuitry 310, the occupancy grid generation circuitry 315, the example occupancy grid storage 320, and / or more generally the neural representation training circuitry 300. In some examples, 912 also implements the neural network 200.

[0072] The programmable circuitry 912 of the illustrated example includes local memory 913 (e.g., cache, registers, etc.). The programmable circuitry 912 of the illustrated example communicates with main memory 914, 916 over a bus 918, which includes volatile memory 914 and non-volatile memory 916. The volatile memory 914 can be implemented by synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), and / or any other type of RAM device. The non-volatile memory 916 can be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 914, 916 of the illustrated example is controlled by a memory controller 917. In some examples, the memory controller 917 can be implemented by one or more integrated circuits, logic circuits, microcontrollers, or any other type of circuitry from any desired family or manufacturer to manage the flow of data to and from the main memory 914, 916. In some examples, the volatile memory 914 implements the training data storage 325. Dynamic random access memory and / or any other type of RAM device. The non-volatile memory 916 can be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 914, 916 of the illustrated example is controlled by a memory controller 917. In some examples, the memory controller 917 can be implemented by one or more integrated circuits, logic circuits, microcontrollers, or any other type of circuitry from any desired family or manufacturer to manage the flow of data to and from the main memory 914, 916. In some examples, the volatile memory 914 implements the training data storage 325.

[0073] The programmable circuitry platform 900 of the illustrated example also includes interface circuitry 920. The interface circuitry 920 can be implemented by hardware in accordance with any type of interface standard, such as an Ethernet interface, a universal serial bus (USB) interface, a Bluetooth® interface, a ZigBee® interface, a Wi-Fi® interface, a WiMAX® interface, a near field communication (NFC) interface, a peripheral component interconnect (PCI) interface, and / or a peripheral component interconnect express (PCIe) interface. Interface, near field communication (NFC) interface, peripheral component interconnect (PCI) interface, and / or peripheral component interconnect express (PCIe) interface.

[0074] In the illustrated example, one or more input devices 922 are connected to the interface circuitry 920. The input device(s) 922 permit(s) a user (e.g., a human user, a machine user, etc.) to input data and / or commands into the programmable circuitry 912. The input device(s) 922 can be implemented, for example, by an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a touchpad, a trackball, an isopoint device, and / or a voice recognition system.

[0075] One or more output devices 924 are also connected to the interface circuitry 920 of the illustrated example. For example, the output devices 924 can be implemented by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT) display, an in-place switching (IPS) display, a touchscreen, etc.), a tactile output device, a printer, and / or a speaker. Thus, the interface circuitry 920 of the illustrated example typically includes graphics driver cards, graphics driver chips, and / or graphics processor circuitry (e.g., GPUs).

[0076] The interface circuitry 920 of the illustrated example also includes a communication device, such as a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and / or a network interface to facilitate exchange of data with external machines (e.g., computing devices of any kind) over a network 926. For example, the communication can be facilitated through an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a super high frequency (SHF) wireless system, a

[0077] The programmable circuitry platform 900 of the illustrated example also includes one or more mass storage disks or devices 928 for storing firmware, software, and / or data. Examples of such mass storage disks or devices 928 include magnetic storage devices (e.g., floppy disks, drives, HDDs, etc.), optical storage devices (e.g., Blu-ray disks, CDs, DVDs, etc.), RAID systems, and / or solid state storage disks or devices (e.g., flash memory devices and / or SSDs). In some examples, the one or more mass storage disks or devices 928 implement the training data storage 325.

[0078] Machine-readable instructions 932 that can be implemented by the machine- readable instructions of FIG. 7 and FIGS. 8A-8B Machine-readable instructions 932 that can be implemented by the machine- readable instructions of

[0079] FIG. 10 areFIG. 9 a block diagram of an example implementation of the programmable circuitry 912 of the FIG. 9 The programmable circuitry 912 of the FIG. 7 and FIGS. 8A-8B some or all of the machine-readable instructions of the flowcharts of the FIG. 3 circuitry of the to be instantiated as logic circuits to perform operations corresponding to those machine-readable instructions. In some such examples, FIG. 3 The circuitry of the to be instantiated by hardware circuits of the microprocessor 1000 in conjunction with the machine-readable instructions. For example, the microprocessor 1000 can be implemented by multi-core hardware circuitry (e.g., CPUs, DSPs, GPUs, XPUs, etc.). Although it can include any number of example cores 1002 (e.g., 1 core), the microprocessor 1000 of the present example is a multi-core semiconductor device including N cores. The cores 1002 of the microprocessor 1000 can operate independently, or can cooperate to execute machine-readable instructions. For example, machine code corresponding to a firmware program, embedded software program, or software program can be executed by one of the cores 1002, or can be executed by multiple ones of the cores 1002 at the same or different times. In some examples, machine code corresponding to a firmware program, embedded software program, or software program is split into threads and executed in parallel by two or more cores 1002. The software program can correspond to the software program of the FIG. 7 and FIGS. 8A-8B some or all of the machine-readable instructions and / or operations represented by the flowcharts of the

[0080] The cores 1002 can communicate over a first example bus 1004. In some examples, the first bus 1004 can be implemented by a communication bus to enable communication associated with cores in the cores 1002. For example, the first bus 1004 can be implemented by at least one of an Inter-Integrated Circuit (I2C) bus, a Serial Peripheral Interface (SPI) bus, a PCI bus, or a PCIe bus. Additionally or alternatively, the first bus 1004 can be implemented by any other type of computing or electrical bus. The cores 1002 can obtain data, instructions, and / or signals from one or more external devices via example interface circuitry 1006. The cores 1002 can output data, instructions, and / or signals to one or more external devices via the interface circuitry 1006. Although the cores 1002 of the present example include example local memory 1020 (e.g., a level one (LI) cache that can be split into an LI data cache and an LI instruction cache), the microprocessor 1000 also includes an example shared memory 1010 that can be shared by the cores (e.g., a level two (L2 cache) to enable fast access to data and / or instructions. Data and / or instructions can be transferred (e.g., shared) by writing to and / or reading from the shared memory 1010. The local memory 1020 of each core of the cores 1002 and the shared memory 1010 can be part of a hierarchy of storage devices that includes multiple levels of cache memory and main memory (e.g., the main memory 914, 916 of FIG. 9 The various levels in the cache hierarchy are managed (e.g., coordinated) by a cache coherency policy.

[0081] Each core 1002 may be referred to as a CPU, DSP, GPU, or any other type of hardware circuitry. Each core 1002 includes a control unit circuitry 1014, an arithmetic and logic (AL) circuitry (sometimes referred to as an ALU) 1016, a plurality of registers 1018, a local memory 1020, and a second example bus 1022. Other configurations are possible. For example, each core 1002 may include a vector unit circuitry, a single instruction multiple data (SIMD) unit circuitry, a load / store unit (LSU) circuitry, a branch / jump unit circuitry, a floating point unit (FPU) circuitry, and the like. The control unit circuitry 1014 includes semiconductor-based circuitry configured to control (e.g., coordinate) data movement within the corresponding core 1002. The AL circuitry 1016 includes semiconductor-based circuitry configured to perform one or more mathematical and / or logical operations on data within the corresponding core 1002. Some examples of the AL circuitry 1016 perform integer-based operations. In other examples, the AL circuitry 1016 also performs floating-point operations. In other examples, AL circuitry 1016 may include a first AL circuitry that performs integer-based operations and a second AL circuitry that performs floating-point operations. In some examples, AL circuitry 1016 may be referred to as an arithmetic logic unit (ALU).

[0082] Registers 1018 are semiconductor-based structures for storing data and / or instructions, such as the results of one or more operations performed by the AL circuitry 1016 of the corresponding core 1002. For example, registers 1018 may include vector registers, SIMD registers, general registers, flag registers, segment registers, machine-specific registers, instruction pointer registers, control registers, debug registers, memory management registers, machine check registers, etc. Registers 1018 may be as follows: FIG. 10 The registers 1018 are shown arranged in banks. Alternatively, the registers 1018 may be organized in any other arrangement, format, or structure, such as by being distributed throughout the core 1002 to reduce access time. The second bus 1022 may be implemented by at least one of an I2C bus, an SPI bus, a PCI bus, or a PCIe bus.

[0083] Each core 1002 and / or, more generally, microprocessor 1000 can include additional and / or alternative structures to those shown and described above. For example, there can be one or more clock circuits, one or more power supplies, one or more power gates, one or more cache home agents (CHAs), one or more aggregate / common mesh stoppers (CMSs), one or more shifters (e.g., barrel shifters), and / or other circuitry. Microprocessor 1000 is a semiconductor device that is fabricated to include many transistors that are interconnected to implement the above-described structures in one or more integrated circuits (ICs) contained in one or more packages.

[0084] Microprocessor 1000 can include and / or cooperate with one or more accelerators (e.g., acceleration circuitry, hardware accelerators, etc.). In some examples, an accelerator is implemented by logic circuitry to perform specific tasks faster and / or more efficiently than a general-purpose processor. Examples of accelerators include ASICs and FPGAs as discussed herein. GPUs, DSPs, and / or other programmable devices can also be accelerators. An accelerator can be on-board microprocessor 1000, in the same chip package as microprocessor 1000, and / or in one or more packages separate from microprocessor 1000.

[0085] FIG. 11 is FIG. 9 a block diagram of another example implementation of programmable circuitry 912 of FIG. 10 In this example, programmable circuitry 912 is implemented by FPGA circuitry 1100. For example, FPGA circuitry 1100 can be implemented by an FPGA. For example, FPGA circuitry 1100 can be used to perform operations that can otherwise be performed by a general-purpose microprocessor executing corresponding machine-readable instructions.

[0086] More specifically, in contrast to microprocessor 1000 in FIG. 10 which is a general-purpose device that can be programmed to execute some or all of the machine-readable instructions represented by the flowcharts of FIG. 7 and FIGS. 8A-8B the interconnect and logic circuitry of FPGA circuitry 1100 in the example can be configured, structured, programmed, and / or interconnected in different ways after fabrication to instantiate, for example, the operations and / or functions corresponding to FIG. 11 FIG. 7 and​FIGS. 8A-8B The FPGA circuit system 1100 may be thought of as an array of logic gates, interconnects, and switches. The switches may be programmed to change the way the logic gates are interconnected via the interconnects, effectively forming one or more dedicated logic circuits (unless and until the FPGA circuit system 1100 is reprogrammed). The configured logic circuits enable the logic gates to work together in different ways to perform different operations on the data received by the input circuit system. Those operations may be related to FIG. 7 and FIGS. 8A-8B Thus, FPGA circuit system 1100 can be configured and / or constructed (for efficient instantiation with FIG. 7 and FIGS. 8A-8B FPGA circuit system 1100 may be a dedicated logic circuit that performs the operations / functions corresponding to the software instructions in a dedicated manner similar to an ASIC. Therefore, FPGA circuit system 1100 may perform operations / functions corresponding to the software instructions faster than a general-purpose microprocessor (performing the same operations / functions). FIG. 11 and FIG. 11 Some or all of the machine-readable instructions correspond to operations / functions.

[0087] exist FIG. 11 In an example, in response to being programmed (and / or reprogrammed one or more times) based on a binary file, the FPGA circuit system 1100 is configured and / or constructed. In some examples, the binary file can be compiled and / or generated based on instructions in a hardware description language (HDL), such as Lucid, Very High Speed ​​Integrated Circuit (VHSIC) Hardware Description Language (VHDL), or Verilog. For example, a user (e.g., a human user, a machine user, etc.) can write code or a program corresponding to one or more operations / functions in the HDL; the code / program can be translated into a low-level language as needed; and the code / program (e.g., code / program in a low-level language) can be converted (e.g., by a compiler, a software application, etc.) into a binary file. In some examples, FIG. 11 The FPGA circuit system 1100 can access and / or load the binary file to make FIG. 11 The FPGA circuit system 1100 is configured and / or constructed to perform one or more operations / functions. For example, a binary file can be implemented by a bit stream (e.g., one or more computer-readable bits, one or more machine-readable bits, etc.), data (e.g., computer-readable data, machine-readable data, etc.), and / or machine-readable instructions, which can be implemented by FIG. 11FPGA circuitry 1100 of the system 1000 to configure and / or structure the FPGA circuitry 1100 of the system 1000 to perform one or more operations / functions. FIG. 11 FPGA circuitry 1100 of the system 1000 to configure and / or structure the FPGA circuitry 1100 of the system 1000 to perform one or more operations / functions.

[0088] In some examples, the binary file is compiled, generated, converted, and / or otherwise output from a unified software platform for programming the FPGA. For example, the unified software platform can convert first instructions (e.g., code or programs) corresponding to one or more operations / functions in a high-level language (e.g., C, C++, Python, etc.) to second instructions corresponding to one or more operations / functions in an HDL. In some such examples, the binary file is compiled, generated, and / or otherwise output from the unified software platform based on the second instructions. In some examples, FIG. 11 FPGA circuitry 1100 of the system 1000 to configure and / or structure the FPGA circuitry 1100 of the system 1000 to perform one or more operations / functions. FIG. 11 FPGA circuitry 1100 of the system 1000 to configure and / or structure the FPGA circuitry 1100 of the system 1000 to perform one or more operations / functions. FIG. 10 FPGA circuitry 1100 of the system 1000 to configure and / or structure the FPGA circuitry 1100 of the system 1000 to perform one or more operations / functions. FIG. 7 FPGA circuitry 1100 of the system 1000 to configure and / or structure the FPGA circuitry 1100 of the system 1000 to perform one or more operations / functions.

[0089] FIGS. 8A-8B The FPGA circuitry 1100 of the system 1000 includes example input / output (I / O) circuitry 1102 to obtain and / or output data to / from example configuration circuitry 1104 and / or external hardware 1106. For example, the configuration circuitry 1104 can be implemented by interface circuitry that can obtain a binary file that can be implemented by a bitstream, data, and / or machine-readable instructions to configure the FPGA circuitry 1100 or portions thereof. In some such examples, the configuration circuitry 1104 can obtain the binary file from a user, a machine, and / or any combination thereof, such as a hardware circuitry (e.g., programmable or dedicated circuitry) that can implement an artificial intelligence / machine learning (AI / ML) model to generate the binary file, etc. In some examples, the external hardware 1106 can be implemented by external hardware circuitry. For example, the external hardware 1106 can be implemented by a microprocessor 1000 of the system 1000. FIG. 11 The FPGA circuitry 1100 of the system 1000 to configure and / or structure the FPGA circuitry 1100 of the system 1000 to perform one or more operations / functions.

[0090] The FPGA circuitry 1100 also includes an array of example logic gate circuitry 1108, a plurality of example configurable interconnects 1110, and example storage circuitry 1112. The logic gate circuitry 1108 and the configurable interconnects 1110 are configurable to instantiate one or more operations / functions that can correspond to at least some of the machine-readable instructions and / or other desired operations of FIG. 11 and FIG. 10 FIG. 11 The logic gate circuitry 1108 is fabricated in blocks or groups as shown in FIG. 11. Each block includes semiconductor-based electrical structures that can be configured as logic circuits. In some examples, the electrical structures include logic gates (e.g., AND gates, OR gates, NOT gates, etc.) that provide basic building blocks for logic circuits. Within each of the logic gate circuitry 1108 are electrical controllable switches (e.g., transistors) to enable the electrical structures and / or logic gates to be configured to form circuits that perform desired operations / functions. The logic gate circuitry 1108 can include other electrical structures, such as look-up tables (LUTs), registers (e.g., flip-flops or latches), multiplexers, etc.

[0091] The configurable interconnects 1110 of the illustrated example are electrically conductive pathways, traces, vias, etc., that can include electrical controllable switches (e.g., transistors) whose state can be changed through programming (e.g., using HDL instruction languages) to activate or deactivate one or more connections between one or more of the logic gate circuitry 1108 to program desired logic circuits.

[0092] The storage circuitry 1112 in the illustrated example is structured to store results of one or more operations performed by corresponding logic gates. The storage circuitry 1112 can be implemented by registers and the like. In the illustrated example, the storage circuitry 1112 is distributed among the logic gate circuitry 1108 to facilitate access and improve execution speed.

[0093] FIG. 9 ​The example FPGA circuitry 1100 also includes example special-purpose operation circuitry 1114. In this example, the special-purpose operation circuitry 1114 includes special-purpose circuitry 1116 that can be invoked to implement commonly used functions to avoid the need to program these functions in the field. Examples of such special-purpose circuitry 1116 include memory (e.g., DRAM) controller circuitry, PCIe controller circuitry, clock circuitry, transceiver circuitry, memory, and multiplier-accumulator circuitry. Other types of special-purpose circuitry can be present. In some examples, the FPGA circuitry 1100 can also include example general-purpose programmable circuitry 1118, e.g., example CPUs 1120 and / or example DSPs 1122. Additionally or alternatively, other general-purpose programmable circuitry 1118 can be present, e.g., GPUs, XPU, etc., which can be programmed to perform other operations.

[0094] While FIG. 10 and FIG. 9 two example implementations of the programmable circuitry 912 of the FPGA circuitry 1100 are shown, many other approaches are also contemplated. For example, the FPGA circuitry can include on-board CPUs, e.g., one or more of the example microprocessors 1000 of the CPU 1120. Thus, the programmable circuitry 912 of the FPGA circuitry 1100 can additionally be implemented by combining at least the example microprocessor 1000 of the CPU 1120 and the example FPGA circuitry 1100 of the FPGA circuitry 1100. FIG. 10 FIG. 11 Thus, it should be understood that some or all of the circuitry in the CPU 1120 can be instantiated at the same or different times as the circuitry in the FPGA circuitry 1100. For example, the CPU 1120 can be instantiated before the FPGA circuitry 1100, after the FPGA circuitry 1100, or concurrently with the FPGA circuitry 1100. FIG. 10 FIG. 7 FIGS. 8A-8B In some such hybrid examples, one or more cores 1002 of the CPU 1120 can execute a first portion of the machine-readable instructions represented by the flowcharts of the CPU 1120 to perform first operations / functions, FIG. 11 FIG. 7 and a second portion of the machine-readable instructions represented by the flowcharts of the FPGA circuitry 1100 to perform second operations / functions corresponding to the second portion of the machine-readable instructions, and / or the ASIC can be configured and / or structured to execute a third portion of the machine-readable instructions represented by the flowcharts of the ASIC to perform third operations / functions corresponding to the third portion of the machine-readable instructions. FIGS. 8A-8B FIG. 7 Thus, it should be understood that some or all of the circuitry in the CPU 1120 can be instantiated at the same or different times as the circuitry in the FPGA circuitry 1100. For example, the CPU 1120 can be instantiated before the FPGA circuitry 1100, after the FPGA circuitry 1100, or concurrently with the FPGA circuitry 1100. FIG. 3 FIG. 10 FIG. 11 FIG. 3

[0095] Thus, it should be understood that some or all of the circuitry in the CPU 1120 can be instantiated at the same or different times as the circuitry in the FPGA circuitry 1100. For example, the CPU 1120 can be instantiated before the FPGA circuitry 1100, after the FPGA circuitry 1100, or concurrently with the FPGA circuitry 1100. FIG. 10 FIG. 11 ​​​​​​​​​​The same and / or different portions of microprocessor 1000 in FIG. 3 The same and / or different portions of FPGA circuitry 1100 in

[0096] In some examples, FIG. 10 Some or all of the circuitry in FIG. 9 Microprocessor 1000 in FIG. 10 FPGA circuitry 1100 in FIG. 11 Some or all of the circuitry in FIG. 9 Within one or more virtual machines and / or containers executing on

[0097] In some examples, FIG. 10 Programmable circuitry 912 of FIG. 11 Microprocessor 1000 of FIG. 11 FPGA circuitry 1100 of FIG. 12 XPU can be implemented by programmable circuitry 912 of FIG. 9 Microprocessor 1000 of FIG. 9 CPU 1120 of FIG. 7 DSP 1122 of FIGS. 8A-8B FPGA circuitry 1100 of

[0098] FIG. 7 A block diagram illustrating an example software distribution platform 1205 for distributing software (e.g., operating system 1210, applications 1215, etc.) to one or more devices is shown. FIGS. 8A-8BExample machine-readable instructions 932) to other hardware devices (e.g., hardware devices owned and / or operated by third parties from the owner and / or operator of the software distribution platform). The example software distribution platform 1205 can be implemented by any computer server, data facility, cloud service, etc. that is capable of storing software and sending software to other computing devices. The third parties can be customers of the entity that owns and / or operates the software distribution platform 1205. For example, the entity that owns and / or operates the software distribution platform 1205 can be the developer, seller, and / or licensor of the example machine-readable instructions 932). The third parties can be consumers, users, retailers, OEMs, etc. that purchase and / or license the software for use and / or resale and / or re-licensing. In the illustrated example, the software distribution platform 1205 includes one or more servers and one or more storage devices. The storage devices store the machine-readable instructions 932, which can correspond to the example machine-readable instructions 932 described above, for example. The one or more servers of the example software distribution platform 1205 communicate with the example network 1210, which can correspond to the Internet and / or any one or more of the example networks described above. In some examples, the one or more servers respond to requests for sending software to requesting parties as part of a commercial business. Payment for the delivery, sale, and / or licensing of the software can be processed by the one or more servers of the software distribution platform and / or a third-party payment entity. The servers enable the purchaser and / or licensor to download the machine-readable instructions 932 from the software distribution platform 1205. For example, the software (which can correspond to the example machine-readable instructions 932 described above, for example) can be downloaded into the example programmable circuitry platform 900 for execution of the machine-readable instructions 932 to implement the neural representation training circuit 300. In some examples, the one or more servers of the software distribution platform 1205 periodically provide, send, and / or force updates to the software (e.g., to the example machine-readable instructions 932) to ensure that improvements, patches, updates, etc. are distributed and applied to the software at the end-user devices. While referred to above as software, the distributed “software” can alternatively be firmware. FIG. 9 ​ ​ ​ ​ ​

[0099] ​​​​​​“Including” and “comprising” (and all forms and tenses thereof) are used herein as open-ended terms. Thus, whenever a claim employs any form of “include” or “comprise” (e.g., includes, included, including, comprises, comprised, comprising, etc.) as a transition term, the disclosure should be interpreted to allow for alternatives, modifications, or equivalents of each of the elements or limitations introduced by the transition term. As used herein, when the phrase “at least” is used as a transitional phrase (e.g., as the preamble of a claim), it is open-ended, in the same manner as the terms “comprising” and “including” are open-ended. The term “and / or” when used in the form “A, B, and / or C” means A, B, C individually or any combination or subset thereof, such as (1) A only, (2) B only, (3) C only, (4) A with B, (5) A with C, (6) B with C, or (7) A, B, and C. As used herein in context with describing structural, compositional, item, object, and / or thing related aspects, the phrase “at least one of A and B” is intended to refer to any implementation of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in context with describing structural, compositional, item, object, and / or thing related aspects, the phrase “at least one of A or B” is intended to refer to any implementation of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. As used herein in context with describing performance or execution of processes, instructions, acts, activities, etc., the phrase “at least one of A and B” is intended to refer to any implementation of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in context with describing performance or execution of processes, instructions, acts, activities, etc., the phrase “at least one of A or B” is intended to refer to any implementation of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0100] As used herein, singular references (e.g., “a,” “an,” “first,” “second,” and the like) are not to be construed as being exclusive or otherwise limiting to a single reference unless expressly indicated from the context. As used herein, the term “a” or “an” object refers to one or more of such objects. The terms “a” (or “an”), “one or more,” and “at least one” are used interchangeably herein. Furthermore, although individually listed, a plurality of elements, components, or acts can be implemented by, e.g., a single entity or object. Additionally, although individual features can be included in different examples or claims, these features can be combined, and the inclusion of them in different examples or claims does not imply that a combination of features is not feasible and / or advantageous.

[0101] As used herein, a connection reference (e.g., attached, coupled, connected, and joined) can include intervening materials and / or relative movement between elements to which the connection reference refers, unless otherwise indicated. Thus, a connection reference does not necessarily imply that two elements are directly connected and / or in fixed relation to one another. As used herein, the definition of stating that any portion is “in contact” with another portion means that there are no intervening portions between the two portions.

[0102] Unless specifically stated otherwise, descriptors used herein, such as “first,” “second,” “third,” etc., do not connote or otherwise imply any priority, physical order, arrangement in a list, and / or order of importance, in any way whatsoever, but are merely used as labels and / or arbitrary names to distinguish elements so as to facilitate an understanding of the disclosed examples. In some examples, a descriptor “first” can be used to refer to an element in the detailed description, while the same element can be referred to in a claim with a different descriptor, such as “second” or “third.” In such instances, it is understood that the descriptors are used merely to clearly identify those elements within the context of the discussion (e.g., in the claims), where the elements can otherwise share the same name, for example.

[0103] As used herein, “approximately” and “about” modify a value / quantity to identify a potential existence of variation that occurs in real-world applications. For example, “approximately” and “about” can modify dimensions that can not be exact due to manufacturing tolerances and / or other real-world defects, as would be understood by one of ordinary skill in the art. For example, “approximately” and “about” can indicate that a dimension can be within a + / - 10% tolerance range, unless otherwise specified herein.

[0104] “Substantially real-time” as used herein refers to occurring in a near-instantaneous manner (recognizing that there can be real-world delays for computation time, transmission, etc.). Thus, unless otherwise specified, “substantially real-time” refers to real-time + 1 second.

[0105] As used herein, the phrase“in communication,” including variants thereof, encompasses direct communication and / or indirect communication through one or more intermediary components and does not require direct physical (e.g., wired) communication and / or continuous communication, but additionally includes selective communication at periodic intervals, at predetermined intervals, at aperiodic intervals, and / or at one-time events.

[0106] “Programmable circuitry” as used herein is defined to include (i) one or more special-purpose electrical circuits (e.g., application-specific circuits (ASICs)) configured to perform specific operations and including one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors); and / or (ii) one or more general-purpose semiconductor-based electrical circuits that are programmable with instructions to perform specific functions and / or operations and including one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors). Examples of programmable circuitry include programmable microprocessors, such as central processing units (CPUs) that can execute first instructions to perform one or more operations and / or functions, field-programmable gate arrays (FPGAs) that can be programmed with second instructions to cause a configuration and / or fabric of the FPGA to instantiate one or more operations and / or functions corresponding to the first instructions, graphics processor units (GPUs) that can execute first instructions to perform one or more operations and / or functions, digital signal processors (DSPs) that can execute first instructions to perform one or more operations and / or functions, XPU, network processing units (NPUs), one or more microcontrollers that can execute first instructions to perform one or more operations and / or functions, and / or integrated circuits (e.g., application-specific integrated circuits (ASICs)). For example, an XPU can be implemented by a heterogeneous computing system that includes multiple types of programmable circuitry (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more NPUs, one or more DSPs, etc., and / or any combination thereof), and orchestration technology (e.g., an application programming interface (API) that can assign computing tasks to any one or more of the multiple types of programmable circuitry that are suitable and available to perform the computing tasks).

[0107] As used herein, integrated circuit / circuitry is defined as one or more semiconductor packages that contain one or more circuit elements, such as transistors, capacitors, inductors, resistors, current paths, diodes, etc. For example, an integrated circuit can be implemented as one or more of an ASIC, an FPGA, a chip, a microchip, programmable circuitry, a semiconductor substrate that couples multiple circuit elements, a system on a chip (SoC), etc.

[0108] In light of the foregoing, it should be appreciated that example systems, apparatuses, articles of manufacture, and methods to implement an occupancy grid of a neural radiance field have been disclosed. The disclosed systems, apparatuses, articles of manufacture, and methods improve efficiency using a computing device by using density data from training light ray travel instead of random sampling to ensure accurate detection of geometry of a scene when generating an occupancy grid for a frame of the scene. The disclosed systems, apparatuses, articles of manufacture, and methods also update values of occupancy grid voxels based on a fixed, predefined number instead of the light density values themselves as in the prior art, which avoids the need for atomic operations when updating grid voxels. Accordingly, the disclosed systems, apparatuses, articles of manufacture, and methods provide one or more improvements in the operation of a machine, such as a computer or other electronic and / or mechanical device.

[0109] Further examples, and combinations thereof, include the following. Example 1 includes an apparatus for generating an occupancy grid of a neural representation for a video frame, the apparatus comprising: interface circuitry; computer-readable instructions; and at least one processor circuit programmed by the computer-readable instructions to: query a neural network for a light density at a sample point along a training light ray, the training light ray being associated with training the neural network to provide a neural representation of the video frame; and generate the occupancy grid for the video frame based on the light density at the sample point along the training light ray, the occupancy grid comprising voxels to indicate whether respective portions of a three-dimensional (3D) volume associated with the neural representation are occupied by geometry.

[0110] Example 2 includes the apparatus of Example 1, wherein to generate the occupancy grid, one or more of the at least one processor circuit is to update a first one of the voxels based on the light density at the sample point along the training light ray, the first one of the voxels corresponding to a portion of the 3D volume that includes the sample point.

[0111] Example 3 includes the apparatus of Example 1 or Example 2, wherein one or more of the at least one processor circuit is to update the first one of the voxels based on a comparison of the light density at the sample point along the training light ray to a threshold value.

[0112] Example 4 includes the apparatus of any of Examples 1-3, wherein the training light ray is a first training light ray, the sample point is a first sample point, and one or more of the at least one processor circuit is to: query the neural network for optical densities at sample points along a plurality of training light rays, the plurality of training light rays associated with a training iteration of the neural network, the plurality of training light rays including the first training light ray, the sample points including the first sample point; identify voxels of the voxels of the occupancy grid associated with optical densities of the optical densities that satisfy a threshold; and set values of the voxels of the voxels to a same value.

[0113] Example 5 includes the apparatus of any of Examples 1-4, wherein the voxels of the occupancy grid include values and flags, the values of the voxels are based on optical densities associated with respective training light ray sample points included in portions of the 3D volume respectively associated with the voxels, and the flags are set to indicate whether the portions of the 3D volume respectively associated with the voxels are occupied by a geometry.

[0114] Example 6 includes the apparatus of any of Examples 1-5, wherein one or more of the at least one processor circuit is to set the flags of the voxels based on comparisons of the values of the voxels to a threshold.

[0115] Example 7 includes the apparatus of any of Examples 1-6, wherein one or more of the at least one processor circuit is to: update the values of the voxels of the occupancy grid in successive training iterations associated with the video frame based on updated optical densities and attenuation values returned by the neural network for updated sample points of updated training light rays associated with the successive training iterations; and update the flags of the voxels of the occupancy grid at intervals corresponding to a number of training iterations.

[0116] Example 8 includes the apparatus of any of Examples 1-7, wherein one or more of the at least one processor circuit is to train the neural network based on the occupancy grid, the training light rays, and training images respectively corresponding to different camera views associated with the video frame.

[0117] Example 9 includes at least one non-transitory computer-readable medium including computer-readable instructions that cause at least one processor circuit system to at least: query a neural network for light density at sample points along training rays associated with training the neural network to provide a neural representation of a video frame; and generate an occupancy grid for the video frame based on the light density at sample points along the training rays, the occupancy grid including voxels indicating whether a corresponding portion of a three-dimensional (3D) volume associated with the neural representation is occupied by a geometric shape.

[0118] Example 10 includes the at least one non-transitory computer-readable medium of Example 9, wherein, to generate the occupancy grid, the computer-readable instructions are configured to cause one or more processor circuits of the at least one processor circuit to update a first voxel of the voxels based on the light density at the sample point along the training ray, the first voxel of the voxels corresponding to a portion of the 3D volume including the sample point.

[0119] Example 11 includes at least one non-transitory computer-readable medium of Example 9 or Example 10, wherein the computer-readable instructions are for causing one or more processor circuits of the at least one processor circuit to update a first voxel of the voxels based on a comparison of the light density at the sample point along the training ray with a threshold value.

[0120] Example 12 includes at least one non-transitory computer-readable medium of any of Examples 9 to 11, wherein the training ray is a first training ray, the sample point is a first sample point, and the computer-readable instructions are for causing one or more of the at least one processor circuits to: query the neural network for light densities at sample points along a plurality of training rays associated with training iterations of the neural network, the plurality of training rays including the first training ray, the sample points including the first sample point; identify voxels of the occupancy grid associated with light densities in the light densities that satisfy a threshold; and set values ​​of the voxels in the voxels to the same value.

[0121] Example 13 includes at least one non-transitory computer-readable medium of any one of Examples 9 to 12, wherein the voxels of the occupancy grid include a value and a flag, the value of the voxel being based on an optical density associated with a corresponding training ray sample point included in the portion of the 3D volume respectively associated with the voxel, and the flag being set to indicate whether the portion of the 3D volume respectively associated with the voxel is occupied by a geometric shape.

[0122] Example 14 includes the at least one non-transitory computer-readable medium of any of examples 9-13, wherein the computer-readable instructions are to cause one or more of the at least one processor circuit to: set a flag of the voxel based on a comparison of a value of the voxel to a threshold; update a value of the voxel of the occupancy grid in a successive training iteration associated with an updated training ray of the video frame based on an updated optical density and attenuation value returned by the neural network for an updated sample point associated with the updated training ray; and update the flags of the voxels of the occupancy grid at intervals corresponding to a number of training iterations.

[0123] Example 15 includes a method for generating an occupancy grid for a neural representation of a video frame, the method comprising: querying a neural network for an optical density at a sample point along a training ray, the training ray associated with training the neural network to provide a neural representation of the video frame; and generating, by at least one processor circuit programmed by at least one instruction, an occupancy grid for the video frame based on the optical density at the sample point along the training ray, the occupancy grid comprising voxels to indicate whether respective portions of a three-dimensional (3D) volume associated with the neural representation are occupied by a geometric shape.

[0124] Example 16 includes the method of example 15, wherein the generating of the occupancy grid comprises updating a first one of the voxels based on the optical density at the sample point along the training ray, the first one of the voxels corresponding to a portion of the 3D volume that includes the sample point.

[0125] Example 17 includes the method of example 15 or example 16, wherein the updating of the first one of the voxels is based on: comparing the optical density at the sample point along the training ray to a threshold.

[0126] Example 18 includes the method of any of examples 15-17, wherein the training ray is a first training ray, the sample point is a first sample point, and the method further comprises: querying the neural network for optical densities at sample points along a plurality of training rays, the plurality of training rays associated with a training iteration of the neural network, the plurality of training rays including the first training ray, the sample points including the first sample point; identifying voxels of the voxels of the occupancy grid associated with optical densities of the optical densities that satisfy a threshold; and setting values of the voxels of the voxels to a same value.

[0127] Example 19 includes the method of any of examples 15-18, wherein the voxels of the occupancy grid include a value and a flag, the value of a voxel is based on optical densities associated with respective training ray sample points included in portions of the 3D volume respectively associated with the voxel, and the flag is set to indicate whether the portion of the 3D volume respectively associated with the voxel is occupied by a geometry.

[0128] Example 20 includes the method of any of examples 15-19, further comprising setting a flag of a voxel based on a comparison of the value of the voxel to a threshold, updating the value of the voxels of the occupancy grid in successive training iterations associated with updated training rays based on updated optical densities and attenuation values returned by the neural network for updated sample points of the updated training rays associated with the successive training iterations associated with the video frame, and updating the flag of the voxels of the occupancy grid at intervals corresponding to a number of training iterations.

[0129] The accompanying drawings are incorporated herein by reference in this Detailed Description. While specific examples of systems, apparatus, articles of manufacture, and methods have been disclosed, those having ordinary skill in the art will readily convey modifications, alterations, permutations, additions, subtractions, substitutions, combinations, and locations of components and functions, which, if not previously disclosed, are also contemplated as falling within the scope of the claims of the patent.

Claims

1. An apparatus for generating an occupancy grid for a neural representation of a video frame, the apparatus comprising: Interface circuit system; computer-readable instructions; as well as at least one processor circuit to be programmed by the computer-readable instructions to: querying a neural network for light density at sample points along training rays associated with training the neural network to provide a neural representation of the video frame; as well as The occupancy grid for the video frame is generated based on the light density at sample points along the training rays, the occupancy grid comprising voxels indicating whether a corresponding portion of a three-dimensional (3D) volume associated with the neural representation is occupied by a geometric shape.

2. The device according to claim 1, wherein To generate the occupancy grid, one or more processor circuits of the at least one processor circuit are configured to update a first voxel of the voxels based on the light density at the sample point along the training ray, the first voxel of the voxels corresponding to a portion of the 3D volume including the sample point.

3. The device according to claim 2, wherein One or more processor circuits of the at least one processor circuit are configured to update a first one of the voxels based on a comparison of the light density at the sample points along the training ray with a threshold value.

4. The device according to any one of claims 1 to 3, wherein: The training light is a first training light, the sample point is a first sample point, and one or more processor circuits of the at least one processor circuit are configured to: querying the neural network for light density at sample points along a plurality of training rays associated with a training iteration of the neural network, the plurality of training rays including the first training ray, and the sample points including the first sample point; identifying voxels of the voxels of the occupancy grid associated with an optical density in the optical densities that satisfies a threshold; as well as Sets the values ​​of the voxels in that voxel to the same value.

5. The device according to any one of claims 1 to 3, wherein: The voxels of the occupancy grid include a value and a flag, the value of the voxel being based on an optical density associated with a corresponding training ray sample point included in a portion of the 3D volume respectively associated with the voxel, and the flag being set to indicate whether the portion of the 3D volume respectively associated with the voxel is occupied by a geometric shape.

6. The device according to claim 5, wherein One or more processor circuits of the at least one processor circuit are configured to set a flag for the voxel based on a comparison of a value of the voxel to a threshold value.

7. The device according to claim 6, wherein One or more processor circuits of the at least one processor circuit are configured to: updating the value of the voxel of the occupancy grid in successive training iterations based on updated optical density and attenuation values ​​returned by the neural network for updated sample points of updated training rays associated with the successive training iterations associated with the video frame; as well as The labels of the voxels of the occupancy grid are updated at intervals corresponding to a number of training iterations.

8. The device according to any one of claims 1 to 3, wherein: One or more processor circuits of the at least one processor circuit are configured to train the neural network based on the occupancy grid, the training rays, and training images corresponding to different camera views associated with the video frames.

9. At least one computer-readable medium comprising computer-readable instructions that cause at least one processor circuitry to at least: querying a neural network for light density at sample points along training rays associated with training the neural network to provide a neural representation of a video frame; and An occupancy grid is generated for the video frame based on the light density at sample points along the training rays, the occupancy grid comprising voxels indicating whether a corresponding portion of a three-dimensional (3D) volume associated with the neural representation is occupied by a geometric shape.

10. The at least one computer-readable medium of claim 9, wherein: To generate the occupancy grid, the computer-readable instructions are for causing one or more of the at least one processor circuits to update a first one of the voxels based on the light density at the sample point along the training ray, the first one of the voxels corresponding to a portion of the 3D volume including the sample point.

11. The at least one computer-readable medium of claim 10, wherein: The computer-readable instructions are for causing one or more of the at least one processor circuits to update a first one of the voxels based on a comparison of the light density at the sample points along the training ray with a threshold value.

12. At least one computer-readable medium according to any one of claims 9 to 11, wherein: The training ray is a first training ray, the sample point is a first sample point, and the computer-readable instructions are configured to cause one or more processor circuits of the at least one processor circuit to: querying the neural network for light density at sample points along a plurality of training rays associated with a training iteration of the neural network, the plurality of training rays including the first training ray, and the sample points including the first sample point; identifying voxels of the voxels of the occupancy grid associated with an optical density in the optical densities that satisfies a threshold; as well as Sets the values ​​of the voxels in that voxel to the same value.

13. At least one computer-readable medium according to any one of claims 9 to 11, wherein: The voxels of the occupancy grid include a value and a flag, the value of the voxel being based on an optical density associated with a corresponding training ray sample point included in a portion of the 3D volume respectively associated with the voxel, and the flag being set to indicate whether the portion of the 3D volume respectively associated with the voxel is occupied by a geometric shape.

14. At least one computer-readable medium according to claim 13, wherein: The computer readable instructions are configured to cause one or more of the at least one processor circuit to: setting a flag of the voxel based on a comparison of a value of the voxel with a threshold; updating the value of the voxel of the occupancy grid in successive training iterations based on updated optical density and attenuation values ​​returned by the neural network for updated sample points of updated training rays associated with the successive training iterations associated with the video frame; as well as The labels of the voxels of the occupancy grid are updated at intervals corresponding to a number of training iterations.

15. A method for generating an occupancy grid for a neural representation of a video frame, the method comprising: querying a neural network for light density at sample points along training rays associated with training the neural network to provide a neural representation of the video frame; as well as At least one processor circuit programmed with at least one instruction generates the occupancy grid for the video frame based on the light density at sample points along the training ray, the occupancy grid comprising voxels indicating whether a corresponding portion of a three-dimensional (3D) volume associated with the neural representation is occupied by a geometric shape.

16. The method according to claim 15, wherein The generating of the occupancy grid includes updating a first one of the voxels based on the light density at the sample point along the training ray, the first one of the voxels corresponding to a portion of the 3D volume including the sample point.

17. The method according to claim 16, wherein The updating of a first one of the voxels is based on comparing the light density at a sample point along the training ray to a threshold value.

18. The method according to any one of claims 15 to 17, wherein: The training light is a first training light, the sample point is a first sample point, and the method further includes: querying the neural network for light density at sample points along a plurality of training rays associated with a training iteration of the neural network, the plurality of training rays including the first training ray, and the sample points including the first sample point; identifying voxels of the voxels of the occupancy grid associated with an optical density in the optical densities that satisfies a threshold; and Sets the values ​​of the voxels in that voxel to the same value.

19. The method according to any one of claims 15 to 18, wherein: The voxels of the occupancy grid include a value and a flag, the value of the voxel being based on an optical density associated with a corresponding training ray sample point included in a portion of the 3D volume respectively associated with the voxel, and the flag being set to indicate whether the portion of the 3D volume respectively associated with the voxel is occupied by a geometric shape.

20. The method according to claim 19, further comprising: setting a flag of the voxel based on a comparison of a value of the voxel with a threshold; updating the value of the voxel of the occupancy grid in successive training iterations based on updated optical density and attenuation values ​​returned by the neural network for updated sample points of updated training rays associated with the successive training iterations associated with the video frame; as well as The labels of the voxels of the occupancy grid are updated at intervals corresponding to a number of training iterations.

Images