System and method of reconstructing a three-dimensional image from twodimensional images

Abstract

A method for reconstructing a three-dimensional image (108) from one or more two-dimensional images using a neural radiance field (102) is provided. The method involves receiving two-dimensional image samples (106) and rendering these samples using the neural radiance field. During the rendering, intermediate computations are stored in a cache (104), where samples related to the computations have an opacity above a predetermined threshold. The cache is queried using interpolation of surrounding cache points (208, 502, 608) of the data stored in the cache, taking each sample's opacity into account. The method includes performing a neural computation on the interpolation results of the querying step to obtain sample data for colour and opacity. Finally, the accumulated neural computation results are used to generate the three-dimensional image.

Description

[0001] SYSTEM AND METHOD OF RECONSTRUCTING A THREE-DIMENSIONAL IMAGE FROM TWO-

[0002] DIMENSIONAL IMAGES

[0003] TECHNICAL FIELD

[0004] The disclosure relates generally to reconstructing a three-dimensional image, and more particularly, the disclosure relates to a method of reconstructing a three-dimensional image from one or more two-dimensional images by carrying out rendering of the two-dimensional image using a neural radiance field (NeRF). Moreover, the disclosure relates to a system for reconstructing the three-dimensional image from the one or more two-dimensional images by carrying out rendering of the two-dimensional image using the neural radiance field.

[0005] BACKGROUND

[0006] In recent years, Neural Radiance Fields, NeRFs, have attracted significant research attention, sparking immense interest. They possess the remarkable capability to reconstruct a scene solely within a neural network from images captured of that scene. Their applications span various fields, including virtual tours, movie production backgrounds, content generation, scanning, and parallel tracking and mapping, PTAM / simultaneous localization and mapping, SLAM, applications, as well as three- dimensional, 3D, diffusion models.

[0007] NeRFs operate on a volume rendering integral principle. For each ray to be rendered, the neural network is sampled multiple times to determine color and opacity, and the integration of these samples creates the color. This process enables the neural network to learn and generate the color of the pixels in an image.

[0008] However, a major drawback of NeRFs is their prolonged evaluation time. Traditional NeRF implementations necessitate 100- 200 samples per ray. Each sample evaluates the neural network, resulting in image generation times of seconds. The NeRF architecture typically comprises a predominantly view-independent component with a smaller view-dependent component to ensure multi- view consistency. Nevertheless, the limited size of the view-dependent part constrains the achievable quality of NeRF reconstructions.

[0009] When aiming to depict expansive scenes with intricate detail, NeRF s encounter limitations due to insufficient network capacity . Even with larger networks, training times escalate. One strategy to address this challenge is segmenting the scene into smaller networks, exemplified by KiloNeRF and Block-NeRF. However, this segmentation introduces the challenge of maintaining consistency across cells or blocks, and may necessitate traversing numerous (transparent) cells to obtain a conclusive result.

[0010] In an existing NeRF, caching computations immediately after the view-independent network segment enables the rendering of images with full quality, thereby disregarding resampling artifacts. In accordance with the original NeRF methodology, this intermediate latent vector storage may comprise 256 values. Such caching permits the complete reuse of the view-independent network segment, which consists of 9 layers in the original NeRF framework. Conversely, the view-dependent segment (which remains uncached) typically necessitates only a single layer. This ensures that cached values can be visualized from any other view direction without compromising quality, thereby facilitating the reuse of approximately 90% of computations. However, storing 256 latent codes may incur excessive costs.

[0011] To mitigate the prolonged training and inference times associated with NeRFs, Instant Neural Graphics Primitives, INGP, propose to transfer portions of the network evaluations into a multi-resolution hash grid structure, where the latent codes are stored and trained before being fed into a compact neural network. The hash grid structure, aligned with the entire scene, needs to encapsulate the entire object. Employing the hash grid structure offers the advantage of maintaining relatively small memory requirements (10 seconds of megabytes, MB) while substantially increasing both training and rendering times. The evaluation of the small neural network can be executed with notable efficiency. Elowever, this approach entails a trade-off, resulting in a loss of quality, which is generally deemed acceptable in light of the significant gains in training and rendering time.

[0012] To attain high-quality NeRFs reconstructions, distributing only a few samples along each ray proves inadequate. Mip-NeRF introduces the utilization of a larger Multi-layer Perceptron, MLP, and employs multiple samples surrounding the ray to effectively capture fine details within the scene. Elowever, this approach significantly escalates both training and rendering times. Although integrating the hash grid structure partially mitigates these costs, the inclusion of extra samples and the utilization of a larger MLP still impose a considerable performance overhead, necessitating approximately 1 second of rendering time on 8 Graphics Processing Units, GPUs. Despite leveraging a hash-map, achieving real-time performance remains unattainable with this approach.

[0013] Another existing method aimed at addressing the prolonged inference time of NeRFs involves devising alternative 3D data structures. These structures store the final network output in a hierarchical or non-hierarchical, scene-aligned 3D grid, facilitating rapid evaluation by loading all data from memory. While this approach significantly increases the rendering speed by circumventing the need for individual network evaluations, it necessitates a substantial amount of memory to store the data in the 3D grid. Elowever, not all view directions can be stored in this 3D grid. Consequently, directional information is typically encoded using spherical harmonics, a lower-resolution directional data structure. Clearly, this approach sacrifices the storage efficiency for speed. Unfortunately, transitioning from a scene representation requiring only a few megabytes (typically 3- 10MB for a traditional NeRF) to multiple gigabytes renders this approach highly inefficient, particularly when dealing with larger scenes encompassing multiple objects. Furthermore, the quality of the rendered output is significantly reduced compared to that achieved by a full NeRF implementation.

[0014] The most analogous solution in the existing methods involves representations that utilize an object space voxel grid structure as an intermediary representation for a radiance field. The object space voxel grid structure represents a three-dimensional grid structure that divides the space surrounding an object into small volumetric elements called voxels, each representing a portion of the object's space. This object space voxel grid structure representation bears similarities to a cache-like structure, suggesting its role in efficiently storing and accessing pertinent data during computation or rendering processes.

[0015] Elowever, the voxel grid structure has a drawback as it lacks adaptability to the current view and can only be viewed as a static cache. Such fixed static grid representation demands extensive memory resources and fails to adjust dynamically to the current view. Consequently, for expansive scenes, this cache may need to encompass the entire scene at the maximum resolution ever required for viewing.

[0016] Further, the technical problems of existing approaches include (i) rendering NeRF-like representations being computationally expensive due to the high volume of computations involved, and (ii) depending on an extensive precomputed "cache" being impractical due to excessive storage and transmission demands. These existing approaches that translate NeRF-like methodologies into the static grid representations struggle to accurately capture view-dependent effects.

[0017] Therefore, there arises a need to address the aforementioned technical problems / drawbacks for reconstructing a three- dimensional image using NeRF.

[0018] SUMMARY

[0019] It is an object of the disclosure to provide a method of reconstructing a three-dimensional image from one or more two- dimensional images by carrying out rendering of the two-dimensional image using a neural radiance field and a system of reconstructing the three-dimensional image from the one or more two-dimensional images by carrying out rendering of the two-dimensional image using the neural radiance field.

[0020] This object is achieved by the features of the independent claims. Further, implementation forms are apparent from the dependent claims, the description, and the figures.

[0021] The disclosure provides a method of reconstructing a three-dimensional image from one or more two-dimensional images by carrying out rendering of the two-dimensional image using a neural radiance field and a system of reconstructing the three- dimensional image from the one or more two-dimensional images by carrying out rendering of the two-dimensional image using the neural radiance field.

[0022] According to a first aspect, there is provided a method of reconstructing a three-dimensional image from one or more two- dimensional images by carrying out rendering of the two-dimensional image using a Neural Radiance Field, NeRF. The method includes receiving a plurality of two-dimensional image samples. The method includes rendering the plurality of two- dimensional image samples using the neural radiance field. During the rendering, intermediate computations are stored in a cache, where samples related to the computations have an opacity above a predetermined threshold. The method includes querying the cache using interpolation of surrounding cache points of the data stored in the cache, considering the opacity of each sample. The method includes carrying out a neural computation of the interpolation results of the querying step to deliver sample data for colour and opacity. The method includes accumulating the results of the neural computation to generate a three- dimensional image.

[0023] The method selectively stores view-independent computations in a cache, focusing only on data relevant to currently visible parts of the scene. Since caching is view-aligned, this "view-alignment" ensures storage requirements stay low, as unnecessary information from hidden areas is not cached. Additionally, cache resolution adapts dynamically based on camera proximity, storing higher resolution data for close-up parts / views and lower resolution for distant parts / objects. This temporary caching eliminates the need for extensive permanent storage, which may result in high storage costs.

[0024] The method achieves significantly faster rendering speeds compared to traditional NeRF implementations, with a potential reduction of 4-100 times. This improvement is achieved while retaining all the benefits of NeRF for scene reconstruction. The method works seamlessly across various NeRF rendering techniques, including vanilla NeRF, instant neural graphics primitives (INGP), Mip-NeRF, and Zip-NeRF. Unlike the traditional NeRF, which requires memory for scene representation, this method only utilizes memory during the rendering process. This makes it suitable for all NeRF rendering application scenarios, including streaming, where memory limitations are often a concern.

[0025] Caching allows for additional samples to be taken during cache generation compared to basic rendering. This leads to higher quality reconstructed images. Furthermore, the caching is automatically adjusted based on camera distance, as it stores samples along view rays.

[0026] Preferably, data from currently visible portions of the three-dimensional image are cached. The intermediate computations include latent codes of a neural rendering pipeline structure. Optionally, the intermediate computations are stored in the cache along view rays, and the samples are stored in the cache in bricks.

[0027] Preferably, the empty bricks are stored in the cache to indicate that there is no radiance to be sampled. A cache structure may correspond to three-dimensional bricks which are cuboids in the perspective space of a cache creating view, and truncated oblique pyramids in world space. Each brick may contain a regular grid of samples generated from a rectangular batch of rays. Preferably, the regular grid of samples captures a fixed number of intermediate latent codes of a neural rendering pipeline. The cache may be resampled by querying the bricks at new sample locations by interpolating the samples in the bricks. Optionally, the interpolating of the samples in the bricks involves tri-linear interpolation or nearest neighbour sampling. Optionally, duplication of samples at the brick boundaries is carried out.

[0028] According to a second aspect, there is provided a system comprising means adapted for carrying out all the steps of the method described above.

[0029] The system selectively stores view-independent computations in a cache, focusing only on data relevant to currently visible parts of the scene. Since caching is view-aligned, this "view-alignment" ensures storage requirements stay low, as unnecessary information from hidden areas is not cached. Additionally, cache resolution adapts dynamically based on camera proximity, storing higher resolution data for close-up parts / views and lower resolution for distant parts / objects. This temporary caching eliminates the need for extensive permanent storage, which may result in high storage costs. The system achieves significantly faster rendering speeds compared to traditional NeRF implementations, with a potential reduction of 4-100 times.

[0030] According to a third aspect, there is provided a computer program including instructions for carrying out all the steps of the above described method, when the computer program is executed on a computer system.

[0031] Therefore, in contradistinction to the existing solutions, the method selectively stores view-independent computations in a cache, focusing only on data relevant to currently visible parts of the scene. Since caching is view-aligned, this "viewalignment" ensures storage requirements stay low, as unnecessary information from hidden areas is not cached. Additionally, cache resolution adapts dynamically based on camera proximity, storing higher resolution data for close-up parts / views and lower resolution for distant parts / objects. This temporary caching eliminates the need for extensive permanent storage.

[0032] These and other aspects of the disclosure will be apparent from the implementation s) described below.

[0033] BRIEF DESCRIPTION OF DRAWINGS

[0034] Implementations of the disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0035] FIG. 1 is a block diagram that illustrates a system of reconstructing a three-dimensional image from one or more two- dimensional images by carrying out rendering of the two-dimensional image using a Neural Radiance Field, NeRF, in accordance with an implementation of the disclosure;

[0036] FIGS. 2A-2B illustrate a cache structure within a system in accordance with an implementation of the disclosure;

[0037] FIGS. 3A-3B illustrate multiple cache views of a cache structure in accordance with an implementation of the disclosure;

[0038] FIG. 4 is a flow diagram that illustrates a method of reconstructing a three-dimensional image from one or more two- dimensional images by carrying out rendering of the two-dimensional image using a neural radiance field in accordance with an implementation of the disclosure;

[0039] FIGS. 5A-5C illustrate exemplary neural radiance fields implementing the method of FIG. 4 in accordance with an implementation of the disclosure; FIGS. 6A-6B illustrate an exemplary cache-driven training and rendering of a neural radiance field (NeRF) in accordance with an implementation of the disclosure; and

[0040] FIG. 7 is an illustration of a computer system in which the various architectures and functionalities of the various previous implementations may be implemented.

[0041] DETAILED DESCRIPTION OF THE DRAWINGS

[0042] Implementations of the disclosure provide a method of reconstructing a three-dimensional image from one or more two- dimensional images by carrying out rendering of the two-dimensional image using a neural radiance field and a system of reconstructing the three-dimensional image from the one or more two-dimensional images by carrying out rendering of the two-dimensional image using the neural radiance field.

[0043] To make solutions of the disclosure more comprehensible for a person skilled in the art, the following implementations of the disclosure are described with reference to the accompanying drawings.

[0044] Terms such as "a first", "a second", "a third", and "a fourth" (if any) in the summary, claims, and foregoing accompanying drawings of the disclosure are used to distinguish between similar objects and are not necessarily used to describe a specific sequence or order. It should be understood that the terms so used are interchangeable under appropriate circumstances, so that the implementations of the disclosure described herein are, for example, capable of being implemented in sequences other than the sequences illustrated or described herein. Furthermore, the terms "include" and "have" and any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, a method, a system, a product, or a device that includes a series of steps or units, is not necessarily limited to expressly listed steps or units but may include other steps or units that are not expressly listed or that are inherent to such process, method, product, or device.

[0045] FIG. 1 is a block diagram that illustrates a system 100 of reconstructing a three-dimensional image 108 from one or more two- dimensional images by carrying out rendering of the two-dimensional image using a Neural Radiance Field, NeRF, 102 in accordance with an implementation of the disclosure. The system 100 receives one or more two-dimensional image samples 106. The system 100 renders the one or more two-dimensional image samples 106 using the neural radiance field 102. During the rendering, the system 100 stores intermediate computations in a cache 104, where two-dimensional image samples 106 related to the computations have an opacity above a predetermined threshold. The intermediate computations include latent codes of a neural rendering pipeline structure. The intermediate computations may be stored in the cache 104 along view rays and the samples are stored in the cache 104 in bricks 110. Each brick may contain a regular grid of samples generated from a rectangular batch of rays. Preferably, the regular grid of samples captures a fixed number of intermediate latent codes of a neural rendering pipeline.

[0046] The system 100 queries the cache 104 using interpolation of surrounding cache points of the data stored in the cache 104, considering the opacity of each sample. The system 100 carries out a neural computation of the interpolation results of the querying step to deliver sample data for colour and opacity. The system 100 accumulates the results of the neural computation to generate a three-dimensional image 108. Preferably, data from currently visible portions of the three-dimensional image 108 are cached. The cache 104 may be resampled by querying the bricks 110 at new sample locations by interpolating the samples in the bricks 110. The interpolation of the samples in the bricks 110 may involve tri-linear interpolation or nearest neighbour sampling. Further, duplication of the samples at the brick boundaries may be carried out.

[0047] The system 100 selectively stores view-independent computations in the cache 104, focusing only on data relevant to currently visible parts of the scene. Since caching is view-aligned, this "view-alignment" ensures storage requirements stay low, as unnecessary information from hidden areas is not cached. Additionally, cache resolution adapts dynamically based on camera proximity, storing higher resolution data for close-up parts / views and lower resolution for distant parts / objects. This temporary caching eliminates the need for extensive permanent storage, which may result in high storage costs.

[0048] The system 100 achieves significantly faster rendering speeds compared to traditional NeRF implementations, with a potential reduction of 4 to 100 times. This improvement is achieved while retaining all the benefits of NeRF for scene reconstruction. The system 100 works seamlessly across various NeRF rendering techniques, including vanilla NeRF, instant neural graphics primitives (INGP), Mip-NeRF, and Zip-NeRF. Unlike the traditional NeRF, which requires memory for scene representation, the system 100 only utilizes memory during the rendering process. This makes it suitable for all NeRF rendering application scenarios, including streaming, where memory limitations are often a concern.

[0049] Caching allows for additional samples to be taken during cache generation compared to basic rendering. This leads to higher quality reconstructed images. Furthermore, the caching is automatically adjusted based on camera distance, as it stores samples along view rays.

[0050] The system 100 efficiently caches computations between views of the two-dimensional image samples 106 by exploiting the high degree of frame-to-frame coherence (similarity) between successive views. Typically, 80-95% of the content remains identical between frames. By caching computations from previous views, the system 100 significantly reduces the computational cost of rendering radiance fields. This allows for a small, on-demand cache to be built on the display device, preserving the advantages of full NeRF representations, such as compactness for streaming. Additionally, the system 100 can easily adapt to view-dependent shading effects.

[0051] The system 100 offers advantages for a wide range of NeRF rendering applications, specifically those involving rendering multiple views or individual rays. The system 100 allows efficient streaming of NeRF representations to small clients, where the entire rendering process occurs on the client device. The system 100 employs a cloud-Based NeRF rendering approach that enables cloud-based rendering of NeRF scenes, with videos streamed to the display device. Here, the system 100 enhances rendering efficiency within the cloud infrastructure itself. Further, NeRFs are valuable for generating realistic environments that are used to train autonomous agents. The caching approach of the system 100 significantly improves the efficiency of these rendering processes for agent training. The system 100 performs a Depth-of-field (DoF) and motion blur rendering, which simulate the camera effects require integrating information across multiple rays. Thus, the system’s 100 caching approach effectively reduces the computational burden of such rendering techniques.

[0052] FIGS. 2A-2B illustrate a cache structure 202 within a system in accordance with an implementation of the disclosure. The cache structure 202 may include three-dimensional bricks 204 which appears as cuboids in the perspective space of a cache creating view, and truncated oblique pyramids in world space. Each brick 204 may contain a regular grid of samples generated from a rectangular batch of rays. Preferably, the regular grid of samples captures a fixed number of intermediate latent codes of a neural rendering pipeline. The system renders one or more two-dimensional image samples using a neural radiance field. During the rendering of the first frame of the one or more two-dimensional image sample, the system stores intermediate computations in the cache along the view rays 206, as shown in FIG. 2A. Specifically, the system retains the two-dimensional image samples that contribute to the final output colour and have an opacity exceeds a predetermined threshold. These high- impact samples are cached, forming a regular grid with denser spacing near the camera (i.e., capturing more detail in close-up areas) and sparser spacing for distant objects. This adaptive approach of the system optimizes caching based on camera perspective, ensuring efficient rendering while preserving details.

[0053] The system stores the intermediate computations in a sparse, view-aligned cache to expedite computations for nearby views. This "view-alignment" allows the system to reuse most computations from previous frames, significantly reducing rendering time. These intermediate results, which can be any type of latent code used in the neural rendering pipeline, are stored directly in 3D space at the same locations where the original rendering samples were taken. This approach maintains consistency with the natural flow of the neural rendering pipeline.

[0054] The system queries the cache using interpolation of surrounding cache points 208 of the data stored in the cache, considering the opacity of each sample. This avoids storing individual samples, which can be inefficient for querying. Since exact cache locations are rarely needed during retrieval, the system queries for slightly different points. This process involves interpolating values from surrounding cache points 208. By considering the opacity of each sample during interpolation, the system ensures accurate results even when querying for in-between locations. Storing the samples in bricks, as shown in FIG. 2B, facilitates efficient querying of cache existence in the 3D space, as well as streamlined data access and interpolation when generating samples along new view rays.

[0055] Ideally, rendering may rely solely on cached data points 208 for maximum efficiency. However, to achieve optimal results, the system needs to efficiently query which samples can be interpolated to avoid network evaluations and identify where new samples need to be generated if they are not part of the cache.

[0056] FIGS. 3A-3B illustrate multiple cache views of a cache structure 302 in accordance with an implementation of the disclosure. Rendering from the cache may involve not only a single view as described in FIGS. 2A & 2B, but also encompass multiple cache views. The cache structure 302 may include three-dimensional bricks 304 and each brick 304 may contain a regular grid of samples generated from a rectangular batch of rays. In FIG. 3A, two cache views (e.g., a first cache view 302A and a second cache view 302B) are positioned around the current camera view 302C. This arrangement ensures that the visible parts of a scene are fully covered, thereby enabling rendering to rely entirely on cached data. The selection of potential views for caching may vary, ranging from simple historical data incorporation (i.e., adding a view to the cache if it lacks significant coverage) to identifying new views that are unique from all other existing cache views. This adaptive approach enables the generation of cache views from unseen / unexplored angles that complement the existing cache views.

[0057] The presence of empty bricks / spaces 304A, as shown in FIG. 3B, within the cache structure 302 refers to areas within the radiance fields which do not have any contribution. For instance, if surfaces 306 block / obstruct the cache generation due to their presence, cache bricks are not generated in the background as they are not sampled. During rendering from the cache, encountering an absence of cache entries raises uncertainty whether there is no radiance to query at that location or if the location simply lacks representation within the cache. The cache structure 302 includes empty bricks 304A to signify the absence of radiance to be sampled.

[0058] Preferably, the empty bricks 304A are stored in the cache to indicate that there is no radiance to be sampled. When bricks represent empty bricks / spaces 304 A, they may not require memory allocation for samples. The system employs a two-layer data structure comprising an acceleration structure that positions the bricks in 3D space and includes a pointer to the actual samples. In case, if no pointer is utilized, the brick signifies the empty space 304A. The acceleration data structure may adopt the form of a regular grid in the perspective space of the cache-creating view or a bounding volume hierarchy in world space. Additionally, the actual bounds of bricks can be stored as triangle meshes, leveraging efficient raytracing hardware for intersection determination.

[0059] FIG. 4 is a flow diagram that illustrates a method of reconstructing a three-dimensional image from one or more two- dimensional images by carrying out rendering of the two-dimensional image using a neural radiance field in accordance with an implementation of the disclosure. At a step 402, one or more two-dimensional image samples are received. At a step 404, the two-dimensional image samples are rendered using the NeRF. During the rendering, intermediate computations are stored in a cache, where samples related to the computations have an opacity above a predetermined threshold. At a step 406, the cache is queried using interpolation of surrounding cache points of the data stored in the cache, considering the opacity of each sample. At a step 408, a neural computation of the interpolation results of the querying step is carried out to deliver sample data for colour and opacity. At a step 410, the results of the neural computation are accumulated to generate a three-dimensional image.

[0060] Preferably, data from currently visible portions of the three-dimensional image are cached. The intermediate computations may include latent codes of a neural rendering pipeline structure. The intermediate computations may be stored in the cache along view rays and the samples are stored in the cache in bricks.

[0061] Preferably, the empty bricks are stored in the cache to indicate that there is no radiance to be sampled. A cache structure may correspond to three-dimensional bricks which appear as cuboids in the perspective space of a cache creating view, and truncated oblique pyramids in world space. Each brick may contain a regular grid of samples generated from a rectangular batch of rays. Preferably, the regular grid of samples captures a fixed number of intermediate latent codes of a neural rendering pipeline. The cache may be resampled by querying the bricks at new sample locations by interpolating the samples in the bricks. The interpolation of the samples in the bricks may involve tri-linear interpolation or nearest neighbour sampling. Further, duplication of samples at the brick boundaries may be carried out.

[0062] The method enables NeRF-like representation rendering on thin devices such as mobile phones, facilitating high-quality rendering with minimal computational resources. Additionally, it supports more efficient NeRF-like streaming from the cloud, optimizing rendering processes for cloud-based applications. This method is particularly beneficial for generating images and videos used in training autonomous agents, including self-driving cars, by providing accurate and detailed visual data. Furthermore, the method enhances the rendering quality for dynamic scenes or product presentations, effectively handling depth of field (DoF) and motion blur, thereby improving the visual realism and effectiveness of such presentations.

[0063] FIGS. 5A-5C illustrates exemplary neural radiance fields implementing the method of FIG. 4 in accordance with an implementation of the disclosure. The method is applicable across various Neural Radiance Field, NeRFs rendering techniques, including vanilla NeRF, Instant Neural Graphics Primitives, INGP, Mip-NeRF, and Zip-NeRF. The method requires memory only during rendering, not for the representation itself, making it suitable for all NeRF rendering application scenarios, including streaming. FIG. 5 A shows a vanilla NeRF representation 500 employing the method of FIG. 4. The method queries the cache by interpolating surrounding cache points 502 of the data stored in the cache, considering the opacity of each sample. The method then performs a neural computation on the interpolation results of the querying step to deliver sample data for colour and opacity and accumulates the results of the neural computation to generate a three-dimensional image.

[0064] In this present method, rendering from within a brick containing cached samples involves interpolating the cached intermediate representation, factoring in the opacity of each sample, and subsequently computing only the remaining portion of the network. This approach conserves approximately 90% of the computation required by vanilla NeRF. While vanilla NeRF suggests storing an intermediate latent vector of 256 values, the method of FIG. 4 indicates that reducing the number of stored values to 8-16 is sufficient to maintain quality while substantially decreasing cache space.

[0065] Applying this method to existing NeRF rendering techniques saves a substantial amount of computation, and the shading is entirely view-dependent. There are no limitations on cache utilization. Unlike existing NeRF, which requires memory for scene representation, this method utilizes memory only during the rendering process. This makes it suitable for all NeRF rendering application scenarios, including streaming, where memory limitations are often a concern. Further, caching allows for additional samples to be taken during cache generation compared to basic rendering, resulting in higher quality reconstructed images. Furthermore, the caching automatically adjusts based on camera distance, as it stores samples along view rays. FIG. 5B shows an instant neural graphics primitives (INGP) rendering representation 504 employing the method of FIG. 4. The method queries the cache by interpolating surrounding cache points of the data stored in the cache, considering the opacity of each sample. The method performs a neural computation on the interpolation results of the querying step to deliver sample data for colour and opacity, accumulates the results of the neural computation to generate a three-dimensional image. Instead of solely relying on a multi-layer perceptron (MLP) for the entire neural rendering pipeline, the method incorporates on a hashgrid to store latent codes in a 3D hash grid and utilize smaller networks to compute final shading. For INGP rendering, the method allocates a substantial amount of time for querying the multi resolution hash grid (typically 3 / 4 of the total time). This method easily introduces caching before the final RGB network of the INGP rendering, resulting in a significant reduction in rendering costs.

[0066] FIG. 5C shows an oracle NeRF rendering representation 506 which employing the method of FIG. 4. The method queries the cache by interpolating surrounding cache points of the data stored in the cache, considering the opacity of each sample. The method carries out a neural computation on the interpolation results of the querying step to deliver sample data for colour and opacity and accumulates the results of the neural computation to generate a three-dimensional image. For example, in a comprehensive proposal network like the oracle NeRF rendering, the method dedicates a significant amount of computation to evaluating the oracle and subsequently computing the view-independent adaptive MLP computations for rendering. Implementing the method just before the final shading network can once more conserve 80-90% of computations.

[0067] FIGS. 6A-6B illustrate an exemplary cache-driven training and rendering of a neural radiance field (NeRF) in accordance with an implementation of the disclosure. The NeRF architecture typically includes a view-independent network 602 with a smaller view-dependent network 604 to ensure multi-view consistency.

[0068] The method uses a first view-dependent network 604 to populate the cache even at a later point in the neural rendering pipeline, potentially storing even smaller latent vectors. Further, scaling down the final post-cache network 606 enhances the present method’s efficiency. By adjusting the architecture, the data required for caching is minimized, allowing the use of an even smaller post-cache network 606. This adjustment also permits the use of the view-dependent network 604 before generating the cache.

[0069] The post-cache network 606 requires to understand both the cache generation direction and the new render direction. However, the cache is now usable only for a limited set of new view directions, close to the original cache-generating view. This limitation is not significant in practice, as the rendered view typically aligns closely with the cache generation view, ensuring similar sampling densities between the generating and rendering views. This caching strategy also aligns well with Zip-NeRF rendering, where multiple samples can be used for the view-independent network 602 and pre-cache view-dependent network 604, combining multiple samples to generate one cache point 608.

[0070] Considering the optimization of the caching approach end-to-end, integrating caching directly into training enables specific network training for extrapolating different view directions. This integration can further decrease network sizes and cached data, as all networks become aware of caching mechanisms and cache resampling. Instead of generating normal sample positions for training, multiple randomly offset sample positions around the actual sample is selected, along with randomly offset viewing directions for cache generation. This approach simulates how arbitrary sample locations may be retrieved from the cache to provide the final input to the Tenderer, effectively simulating the caching, interpolation, and extrapolation approach of cached rendering. Particularly, this method is useful for training the network on how to handle interpolation, especially where some samples have zero contribution (zero opacity).

[0071] FIG. 7 is an illustration of a computer system in which the various architectures and functionalities of the various previous implementations may be implemented. As shown, the computer system 700 includes at least one processor 704 that is connected to a bus 702, wherein the computer system 700 may be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), Hyper Transport, or any other bus or point-to-point communication protocol (s). The computer system 700 also includes a memory 706.

[0072] Control logic (software) and data are stored in the memory 706 which may take a form of random-access memory (RAM). In the disclosure, a single semiconductor platform may refer to a sole unitary semiconductor-based integrated circuit or chip. It should be noted that the term single semiconductor platform may also refer to multi-chip modules with increased connectivity which simulate on-chip modules with increased connectivity which simulate on-chip operation and make substantial improvements over utilizing a conventional central processing unit (CPU) and bus implementation. Of course, the various modules may also be situated separately or in various combinations of semiconductor platforms per the desires of the user.

[0073] The computer system 700 may also include a secondary storage 710. The secondary storage 710 includes, for example, a hard disk drive and a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, digital versatile disk (DVD) drive, recording device, universal serial bus (USB) flash memory. The removable storage drive at least one of reads from and writes to a removable storage unit in a well-known manner.

[0074] Computer programs, or computer control logic algorithms, may be stored in at least one of the memory 706 and the secondary storage 710. Such computer programs, when executed, enable the computer system 700 to perform various functions as described in the foregoing. The memory 706, the secondary storage 710, and any other storage are possible examples of computer-readable media.

[0075] In an implementation, the architectures and functionalities depicted in the various previous figures may be implemented in the context of the processor 704, a graphics processor coupled to a communication interface 712, an integrated circuit (not shown) that is capable of at least a portion of the capabilities of both the processor 704 and a graphics processor, a chipset (namely, a group of integrated circuits designed to work and sold as a unit for performing related functions, and so forth).

[0076] Furthermore, the architectures and functionalities depicted in the various previous-described figures may be implemented in a context of a general computer system, a circuit board system, a game console system dedicated for entertainment purposes, an application-specific system. For example, the computer system 700 may take the form of a desktop computer, a laptop computer, a server, a workstation, a game console, an embedded system.

[0077] Furthermore, the computer system 700 may take the form of various other devices including, but not limited to a personal digital assistant (PDA) device, a mobile phone device, a smart phone, a television, and so forth. Additionally, although not shown, the computer system 700 may be coupled to a network (for example, a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, or the like) for communication purposes through an RO interface 708.

[0078] It should be understood that the arrangement of components illustrated in the figures described are exemplary and that other arrangement may be possible. It should also be understood that the various system components (and means) defined by the claims, described below, and illustrated in the various block diagrams represent components in some systems configured according to the subject matter disclosed herein. For example, one or more of these system components (and means) may be realized, in whole or in part, by at least some of the components illustrated in the arrangements illustrated in the described figures.

[0079] In addition, while at least one of these components are implemented at least partially as an electronic hardware component, and therefore constitutes a machine, the other components may be implemented in software that when included in an execution environment constitutes a machine, hardware, or a combination of software and hardware. Although the disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions, and alterations can be made herein without departing from the spirit and scope of the disclosure as defined by the appended claims.

Claims

CLAIMS1. A method of reconstructing a three-dimensional image (108) from a plurality of two-dimensional images by carrying out rendering of the plurality of two-dimensional image using a neural radiance field (102), comprising steps of: receiving a plurality of two-dimensional image samples (106); rendering the plurality of two-dimensional image samples using the neural radiance field, wherein during the rendering, intermediate computations are stored in a cache (104), where samples related to the computations have an opacity above a predetermined threshold; querying the cache using interpolation of surrounding cache points (208, 502, 608) of the data stored in the cache, considering the opacity of each sample; carrying out a neural computation of the interpolation results of the querying step to deliver sample data for colour and opacity; and accumulating the results of the neural computation to generate a three-dimensional image.

2. The method of claim 1, wherein data from currently visible portions of the three-dimensional image are cached.

3. The method of any preceding claim, wherein the intermediate computations are latent codes of a neural rendering pipeline structure.

4. The method of any preceding claim, wherein the intermediate computations are stored in the cache along view rays (206).

5. The method of any preceding claim, wherein the samples are stored in the cache in bricks (110, 304).

6. The method of claim 5, wherein the empty bricks (304A) are stored in the cache to indicate that there is no radiance to be sampled.

7. The method of claim 5, wherein a cache structure (202, 302) corresponds to three-dimensional bricks (204) which are cuboids in the perspective space of a cache creating view, and truncated oblique pyramids in world space.

8. The method of claim 5, wherein each brick contains a regular grid of samples generated from a rectangular batch of rays.

9. The method of claim 8, wherein the regular grid of samples captures a fixed number of intermediate latent codes of a neural rendering pipeline.

10. The method of claim 5, wherein the cache is resampled by querying the bricks at new sample locations by interpolating the samples in the bricks.

11. The method of claim 10, wherein the interpolating of the samples in the bricks involves tri-linear interpolation.

12. The method of claim 10, wherein the interpolating of the samples in the bricks involves nearest neighbour sampling.

13. The method of claim 10, wherein duplication of samples at the brick boundaries is carried out.

14. A system (100) comprising means adapted for carrying out all the steps of the method according to any preceding method claim.

15. A computer program comprising instructions for carrying out all the steps of the method according to any preceding method claim, when said computer program is executed on a computer system.

Images