Synthesis of new views from sparse volume data structures

Abstract

A computer-implemented method for transforming a neural radiation field model is described. A plurality of inputs are provided to a neural radiation field (NeRF) model representing a 3-dimensional space having an object, where each of the plurality of inputs includes a location and a view direction and corresponds to a respective color representing voxels of the 3-dimensional space. Spectral analysis is performed on a plurality of outputs of the NeRF model based on the plurality of inputs, wherein the plurality of outputs include respective colors of voxels. Frequency components are extracted that represent a spectral analysis of colors of at least some of the voxels. A sparse volume data structure is generated that represents respective colors of the 3-dimensional space and at least some of the voxels.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application Ser. No. 17 / 806,410, filed on Jun. 10, 2022, and entitled “NOVEL VIEW SYNTHESIS FROMSPARSE VOLUME DATA STRUCTURE,” the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] Three-dimensional scene reconstruction has attracted much attention in computer vision technology and has played an important role in digital assets, augmented reality, virtual reality, games, movies and other applications. In recent years, with the rapid development of deep learning, deep learning-based 3D scene reconstruction techniques have emerged, including a class of implicit scene representations represented by neural radiance fields (NeRF). The NeRF system uses a neural network model to learn a representation of a scene based on several images of the scene acquired from different views (e.g., top view, side view, etc.), with the goal of providing a rendering of a realistic image of the scene from an unobserved viewpoint. However, NeRF is computationally demanding and poses a challenge to real-time applications. Typically, rendering a view from a trained NeRF requires up to millions of inference calculations by the neural network model, and rendering an 800×600 image may take more than 10 minutes on some hardware. Although some subsequent studies based on NeRF have found that rendering can be accelerated by pre-computation, this method usually requires additional memory space to store these pre-computation results. For an 800×600 image, its runtime memory consumption can be as high as 6 gigabits, so it may not be executable on ordinary mobile phones.

[0004] Embodiments have been described with respect to these and other general considerations. Also, although relatively specific problems have been discussed, it should be understood that the embodiments should not be limited to solving the specific problems identified in the background. Summary of the invention

[0005] Various aspects of the present invention relate to neural network models for synthesizing novel views.

[0006] In one aspect, a computer-implemented method for transforming a neural radiation field model is provided. The method includes: providing multiple inputs to a neural radiation field (NeRF) model representing a 3-dimensional space having an object, wherein each of the multiple inputs includes a location and a view direction and corresponds to a corresponding color of a voxel representing the 3-dimensional space; performing spectral analysis on multiple outputs of the NeRF model based on the multiple inputs, wherein the multiple outputs include the corresponding colors of the voxels; extracting frequency components that represent the spectral analysis of the colors of at least some of the voxels; and generating a sparse volume data structure that represents the 3-dimensional space and the corresponding colors of at least some of the voxels.

[0007] In another aspect, a computer-implemented method for rendering an image of an object in a 3-dimensional space is provided. The method includes: processing a ray from a viewpoint through pixels of an image into the 3-dimensional space, the 3-dimensional space being represented by a sparse volume data structure and a residual neural network; sampling voxels along the ray and accumulating the colors of the voxels according to the color values ​​stored in the sparse volume data structure; providing the location and perspective of the viewpoint to the residual neural network to obtain residual colors of pixels of the image; and rendering the image by combining the accumulated color and the residual color of each pixel of the image.

[0008] In yet another aspect, a non-transitory computer-readable storage medium comprising instructions is provided, the instructions being executable by one or more processors. When executed by the one or more processors, the instructions cause the one or more processors to: provide a plurality of inputs to a neural radiance field (NeRF) model representing a 3-dimensional space having an object, wherein each of the plurality of inputs comprises a location and a view direction and corresponds to a corresponding color of a voxel representing the 3-dimensional space; perform spectral analysis on a plurality of outputs of the NeRF model based on the plurality of inputs, wherein the plurality of outputs comprises the corresponding colors of the voxels; extract frequency components representing the spectral analysis of the colors of at least some of the voxels; and generate a sparse volume data structure representing the 3-dimensional space and the corresponding colors of at least some of the voxels.

[0009] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0011] Figure 1 A block diagram of an example of a system for new view synthesis according to an example embodiment is shown.

[0012] Figure 2 A diagram showing an example of a prior art neural radiance field training process.

[0013] Figure 3A A diagram of an example sparse data processor for generating a sparse volumetric data structure according to an example embodiment is shown.

[0014] Figure 3B A diagram showing an example sparse volume data structure and corresponding first and second textures according to an example embodiment is shown.

[0015] Figure 4 A diagram showing an example graph for converting a NeRF-like model into a sparse volume data structure according to an example embodiment.

[0016] Figure 5 A flow chart of an example method for transforming a neural radiation field model according to an example embodiment is shown.

[0017] Figure 6 A flow chart of an example method for rendering an image of an object in 3-dimensional space according to an example embodiment is shown.

[0018] Figure 7 is a block diagram illustrating example physical components of a computing device in which various aspects of the disclosure may be practiced.

[0019] Figure 8 and Fig. 9 is a simplified block diagram of a mobile computing device in which various aspects of the disclosure may be practiced. DETAILED DESCRIPTION

[0020] In the following detailed description, reference is made to the accompanying drawings that form a part thereof and illustrate specific embodiments or examples by way of illustration. These aspects may be combined, other aspects may be used, and structural changes may be made without departing from the present disclosure. The embodiments may be put into practice as methods, systems, or devices. Therefore, the embodiments may take the form of hardware implementation, complete software implementation, or a combination of hardware and software implementations. Therefore, the following detailed description should not be considered restrictive, and the scope of the present disclosure is limited by the appended claims and their equivalents.

[0021] The present disclosure describes various examples of a computing device having a processor configured to perform new view synthesis (NVS). The computing device is configured to pre-compute a neural radiation field (NeRF) model of a scene based on a predictive calculation, and transform the NeRF model into a sparse volume data structure. Compared with the NeRF model, the sparse volume data structure is greatly reduced in size without reducing the accuracy and speed of the calculation, and ensures that the runtime memory and video card memory consumption are controllable. Moreover, the computing device is configured to connect the calculation data stream and the rendering data stream for processing by a graphics processing unit (GPU), thereby achieving real-time rendering on a mobile phone or smart phone. To achieve this, the computing device is configured to execute a pre-computation method of a NeRF-like algorithm for converting the NeRF model into a sparse volume data structure, and perform a GPU calculation and rendering process that is effective for the sparse volume data structure. The transformed model representation (i.e., a sparse volume data structure) preserves the NeRF model's ability to render fine geometric details and view-dependent appearance (e.g., specularity, reflections), is compact (less than 100 megabytes per scene on average), and can be rendered in real time on consumer hardware (e.g., 30 frames per second on an iPhone) when configured with the rendering engine described in this article.

[0022] This embodiment and many other embodiments of a computing device are described herein. For example, Figure 1 A block diagram of an example of a system 100 for synthesizing new views is shown according to an example embodiment. System 100 includes a computing device 110, which is configured to train a neural network model, such as neural network model 118 or neural network model 128, using a source image 130. Computing device 110 includes a view processor 111, which is configured to perform view processing to synthesize a new view of an object or scene based on neural network model 118 or 128. In some examples, system 100 may also include a data repository 120 communicatively coupled to computing device 110 via network 140. In some examples, computing device 110 includes a first neural network model 118 for a NeRF model and a second neural network model (not shown) for a residual neural network model, as described below.

[0023] Typically, source image 130 is an image representing a scene, an object or group of objects, a person or crowd, or other suitable object. In some examples, source image 130 is an image captured by a digital camera, a digital image processor, an image capture module (e.g., of a webcam or smartphone), or other suitable image capture device. In other examples, some or all of source image 130 is generated by a 2D or 3D rendering module, and a snapshot or screenshot is captured based on the output of the rendering module or the content on a display (e.g., a computer monitor or smartphone screen).

[0024] Source images 130 are referred to as "sparse" source images because they do not include an image of every possible view of a scene. For example, source images 130 may include a top view of an object, a left side view of an object, and a right perspective view of an object, but omit the left perspective view of an object, the right side view of an object, etc. In some examples, source images 130 include two, three, four, or more images of an object, and may include tens, hundreds, or thousands of images from different locations (e.g., x, y, and z coordinates in 3D space) and orientations (e.g., polar angle θ and azimuth angle φ). Figure 2 The training of the neural network model 118 or 128 is described.

[0025] Computing device 110 may be any type of computing device, including a smartphone, a mobile computer, or a mobile computing device (e.g., Devices, laptops, notebooks, such as Apple iPad TM The computing device 110 may be a tablet computer, a netbook, etc., or a stationary computing device such as a desktop computer or a PC (personal computer). The computing device 110 may be configured to communicate with a social media platform, a cloud processing provider, software as a service provider, or other suitable entity, for example, using social media software and a suitable communication network. The computing device 110 may be configured to execute one or more software applications (or, "applications") and / or services and / or manage hardware resources (e.g., processor, memory, etc.) that may be used by a user of the computing device 110.

[0026] exist Figure 1 In the example shown, the view processor 111 includes a NeRF processor 112, a sparse data processor 114, and a rendering engine 116. In other examples, one or more of the NeRF processor 112, the sparse data processor 114, or the rendering engine 116 may be formed as a combined processor. In some examples, at least some portions of the view processor 111 may be combined with the neural network model 118, for example by including a neural network processor or other suitable processor configured to implement the neural network model. In other words, the neural network model 118 may be integrated with the view processor 111 and implemented with or as a neural network processor. In some examples, the neural network model 118 is omitted from the computing device 110, and a neural network model 128 is used instead.

[0027] In some examples, one or more of the NeRF processor 112, the sparse data processor 114, or the rendering engine 116 are located in other computing devices (not shown). In one such example, a first computing device includes the NeRF processor 112 and the sparse data processor 114, and a second computing device includes the rendering engine 116. In another example, a first computing device includes the NeRF processor 112, a second computing device includes the sparse data processor 114, and a third computing device includes the rendering engine 116.

[0028] The NeRF processor 112 is configured to train the neural network model 118 to synthesize views based on source images, such as source image 130. In some examples, the NeRF processor 112 performs a neural radiation field training process 200, as described below with respect to Figure 2 Just as described.

[0029] The sparse data processor 114 is configured to convert the neural network model 118 or 128 (e.g., a NeRF model or a NeRF-like model) into a sparse volume data structure representing the neural network model. Typically, a NeRF-like model is trained to model the neural radiation field of a view of an object within a 3-dimensional space or scene as a continuous multivariate function. However, for most real scenes, the geometric distribution is mainly composed of the surfaces of objects, which means that most areas in the neural radiation field are actually blank and no value operations need to be performed on these blank areas. Moreover, since the rendering of the view itself is constrained by a discrete resolution, a completely continuous radiation field is also unnecessary. In other words, rendering can be completed as long as value operations can be performed on a finite number of points in space.

[0030] The sparse data processor 114 exploits these concepts by first discretizing the target space of the NeRF-like model and modeling the target space as a collection of small voxels. A voxel represents a value on a regular grid in three-dimensional space, but its position is typically not explicitly encoded. Instead, a rendering engine such as the rendering engine 116 infers the position of the voxel based on the position of the voxel relative to other voxels. In other words, the rendering engine 116 infers the position of the voxel based on the position of the voxel in a sparse volume data structure representing the object or scene. Using the voxel representation of the NeRF-like model, the results of the neural radiation field at the location associated with each voxel can then be calculated offline and saved.

[0031] For a given spatial location or voxel, its color is related to the viewing direction. If the results of each angle are stored for each voxel, the memory storage overhead will be very large. In order to improve storage efficiency, the sparse data processor 114 is configured to decompose the color change related to the viewing direction into a diffuse component, a low-frequency component, and a high-frequency component. The decomposition is described below.

[0032] The rendering engine 116 is configured to render a view of the object based on the sparse volume data structure. In some examples, the rendering engine 116 is configured as one or more software modules or firmware modules executed on a graphics processing unit (GPU), a central processing unit (CPU), or other suitable processor.

[0033] Neural network model 118 is trained using view processor 111 (e.g., via NeRF processor 112), and is configured to process input coordinates and orientations to provide a synthetic view of an object. In some examples, neural network model 118 is implemented as a deep fully connected neural network without convolutional layers, such as a multi-layer perceptron (MLP) model. Neural network model 128 is generally similar to neural network model 118, but is stored remotely from computing device 110 (e.g., at data repository 120).

[0034] Data repository 120 may include one or more of any type of storage mechanism, including a disk (e.g., in a hard drive), an optical disk (e.g., in an optical drive), a magnetic tape (e.g., in a magnetic tape drive), a memory device such as a RAM device, a ROM device, and / or any other suitable type of storage medium. For example, data repository 120 may store neural network model 128 and / or source images 130 (e.g., images used to train neural network model 118 and / or 128). In some examples, data repository 120 provides source images 130 to view processor 111 for training neural network model 118 and / or neural network model 128. In some examples, one or more data repositories 120 may be co-located (e.g., housed in one or more nearby buildings with associated components such as backup power, redundant data communications, environmental controls, etc.) to form a data center, or may be arranged in other ways. Thus, in embodiments, one or more data repositories 120 may be a data center in a distributed collection of data centers.

[0035] The network 140 may include one or more networks, such as a local area network (LAN), a wide area network (WAN), an enterprise network, the Internet, etc., and may include one or more wired and / or wireless portions. The computing device 110 and the data repository 120 may include at least one wired or wireless network interface that enables communication with each other (or with an intermediate device, such as a web server or a database server) via the network 140. Examples of such network interfaces include, but are not limited to, an IEEE 802.11 wireless LAN (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth® ... TMAn interface or a near field communication (NFC) interface. Examples of network 140 include a local area network (LAN), a wide area network (WAN), a personal area network (PAN), the Internet, and / or any combination thereof.

[0036] Figure 2 A diagram of an example of a prior art neural radiance field training process 200 is shown. The process 200 generally describes synthesizing an image by sampling 5-dimensional input coordinates along camera rays (202). The input coordinates generally indicate a location (x, y, z) in 3-dimensional space and a viewing direction or orientation in spherical coordinates (θ, φ) that can be combined as (x, y, z, θ, φ). The input coordinates are provided to a multi-layer perceptron (MLP) neural network model (F Θ , such as a neural network model 118 or 128) to produce (204) color (RGB) and volume density σ. Volume rendering techniques composite (206) the RGB values ​​and volume density σ into an image. The rendering function 204 is differentiable, so the neural network model can be trained (208) using a loss function that minimizes the residual between the synthesized image and the ground truth ("gt") image. Mildenhall, B. et al., "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis," published in Springer Cham's European Conference on Computer Vision on August 23, 2020, pp. 405-421, describes further details of the neural radiance field training process. Although process 200 is described with respect to an MLP neural network model, in other examples, other neural network models or NeRF-like models may be used.

[0037] In general, process 200 abstracts a scene as a radiation field, i.e., for any point in space, the color and transparency of light emitted in any direction can be calculated. When such a radiation field is available, then synthesizing an image from any viewing direction is a simple numerical accumulation process. NeRF uses a neural network model (e.g., neural network model 118 or 128) to represent the radiation field. The input of neural network model 118 is a location (x, y, z) in 3D space and an viewing direction or orientation in spherical coordinates (θ, φ), from which a view (e.g., a virtual camera location) should be generated, and the output is the color and transparency of a voxel in 3D space. Input images from different viewing directions can be provided to train the neural network, and NVS can be performed after the training is completed. Because the method is simple and effective, a large number of investigators have conducted subsequent investigations based on NeRF, forming a large class of NVS solutions. However, the use of NeRF requires a complete neural network inference for each position in space, which makes its execution inefficient and difficult to implement.

[0038] Figure 3A A diagram of an example sparse data processor 300 for generating a sparse volume data structure according to an example embodiment is shown. The sparse data processor 300 generally corresponds to the sparse data processor 114 and is configured to process an input 310 (such as a NeRF-like model) and convert it into an output 320 (such as a sparse volume data structure). As described above, the sparse data processor 300 is configured to convert the neural network model 118 or 128 into a sparse volume data structure representing the neural network model by decomposing the color change of each voxel related to the viewing direction into a diffuse component, a low-frequency component, and a high-frequency component.

[0039] The diffusion component generally represents the color component of the voxel that is independent of the viewing direction. For the target location (i.e., voxel), the sparse data processor 300 uses the trained neural network model 118 to calculate multiple colors in different directions. In one example, the multiple colors include a color for each viewing direction (θ, φ) at 5° intervals, such as (0°, 0°), (0°, 5°), (0°, 10°), ... (0°, 355°), (5°, 0°), etc. In other examples, different intervals are used, such as 1°, 2°, 10°, or another suitable interval. In some examples, for some viewing directions, the interval is larger to reduce the computational cost, while for other viewing directions, the interval is smaller to improve color accuracy. For example, viewing directions that are unlikely to be used (e.g., very low downward angles, very high upward angles) can use larger intervals, while viewing directions that are commonly used (e.g., usually horizontal angles) can use smaller intervals.

[0040] The sparse data processor 300 includes a spherical harmonics processor 342 that performs a spherical-based spectral analysis technique (e.g., spherical harmonics using Fourier transforms) on a plurality of colors, extracts a first component having a frequency of 0 as a diffuse component at the target location, and stores corresponding RGB values ​​as diffuse components. The RGB values ​​may be stored as a set of values ​​for each color at the location, such as (235, 117, 21) for orange, with a range from 0 to 255 for 8-bit colors, 0 to 65,535 for 16-bit colors, or another suitable range. In other examples, the RGB values ​​are stored as hexadecimal values, such as #eb7515 for orange. Similar to the diffuse component calculation, the sparse data processor 300 extracts low-frequency components having the first two frequencies of the spherical harmonics, and stores 9 corresponding RGB values, such as the corresponding RGB values ​​for the following:

[0041]

[0042] In other examples, the sparse data processor 300 may extract a different number of low frequency components, such as the first three frequencies, and store 16 corresponding RGB values, although additional low frequency components generally require more processing resources.

[0043] In the spectral analysis results obtained by spherical harmonics, the recovery of high-frequency information often requires the use of many components (>100) to meet certain accuracy requirements. Storing them will consume a lot of memory, but the contribution from their final rendered color (for example, when combined with the diffuse component and the low-frequency component) is often weak, so the benefit of direct storage is very low. Due to the small contribution, the sparse data processor 300 defines the contribution of the high-frequency component to the final rendered color as a color residual. In fact, the distribution of color residuals in the same scene has certain patterns. For example, the lighting in the same scene is usually consistent. The material distribution is roughly continuous, so the sparse data processor 300 is configured to use a simpler residual neural network model 344 to model the color residual. The input of the residual neural network model 344 is also 3D coordinates and 2D direction information (x, y, z, θ, φ), and the output is the color residual. The training data of the residual neural network model 344 is obtained by subtracting the diffuse component and the low-frequency component from the NeRF inference result. In some examples, the residual neural network model 344 has only one layer and can be trained in less than 100 samples. The weights of the trained residual neural network model 344 are stored as a grayscale image. In this way, the sparse data processor 300 provides diffusion volume data and low frequency volume data and a grayscale image representing the residual neural network model 344 based on the NeRF model.

[0044] Figure 3BA diagram of an example data structure 350 and corresponding first texture 352 and second texture 354 according to an example embodiment is shown. For ease of explanation, data structure 350 is 2-dimensional with grid coordinates (A to D and 1 to 4) and a single color value, but can be easily extrapolated to a 3-dimensional space with multiple color values ​​(i.e., diffuse color components and low-frequency color components). Data structure 350 is generally blank, but includes orange at A3 and A4, red at B4 and C4, pink at C1, green at C3, and yellow at D1 and D2. Further explanation of first texture 352 and second texture 354 is provided below.

[0045] Figure 4 A diagram of an example graph 400 for converting a NeRF-like model 410 of an object (forklift) in 2-dimensional space into a sparse volume data structure 420 according to an example embodiment is shown. Graph 400 includes a sparse data processor 300. Although the sparse data processor 300 provides two volumes of data, i.e., diffuse volume data and low-frequency volume data, for most scenarios, both volumes of data are very sparse, and useful data is typically less than 5% of the total. If the volume data is stored as a conventional square lattice (or, in the 3-dimensional example, a cubic lattice), the space utilization will be very low. To solve this problem, the sparse data processor 300 includes a block partitioning processor 346, which is configured to divide the space (i.e., x, y space for 2-D, x, y, z space for 3-D) into blocks of inconsistent sizes, so that large blank areas (i.e., no objects or not adjacent to objects) can be represented by only one large block (e.g., block 422) or a medium-sized block (e.g., block 424), and then the blocks where valid data are located (e.g., adjacent to the surface of the object) can be represented by smaller blocks (e.g., block 426), which can effectively compress the size of the data. Therefore, the NeRF model 410 is converted into two sparse volume data structures 420 (i.e., for diffuse color components and low-frequency components) and a grayscale image representing a color residual network. In some examples, the two sparse volume data structures are combined into a single sparse volume data structure. Figure 4 In the example shown, large block 422 and medium block 424 are shown as covering the forklift, but small block 426 is not shown for clarity. Typically, small block 426 will cover the rest of data structure 420.

[0046] Typically, the resources on a mobile phone or smartphone used to render a view (i.e., for the rendering engine 116) are limited, such as the size of the graphics processing memory, the computational style (e.g., available processing instructions), and the computational speed (e.g., clock frequency). Using the above-described sparse volume data structure, the compressed model volume is typically less than 100 megabytes, which typically overcomes the limitations of the graphics processing memory of the rendering engine 116. To improve processing on typical smartphone processing hardware, the rendering engine 116 implements rendering on the GPU side using the shading language of the graphics API, such as using OpenGLES 3.0 or the iOS Metal framework.

[0047] There are specific patterns of supported or even preferred computing modes for the graphics API of smartphones. Typically, the GPU on a smartphone operates on regularly organized graphics data (e.g., textures and geometric shapes) and does not originally support the above-mentioned sparse volume data structure 420. In order to overcome the lack of native support for sparse volume data structures, the rendering engine 116 is configured to use two textures to represent the sparse volume data structure, represented as a first texture and a second texture, wherein the first texture records the data of valid points in the sparse volume data structure, and the second texture records the connection relationship between the blocks of the sparse volume data structure (e.g., a tree structure describing the topology of the connected blocks). In this way, the rendering engine 116 uses the first texture and the second texture to implement the operation of the tree structure using a shading language (i.e., texture), which most graphics APIs already support and can effectively process. In some scenarios, the sparse volume data structure 420 is expanded into two 2-dimensional textures, and the sparsity of the two textures is increased. In some examples, the sparse data processor 114 is configured to use 2D texture compression (such as, using the KTX format or other suitable compression format) to reduce the consumption of video card memory.

[0048] Refer again Figure 3B , data structure 350 can be represented as a tree structure where a single node representing a larger first block, which covers A1, A2, B1, and B2, does not contain any data. Figure 4 The tree structure shown with nodes representing blocks of different sizes is useful for representing NeRF models, but the approach is more challenging to implement within a typical GPU. The texture generator 348 is configured to convert the tree structure into a first texture 352 that records data of valid points and a second texture 354 that records the connection relationship between blocks. Figure 3BIn the example shown, the first texture 352 is a 2-dimensional texture having a first dimension corresponding to the index of the block (i.e., the grid structure of the data structure 350) and a second dimension corresponding to the color of the corresponding block (O for orange, R for red, etc.). The second texture 354 is a 2-dimensional texture having a first dimension corresponding to the index of the block and a second dimension corresponding to the relative location of the adjacent blocks: top (T), top right (TR), right (R), bottom right (BR), bottom (B), bottom left (BL), left (L), and top left (TL), where the values ​​correspond to the index or pointer to the corresponding block. As an example, cell A3 is adjacent to cells A2, B2, B3, B4, and A4, but cells A2, B2, and B3 do not contain color data and can be omitted from the first texture 352. However, the positional relationship between cell A3 and other cells that have data (such as, cells A4 and B4) is provided by the second texture 354, where the bottom right (BR) value refers to B4 and the bottom (B) value refers to A4.

[0049] To synthesize a new view, the rendering engine 116 processes a ray from the camera's viewpoint through each pixel and samples several points on the ray. For each point, the rendering engine 116 calculates the corresponding color of the pixel using the sparse volume data structure and the residual neural network model 344, and accumulates these values ​​along the ray to calculate the final color of the pixel. Even when the sparse volume data structure is already sparse, no calculation is required in the blank area, but sampling of the blank area cannot be avoided. In order to reduce the sampling of the blank area during rendering, the rendering engine 116 is configured to generate multiple mipmap sets for the first texture and the second texture representing the sparse volume data structure. Each color calculation starts from the bottom of the mipmap to determine whether it is blank, so that the blank area can be passed quickly and the calculation of valid data can be started faster. Although mipmaps can use additional memory, in some examples, the increase in calculation speed using mipmaps increases the overall processing speed by 80%.

[0050] Figure 5 Flowchart of an example method 500 for transforming a neural radiation field model according to an example embodiment is shown. Unless otherwise indicated, the technical process shown in these figures will be performed automatically. In any given embodiment, some steps of the process may be repeated, possibly with different parameters or data. The steps in the embodiment may also be performed in the same manner as in the example embodiment. Figure 5The steps may be performed in a sequence different from the top-to-bottom sequence specified in the method 500. The steps may be performed serially with partial overlap, or completely in parallel. Therefore, the order in which the steps of method 500 are performed may change from one execution of the process to another execution of the process. Steps may also be omitted, combined, renamed, regrouped, performed on one or more machines, or otherwise depart from the illustrated flow if the performed process is operable and complies with at least one claim. Figure 5 Step A may be performed by computing device 110 (e.g., via view processor 111, neural network model 118) or other suitable computing device. In some examples, neural network model 128 may be used to perform Figure 5 steps.

[0051] The method 500 begins with step 502. In step 502, a plurality of inputs are provided to a neural radiance field (NeRF) model representing a 3-dimensional space having an object, such as the NeRF-like model 410. Each of the plurality of inputs includes a location and a view direction, and corresponds to a respective color of a voxel representing the 3-dimensional space (e.g., as an output of the NeRF-like model 410).

[0052] In step 504, a spectral analysis is performed on a plurality of outputs of the NeRF model based on the plurality of inputs. The plurality of outputs include respective colors of the voxels. In some examples, the spectral analysis is a Fourier analysis of colors across input view directions and locations.

[0053] In step 506, frequency components are extracted by spectral analysis, which frequency components represent the color of at least some of the voxels. In some examples, the diffuse color component is extracted as a fundamental frequency component, and the low frequency component is extracted as a first harmonic component and a second harmonic component. In some examples, step 506 also includes training a residual neural network to represent the high frequency components of the spectral analysis (e.g., using the residual neural network model 344). In one such example, the residual neural network is a single-layer neural network, and the training data of the residual neural network model is obtained by subtracting the diffuse color component and the low frequency component from the corresponding output of the NeRF model. The weights of the residual neural network model can be stored as a grayscale image and provided to the neural network model to render the high frequency component based on the input location and orientation.

[0054] In step 508 , a sparse volume data structure is generated, which represents the 3-dimensional space and the corresponding colors of at least some of the voxels.

[0055] In some examples, method 500 also includes dividing the 3D space into blocks of inconsistent sizes and storing colors associated with frequency components of voxels within corresponding blocks. For example, block partitioning processor 346 divides the 3D space into blocks 422, 424, and 426. In one example, storing colors includes omitting colors within blocks that are not adjacent to the surface of the object. In other words, data of blank blocks within the 3D space is not stored. In some examples, dividing the 3D space includes using larger-sized blocks (e.g., block 422) for areas of the 3D space farther from the object, and using smaller-sized blocks (e.g., block 424) for areas of the 3D space closer to the object.

[0056] In some examples, storing the color includes storing the color within the block as a first texture and storing the relative position within the block as a second texture.For example, texture generator 348 generates first texture 352 and second texture 354 that may be stored to represent data structure 350 .

[0057] In some examples, the sparse volume data structure is a tree data structure having nodes corresponding to blocks of inconsistent sizes.

[0058] Figure 6 Flowchart of an example method 600 for rendering an image of an object in 3-dimensional space according to an example embodiment is shown. Unless otherwise indicated, the technical process shown in these figures will be performed automatically. In any given embodiment, some steps of the process may be repeated, and some steps of the process may be operated with different parameters or data. The steps in the embodiment may also be performed in the same manner as in the example embodiment. Figure 6 The steps may be performed in a sequence different from the top-to-bottom sequence specified in the method 600. The steps may be performed serially with partial overlap, or completely in parallel. Therefore, the order in which the steps of method 600 are performed may change from one execution of the process to another execution of the process. Steps may also be omitted, combined, renamed, regrouped, performed on one or more machines, or otherwise depart from the illustrated flow if the performed process is operable and complies with at least one claim. Figure 6 The steps of may be performed by computing device 110 (e.g., via view processor 111, neural network model 118) or other suitable computing devices. In some examples, neural network model 128 may be used to perform Figure 6 steps.

[0059] The method 600 begins with step 602. In step 602, rays are processed from a viewpoint through pixels of an image into a 3-dimensional space represented by a sparse volume data structure and a residual neural network.

[0060] In step 604, voxels are sampled along the ray and the colors of the voxels are accumulated according to the color values ​​stored in the sparse volume data structure.

[0061] In step 606, the location and perspective of the viewpoint are provided to a residual neural network to obtain residual colors of pixels of the image.

[0062] In step 608, the image is rendered by combining the accumulated color and the residual color for each pixel of the image.

[0063] Figure 7 is a block diagram illustrating the physical components (e.g., hardware) of a computing device 700 in which various aspects of the present disclosure may be practiced. The computing device components described below may have computer-executable instructions for implementing a view rendering application 720 on a computing device (e.g., computing device 110), including computer-executable instructions for the view rendering application 720 that may be executed to implement the methods disclosed herein. In a basic configuration, the computing device 700 may include at least one processing unit 702 and a system memory 704. Depending on the configuration and type of the computing device, the system memory 704 may include, but is not limited to, volatile storage devices (e.g., random access memory), non-volatile storage devices (e.g., read-only memory), flash memory, or any combination of such memories. The system memory 704 may include an operating system 705 and one or more program modules 706 suitable for running the view rendering application 720, such as those related to the system memory 704. Figure 1 and one or more components of 3, and specifically, a NeRF processor 721 (e.g., corresponding to the NeRF processor 112), a sparse data processor 722 (e.g., corresponding to the sparse data processor 114), and a rendering engine 723 (e.g., corresponding to the rendering engine 116).

[0064] For example, operating system 705 may be suitable for controlling the operation of computing device 700. In addition, examples of the present disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application program, and are not limited to any particular application or system. Figure 7 708 by these components within dashed line 708. Computing device 700 may have additional features or functionality. For example, computing device 700 may also include additional data storage devices (removable and / or non-removable), such as, for example, magnetic disks, optical disks, or tapes. Such additional storage devices may be Figure 7 The example is illustrated by a removable storage device 709 and a non-removable storage device 710.

[0065] As explained above, a plurality of program modules and data files may be stored in the system memory 704. When executed on the processing unit 702, the program module 706 (e.g., the view rendering application 720) may perform processes including, but not limited to, the various aspects described herein. Other program modules that may be used according to various aspects of the present disclosure and in particular for generating views may include a NeRF processor 721, a sparse data processor 722, and a rendering engine 723.

[0066] In addition, examples of the present disclosure may be practiced in circuits including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits using microprocessors, or on a single chip containing electronic components or microprocessors. For example, examples of the present disclosure may be practiced via a system on a chip (SOC), wherein Figure 7 Each or many of the illustrated components may be integrated onto a single integrated circuit. Such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functionalities, all of which are integrated (or, "burned") onto a chip substrate as a single integrated circuit. When operated via a SOC, the functionality described herein with respect to the ability of a client to switch protocols may be operated via dedicated logic integrated with other components of the computing device 800 on a single integrated circuit (chip). Examples of the present disclosure may also be practiced using other technologies capable of performing logical operations (such as, for example, AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, examples of the present disclosure may be practiced within a general-purpose computer or in any other circuit or system.

[0067] The computing device 700 may also have one or more (multiple) input devices 712, such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or sliding input device, etc. Output device(s) 714 may also be included, such as a display, a speaker, a printer, etc. The above devices are examples, and other devices may be used. The computing device 700 may include one or more communication connections 716 that allow communication with other computing devices 750. Examples of suitable communication connections 716 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial bus (USB), parallel and / or serial ports.

[0068] The term computer-readable medium used herein may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as, computer-readable instructions, data structures or program modules). System memory 704, removable storage device 709 and non-removable storage device 710 are all examples of computer storage media (e.g., memory storage devices). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device or any other product that can be used to store information and can be accessed by computing device 700. Any such computer storage medium may be part of computing device 700. Computer storage media does not include carrier waves or other propagation or modulated data signals.

[0069] Communication media may be implemented by computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics set or changed in such a manner as to encode the information in the signal. By way of example and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared and other wireless media.

[0070] Figure 8 and Fig. 9 The mobile computing device 800 on which examples of the present disclosure may be practiced is illustrated, such as a mobile phone, a smart phone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, etc. In some aspects, the client may be a mobile computing device. Figure 8, illustrates one aspect of a mobile computing device 800 for implementing these aspects. In a basic configuration, the mobile computing device 800 is a handheld computer with input elements and output elements. The mobile computing device 800 typically includes a display 805 and one or more input buttons 810 that allow a user to enter information into the mobile computing device 800. The display 805 of the mobile computing device 800 can also act as an input device (e.g., a touch screen display). If included, the optional side input element 815 allows further user input. The side input element 815 can be a rotary switch, a button, or any other type of manual input element. In alternative aspects, the mobile computing device 800 can include more or fewer input elements. For example, in some examples, the display 805 may not be a touch screen. In yet another alternative example, the mobile computing device 800 is a portable telephone system, such as a cellular phone. The mobile computing device 800 may include a front-facing camera 830. The mobile computing device 800 may also include an optional keypad 835. The optional keypad 835 may be a physical keypad or a "soft" keypad generated on the touch screen display. In various examples, output elements include a display 805 for showing a graphical user interface (GUI), a visual indicator 820 (e.g., a light emitting diode), and / or an audio transducer 825 (e.g., a speaker). In some aspects, the mobile computing device 800 includes a vibration transducer for providing tactile feedback to the user. In yet another aspect, the mobile computing device 800 includes input and / or output ports for sending or receiving signals to or from external devices, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port).

[0071] Fig. 9 902 is a block diagram illustrating an architecture of one aspect of a mobile computing device. That is, the mobile computing device 800 may include a system (e.g., architecture) 902 to implement some aspects. In one example, the system 902 is implemented as a "smart phone" capable of running one or more applications (e.g., browsers, emails, calendars, contact managers, messaging clients, games, and media clients / players). In some aspects, the system 902 is integrated into a computing device, such as an integrated personal digital assistant (PDA) and a wireless phone. The system 902 may include a display 905 (similar to the display 805), such as a touch screen display or other suitable user interface. The system 902 may also include an optional keypad 935 (similar to the keypad 835) and one or more peripheral device ports 930, such as input and / or output ports for audio, video, control signals, or other suitable signals.

[0072] In some examples, the system 902 may include a processor 960 coupled to a memory 962. The system 902 may also include a special purpose processor 961, such as a neural network processor. One or more application programs 966 may be loaded into the memory 962 and run on or in association with the operating system 964. Examples of application programs include a phone dialer program, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. The system 902 also includes a non-volatile storage area 962 within the memory 968. The non-volatile storage area 968 may be used to store persistent information that should not be lost if the system 902 is powered off. The application 966 may use and store information in the non-volatile storage area 968, such as emails or other messages used by an email application. A synchronization application (not shown) also resides on the system 902 and is programmed to interact with a corresponding synchronization application residing on a host computer to synchronize the information stored in the non-volatile storage area 968 with the corresponding information stored at the host computer.

[0073] System 902 has a power source 970, which can be implemented as one or more batteries. Power source 970 can also include an external power source, such as an AC adapter or a powered docking cradle to replenish or recharge the batteries.

[0074] The system 902 may also include a radio interface layer 972 that performs the functions of transmitting and receiving radio frequency communications. The radio interface layer 972 facilitates wireless connectivity between the system 902 and the "outside world" via a communications carrier or service provider. Transmissions to and from the radio interface layer 972 are performed under the control of the operating system 964. In other words, communications received by the radio interface layer 972 can be disseminated to the application 966 via the operating system 964, and vice versa.

[0075] Visual indicator 920 can be used to provide visual notification, and / or audio interface 974 can be used to communicate via audio transducer 825 (eg, Figure 8The audio transducer 825 shown in the figure produces an audible notification. In the example shown in the figure, the visual indicator 920 is a light emitting diode (LED), and the audio transducer 825 can be a speaker. These devices can be directly coupled to the power supply 970 so that when activated, even if the processor 960 and other components may be turned off to save battery power, they will remain turned on for a duration specified by the notification mechanism. The LED can be programmed to remain turned on indefinitely until the user takes action to indicate the power-on state of the device. The audio interface 974 is used to provide an audible signal to the user and receive an audible signal from the user. For example, in addition to being coupled to the audio transducer 825, the audio interface 974 can also be coupled to a microphone to receive audible input, such as promoting a telephone conversation. According to the example of the present disclosure, the microphone can also be used as an audio sensor to facilitate the control of the notification, as described below. The system 902 can also include a video interface 976, which enables the operation of the peripheral device 930 (e.g., a vehicle-mounted camera) to record still images, video streams, etc.

[0076] The mobile computing device 800 implementing the system 902 may have additional features or functionalities. For example, the mobile computing device 800 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage devices may be used to store data or to store information. Fig. 9 This is illustrated by the non-volatile storage area 968.

[0077] Data / information generated or captured by the mobile computing device 800 and stored via the system 902 may be stored locally on the mobile computing device 800, as described above, or the data may be stored on any number of storage media that may be accessed by the device via the radio interface layer 972 or via a wired connection between the mobile computing device 800 and a separate computing device associated with the mobile computing device 800 (e.g., a server computer in a distributed computing network such as the Internet). As should be appreciated, such data / information may be accessed via the mobile computing device 800, via the radio interface layer 972, or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use in accordance with well-known data / information transmission and storage components, including email and collaborative data / information sharing systems.

[0078] As you should know, Figure 8 and Fig. 9 It is described for the purpose of illustrating the present methods and systems and is not intended to limit the present disclosure to a specific sequence of steps or a specific combination of hardware or software components.

[0079] The phrases "at least one", "one or more", "or", and "and / or" are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions "at least one of A, B, and C", "at least one of A, B, or C", "one or more of A, B, and C", "one or more of A, B, or C", "A, B and / or C", and "A, B, or C" refers to A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0080] The term "a" or "an" entity refers to one or more of that entity. Therefore, the terms "a" (or, "an"), "one or more" and "at least one" can be used interchangeably herein. It should also be mentioned that the terms "including", "comprising" and "having" can be used interchangeably.

[0081] As used herein, the term "automatic" and variations thereof refer to any process or operation, which is generally continuous or semi-continuous, that is performed without substantial human input while the process or operation is being performed. However, a process or operation may be automatic even if the performance of the process or operation uses substantial or insubstantial human input if input is received prior to the performance of the process or operation. Human input is considered substantial if such input affects the manner in which the process or operation is performed. Human input that consents to the performance of the process or operation is not considered "substantial."

[0082] Any of the steps, functions and operations discussed herein may be performed continuously and automatically.

[0083] The exemplary systems and methods of the present disclosure have been described with respect to computing devices. However, in order to avoid unnecessarily obscuring the present disclosure, the previous description omits several known structures and devices. This omission should not be construed as limiting. Specific details are set forth to provide an understanding of the present disclosure. However, it should be understood that the present disclosure can be practiced in a variety of ways beyond the specific details set forth herein.

[0084] In addition, although the exemplary aspects illustrated herein show various components of the system being collocated, certain components of the system may be located remotely, at a remote portion of a distributed network (such as a LAN and / or the Internet), or within a dedicated system. Thus, it should be appreciated that the components of the system may be combined into one or more devices, such as a server, a communication device, or collocated at a specific node of a distributed network, such as an analog and / or digital telecommunications network, a packet switching network, or a circuit switching network. It will be appreciated from the previous description, and for reasons of computational efficiency, that the components of the system may be arranged at any location within the distributed network of components without affecting the operation of the system.

[0085] In addition, it should be understood that the various links connecting the elements can be wired or wireless links or any combination thereof or any other known or later developed (multiple) elements that can supply and / or transfer data between the connected elements. These wired or wireless links can also be secure links and may be able to transfer encrypted information. For example, the transmission medium used as a link can be any suitable electrical signal carrier, including coaxial cable, copper wire and optical fiber, and can take the form of sound waves or light waves, such as those generated during radio wave and infrared data communications.

[0086] While the flow diagrams have been discussed and illustrated with respect to a particular sequence of events, it should be appreciated that changes, additions, and omissions to this sequence may occur without materially affecting the operation of the disclosed configurations and aspects.

[0087] Several variations and modifications of the present disclosure may be used. It would be possible to provide some features of the present disclosure without providing others.

[0088] In other configurations, the systems and methods of the present disclosure may be implemented in conjunction with a special purpose computer, a programmable microprocessor or microcontroller and (multiple) peripheral integrated circuit elements, an ASIC or other integrated circuit, a digital signal processor, a hardwired electronic or logic circuit (such as, discrete element circuit), a programmable logic device or gate array (such as, PLD, PLA, FPGA, PAL), a special purpose computer, any comparable components, etc. In general, any device or component that can implement the methods illustrated herein may be used to implement various aspects of the present disclosure. Exemplary hardware that may be used for the present disclosure includes computers, handheld devices, phones (e.g., cellular, Internet enabled, digital, analog, hybrid, and other) and other hardware known in the art. Some of these devices include processors (e.g., single or multiple microprocessors), memory, non-volatile storage, input devices, and output devices. In addition, alternative software implementations including but not limited to distributed processing or component / object distributed processing, parallel processing, or virtual machine processing may also be constructed to implement the methods described herein.

[0089] In yet another configuration, the disclosed methods can be readily implemented in conjunction with software using objects or object-oriented software development environments that provide portable source code that can be used on a variety of computer or workstation platforms. Alternatively, the disclosed systems can be partially or completely implemented in hardware using standard logic circuits or VLSI designs. Whether software or hardware is used to implement a system according to the present disclosure depends on the speed and / or efficiency requirements of the system, the specific functionality, and the specific software or hardware systems or microprocessor or microcomputer systems used.

[0090] In yet another configuration, the disclosed methods may be implemented in part in software, which may be stored on a storage medium and executed on a programmed general purpose computer in cooperation with a controller and memory, a dedicated computer, a microprocessor, etc. In these examples, the systems and methods of the present disclosure may be implemented as a program embedded on a personal computer, such as an applet, or CGI scripts, implemented as resources residing on a server or computer workstation, implemented as routines embedded in a dedicated measurement system, system component, etc. The system and / or method can also be implemented by physically incorporating it into a software and / or hardware system.

[0091] The present disclosure is not limited to standards and protocols, if described. Other similar standards and protocols not mentioned herein exist and are included in the present disclosure. Moreover, the standards and protocols mentioned herein and other similar standards and protocols not mentioned herein are periodically replaced by faster or more efficient equivalents having substantially the same functions. Such replacement standards and protocols having the same functions are considered to be equivalents included in the present disclosure.

[0092] In various configurations and aspects, the present disclosure includes components, methods, processes, systems and / or devices substantially as depicted and described herein, including various combinations, subcombinations and subsets thereof. After understanding the present disclosure, those skilled in the art will understand how to proceed and use the systems and methods disclosed herein. In various configurations and aspects, the present disclosure includes providing devices and processes in the absence of items not depicted and / or described herein or in its various configurations or aspects, including in the absence of such items that may have been used in previous devices or processes, for example, to improve performance, achieve ease of use and / or reduce implementation costs.

[0093] The present disclosure relates to systems and methods for new view synthesis according to at least the examples provided in the following sections:

[0094] (A1) In one aspect, some examples include providing a computer-implemented method for transforming a neural radiation field model. The method includes: providing multiple inputs to a neural radiation field (NeRF) model representing a 3-dimensional space having an object, wherein each of the multiple inputs includes a location and a view direction and corresponds to a corresponding color of a voxel representing the 3-dimensional space; performing spectral analysis on multiple outputs of the NeRF model based on the multiple inputs, wherein the multiple outputs include the corresponding colors of the voxels; extracting frequency components that represent the spectral analysis of the colors of at least some of the voxels; and generating a sparse volume data structure that represents the 3-dimensional space and the corresponding colors of at least some of the voxels.

[0095] (A2) In some examples of A1, extracting the frequency component includes: extracting the diffuse color component as a fundamental frequency component; and extracting the low frequency component as a first harmonic component and a second harmonic component.

[0096] (A3) In some examples of A1 to A2, extracting the frequency component also includes training a residual neural network to represent the high-frequency component of the spectral analysis.

[0097] (A4) In some examples of A1 to A3, the residual neural network is a single-layer neural network, and the training data of the residual neural network model is obtained by subtracting the diffuse color component and the low-frequency component from the corresponding output of the NeRF model.

[0098] (A5) In some examples of A1 to A4, the method also includes storing the weights of the residual neural network model as a grayscale image.

[0099] (A6) In some examples of A1 to A5, the method further includes: dividing the 3-dimensional space into blocks of inconsistent sizes; and storing colors associated with frequency components of voxels within the corresponding blocks.

[0100] (A7) In some examples of A1 to A6, storing the colors includes omitting colors within blocks that are not adjacent to a surface of the object.

[0101] (A8) In some examples of A1 to A7, dividing the 3D space includes using larger-sized blocks for a region of the 3D space farther from the object, and using smaller-sized blocks for a region of the 3D space closer to the object.

[0102] (A9) In some examples of A1 to A8, storing the color includes storing the color within the block as a first texture and storing the relative position within the block as a second texture.

[0103] (A10) In some examples of A1 to A9, the sparse volume data structure is a tree data structure having nodes corresponding to blocks of inconsistent sizes.

[0104] (B1) In one aspect, some examples include a computer-implemented method for rendering an image of an object in a 3-dimensional space. The method includes: processing a ray from a viewpoint through pixels of an image into the 3-dimensional space, the 3-dimensional space being represented by a sparse volume data structure and a residual neural network; sampling voxels along the ray and accumulating the colors of the voxels according to the color values ​​stored in the sparse volume data structure; providing the location and perspective of the viewpoint to the residual neural network to obtain residual colors of pixels of the image; and rendering the image by combining the accumulated color and the residual color of each pixel of the image.

[0105] (B2) In some examples of (B1), the sparse volume data structure is a tree data structure having nodes corresponding to blocks of inconsistent sizes in a 3-dimensional space.

[0106] (B3) In some examples of (B1) to (B2), a larger-sized block corresponds to a region of the 3-dimensional space farther from the object, and a smaller-sized block corresponds to a region of the 3-dimensional space closer to the object.

[0107] In yet another aspect, some examples include a computing system comprising one or more processors and a memory coupled to the one or more processors, the memory storing one or more instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the methods described herein (e.g., method 500 or method 600 described above, A1 to A10 or B1 to B3 described above).

[0108] In yet another aspect, some examples include a non-volatile computer-readable storage medium storing one or more programs for execution by one or more processors of a storage device, the one or more programs including instructions for performing any of the methods described herein (e.g., method 500 or method 600 described above).

[0109] For example, various aspects of the present disclosure are described above with reference to block diagrams and / or operational diagrams of methods, systems, and computer program products according to various aspects of the present disclosure. The functions / actions mentioned in the blocks may not occur in the order shown in any flow chart. For example, in fact, two blocks shown in succession may be executed substantially concurrently, or the blocks may sometimes be executed in reverse order, depending on the functionality / actions involved.

[0110] The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure claimed in any way. The aspects, examples and details provided in this application are considered to be sufficient to convey ownership, and enable others to carry out and use the best mode of the disclosure claimed. The disclosure claimed should not be interpreted as being limited to any aspect, example or detail provided in this application. Regardless of whether combined or individually shown and described, various features (structural and methodological) are intended to be selectively included or omitted to produce examples with specific feature sets. Description and illustration of the application have been provided, and those skilled in the art can envision changes, modifications and alternating aspects within the spirit of the broader aspects of the general inventive concept implemented in this application, which do not depart from the broader scope of the disclosure claimed.

Claims

1. A computer-implemented method for transforming a neural radiation field model, the method comprising: include: providing a plurality of inputs to a neural radiance field (NeRF) model representing a 3-dimensional space having an object, wherein each of the plurality of inputs comprises a location and a view direction and corresponds to a respective color of a voxel representing the 3-dimensional space; performing spectral analysis on a plurality of outputs of the NeRF model based on the plurality of inputs, wherein the plurality of outputs include the corresponding colors of the voxels; extracting frequency components representing the spectral analysis of the colors of at least some of the voxels; as well as A sparse volume data structure is generated, the sparse volume data structure representing the 3-dimensional space and the corresponding colors for the at least some of the voxels.

2. The method according to claim 1, wherein the frequency component is extracted include: Extracting diffuse color components as fundamental frequency components; as well as The low frequency components are extracted as the first harmonic component and the second harmonic component.

3. The method according to claim 2, wherein extracting the frequency component further comprises: include: A residual neural network is trained to represent the high frequency components of the spectral analysis.

4. The method according to claim 3, wherein the residual neural network is a single-layer neural network, and the training data for the residual neural network model is obtained by subtracting the diffuse color component and the low-frequency component from the corresponding output of the NeRF model.

5. The method according to claim 3, further comprising: include: The weights of the residual neural network model are stored as a grayscale image.

6. The method according to claim 1, further comprising: include: Dividing the 3D space into blocks of inconsistent sizes; as well as Colors associated with the frequency components for voxels within the corresponding block are stored.

7. The method according to claim 6, wherein the color is stored include: Colors within blocks that are not adjacent to the surface of the object are omitted.

8. The method according to claim 6, wherein the 3-dimensional space is divided include: A larger-sized block is used for a region of the 3-dimensional space that is farther from the object, and a smaller-sized block is used for a region of the 3-dimensional space that is closer to the object.

9. The method according to claim 6, wherein the color is stored include: The colors within the blocks are stored as a first texture, and the relative positions within the blocks are stored as a second texture.

10. The method of claim 6, wherein the sparse volume data structure is a tree data structure having nodes corresponding to the inconsistently sized blocks.

11. A computer-implemented method for rendering an image of an object in 3-dimensional space, the method include: Processing rays from a viewpoint through pixels of the image into the 3-dimensional space, the 3-dimensional space being represented by a sparse volume data structure and a residual neural network; sampling voxels along the ray and accumulating a color for the voxel based on a color value stored in the sparse volume data structure; providing the location and perspective of the viewpoint to the residual neural network to obtain a residual color for the pixel of the image; and The image is rendered by combining the accumulated color and the residual color for each pixel of the image.

12. The method of claim 11, wherein the sparse volume data structure is a tree data structure having nodes corresponding to blocks of inconsistent sizes in the 3-dimensional space. 13 . The method of claim 12 , wherein a block of a larger size corresponds to a region of the 3-dimensional space farther from the object, and a block of a smaller size corresponds to a region of the 3-dimensional space closer to the object.

14. A non-transitory computer-readable storage medium comprising instructions executable by one or more processors, the instructions, when executed by the one or more processors, causing the one or more processors to: providing a plurality of inputs to a neural radiance field (NeRF) model representing a 3-dimensional space having an object, wherein each of the plurality of inputs comprises a location and a view direction and corresponds to a respective color of a voxel representing the 3-dimensional space; performing spectral analysis on a plurality of outputs of the NeRF model based on the plurality of inputs, wherein the plurality of outputs include the corresponding colors of the voxels; extracting frequency components representing the spectral analysis of the colors of at least some of the voxels; as well as A sparse volume data structure is generated, the sparse volume data structure representing the 3-dimensional space and the corresponding colors for the at least some of the voxels.

15. The computer-readable storage medium of claim 14, wherein the instructions are executable by the one or more processors to cause the one or more processors to: extracting a diffuse color component as a fundamental frequency component; and The low frequency components are extracted as the first harmonic component and the second harmonic component.

16. The computer-readable storage medium of claim 15, wherein the instructions are executable by the one or more processors to cause the one or more processors to: A residual neural network is trained to represent the high frequency components of the spectral analysis.

17. The computer-readable storage medium of claim 16, wherein the residual neural network is a single-layer neural network, and training data for the residual neural network model is obtained by subtracting the diffuse color component and the low-frequency component from corresponding outputs of the NeRF model.

18. The computer-readable storage medium of claim 14, wherein the instructions are executable by the one or more processors to cause the one or more processors to: Dividing the 3D space into blocks of inconsistent sizes; and Colors associated with the frequency components for voxels within the corresponding block are stored.

19. The computer-readable storage medium of claim 18, wherein the instructions are executable by the one or more processors to cause the one or more processors to: Colors within blocks that are not adjacent to the surface of the object are omitted.

20. The computer-readable storage medium of claim 18, wherein the instructions are executable by the one or more processors to cause the one or more processors to: Divide the 3D space include: A larger-sized block is used for a region of the 3-dimensional space that is farther from the object, and a smaller-sized block is used for a region of the 3-dimensional space that is closer to the object.

Images