Information processor and information processing method and program

Abstract

To reduce the calculation cost in a technique of acquiring a feature quantity by performing the ray tracing on a grid.SOLUTION: An information processor that performs the ray tracing on a grid composed of multiple cells to acquire a feature quantity at a sampling position along a ray comprises an operation unit that performs the interpolation calculation at the sampling position of a target cell subject to feature quantity interpolation calculation using the respective feature quantities at multiple vertices of the target cell and writes the interpolated feature quantity to a buffer.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a technique for acquiring feature quantities by performing ray tracing on a grid. [Background technology]

[0002] Neural Radiance Fields (NeRF) is a well-known technique for generating images from new viewpoints from multiple image data captured from various angles.

[0003] Also, rendering methods that can perform processing faster than NeRF include Plenoxels and DVGO. Rendering methods such as Plenoxels and DVGO store features in a grid and perform ray tracing to obtain the features at each sampling position using trilinear interpolation. This method is called a grid-based method. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2000-207576 Summary of the Invention [Problem to be solved by the invention]

[0005] Conventional grid-based methods have the problem of high computational costs for memory access and interpolation to obtain features.

[0006] The present invention has been made in view of the above points, and aims to reduce the calculation cost in a technique for acquiring feature quantities by performing ray tracing on a grid. [Means for solving the problem]

[0007] According to the disclosed technology, there is provided an information processing apparatus that performs ray tracing on a grid consisting of a plurality of cells to acquire feature quantities at sampling positions along a ray, the information processing apparatus comprising: a calculation unit that performs an interpolation calculation using the feature values ​​at each of a plurality of vertices of a target cell at a sampling position of the target cell, which is a cell for which an interpolation calculation of a feature value is performed, and executes a process of writing the interpolated feature value into a buffer; An information processing device comprising: [Effects of the Invention]

[0008] According to the disclosed technology, it is possible to reduce the calculation cost in a technology for acquiring feature quantities by performing ray tracing on a grid. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an overview of processing in NeRF. [Figure 2] 10 is a flowchart illustrating a process for determining sampling positions in a conventional method. [Figure 3] FIG. 1 shows two adjacent cells. [Figure 4] FIG. 1 is a configuration diagram of an information processing device 100. [Figure 5] 1 is a flowchart showing an example of operation of the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating a processing example 1 of a process for saving a feature amount to a register. [Figure 7] FIG. 10 is a diagram illustrating a processing example 1 of a process for saving a feature amount to a register. [Figure 8] FIG. 10 is a diagram illustrating a second example of a process for saving a feature amount to a register. [Figure 9] FIG. 10 is a diagram illustrating a second example of a process for saving a feature amount to a register. [Figure 10] 10 is a flowchart showing an example of operation of the second embodiment. [Figure 11]FIG. 10 is a diagram illustrating an image of layering processing. [Figure 12] 1 is a diagram illustrating an example of a hardware configuration of an information processing device 100. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0011] In the following, the conventional technology and its problems will be described in more detail first, and then the technology according to the present embodiment will be described. Note that while the conventional technology itself is publicly known, the analysis of the conventional technology and the explanation of the problems are not publicly known.

[0012] (Regarding the prior art) As mentioned above, NeRF is a well-known technology for generating images from new viewpoints from multiple image data captured from various angles. Figure 1 shows an overview of the processing in NeRF.

[0013] NeRF first learns a Radiance Field that takes three-dimensional coordinates ((x, y, z) in Figure 1) and viewing direction ((θ, φ) in Figure 1) as input and outputs color and intensity. When rendering to generate an image from a new viewpoint, the color (r, g, b) and intensity (σ) for each point sampled by ray tracing are obtained from the Radiance Field, and these are accumulated to determine the color (r, g, b) of the pixel corresponding to the ray.

[0014] NeRF has the problem of taking a long time to train and render. Therefore, grid-based methods that represent radiance fields as 3D feature grids based on NeRF have been widely studied as fast rendering methods. Examples of such fast rendering methods include Plenoxels and DVGO.

[0015] Plenoxels stores spherical harmonics in a grid, interpolates the coefficients of the spherical harmonics at the queried 3D coordinates using trilinear interpolation, and then determines the color corresponding to the viewing direction.

[0016] In DVGO and other methods, features are stored in a grid, and the features at the queried 3D coordinates are interpolated using trilinear interpolation. The interpolated values ​​are then input into an MLP (Multilayer Perceptron) along with the gaze direction to determine color and intensity.

[0017] As described above, in many grid-based methods, the features of each point sampled by ray tracing are collected from a 3D grid using trilinear interpolation, and the color and density are calculated by processing each feature independently (such as calculating spherical harmonics or MLP), and the pixel color is determined by accumulating these.

[0018] (About the assignment) In the trilinear-interpolation of the grid-based method, it is necessary to access eight entries in memory per sample to obtain feature values ​​and perform interpolation calculations. Therefore, the grid-based method has the problem of high memory access costs for obtaining feature values ​​and the calculation costs for the interpolation.

[0019] In addition, grids of multiple resolutions are often used simultaneously to improve expressiveness, which also leads to an increase in computational costs. Specifically, power-of-two resolutions are often used as multiple resolutions (e.g., preparing 64x64x64, 32x32x32, and 16x16x16 grids and concatenating the features obtained from each). Conventional methods perform processing independently for each resolution grid, so the computational cost increases in proportion to the number of grids.

[0020] In addition to the above, some grid-based methods may be designed so that the learned 3D feature grid is sparse. In this case, they exploit this sparsity when determining sampling locations, and save memory and computation time by avoiding sampling in vacuum areas.

[0021] More specifically, a technique called Occupancy Grid reduces computational costs by pre-memorizing cells with a density above a threshold and sampling only from those cells. This technique first enumerates the cells that each ray passes through within the grid, leaving only those with a density above a threshold, and then performs sampling within those cells. Related to this cell enumeration, methods such as DDA (Digital Differential Analyzer) and HDDA (Hierarchical Digital Differential Analyzer) are known to efficiently enumerate voxels that are crossed by a line drawn on the grid.

[0022] Unlike volume rendering, NeRF-based methods require feature values ​​corresponding to the sampling position to be stored in a buffer. In other words, volume rendering in the CG field obtains color and density directly from the Radiance Field, rather than feature values, so color can be cumulatively calculated while sampling. In contrast, NeRF-based methods require the obtained feature values ​​to be first processed using MLP or similar. Therefore, in order to reduce computational overhead through batching, it is necessary to temporarily store feature values ​​at each position in a buffer.

[0023] Regarding the storage of the above features, conventional NeRF-based methods first enumerate the sampling positions, then perform batch processing to obtain the features (trilinear interpolation), and store the obtained features in a buffer. Since the process of enumerating the sampling positions is included, the computational cost is high.

[0024] The process of determining sampling positions in the conventional method will be described with reference to the flowchart in Fig. 2. Here, the device that performs the process will be referred to as "information processing device 100." The configuration of information processing device 100 that performs the process according to this embodiment will be described later.

[0025] A "grid" is a set of one or more cells (which may also be called voxels). A "cell" is a cube with features at its eight vertices. A "test cell" is the cell under consideration. A "test cell" may also be called a target cell.

[0026] The grid is G, and the initial sampling position is P. In S1 (step 1), the information processing device 100 determines the next test cell using DDA. In S2, the information processing device 100 determines whether the test cell is inside G. If Yes, the process proceeds to S3, and if No, the process ends.

[0027] In S3, the information processing apparatus 100 determines whether the test cell can be skipped, and if Yes, the process returns to S1, and if No, the process proceeds to S4.

[0028] In S4, the information processing device 100 updates the sampling position P. In S5, the information processing device determines whether the ray (sampling coordinate P) is inside the test cell, and if Yes, proceeds to S6, and if No, returns to S1.

[0029] In S6, the information processing apparatus 100 writes the coordinate P into the output buffer, and then returns to S4.

[0030] Through the above process, the information processing apparatus 100 repeats the process of recording the coordinates while advancing little by little along the ray, and then, for a cell to be skipped, using the DDA to advance in one go to the coordinates of the cell with which the ray will next collide.

[0031] In the above process, the feature values ​​corresponding to the coordinates sampled at nearby positions are obtained by performing trilinear interpolation using entries obtained from the same cell in subsequent processing. This results in multiple array accesses to the same index, which increases the computational cost.

[0032] (Outline of the embodiment) In order to solve the above-mentioned problem, in this embodiment, the information processing device 100 stores the interpolated feature directly in the output buffer without going through enumeration of the sample positions.

[0033] In the flow of Fig. 2 described above, feature quantities corresponding to coordinates falling into the same test cell are calculated by trilinear interpolation from the same vertex. Therefore, the information processing device 100 in this embodiment stores feature quantities necessary for interpolation in a register, which has faster access speed than memory, in the loop that performs "updating and recording coordinate P inside the test cell" in the flow of Fig. 2, and calculates the interpolated feature quantities at this point.

[0034] As shown in Figure 3, adjacent cells share four vertices. Therefore, when performing trilinear interpolation in ray tracing across the two cells, four of the eight feature values ​​are common, so memory access can be omitted. The white circles in Figure 3 are the newly accessed feature values. This makes it possible to halve memory accesses while searching for valid adjacent cells. The specific processing method will be described later.

[0035] (DDA processing example) Here, we will explain an example of the DDA processing executed by the information processing device 100. As an example, it is assumed that the information processing device 100 scans a cell (0≦x≦1 and 0≦y≦1 and 0≦z≦1) with a ray (vx, vy, vz are all positive vectors v). The vector v corresponds to the velocity of the ray.

[0036] When the current sample point coordinates are (x,y,z) and are inside a cell, one of the following three conditions is true when this ray travels from the sample point and exits the cell:

[0037] The condition [0≦x≦1] is broken: the ray advances for the time (1-x) / vx, and x>1.

[0038] The condition [0≦y≦1] is broken: the ray advances by the time (1-y) / vy, and y>1.

[0039] The condition [0≦z≦1] is broken: the ray moves forward for a time (1-z) / vz, and z>1.

[0040] The one that actually holds is the one that occurred the earliest out of the three, so the ray is min((1-x) / vx,(1-y) / vy,(1-z) / vz) The cell exits when the

[0041] Furthermore, if the ray leaves a cell when x>1, it will next enter the cell next to it in the x-axis direction (1≦x≦2 and 0≦y≦1 and 0≦z≦1). The timing at which the ray leaves that cell is min(1 / vx,(1-y) / vy-(1-x) / vx,(1-z) / vz-(1-x) / vx) This is when Ray moves forward for that amount of time.

[0042] By repeatedly updating while remembering the "time it takes for the ray to cross the boundary of each axis," it is possible to enumerate the cells that the ray passes through in order.

[0043] (Device configuration example) FIG. 4 shows an example configuration of an information processing device 100 according to this embodiment. The information processing device 100 executes a process of acquiring feature amounts at sampling positions along a ray by performing ray tracing on a grid consisting of a plurality of cells. The information processing device 100 may perform rendering processing using the acquired feature amounts. Furthermore, the information processing device 100 may output the acquired feature amounts to an external device. In this case, for example, a function or device external to the information processing device 100 executes rendering processing using the feature amounts. Furthermore, the feature amounts interpolated and calculated by the information processing device 100 may be used for processing other than rendering.

[0044] As shown in FIG. 4, the information processing device 100 includes an input unit 110, a calculation unit 120, and an output unit .

[0045] For example, grid information and initial sampling positions are input from the input unit 110. The calculation unit 120 includes a memory and a register, and executes the processes of the first and second embodiments described below. The output unit 130 may output feature amounts obtained by performing interpolation processing, or may output rendering results.

[0046] Hereinafter, a first embodiment and a second embodiment will be described as specific examples of the operation of the information processing device 100.

[0047] Example 1 The operation of the information processing device 100 in the first embodiment will be described with reference to the flowchart of FIG.

[0048] The grid is designated as G, and the sampling position is designated as P. In S101, the information processing apparatus 100 determines the next test cell using the DDA. Note that the test cell that is the target of the feature interpolation calculation may be called a target cell.

[0049] In S102, the information processing apparatus 100 determines whether the test cell is inside G. If Yes, the process proceeds to S103, and if No, the process ends.

[0050] In S103, the information processing apparatus 100 determines whether a test cell can be skipped. If Yes, it returns to S101; if No, it proceeds to S104.

[0051] In S104, the information processing apparatus 100 calculates newly required vertex coordinates. Here, the newly required number of vertices is either 4 or 8. If the skipping of the test cell did not occur immediately before (i.e., if the current cell is adjacent to the previous search cell), the number of new vertices is 4.

[0052] In S105, the information processing apparatus 100 acquires the feature amounts of new vertices from the memory.

[0053] In S106, the information processing apparatus 100 stores the feature amounts necessary for trilinear interpolation in registers R[0]…R[7].

[0054] In S107, the information processing apparatus 100 updates the sampling position P. In S108, the information processing apparatus determines whether the ray (coordinate P) is inside the test cell. If Yes, it proceeds to S109; if No, it proceeds to S101.

[0055] In S109, the information processing apparatus 100 calculates the feature amount at P by trilinear interpolation from R[0]…R[7]. In S110, the information processing apparatus 100 writes the interpolated feature amount to the output buffer.

[0056] <Details of S106> Here, as processing examples 1 and 2 for storing the feature amounts in the registers in S106, the processing of storing the four newly required vertices in the registers when the ray moves from a certain cell to an adjacent cell, as shown in FIG. 3, will be described.

[0057] <Processing Example 1> In processing example 1, for example, if a ray advances from a cell that is not adjacent to cell 1 shown in Fig. 6(a) to cell 1 shown in Fig. 6(a), the information processing device 100 loads the feature amounts of each of the eight vertices from memory and stores them in the registers corresponding to the vertices, as shown in the upper part of Fig. 7. The information processing device 100 calculates the interpolation result from the eight feature amounts stored in the registers and the coordinates of the sample points.

[0058] The information processing apparatus 100 determines the register corresponding to each of the eight vertices based on the relative position of the vertex with respect to the direction of the ray.

[0059] For example, as shown in FIG. 6(b), when the sampling coordinates move from cell 1 to its neighboring cell 2 on the right, the features P5 to P8 of the shared vertices on the right side of cell 1 are moved to the left side of the next cell, cell 2. Therefore, the information processing device 100 moves P5 to P8 that were in the register to another register, as shown in the lower part of FIG. 7. After this movement is complete, new data P9 to P12 are loaded from memory into the register.

[0060] In the processing example 1, new data P9 to P12 cannot be loaded from memory to the registers until the feature amounts have been transferred between the registers, which takes time.

[0061] <Processing example 2> Next, processing example 2 will be described. Processing example 2 is a processing example in which processing is more efficient and processing time is reduced than processing example 1. In processing example 2, when the sampling coordinates move from cell 1 to its adjacent cell 2 on the right, the information processing device 100 reflects the direction of the ray on the shared surface, as shown in FIG. 8(b). This makes it possible to change the storage destination of P9 to P12, and as shown in FIG. 9, it is possible to eliminate the need to move the data of P5 to P8 to the register.

[0062] In the processing example 2, since there is no register movement, memory loading can be completed quickly. The trilinear interpolation process can be the same as in the processing example 1.

[0063] Example 2 Next, a description will be given of Example 2. Example 2 applies the processing described in Example 1 to a hierarchical grid. Example 2 targets a grid with multiple resolutions and a width that is a power of two.

[0064] More specifically, in the second embodiment, a radiance field is targeted in which a grid with a resolution of 2P×2P×2P overlaps with a grid with a resolution of P×P×P. The information processing device 100 samples features along a ray passing through the radiance field and stores the features in a buffer.

[0065] The grid with the higher hierarchy is called G 2 Let's say. G 2 The cell in has twice the size of the cell used in Example 1 and contains 2x2x2 cells. When all the skip flags of the internal cells are true, 2 The flag for this cell in is also set to true, allowing ray calculations to be omitted at larger widths.

[0066] The operation of the information processing device 100 in the second embodiment will be described with reference to the flowchart of FIG.

[0067] Grid G 2 and the initial sampling position is P. In S201, the information processing apparatus 100 determines the next test cell using DDA. In S202, the information processing apparatus 100 determines whether the test cell is G 2 If it is Yes, the process proceeds to S203, and if it is No, the process ends.

[0068] In S203, the information processing apparatus 100 determines whether the test cell can be skipped, and if Yes, the process returns to S201, and if No, the process proceeds to S204.

[0069] In S204, the information processing apparatus 100 calculates the coordinates of newly required vertices. Here, the number of newly required vertices is either 4 or 8. If a test cell has not been skipped immediately before (if the current cell is adjacent to the previous search cell), the new number of vertices will be 4.

[0070] In S205, the information processing device 100 acquires the feature quantities of the new vertices from the memory. In S206, the information processing device 100 stores the feature quantities necessary for trilinear interpolation in registers R2[0]...R2[7]. As the storage method here, processing example 1 or processing example 2 described in the first embodiment can be used.

[0071] In S207, the information processing device 100 limits the target grid to the inside of the test cell and starts the processing of the first embodiment (FIG. 5). In the processing of the first embodiment, the information processing device 100 performs trilinear interpolation calculation, and when writing the obtained feature values ​​to the output buffer, the feature values ​​stored in register R2 are also used in the interpolation calculation.

[0072] An image of the layering process is shown in Figure 11. In the example in Figure 11, the black circles indicate that cells have been skipped, and feature interpolation is performed within the small cell indicated by A.

[0073] (Example of hardware configuration) The information processing device 100 described in this embodiment can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0074] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0075] Fig. 12 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 12 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., which are all interconnected by a bus B. The computer may further include a GPU. The memory device 1003 corresponds to the memory described above. A register exists in the CPU 1004. A buffer is, for example, a data storage area in the memory device 1003.

[0076] A program for realizing processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0077] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes the functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0078] (Effects of the embodiment) As described above, the technology described in this embodiment makes it possible to reduce the calculation cost in the technology for acquiring feature amounts by performing ray tracing on a grid.

[0079] The following additional notes are provided regarding the above-described embodiments.

[0080] <Additional Notes> (Additional note 1) An information processing apparatus that performs ray tracing on a grid consisting of a plurality of cells to acquire feature quantities at sampling positions along a ray, comprising: a calculation unit that performs an interpolation calculation using the feature values ​​at each of a plurality of vertices of a target cell at a sampling position of the target cell, which is a cell for which an interpolation calculation of a feature value is performed, and executes a process of writing the interpolated feature value into a buffer; An information processing device comprising: (Additional note 2) The calculation unit determines the target cell using a DDA (Digital Differential Analyzer). Item 1. An information processing device according to item 1. (Additional note 3) Each cell has eight vertices, and if the calculation unit skips the cell immediately before reaching the target cell, it acquires the feature values ​​of the eight vertices from memory, and if the calculation unit does not skip the cell immediately before reaching the target cell, it acquires the feature values ​​of the four vertices from memory. Item 1. An information processing device according to item 1. (Additional note 4) When the target cell moves from a first cell to a second cell adjacent to the first cell, the calculation unit changes the direction of the ray to a direction in which the ray is reflected by a shared surface between the first cell and the second cell, and uses the feature amounts of the four vertices stored in the register when the first cell was the target cell for interpolation calculation in the second cell without moving them. Item 3. An information processing device according to item 3. (Additional note 5) The calculation unit executes the process on a hierarchical grid. Item 1. An information processing device according to item 1. (Additional note 6) 1. An information processing method executed by an information processing device that acquires feature quantities at sampling positions along a ray by performing ray tracing on a grid consisting of a plurality of cells, the method comprising: At a sampling position of a target cell, which is a cell for which interpolation calculation of a feature is to be performed, an interpolation calculation is performed using the feature values ​​at each of a plurality of vertices of the target cell, and the interpolated feature values ​​are written to a buffer. Information processing methods. (Additional note 7) A non-transitory storage medium storing a program for causing a computer to function as a calculation unit in the information processing device according to any one of claims 1 to 5.

[0081] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0082] 100 Information processing device 110 Input section 120 Arithmetic section 130 Output section 1000 Drive Device 1001 Recording media 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output Device

Claims

1. An information processing apparatus that performs ray tracing on a grid consisting of a plurality of cells to acquire feature quantities at sampling positions along a ray, comprising: a calculation unit that performs an interpolation calculation using the feature values ​​at each of a plurality of vertices of a target cell at a sampling position of the target cell, which is a cell for which an interpolation calculation of a feature value is performed, and executes a process of writing the interpolated feature value into a buffer; An information processing device comprising:

2. The calculation unit determines the target cell using a DDA (Digital Differential Analyzer). The information processing device according to claim 1 .

3. Each cell has eight vertices, and if the calculation unit skips the cell immediately before reaching the target cell, it acquires the feature values ​​of the eight vertices from memory, and if the calculation unit does not skip the cell immediately before reaching the target cell, it acquires the feature values ​​of the four vertices from memory. The information processing device according to claim 1 .

4. When the target cell moves from a first cell to a second cell adjacent to the first cell, the calculation unit changes the direction of the ray to a direction in which the ray is reflected by a shared surface between the first cell and the second cell, and uses the feature amounts of the four vertices stored in the register when the first cell was the target cell for interpolation calculation in the second cell without moving them. The information processing device according to claim 3 .

5. The calculation unit executes the process on a hierarchical grid. The information processing device according to claim 1 .

6. 1. An information processing method executed by an information processing device that acquires feature quantities at sampling positions along a ray by performing ray tracing on a grid consisting of a plurality of cells, the method comprising: At a sampling position of a target cell, which is a cell for which interpolation calculation of a feature is to be performed, an interpolation calculation is performed using the feature values ​​at each of a plurality of vertices of the target cell, and the interpolated feature values ​​are written to a buffer. Information processing methods.

7. A program for causing a computer to function as the calculation unit in the information processing device according to any one of claims 1 to 5.

Images