Information processing device and method, and proxy voxel grid

Abstract

The present disclosure pertains to an information processing device and method and to a proxy voxel grid, with which it is possible to suppress an increase in an amount of proxy data of a neural radiance field. The proxy voxel grid is applied as proxy data for editing a neural radiance field, which is a neural network approximating a radiance field function expressing a scene of a space including an object, the proxy voxel grid comprising: occupied cell index information composed of indexes of all occupied cells; and UVW coordinate information composed of UVW coordinates indicating initial values of coordinates of vertices of all occupied cells. The proxy voxel grid thereof is generated. The proxy voxel grid thereof is edited. The proxy voxel grid thereof is encoded. A bit stream is decoded, and the proxy voxel grid thereof is generated. Rendering is then carried out using the proxy voxel grid thereof. The present disclosure is applicable to, for example, an information processing device, an electronic apparatus, an information processing method, a program, proxy data, and the like.

Description

Information processing device and method, and proxy voxel grid

[0001] The present disclosure relates to an information processing device and method, and a proxy voxel grid, and more particularly to an information processing device and method, and a proxy voxel grid that are capable of suppressing an increase in the amount of proxy data of a neural luminance field.

[0002] Previously, there was a neural radiance field (NeRF) approach that used a neural network to approximate a luminance field function that models complex 3D scenes in which the color of an object's surface changes depending on the position and viewing direction due to reflection phenomena, etc. Because NeRF is an implicit function representation, the scene information (information on the luminance field function) is stored in a black box-like manner as a parameter of the neural network. As a result, editing operations such as deleting or moving objects in the scene cannot be performed as is.

[0003] NeRFShop is a method for editing this neural luminance field (see, for example, Non-Patent Document 1). NeRFShop uses proxy data for editing to execute NeRF editing. In the case of NeRFShop, a tetrahedral mesh is applied as the proxy data. By moving or deleting the vertices of this tetrahedral mesh, a rendering image that reflects the editing can be obtained. In other words, the area corresponding to the moved or deleted vertex is moved or deleted in the rendering image.

[0004] CLEMENT JAMBON, BERNHARD KERBL, GEORGIOS KOPANAS, STAVROS DIOLATZIS, THOMAS LEIMKUHLER, GEORGE DRETTAKIS, "NeRFshop: Interactive Editing of Neural Radiance Fields", Proceedings of the ACM on Computer Graphics and Interactive Techniques, Volume 6, Number 1, https: / / www-sop.inria.fr / reves / Basilic / 2023 / JKKDLD23 / nerfshop.pdf, May 2023

[0005] However, in order to perform more accurate editing operations, it was necessary to make the proxy data more precise. In other words, a tetrahedral mesh with higher resolution was required. Increasing the resolution of the tetrahedral mesh increases the number of tetrahedrons in the proxy data. In general, the data volume of a tetrahedral mesh is roughly proportional to the number of tetrahedrons. In other words, if a tetrahedral mesh with a sufficiently high resolution for editing operations were used as proxy data, there was a risk that the data volume would increase.

[0006] The present disclosure has been made in view of such circumstances, and makes it possible to suppress an increase in the amount of proxy data for neural luminance fields.

[0007] An information processing device according to one aspect of the present technology is an information processing device that includes a proxy voxel grid generation unit that generates a proxy voxel grid using a voxel grid as proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object.

[0008] An information processing method according to one aspect of the present technology is an information processing method for generating a proxy voxel grid using a voxel grid as proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object.

[0009] An information processing device according to another aspect of the present technology includes a rendering unit that performs rendering using a proxy voxel grid, which is proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object, and the proxy voxel grid is the proxy data using the voxel grid.

[0010] Another aspect of the present technology is an information processing method that performs rendering using a proxy voxel grid, which is proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object, and the proxy voxel grid is the proxy data using a voxel grid.

[0011] A proxy voxel grid according to yet another aspect of the present technology is proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object, and has occupied cell index information composed of indices of all occupied cells, and UVW coordinate information composed of UVW coordinates that indicate initial values ​​of coordinates of vertices of all the occupied cells, and the occupied cells are proxy voxel grids that are voxels that make up the voxel grid and have geometry of the object present inside.

[0012] In an information processing device and method according to one aspect of the present technology, a proxy voxel grid is generated using a voxel grid as proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object.

[0013] In an information processing device and method according to another aspect of the present technology, rendering is performed using a proxy voxel grid, which is proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object.

[0014] In yet another aspect of the present technology, a proxy voxel grid is proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object, and includes occupied cell index information composed of indices of all occupied cells, and UVW coordinate information composed of UVW coordinates that indicate initial values ​​of the coordinates of the vertices of all occupied cells.

[0015] 1 is a diagram for explaining a neural luminance field; FIG. 1 is a diagram for explaining NeRFShop; FIG. 1 is a diagram for explaining rendering after editing; FIG. 1 is a diagram for explaining example proxy data of a neural luminance field; FIG. 1 is a diagram for explaining a proxy voxel grid; FIG. 1 is a diagram for explaining a proxy voxel grid; FIG. 1 is a diagram for explaining a proxy voxel grid; A block diagram showing an example of the main configuration of an information processing system; A block diagram showing an example of the main configuration of a proxy voxel grid encoding unit; A block diagram showing an example of the main configuration of a UVW coordinate encoding unit; A block diagram showing an example of the main configuration of a proxy voxel grid decoding unit; A block diagram showing an example of the main configuration of a UVW coordinate decoding unit; A flowchart showing an example of the flow of a neural luminance field editing process; A flowchart showing an example of the flow of a proxy voxel grid encoding process; A flowchart showing an example of the flow of a UVW coordinate encoding process; A flowchart showing an example of the flow of a proxy voxel grid decoding process; A flowchart showing an example of the flow of a UVW coordinate decoding process; A flowchart showing an example of the flow of a rendering process; A block diagram showing an example of the main configuration of an information processing system; FIG. 1 is a flowchart illustrating an example of the flow of a neural luminance field editing process. FIG. 2 is a block diagram illustrating an example of the main configuration of a proxy voxel grid encoding unit. FIG. 3 is a block diagram illustrating an example of the main configuration of a proxy voxel grid decoding unit. FIG. 4 is a flowchart illustrating an example of the flow of a proxy voxel grid generation process. FIG. 5 is a flowchart illustrating an example of the flow of a proxy voxel grid editing process. FIG. 6 is a flowchart illustrating an example of the flow of a proxy voxel grid encoding process. FIG. 7 is a flowchart illustrating an example of the flow of a proxy voxel grid decoding process. FIG. 8 is a block diagram illustrating an example of the main configuration of a computer.

[0016] Below, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. Literature etc. supporting technical content and technical terminology 2. Proxy data for neural luminance field 3. Application of voxel grid 4. First embodiment (information processing system) 5. Second embodiment (information processing system) 6. Third embodiment (application of attributes) 7. Supplementary notes

[0017] <1. Literature, etc. supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the content described in the embodiments, but also the content described in the following non-patent documents, etc. that were publicly known at the time of filing, and the content of other documents referenced in the following non-patent documents.

[0018] Non-Patent Document 1: (mentioned above) Non-Patent Document 2: S. Gumhold, S. Guthe and W. Strasser, "Tetrahedral mesh compression with the cut-border machine", Proceedings Visualization '99 (Cat. No.99CB37067), San Francisco, CA, USA, 1999, pp. 51-509, doi: 10.1109 / VISUAL.1999.809868.https: / / ieeexplore.ieee.org / document / 809868, 22 March 2018

[0019] In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.

[0020] <2. Neural Radiance Field Surrogate Data> <Neural Radiance Field> A function F: (x, y, z, θ, φ) → (r, g, b, σ) that outputs density and color (r, g, b, σ) when 3D coordinates (x, y, z) and gaze direction (θ, φ) are input is called a radiance field function or luminance field. However, density σ does not depend on gaze direction (θ, φ) but is determined by coordinates (x, y, z). This is because geometry depends only on position, not gaze direction. Complex 3D scenes in which the color of an object surface changes depending on position and gaze direction due to reflection phenomena, etc., can be modeled as a radiance field function.

[0021] Because the radiance field function F is a high-dimensional function, it may be approximated using a function Fθ with some compact parameter θ. Neural radiance field (NeRF) is a method for approximating the radiance field function F using a neural network Fθ, where θ is a parameter of the neural network. As shown in Figure 1A, the neural network Fθ inputs three-dimensional coordinates and a gaze direction vector (x, y, z, θ, φ) and outputs a density and color vector (r, g, b, σ).

[0022] <Volume Rendering> To render NeRF (or a general luminance field) as an image, a rendering algorithm called volume rendering is used. The process for calculating the color of a pixel in a rendered image using volume rendering is outlined below. 1: As shown in Figure 1B, a ray is generated from the viewpoint toward the pixel. 2: Points on the ray are sampled. One sampling method is to select points at equal intervals on the ray. In Figure 1B, the black dots on the ray indicate sample points. 3: The three-dimensional coordinates (x, y, z) of each sample point and the viewing direction (θ, φ) of the ray are input into the neural network Fθ, which derives the density and color (r, g, b, σ) of the sample point. 4: The weighted sum of the colors of all sample points on the ray determines the color of the final pixel (the white dot on the ray in Figure 1B). The weight of the sample point is the visibility factor, which indicates the degree to which the sample point is visible from the viewpoint. For example, if there is an obstruction between the two points, the visibility of that sample point will be close to 0. The visibility is calculated by integrating the nonlinearly transformed density.

[0023] <NeRFShop> Because NeRF is an implicit representation, scene information (luminance field information) is stored in a black box as neural network parameters. In other words, it is not clear how a certain region in the scene is related to which parameter in the neural network. As a result, it is not possible to perform editing operations such as deleting or moving objects in the scene as is.

[0024] Therefore, as described in Non-Patent Document 1, for example, a method (NeRFShop) for editing NeRF using proxy data for editing has been devised. In NeRFShop, a tetrahedral mesh (also called a proxy tetrahedral mesh) is used as the proxy data. By moving or deleting vertices of the proxy tetrahedral mesh, the rendered image changes to reflect the editing. For example, deleting some vertices of the proxy tetrahedral mesh deletes the corresponding area in the rendered image.

[0025] A tetrahedral mesh is a collection of tetrahedrons. As shown in Figure 2A, each tetrahedron consists of four vertices and connectivity information between them. In a proxy tetrahedral mesh, XYZ and UVW coordinates are assigned to each vertex as attributes. The XYZ coordinates represent the vertex position (x, y, z) in world space. The XYZ coordinates change when a vertex is moved due to editing, etc. The UVW coordinates represent the vertex position (u, v, w) before editing. This vertex position before editing is also called the initial vertex position. In other words, in the initial state (before editing), (x, y, z) = (u, v, w) for all vertices. The UVW coordinates do not change even if the XYZ coordinates are moved due to editing. The connectivity information represents the indices (i1, i2, i3, i4) of the four vertices of the tetrahedron.

[0026] A proxy tetrahedral mesh, as shown in Fig. 2B, uses a collection of such tetrahedrons to represent objects in a scene, and can be deformed, added, or deleted according to editing operations. For example, if the number of vertices is V and the number of tetrahedrons is T, the tetrahedral mesh is composed of V XYZ coordinates, V UVW coordinates, and T connectivity information.

[0027] <Rendering of edited results> When rendering the edited results, part of the volume rendering method described above is modified to obtain a rendered image that reflects the editing operations on the proxy tetrahedral mesh. Specifically, the process of calculating density and color from the coordinates of the sample point and the line of sight of the ray is modified as follows: [Before modification] The neural network Fθ inputs the XYZ coordinates (x, y, z) of the sample point and the line of sight of the ray (θ, φ). [After modification] The neural network Fθ inputs the UVW coordinates (u, v, w) of the sample point and the line of sight after rotation (θ', φ').

[0028] The UVW coordinates (u, v, w) of the sample point and the viewing direction after rotation (θ', φ') are derived as follows: 1: From the coordinates (x, y, z) of the sample point, search for a tetrahedron that contains the sample point. If not found, terminate. 2: Complement the UVW coordinates of the four vertices of the tetrahedron to derive the UVW coordinates (u, v, w) of the sample point. 3: Derive the rotation matrix within the tetrahedron to derive the viewing direction after rotation (θ', φ').

[0029] That is, as shown in Figure 3, density and color are derived by inputting the sample point positions and viewing direction for the initial proxy tetrahedral mesh, which correspond to the sample point positions and viewing direction for the proxy tetrahedral mesh after editing, into the neural network Fθ. In Figure 3, the left side shows the object in the XYZ coordinate space, and the right side shows the object in the UVW coordinate space. By using UVW coordinates instead of XYZ coordinates during rendering, rendering results that reflect the editing can be obtained. Note that in the initial state, (u, v, w, θ', φ') = (x, y, z, θ, φ), so a normal rendering image is obtained.

[0030] Editing operations on a tetrahedral mesh include, for example, deleting, duplicating, and moving a tetrahedron. When these editing operations are performed on a tetrahedron by performing rendering as described above, the editing operations are reflected in the rendered image. For example, when a tetrahedron is deleted, the area included in the tetrahedron is deleted in the rendered image. When a tetrahedron is duplicated, the area included in the tetrahedron is duplicated in the rendered image. When a tetrahedron is moved, the area included in the tetrahedron moves in the rendered image.

[0031] <Data volume of proxy tetrahedral mesh> In recent years, there has been a demand for more precise editing operations. However, in order to perform more precise editing operations, it is necessary to reduce the editable unit. In other words, it has been necessary to make the proxy data more precise. For example, when a proxy tetrahedral mesh is used as proxy data, as in NeRFShop, the editable unit corresponds to one tetrahedron. In other words, in order to perform more precise editing operations, it has been necessary to increase the number of tetrahedrons in the proxy tetrahedral mesh.

[0032] Generally, the amount of data for a tetrahedral mesh includes portions proportional to the number of vertices and tetrahedrons, as follows: Data amount for a tetrahedral mesh = (number of bits for XYZ coordinates) x 3 x (number of vertices) + (number of bits for UVW coordinates) x 3 x (number of vertices) + (number of bits for vertex index) x 4 x (number of tetrahedrons).

[0033] Therefore, when a proxy tetrahedral mesh with a sufficiently high resolution for editing operations is used as proxy data, the amount of proxy data may increase as the editing operations become more precise. The increased amount of proxy data may increase the processing load for generating, editing, encoding, decoding, rendering, and the like of the proxy data. Furthermore, when transmitting proxy data between a server and a client, for example, the amount of data transmitted may increase, and the transmission-related load may also increase. These increased loads may result in increased costs and processing times.

[0034] Another method for encoding tetrahedral meshes is described in, for example, Non-Patent Document 2. In this method, all tetrahedrons in the target tetrahedral mesh are searched, and all connection information, vertex coordinates, and vertex attributes are sequentially encoded. Vertex coordinates are encoded using relatively simple differential encoding, such as encoding the difference from the center of gravity of a nearby triangle. Vertex attributes are encoded using differential encoding with a linearly interpolated value of four vertex attribute values ​​in a nearby tetrahedron.

[0035] Therefore, it was necessary to encode connection information in addition to the vertex coordinates. Furthermore, when encoding vertex coordinates, the vertices to be encoded generally do not exist on the selected triangle, so the difference to be encoded does not become zero. Furthermore, when encoding vertex attributes, prediction is performed using only four vertex attribute values, which can lead to numerical instability and large difference values ​​to be encoded in the case of noisy or distorted tetrahedrons. Therefore, it was difficult to achieve sufficiently high encoding efficiency when encoding a proxy tetrahedral mesh. In other words, it was difficult to suppress the increase in data volume even when encoding a proxy tetrahedral mesh.

[0036] 3. Application of Voxel Grid Method 1 Therefore, as shown in the top row of the table in FIG. 4, a voxel grid is used as proxy data for editing the neural luminance field (Method 1).

[0037] For example, a proxy voxel grid is applied as proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object. The proxy data includes occupied cell index information composed of the indexes of all occupied cells and UVW coordinate information composed of UVW coordinates that indicate the initial values ​​of the vertices of all occupied cells. Note that an occupied cell is a voxel that contains the geometry of an object among the voxels that make up the voxel grid.

[0038] A voxel grid is a data structure in which voxels (cubic cells) are arranged in a three-dimensional grid. If the resolution of the voxel grid is N^3 (e.g., 512^3), each voxel corresponds one-to-one to an integer index (i, j, k) (i, j, k = 0, ..., N-1). In the case of a proxy voxel grid, the voxel corresponding to the maximum resolution of this voxel grid (i.e., the smallest voxel) is the editable unit. Note that in this specification, superscripts such as exponents are indicated by "^". For example, in "A^B", "B" indicates a superscript (e.g., an exponent) attached to "A".

[0039] All voxels that make up the voxel grid are divided into occupied cells and unoccupied cells. An occupied cell is a voxel that has geometry inside it. In other words, an occupied cell contains coordinates within it where the NeRF power density is greater than or equal to a threshold. An unoccupied cell is a cell other than an occupied cell. In other words, an unoccupied cell is a voxel that has no geometry inside it and does not contain coordinates within it where the NeRF power density is greater than or equal to a threshold.

[0040] The occupied cell index information is information consisting of indices (also called occupied cell indices) corresponding to each occupied cell. Empirically, occupied cells are much fewer than unoccupied cells. Therefore, as shown in FIG. 5, the proxy voxel grid only needs to include a set of indices for occupied cells (it does not include indices for unoccupied cells). The indices are one-to-one information for occupied cells. Therefore, the occupied cell index information can reduce the amount of data compared to the connectivity information indicating the connections of each vertex of the tetrahedral mesh. In other words, by applying the proxy voxel grid, the increase in the amount of proxy data for the neural intensity field can be suppressed compared to when using a proxy tetrahedral mesh.

[0041] Furthermore, as shown in the example in Figure 5, empirically, the distribution of occupied cells is biased toward a certain portion of space. Therefore, the increase in data volume can be suppressed by organizing the set of indices of occupied cells into an octree. This octree assumes that voxels are obtained by recursively dividing space into eight parts (dividing each part into two along mutually perpendicular axes). The indices of each occupied cell are represented as a tree structure according to the hierarchical structure of the voxels. This octree structure allows the indices of regions where all voxels are occupied cells to be combined (represented by a single index). Therefore, the data volume of the index set can be reduced compared to when the entire space is represented using the minimum voxels (maximum resolution voxels). In other words, applying this octree structure can further suppress the increase in data volume of the surrogate data for the neural intensity field. In contrast, it is difficult to create such a tree structure for a surrogate tetrahedral mesh, making it difficult to suppress the increase in data volume.

[0042] Because voxels are rectangular, an occupied cell has eight vertices, as shown in Figure 6. Each vertex has UVW coordinates. The UVW coordinate information consists of the initial coordinate values ​​(UVW coordinates) of each vertex of all occupied cells in space. Note that in UVW coordinate information, vertices are managed independently of voxels. That is, for example, if there are adjacent occupied cells, the common vertices between them are shared. In other words, if vertices overlap between occupied cells, they are considered to be a single vertex. For example, if there are A occupied cells and B vertices in a space, the number of UVW coordinates included in the UVW coordinate information is B, not A x 8. Similar to the UVW coordinates in NeRFShop, they are initialized to the initial position of the vertex. Note that vertex coordinates can change due to edits to the proxy voxel grid (for example, adding noise).

[0043] As described above, the proxy voxel grid has occupied cell index information instead of the connectivity information of the proxy tetrahedral mesh. Therefore, by applying the proxy voxel grid, it is possible to suppress the increase in the amount of proxy data for the neural intensity field compared to when a proxy tetrahedral mesh is applied.

[0044] Therefore, it is possible to suppress an increase in the processing load of generating, editing, encoding, decoding, rendering, etc. of proxy data. Furthermore, when transmitting proxy data between a server and a client, for example, it is possible to suppress an increase in the transmission volume, and it is also possible to suppress an increase in the transmission-related load. Furthermore, by suppressing these increases in load, it is possible to suppress an increase in costs and an increase in processing time.

[0045] Furthermore, by applying a proxy voxel grid, it is possible to suppress the reduction in coding efficiency more than in the case of a proxy tetrahedral mesh.

[0046] The proxy voxel grid may include any information other than those described above. For example, the proxy voxel grid may further include active flag information consisting of active flags set for all occupied cells. The active flag is flag information indicating whether or not an object geometry exists after editing the occupied cell. As shown in FIG. 6 , one active flag is set for each occupied cell. For example, when this active flag is ON (true), the occupied cell is occupied (geometry exists inside). When this active flag is OFF (false), the occupied cell is unoccupied (no geometry exists inside). The value of this active flag can be updated by editing the proxy voxel grid. For example, when deleting an occupied cell in a region during editing of the proxy voxel grid, the active flag of the occupied cell may be set OFF instead of deleting the corresponding index from the octree of the occupied cell index information. Note that, in rendering, if the active flag of an occupied cell is OFF, the occupied cell is treated the same as an unoccupied cell. This allows the editing results to be reflected in the proxy voxel grid without updating the occupied cell index information. This can speed up editing operations.

[0047] The proxy voxel grid may also have a transformation matrix that indicates the correspondence between the index of an occupied cell and the position of the occupied cell after editing. The transformation matrix H is a 4x4 homogeneous transformation matrix that holds the transformation (e.g., translation, rotation, scale transformation, shear, etc.) from the index space (i, j, k) to the world space (x, y, z). For example, the index (i, j, k) of a certain occupied cell and the XYZ coordinates (x, y, z) of the vertex closest to the origin of that cell have the following relationship. Note that the superscript T represents the transpose of a vector or matrix: (x, y, z, 1)^T = H × (i, j, k, 1)^T

[0048] For example, if the number of occupied cells is O and the number of vertices is V, the proxy voxel grid consists of O indices (stored as an octree), O active flags, V UVW coordinates, and one transformation matrix.

[0049] The proxy voxel grid may also have cell attribute information consisting of attributes added to occupied cells. For example, an average density value (floating-point value) within a cell or an average color within a cell (a three-dimensional vector in the case of RGB color) may be added as an attribute to an occupied cell. The proxy voxel grid may also have cell attribute information indicating these attributes. This allows the proxy voxel grid to support editing of cell attributes.

[0050] <Method 1-1> When the above-described method 1 is applied, a proxy voxel grid may be generated as shown in the second row from the top of the table in FIG. 4 (method 1-1).

[0051] For example, a first information processing device (e.g., the terminal device 111 in FIG. 8 or the server 312 in FIG. 21) may include a proxy voxel grid generation unit (e.g., the proxy voxel grid generation unit 121 in FIG. 8 or the proxy voxel grid generation unit 321 in FIG. 21) that generates a proxy voxel grid using a voxel grid as proxy data for editing a neural luminance field that is a neural network that approximates a luminance field function that represents a spatial scene including an object. Furthermore, in a first information processing method (e.g., the neural luminance field editing process in FIG. 13 or the neural luminance field editing process in FIG. 22) executed by the first information processing device, a proxy voxel grid using a voxel grid may be generated as proxy data for editing a neural luminance field that is a neural network that approximates a luminance field function that represents a spatial scene including an object.

[0052] By generating a proxy voxel grid as proxy data in this way, the first information processing apparatus can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0053] For example, the proxy voxel grid generator may generate, as a proxy voxel grid, occupied cell index information composed of indices of all occupied cells and UVW coordinate information composed of UVW coordinates indicating initial values ​​of the coordinates of the vertices of all occupied cells, as in the example shown in Fig. 14. By doing so, the first information processing device can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0054] 14, the proxy voxel grid generator may generate a voxel grid by dividing space into voxels, determine whether each voxel constituting the voxel grid is an occupied cell or an unoccupied cell, and generate occupied cell index information by organizing the indices of the occupied cells into an octree. By doing so, the first information processing device can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0055] Alternatively, the proxy voxel grid generator may derive the density of the voxel centers based on the neural luminance field, and determine the voxel as an occupied cell if the density is greater than a predetermined threshold, and determine the voxel as an unoccupied cell if the density is not greater than the threshold. This allows the proxy voxel grid generator to easily determine whether each voxel constituting the voxel grid is an occupied cell or an unoccupied cell. Therefore, the first information processing device can suppress an increase in the amount of proxy data for the neural luminance field, as described above in <Method 1>.

[0056] Furthermore, the proxy voxel grid generating unit may generate active flag information consisting of active flags set for all occupied cells as the proxy voxel grid, as in the example shown in Fig. 14. By doing so, the first information processing device can speed up the editing operation, as described above in <Method 1>.

[0057] Furthermore, the proxy voxel grid generation unit may generate, as the proxy voxel grid, cell attribute information constituted by attributes to be added to each occupied cell, as in the example shown in Fig. 25. By doing so, the first information processing device can make the proxy voxel grid compatible with editing of cell attributes, as described above in <Method 1>.

[0058] <Method 1-2> When the above-described method 1 is applied, the proxy voxel grid may be edited as shown in the third row from the top of the table in FIG. 4 (method 1-2).

[0059] For example, a second information processing device (e.g., the terminal device 111 in FIG. 8 or the terminal device 311 in FIG. 21) may include a proxy voxel grid editing unit (e.g., the proxy voxel grid editing unit 122 in FIG. 8 or the proxy voxel grid editing unit 332 in FIG. 21) that edits proxy voxel grids that are proxy data for editing a neural luminance field that is a neural network that approximates a luminance field function that represents a spatial scene including an object. Furthermore, a second information processing method (e.g., the neural luminance field editing process in FIG. 13 or the neural luminance field editing process in FIG. 22) executed by the second information processing device may edit proxy voxel grids that are proxy data for editing a neural luminance field that is a neural network that approximates a luminance field function that represents a spatial scene including an object. Here, the proxy voxel grid is proxy data using a voxel grid.

[0060] By editing the proxy voxel grid in this way, the second information processing device can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0061] For example, the proxy voxel grid editing unit may generate or edit a transformation matrix that indicates the correspondence between the index of an occupied cell and the position of the occupied cell after editing, as in the example shown in Fig. 15. By doing so, the second information processing device can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0062] Furthermore, the proxy voxel grid editing unit may edit the coordinates of the vertices of the occupied cells that make up the voxel grid, as in the example shown in Fig. 15. By doing so, the second information processing device can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0063] Furthermore, the proxy voxel grid editing unit may edit an active flag, which is flag information indicating whether or not an object has geometry after editing of the occupied cells, as in the example shown in Fig. 15. By doing so, the second information processing device can speed up the editing operation, as described above in <Method 1>.

[0064] Furthermore, the proxy voxel grid editing unit may edit attributes added to occupied cells as in the example shown in Fig. 26. By doing so, the second information processing device can make the proxy voxel grid compatible with editing of cell attributes as described above in <Method 1>.

[0065] <Method 1-3> When the above-described method 1 is applied, the proxy voxel grid may be coded as shown in the fourth row from the top of the table in FIG. 4 (method 1-3).

[0066] For example, a third information processing device (e.g., the terminal device 111 in FIG. 8 , the terminal device 311 in FIG. 21 , or the server 312 in FIG. 21 ) may include a proxy voxel grid encoding unit (e.g., the proxy voxel grid encoding unit 123 in FIG. 8 , the proxy voxel grid encoding unit 322 in FIG. 21 , or the proxy voxel grid encoding unit 333 in FIG. 21 ) that encodes a proxy voxel grid that is proxy data for editing a neural luminance field that is a neural network that approximates a luminance field function that represents a spatial scene including an object. Furthermore, a third information processing method (e.g., the neural luminance field editing process in FIG. 13 or the neural luminance field editing process in FIG. 22 ) executed by the third information processing device may encode a proxy voxel grid that is proxy data for editing a neural luminance field that is a neural network that approximates a luminance field function that represents a spatial scene including an object. Here, the proxy voxel grid is proxy data using a voxel grid.

[0067] By encoding the proxy voxel grid in this way, the third information processing apparatus can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0068] For example, the proxy voxel grid encoding unit may include an occupied cell index encoding unit (e.g., the occupied cell index encoding unit 152 in FIG. 9 ) that encodes occupied cell index information composed of indexes of all occupied cells included in the proxy voxel grid, and a UVW coordinate encoding unit (e.g., the UVW coordinate encoding unit 153 in FIG. 9 ) that encodes UVW coordinate information composed of UVW coordinates indicating initial values ​​of the coordinates of the vertices of all occupied cells included in the proxy voxel grid. By doing so, the third information processing apparatus can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0069] The occupied cell index encoding unit may losslessly encode the occupied cell index information in which the indexes of the occupied cells are organized into an octree. By doing so, the third information processing device can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0070] The UVW coordinate encoding unit may also encode the difference between the predicted and actual UVW coordinate values, as in the example shown in Fig. 10. By doing so, the third information processing device can suppress an increase in the amount of proxy data for the neural luminance field, as described above in <Method 1>.

[0071] Furthermore, the UVW coordinate encoding unit may derive predicted values ​​of UVW coordinates using the edited coordinates of the vertices of the occupied cells and prediction parameters, as in the example shown in Fig. 10. By doing so, the third information processing apparatus can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0072] Furthermore, the UVW coordinate encoding unit may update the prediction parameters for each vertex using the difference between the predicted and actual values ​​of the UVW coordinates, as in the example shown in Fig. 10. By doing so, the third information processing device can suppress an increase in the amount of proxy data for the neural luminance field, as described above in <Method 1>.

[0073] The UVW coordinate encoding unit may encode the UVW coordinates of each vertex in Morton order, thereby enabling the third information processing apparatus to suppress an increase in the amount of proxy data for the neural luminance field, as described above in <Method 1>.

[0074] The proxy voxel grid encoding unit may further include an active flag encoding unit (for example, the active flag encoding unit 154 in FIG. 9 ) that encodes active flag information configured by active flags that are set for all occupied cells included in the proxy voxel grid. In this way, the third information processing apparatus can speed up editing operations, as described above in <Method 1>.

[0075] The proxy voxel grid encoding unit may further include a transformation matrix encoding unit (for example, the transformation matrix encoding unit 151 in FIG. 9 ) that encodes a transformation matrix indicating the correspondence between the index of an occupied cell and the position of the occupied cell after editing. By doing so, the third information processing apparatus can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0076] The proxy voxel grid encoding unit may further include a cell attribute encoding unit (for example, the cell attribute encoding unit 451 in FIG. 23) that encodes cell attribute information made up of attributes added to each of the occupied cells included in the proxy voxel grid. In this way, the third information processing apparatus can make the proxy voxel grid compatible with editing of cell attributes, as described above in <Method 1>.

[0077] <Method 1-4> When the above-described method 1 is applied, the bitstream of the proxy voxel grid may be decoded as shown in the fifth row from the top of the table in FIG. 4 (Method 1-4).

[0078] For example, a fourth information processing device (e.g., the server 112 in FIG. 8 , the terminal device 311 in FIG. 21 , or the server 312 in FIG. 21 ) may include a proxy voxel grid decoding unit (e.g., the proxy voxel grid decoding unit 131 in FIG. 8 , the proxy voxel grid decoding unit 323 in FIG. 21 , or the proxy voxel grid decoding unit 331 in FIG. 21 ) that decodes a bitstream and generates a proxy voxel grid that is proxy data for editing a neural luminance field that is a neural network that approximates a luminance field function that represents a spatial scene including an object. Furthermore, a fourth information processing method (e.g., the neural luminance field editing process in FIG. 13 or the neural luminance field editing process in FIG. 22 ) executed by the fourth information processing device may decode a bitstream and generate a proxy voxel grid that is proxy data for editing a neural luminance field that is a neural network that approximates a luminance field function that represents a spatial scene including an object. Here, the proxy voxel grid is proxy data using a voxel grid.

[0079] By encoding the proxy voxel grid in this way, the fourth information processing apparatus can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0080] For example, the proxy voxel grid decoding unit may include an occupied cell index decoding unit (e.g., the occupied cell index decoding unit 172 in FIG. 11 ) that decodes the bit stream and generates occupied cell index information consisting of indexes of all occupied cells included in the proxy voxel grid, and a UVW coordinate decoding unit (e.g., the UVW coordinate decoding unit 173 in FIG. 11 ) that decodes the bit stream and generates UVW coordinate information consisting of UVW coordinates indicating initial values ​​of the coordinates of the vertices of all occupied cells included in the proxy voxel grid. By doing so, the fourth information processing apparatus can suppress an increase in the data amount of the proxy data of the neural luminance field, as described above in <Method 1>.

[0081] The occupied cell index decoding unit may losslessly decode the bitstream to generate occupied cell index information in which the indexes of the occupied cells are organized into an octree. By doing so, the fourth information processing device can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0082] 12, the UVW coordinate decoding unit may decode the bitstream to generate a difference between a predicted value and an actual value of the UVW coordinate, derive the predicted value, and add the derived predicted value to the difference value to derive the UVW coordinate. By doing so, the fourth information processing apparatus can suppress an increase in the amount of data of the proxy data of the neural luminance field, as described above in <Method 1>.

[0083] Furthermore, the UVW coordinate decoding unit may derive predicted values ​​of UVW coordinates using the edited coordinates of the vertices of the occupied cells and the prediction parameters, as in the example shown in Fig. 12. By doing so, the fourth information processing apparatus can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0084] Furthermore, the UVW coordinate decoding unit may derive the edited coordinates of the vertices of the occupied cells using occupied cell index information and a transformation matrix, as shown in the example of Figure 12. Note that this transformation matrix indicates the correspondence between the index of the occupied cell and the position of the occupied cell after editing. By doing so, the fourth information processing device can suppress an increase in the amount of proxy data for the neural luminance field, as described above in <Method 1>.

[0085] Furthermore, the UVW coordinate decoding unit may update the prediction parameters for each vertex using the edited coordinates of the vertices of the occupied cell and the difference between the predicted and actual values ​​of the UVW coordinates, as in the example shown in Fig. 12. By doing so, the fourth information processing apparatus can suppress an increase in the amount of proxy data for the neural luminance field, as described above in <Method 1>.

[0086] The proxy voxel grid decoding unit may further include an active flag decoding unit (for example, the active flag decoding unit 174 in FIG. 11 ) that decodes the bitstream and generates active flag information made up of active flags that are set for all occupied cells included in the proxy voxel grid. In this way, the fourth information processing apparatus can speed up editing operations, as described above in <Method 1>.

[0087] The proxy voxel grid decoding unit may further include a transformation matrix decoding unit (e.g., the transformation matrix decoding unit 171 in FIG. 11 ) that decodes the bitstream and generates a transformation matrix that indicates the correspondence between the indexes of the occupied cells and the positions of the occupied cells after editing. By doing so, the fourth information processing device can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0088] The proxy voxel grid encoding unit may further include a cell attribute decoding unit (for example, the cell attribute decoding unit 471 in FIG. 24 ) that decodes the bitstream and generates cell attribute information made up of attributes that are added to each of the occupied cells included in the proxy voxel grid. In this way, the fourth information processing device can make the proxy voxel grid compatible with editing of cell attributes, as described above in <Method 1>.

[0089] <Method 1-5> When the above-described method 1 is applied, rendering may be performed using a proxy voxel grid, as shown in the bottom row of the table in FIG. 4 (Method 1-5).

[0090] For example, a fifth information processing device (e.g., the server 112 in FIG. 8 or the server 312 in FIG. 21) may include a rendering unit (e.g., the rendering unit 132 in FIG. 8 or the rendering unit 324 in FIG. 21) that performs rendering using a proxy voxel grid that is proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object. Furthermore, in a fifth information processing method (e.g., the neural luminance field editing process in FIG. 13 or the neural luminance field editing process in FIG. 22) executed by the fifth information processing device, rendering may be performed using a proxy voxel grid that is proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object. Here, the proxy voxel grid is proxy data that uses a voxel grid.

[0091] By performing rendering using a proxy voxel grid in this way, the fifth information processing apparatus can suppress an increase in the amount of proxy data of the neural luminance field, as described above in <Method 1>.

[0092] For example, the rendering unit may generate a rendering image using the UVW coordinates of the sample points, the gaze direction after rotation, and the neural luminance field. Here, the UVW coordinates of the sample points indicate the initial coordinate values ​​of the sample points. By doing so, the fifth information processing apparatus can suppress an increase in the amount of proxy data for the neural luminance field, as described above in <Method 1>.

[0093] Alternatively, the rendering unit may search for an occupied cell that contains the sample point, derive the UVW coordinates of the sample point using the coordinates of the vertices of the occupied cell, and derive the gaze direction after rotation using the rotation matrix of the sample point, as in the example shown in Figure 7. By doing so, the fifth information processing device can suppress an increase in the amount of proxy data for the neural luminance field, as described above in <Method 1>.

[0094] <Combination> Each of the above-described methods may be applied in combination with any other method as long as no contradiction occurs. Three or more methods may be applied in combination. For example, methods 1-1 to 1-5 may be applied in appropriate combination. Furthermore, techniques that can be combined are not limited to those shown in the table of FIG. 4 as "methods," but may include all of the elements described above in <3. Application of voxel grid>. Furthermore, each of the above-described methods may be applied in combination with methods other than those described above.

[0095] In this specification, a description of a higher-level method also applies to lower-level methods within that higher-level method, unless a contradiction arises. For example, if it is described that "Method 1 may be applied," then each of Methods 1-1 to 1-5 can be applied.

[0096] 8 is a block diagram showing a main configuration example of an information processing system to which the present technology described above in <3. Application of Voxel Grid> is applied. The information processing system 100 shown in Fig. 8 is a system that edits proxy data of a neural luminance field and generates a rendering image corresponding to the editing result.

[0097] 8, the information processing system 100 includes a terminal device 111 and a server 112. The terminal device 111 and the server 112 are connected to each other via a network (not shown) so that they can communicate with each other. For example, the terminal device 111 operates as a client of the server 112.

[0098] The terminal device 111 generates a proxy voxel grid using a voxel grid as proxy data corresponding to the neural luminance field (NeRF data), edits the proxy voxel grid, encodes the edited proxy voxel grid, and supplies the bit stream to the server 112. In response, the server 112 decodes the supplied bit stream to generate a proxy voxel grid, and performs rendering using the decoded proxy voxel grid to generate a rendered image.

[0099] The configurations of the terminal device 111 and the server 112 are each arbitrary. For example, the information processing system 100 may be provided with a plurality of terminal devices 111 and a plurality of servers 112. Furthermore, the number of terminal devices 111 and the number of servers 112 do not have to be the same. Furthermore, the terminal device 111 may be configured with a plurality of information processing devices. Furthermore, the server 112 may be configured with a plurality of information processing devices. For example, the server 112 may be configured as a cloud server, the device configuration of which is not limited.

[0100] 8, the terminal device 111 includes a proxy voxel grid generator 121, a proxy voxel grid editor 122, and a proxy voxel grid encoder 123. The server 112 includes a proxy voxel grid decoder 131 and a rendering unit 132.

[0101] The proxy voxel grid generator 121 of the terminal device 111 executes processing related to generating a proxy voxel grid. For example, the proxy voxel grid generator 121 may acquire NeRF data. The NeRF data is neural network (neural luminance field) data that inputs three-dimensional coordinates and a gaze direction vector (x, y, z, θ, φ) and outputs density and color vectors (r, g, b, σ). For example, the NeRF data may include parameters such as a weight matrix of the neural network (neural luminance field). The neural network only needs to specify its inputs and outputs, and the specific processing content is not limited. In other words, any NeRF that includes a neural network with the above-described inputs and outputs is acceptable. Note that, in this specification, the NeRF data is assumed to be shared in advance between the cloud environment and the local environment. In other words, the terminal device 111 and the server 112 already have the NeRF data, and the NeRF data is not transmitted between the terminal device 111 and the server 112.

[0102] The proxy voxel grid generator 121 may also generate a proxy voxel grid using the NeRF data. In generating the proxy voxel grid, for example, the proxy voxel grid generator 121 may initialize an NxNxN voxel grid (where N is the resolution of the voxel grid). The proxy voxel grid generator 121 may also determine whether each voxel in the voxel grid is an occupied cell or an unoccupied cell. For example, in this determination, the proxy voxel grid generator 121 may input the XYZ coordinates (x, y, z) of the voxel center and a dummy line of sight direction (θ, φ) = (0, 0) into a neural network to derive the density σ of the voxel center. The proxy voxel grid generator 121 may then determine the voxel as an occupied cell if the density σ is greater than a threshold. Otherwise, the proxy voxel grid generator 121 may determine that the voxel is an unoccupied cell and delete the voxel. That is, the proxy voxel grid generator 121 may leave only occupied cells from the initialized voxel grid, as in the example shown in FIG. 5. Note that this threshold may be determined in advance. In this manner, the proxy voxel grid generator 121 may generate occupied cell index information.

[0103] Furthermore, the proxy voxel grid generator 121 may set an active flag with a value of "ON" for all occupied cells for which the above-described determination has been made.

[0104] 6, the proxy voxel grid generator 121 may initialize the UVW coordinates for all vertices of all occupied cells with the XYZ coordinates of those vertices. In other words, the proxy voxel grid generator 121 may generate UVW coordinate information. The proxy voxel grid generator 121 may supply the proxy voxel grid generated as described above to the proxy voxel grid editor 122.

[0105] The proxy voxel grid editing unit 122 executes processing related to editing the proxy voxel grid. For example, the proxy voxel grid editing unit 122 may obtain a proxy voxel grid supplied from the proxy voxel grid generation unit 121. The proxy voxel grid editing unit 122 may also execute editing operations on the proxy voxel grid. Editing operations on the proxy voxel grid may include, for example, editing a transformation matrix, editing UVW coordinates, editing an active flag, and the like.

[0106] The proxy voxel grid editing unit 122 can translate, rotate, scale, and shear the proxy voxel grid in world space by editing the transformation matrix added to the proxy voxel grid. As a result of this editing, the position, orientation, scale, shear, and so on of the entire scene can change in the rendered image.

[0107] The proxy voxel grid editor 122 can add noise to the UVW coordinates of some vertices of the occupied cells by editing the UVW coordinates, which can result in local deformation of part of the scene in the rendered image.

[0108] The proxy voxel grid editor 122 can edit the active flag to, for example, turn off the active flag of a desired region. As a result of such editing, a part of the scene may be deleted from the rendered image as well.

[0109] The "edited proxy voxel grid" generated by the above editing operation is used to generate a rendering image that reflects the editing. The proxy voxel grid editing unit 122 may supply the "edited proxy voxel grid" to the proxy voxel grid encoding unit 123.

[0110] The proxy voxel grid encoding unit 123 executes processing related to encoding of the proxy voxel grid. For example, the proxy voxel grid encoding unit 123 may acquire an "edited proxy voxel grid" supplied from the proxy voxel grid editing unit 122. The proxy voxel grid encoding unit 123 may also encode the "edited proxy voxel grid" and generate a bitstream thereof. The proxy voxel grid encoding unit 123 may supply (transmit) the generated bitstream (the bitstream of the "edited proxy voxel grid") to the server 112.

[0111] The proxy voxel grid decoding unit 131 of the server 112 performs processing related to decoding of the bit stream of the proxy voxel grid. For example, the proxy voxel grid decoding unit 131 may acquire a bit stream supplied from (the proxy voxel grid encoding unit 123 of) the terminal device 111. The proxy voxel grid decoding unit 131 may also decode the bit stream to generate a "decoded proxy voxel grid." This "decoded proxy voxel grid" is a restored version of the above-mentioned "edited proxy voxel grid." In other words, the "decoded proxy voxel grid" and the "edited proxy voxel grid" correspond to each other and represent the same scene. Note that the "decoded proxy voxel grid" may include coding distortion. In other words, the "decoded proxy voxel grid" and the "edited proxy voxel grid" may or may not completely match each other. The proxy voxel grid decoding unit 131 may supply (transmit) the generated “decoded proxy voxel grid” to the rendering unit 132 .

[0112] The rendering unit 132 executes processing related to rendering. For example, the rendering unit 132 may acquire a "decoded proxy voxel grid" supplied from the proxy voxel grid decoding unit 131. Furthermore, the rendering unit 132 may execute rendering using the "decoded proxy voxel grid" and NeRF data to generate a rendered image.

[0113] In this case, the rendering unit 132 modifies the process of deriving density and color from the coordinates of the sample point and the line of sight of the ray in the case of NeRFShop as follows: [Before modification] The neural network Fθ receives the XYZ coordinates (x, y, z) of the sample point and the line of sight of the ray (θ, φ). [After modification] The neural network Fθ receives the UVW coordinates (u, v, w) of the sample point and the line of sight after rotation (θ', φ').

[0114] Specifically, the UVW coordinates (u, v, w) of the sample point and the viewing direction after rotation (θ', φ') are derived as follows: 1: Search for an occupied cell that contains the sample point from the coordinates (x, y, z) of the sample point. If not found, terminate. 2: If the active flag of that cell is OFF, terminate. 3: As shown in the example in Figure 7, calculate the UVW coordinates (u, v, w) of the sample point by trilinear interpolation (also called trilinear interpolation) of the UVW coordinates of the eight vertices of that voxel. 4: Rotate the viewing direction (θ, φ) using the rotation matrix R at the sample point as shown in the following formula, and derive the viewing direction after rotation (θ', φ'). (θ', φ')^T = R × (θ, φ)^T

[0115] The rotation matrix R at the sample points is derived by singular value decomposition of the Jacobian matrix J below.

[0116] Here, (u, v, w) are UVW coordinates after trilinear interpolation, and (x, y, z) are XYZ coordinates of the sample point. The Jacobian matrix J can be calculated using, for example, an automatic differentiation algorithm.

[0117] When the proxy voxel grid is edited by the proxy voxel grid editing unit 122, the edited results are reflected in the rendered image. The rendering unit 132 may output the generated rendered image. The output destination is arbitrary. For example, the rendered image may be displayed as visual information, or may be supplied to another device.

[0118] <Proxy Voxel Grid Encoding Unit> Fig. 9 is a block diagram showing an example of the main configuration of the proxy voxel grid encoding unit 123 of Fig. 8. As shown in Fig. 9, the proxy voxel grid encoding unit 123 may have a transformation matrix encoding unit 151, an occupied cell index encoding unit 152, a UVW coordinate encoding unit 153, and an active flag encoding unit 154.

[0119] The transformation matrix encoding unit 151 executes processing related to encoding of a transformation matrix. For example, the transformation matrix encoding unit 151 may encode a transformation matrix included in a proxy voxel grid and generate a bitstream thereof (also referred to as a transformation matrix bitstream). Any method for encoding the transformation matrix may be used. Furthermore, since the amount of data for the transformation matrix is ​​small, this encoding may be omitted. The transformation matrix encoding unit 151 may supply the generated transformation matrix bitstream to the server 112 (the proxy voxel grid decoding unit 131) as a "bitstream of the edited proxy voxel grid."

[0120] The occupied cell index encoding unit 152 performs processing related to encoding of occupied cell index information. For example, the occupied cell index encoding unit 152 may encode the occupied cell index information included in the proxy voxel grid and generate a bitstream (also referred to as an occupied cell index bitstream). In this case, the occupied cell index encoding unit 152 may losslessly encode an octree of a set of indexes of occupied cells in the proxy voxel grid. In other words, the occupied cell index encoding unit 152 may losslessly encode a set of indexes (i, j, k) of all occupied cells. Note that any method may be used to encode this octree. For example, G-PCC (Geometry-based Point Cloud Compression) encoding may be applied. The occupied cell index encoding unit 152 may supply the generated occupied cell index bitstream to the server 112 (the proxy voxel grid decoding unit 131) as a "bitstream of the edited proxy voxel grid."

[0121] The UVW coordinate encoding unit 153 executes processing related to encoding of UVW coordinate information. For example, the UVW coordinate encoding unit 153 may encode UVW coordinate information included in a proxy voxel grid and generate a bitstream (also referred to as a UVW coordinate bitstream). In this case, the UVW coordinate encoding unit 153 may encode a set of UVW coordinates of the vertices of all occupied cells. In this case, the UVW coordinate encoding unit 153 may encode each UVW coordinate using predictive encoding, which encodes the difference between a predicted value and an actual value. The UVW coordinate encoding unit 153 may supply the generated UVW coordinate bitstream to the server 112 (the proxy voxel grid decoding unit 131) as a "bitstream of the proxy voxel grid after editing."

[0122] The active flag encoding unit 154 performs processing related to encoding of active flags. For example, the active flag encoding unit 154 may encode active flags included in a proxy voxel grid and generate a bit stream thereof (also referred to as an active flag bit stream). In this case, the active flag encoding unit 154 may losslessly encode a set of active flags of all occupied cells to generate the active flag bit stream. In this case, the active flag encoding unit 154 may arrange the occupied cells in a row in Morton order of indexes and then entropy encode the active flags in order from the front.

[0123] <UVW Coordinate Encoding Unit> Fig. 10 is a block diagram showing an example of the main configuration of the UVW coordinate encoding unit 153 in Fig. 9. As shown in Fig. 10, the UVW coordinate encoding unit 153 may include a prediction parameter initialization unit 161, a coordinate supply unit 162, a prediction value derivation unit 163, a difference encoding unit 164, and a prediction parameter update unit 165.

[0124] The prediction parameter initialization unit 161 executes processing related to initialization of prediction parameters. For example, the prediction parameter initialization unit 161 may initialize prediction parameters (A, b) used in the prediction value derivation unit 163. For example, the prediction parameter initialization unit 161 may initialize the prediction parameter A as a 3x3 identity matrix. Furthermore, the prediction parameter initialization unit 161 may initialize the prediction parameter b as a three-dimensional zero vector. In other words, the prediction parameter initialization unit 161 may supply the initial values ​​of the prediction parameters (A, b) to the prediction value derivation unit 163.

[0125] The coordinate supply unit 162 executes processing related to the supply of vertex coordinates. For example, the coordinate supply unit 162 may arrange all vertices of all occupied cells in the proxy voxel grid in a line in Morton order of indexes, and then supply the XYZ coordinates of each vertex in order from the front to the predicted value derivation unit 163. The coordinate supply unit 162 may also supply the UVW coordinates of each vertex in that order to the differential encoding unit 164.

[0126] The predicted value derivation unit 163 executes processing related to the derivation of predicted values ​​of UVW coordinates of vertices. For example, the predicted value derivation unit 163 may derive predicted values ​​(u', v', w') of UVW coordinates from XYZ coordinates (x, y, z) and prediction parameters (A, b), and supply the derived predicted values ​​to the differential encoding unit 164. Note that the prediction parameter A may be a 3x3 matrix. The prediction parameter b may be a three-dimensional vector. The predicted value derivation unit 163 may derive the predicted values ​​using the following linear prediction: (u', v', w')^T ← A×(x, y, z)^T + b

[0127] The differential encoding unit 164 performs processing related to differential encoding of UVW coordinates. For example, the differential encoding unit 164 may derive differential values ​​(du, dv, dw) from predicted values ​​(u', v', w') and actual values ​​(u, v, w) as follows: (du, dv, dw) ← (u', v', w') - (u, v, w)

[0128] The differential encoding unit 164 may then entropy encode the derived differential value to generate a bitstream (UVW coordinate bitstream). If the predicted value is close to the actual value, the difference will be close to 0, improving encoding efficiency. The differential encoding unit 164 may supply the generated UVW coordinate bitstream to the server 112 (the proxy voxel grid decoding unit 131). The differential encoding unit 164 may also supply the derived differential value to the prediction parameter update unit 165.

[0129] The prediction parameter updating unit 165 performs processing related to updating the prediction parameters. For example, the prediction parameter updating unit 165 may use the difference values ​​(du, dv, dw) to update the prediction parameters (A, b) used in the prediction value derivation unit 163. In other words, the prediction parameter updating unit 165 may update the prediction parameters (A, b) using the difference values ​​(du, dv, dw) and supply the updated prediction parameters (A, b) to the prediction value derivation unit 163.

[0130] The purpose of this update is to reduce the prediction error of the UVW coordinates to be coded next. For example, the prediction parameter update unit 165 may update the prediction parameters (A, b) in a direction that minimizes the prediction error function L as follows: η is a coefficient that is set in advance. A ← A - η · ∂L / ∂A b ← b - η · ∂L / ∂b

[0131] Here, ∂L / ∂A is the differential value with respect to the prediction parameter A and may be a 3x3 matrix. Also, ∂L / ∂b is the differential value with respect to the prediction parameter b and may be a three-dimensional vector. Note that a squared error function may be applied as the prediction error function L. In this case, L, ∂L / ∂A, and ∂L / ∂b are as follows: L = 1 / 2 · (du, dv, dw) · (du, dv, dw)^T ∂L / ∂A = (du, dv, dw)^T · (x, y, z) ∂L / ∂b = (du, dv, dw)^T

[0132] <Proxy Voxel Grid Decoding Unit> Fig. 11 is a block diagram showing an example of the main configuration of the proxy voxel grid decoding unit 131 of Fig. 8. As shown in Fig. 11, the proxy voxel grid decoding unit 131 may have a transformation matrix decoding unit 171, an occupied cell index decoding unit 172, a UVW coordinate decoding unit 173, an active flag decoding unit 174, and a proxy voxel grid construction unit 175.

[0133] The transformation matrix decoding unit 171 performs processing related to the decoding of the transformation matrix bitstream. For example, the transformation matrix decoding unit 171 may decode the transformation matrix bitstream to generate a transformation matrix. Any method for decoding this transformation matrix bitstream may be used. Note that if the transformation matrix is ​​transmitted without being coded, this decoding may be omitted. The transformation matrix decoding unit 171 may supply the generated transformation matrix to the proxy voxel grid construction unit 175.

[0134] The occupied cell index decoding unit 172 performs processing related to decoding of the occupied cell index bitstream. For example, the occupied cell index decoding unit 172 may decode the occupied cell index bitstream to generate occupied cell index information. In this case, the occupied cell index decoding unit 172 may losslessly decode the occupied cell index bitstream to generate an octree of a set of indexes of occupied cells in the proxy voxel grid. That is, the occupied cell index decoding unit 172 may generate a set of indexes (i, j, k) of all occupied cells by lossless decoding. Note that any method for decoding this octree may be used. For example, G-PCC (Geometry-based Point Cloud Compression) decoding may be applied. The occupied cell index decoding unit 172 may supply the generated occupied cell index information to the proxy voxel grid construction unit 175.

[0135] The UVW coordinate decoding unit 173 performs processing related to decoding of the UVW coordinate bitstream. For example, the UVW coordinate decoding unit 173 may decode the UVW coordinate bitstream to generate UVW coordinate information. In this case, the UVW coordinate decoding unit 173 may decode the UVW coordinate bitstream to generate a set of UVW coordinates for the vertices of all occupied cells. In this case, the UVW coordinate decoding unit 173 may generate each UVW coordinate by predictive decoding, which decodes the bitstream and generates the difference between a predicted value and an actual value. The UVW coordinate decoding unit 173 may supply the generated UVW coordinate information to the proxy voxel grid construction unit 175.

[0136] The active flag decoder 174 performs processing related to decoding of the active flag bit stream. For example, the active flag decoder 174 may decode the active flag bit stream to generate a set of active flags. In this case, the active flag decoder 174 may losslessly decode the active flag bit stream to generate a set of active flags for all occupied cells. In this case, the active flag decoder 174 may decode the bit stream and generate active flags in the order in which the occupied cells are arranged in a row in Morton order of indexes.

[0137] The proxy voxel grid construction unit 175 executes processing related to the construction of a proxy voxel grid. For example, the proxy voxel grid construction unit 175 may construct a proxy voxel grid using the decoded transformation matrix, occupied cell index information, UVW coordinate information, and a set of active flags. The UVW coordinate information has UVW coordinates arranged in the Morton order of the vertices. The set of active flags has active flags arranged in the Morton order of the cells. Therefore, the proxy voxel grid construction unit 175 may copy these values ​​to the attributes of the vertices or cells in order from the front. The proxy voxel grid construction unit 175 may supply the constructed proxy voxel grid to the rendering unit 132.

[0138] <UVW Coordinate Decoding Unit> Fig. 12 is a block diagram showing an example of the main configuration of the UVW coordinate decoding unit 173 in Fig. 11. As shown in Fig. 12, the UVW coordinate decoding unit 173 may include a prediction parameter initialization unit 181, an XYZ coordinate derivation unit 182, a predicted value derivation unit 183, a difference decoding unit 184, a predicted value addition unit 185, and a prediction parameter update unit 186.

[0139] The prediction parameter initialization unit 181 executes processing related to initialization of prediction parameters. For example, the prediction parameter initialization unit 181 may initialize prediction parameters (A, b) used in the prediction value derivation unit 183. For example, the prediction parameter initialization unit 181 may initialize the prediction parameter A as a 3x3 identity matrix. Furthermore, the prediction parameter initialization unit 181 may initialize the prediction parameter b as a three-dimensional zero vector. In other words, the prediction parameter initialization unit 181 may supply the initial values ​​of the prediction parameters (A, b) to the prediction value derivation unit 183.

[0140] The XYZ coordinate derivation unit 182 performs processing related to the derivation of the XYZ coordinates of the vertices. For example, the XYZ coordinate derivation unit 182 may arrange all the vertices of the occupied cells in a line in the Morton order of their indices, and then derive the XYZ coordinates of each vertex from the front to the back. The XYZ coordinate derivation unit 182 may derive the XYZ coordinates of each vertex using the index of the occupied cell corresponding to the XYZ coordinate and a transformation matrix. The XYZ coordinate derivation unit 182 may supply the derived XYZ coordinates of each vertex to the predicted value derivation unit 183 and the predicted parameter update unit 186.

[0141] The predicted value derivation unit 183 executes processing related to the derivation of predicted values ​​of the UVW coordinates of vertices. For example, the predicted value derivation unit 183 may derive predicted values ​​(u', v', w') of the UVW coordinates from the XYZ coordinates (x, y, z) and prediction parameters (A, b), and supply the derived predicted values ​​to the predicted value addition unit 185. Note that the prediction parameter A may be a 3x3 matrix. The prediction parameter b may be a three-dimensional vector. The predicted value derivation unit 183 may derive the predicted values ​​using the following linear prediction: (u', v', w')^T ← A×(x, y, z)^T + b

[0142] The differential decoding unit 184 performs processing related to differential decoding of the UVW coordinate bitstream. For example, the differential decoding unit 184 may entropy decode the UVW coordinate bitstream to derive difference values ​​(du, dv, dw) between the predicted values ​​(u', v', w') and the actual values ​​(u, v, w). The differential decoding unit 184 may supply the derived difference values ​​(du, dv, dw) to the predicted value adding unit 185 and the prediction parameter updating unit 186.

[0143] The predicted value addition unit 185 executes processing related to addition of predicted values. For example, the predicted value addition unit 185 adds the difference values ​​(du, dv, dw) and the predicted values ​​(u', v', w') to derive the decoded UVW coordinates (u, v, w). (u, v, w) ← (u', v', w') + (du, dv, dw)

[0144] The predicted value addition unit 185 supplies the derived “decoded UVW coordinates (u, v, w)” to the proxy voxel grid construction unit 175 .

[0145] The prediction parameter updating unit 186 performs processing related to updating of prediction parameters. For example, the prediction parameter updating unit 186 may update the prediction parameters (A, b) used in the prediction value derivation unit 183 by using the difference values ​​(du, dv, dw) and the XYZ coordinates (x, y, z). In other words, the prediction parameter updating unit 186 may update the prediction parameters (A, b) by using the difference values ​​(du, dv, dw) and the XYZ coordinates (x, y, z) and supply the updated prediction parameters (A, b) to the prediction value derivation unit 183.

[0146] The purpose of this update is to reduce the prediction error of the UVW coordinates to be added to the next difference value. For example, the prediction parameter update unit 186 may update the prediction parameters (A, b) in a direction that minimizes the prediction error function L, using a method similar to that used by the prediction parameter update unit 165.

[0147] With the above-described configuration, the information processing system 100 (the terminal device 111 and the server 112) can apply the present technology described in <3. Application of Voxel Grid>, thereby suppressing an increase in the amount of proxy data of the neural luminance field. Therefore, an increase in the processing load of generating, editing, encoding, decoding, rendering, and the like of proxy data can be suppressed. Furthermore, when transmitting proxy data between a server and a client, for example, an increase in the transmission volume can be suppressed, and an increase in the transmission-related load can also be suppressed. Furthermore, by suppressing these increases in load, an increase in cost and an increase in processing time can be suppressed. Furthermore, by applying a proxy voxel grid, a decrease in coding efficiency can be suppressed compared to the case of a proxy tetrahedral mesh.

[0148] <Flow of Neural Luminance Field Editing Process> An example of the flow of neural luminance field editing process executed by the information processing system 100 will be described with reference to the flowchart of FIG.

[0149] When the process starts, the proxy voxel grid generator 121 of the terminal device 111 generates a proxy voxel grid corresponding to the NeRF data in step S101.

[0150] In step S102, the proxy voxel grid editing unit 122 edits the proxy voxel grid.

[0151] In step S103, the proxy voxel grid encoding unit 123 encodes the edited proxy voxel grid to generate a bitstream.

[0152] In step S104 , the proxy voxel grid encoding unit 123 supplies the generated proxy voxel grid bit stream to the server 112 .

[0153] In step S111, the proxy voxel grid decoding unit 131 of the server 112 acquires the bitstream.

[0154] In step S112, the proxy voxel grid decoding unit 131 decodes the bit stream and generates a decoded proxy voxel grid.

[0155] In step S113, the rendering unit 132 performs rendering using the decoded proxy voxel grid to generate a rendered image.

[0156] When the process of step S113 is completed, the editing process of the neural luminance field is completed.

[0157] <Flow of Proxy Voxel Grid Generation Processing> Next, an example of the flow of the proxy voxel grid generation processing executed in step S101 of FIG. 13 will be described with reference to the flowchart of FIG.

[0158] When the proxy voxel grid generation process starts, the proxy voxel grid generation unit 121 divides a space and generates a voxel grid in step S121.

[0159] In step S122, the proxy voxel grid generation unit 121 determines whether each voxel in the voxel grid is occupied or unoccupied.

[0160] In step S123, the proxy voxel grid generator 121 generates occupied cell indices based on the determination result, and generates an octree of the occupied cell indices.

[0161] In step S124, the proxy voxel grid generator 121 generates initial vertex coordinates (UVW coordinates) for all vertices of all occupied cells.

[0162] In step S125, the proxy voxel grid generator 121 generates an active flag for each occupied cell.

[0163] When the processing in step S125 is completed, the proxy voxel grid generation processing is completed, and the processing returns to FIG.

[0164] <Flow of Proxy Voxel Grid Editing Process> Next, an example of the flow of the proxy voxel grid editing process executed in step S102 of FIG. 13 will be described with reference to the flowchart of FIG.

[0165] When the proxy voxel grid editing process is started, the proxy voxel grid editing unit 122 receives an editing input in step S131.

[0166] In step S132, the proxy voxel grid editing unit 122 generates or edits a transformation matrix in accordance with the editing input.

[0167] In step S133, the proxy voxel grid editing unit 122 edits the vertex coordinates in accordance with the editing input.

[0168] In step S134, the proxy voxel grid editing unit 122 edits the active flag in accordance with the editing input.

[0169] When the processing in step S134 ends, the proxy voxel grid editing processing ends, and the processing returns to FIG.

[0170] <Flow of Proxy Voxel Grid Encoding Process> Next, an example of the flow of the proxy voxel grid encoding process executed in step S103 of FIG. 13 will be described with reference to the flowchart of FIG.

[0171] When the proxy voxel grid encoding process starts, the transformation matrix encoding unit 151 of the proxy voxel grid encoding unit 123 encodes the transformation matrix in step S141 to generate a transformation matrix bitstream.

[0172] In step S142, the occupied cell index encoding unit 152 encodes the occupied cell index information to generate an occupied cell index bitstream.

[0173] In step S143, the UVW coordinate encoding unit 153 encodes the UVW coordinate information to generate a UVW coordinate bitstream.

[0174] In step S144, the active flag encoding unit 154 encodes the active flag to generate an active flag bit stream.

[0175] When the processing in step S144 is completed, the proxy voxel grid encoding processing is completed, and the processing returns to FIG.

[0176] The processing order of steps S141 to S144 is arbitrary and is not limited to the above example.

[0177] <Flow of UVW Coordinate Encoding Process> Next, an example of the flow of the UVW coordinate encoding process executed in step S143 of FIG. 16 will be described with reference to the flowchart of FIG.

[0178] When the UVW coordinate encoding process starts, the differential encoding unit 164 initializes the UVW coordinate bitstream in step S151.

[0179] In step S152, the prediction parameter initialization unit 161 initializes the prediction parameters.

[0180] In step S153, the coordinate supply unit 162 processes the XYZ coordinates and UVW coordinates of the next vertex.

[0181] In step S154, the predicted value derivation unit 163 derives a predicted value using the XYZ coordinates and the prediction parameters.

[0182] In step S155, the differential encoding unit 164 derives a differential value using the UVW coordinates and the predicted value.

[0183] In step S156, the differential encoding unit 164 encodes the differential value and adds it to the UVW coordinate bitstream.

[0184] In step S157, the prediction parameter update unit 165 updates the prediction parameters using the difference value.

[0185] In step S158, the differential encoding unit 164 determines whether all vertices have been processed. If it is determined that an unprocessed vertex exists, the process returns to step S153. The processes of steps S153 to S158 are performed for each vertex, and if it is determined in step S158 that all vertices have been processed, the process proceeds to step S159.

[0186] In step S159, the differential encoding unit 164 outputs a UVW coordinate bitstream.

[0187] When the process of step S159 ends, the UVW coordinate encoding process ends, and the process returns to FIG.

[0188] <Flow of Proxy Voxel Grid Decoding Process> Next, an example of the flow of the proxy voxel grid decoding process executed in step S112 of FIG. 13 will be described with reference to the flowchart of FIG.

[0189] When the proxy voxel grid decoding process starts, the transformation matrix decoding unit 171 of the proxy voxel grid decoding unit 131 decodes the transformation matrix bit stream and generates a transformation matrix in step S161.

[0190] In step S162, the occupied cell index decoder 172 decodes the occupied cell index bit stream to generate occupied cell index information.

[0191] In step S163, the UVW coordinate decoding unit 173 encodes the UVW coordinate bit stream using the generated transformation matrix and occupied cell index information to generate UVW coordinate information.

[0192] In step S164, the active flag decoder 174 decodes the active flag bit stream to generate a set of active flags.

[0193] In step S165, the proxy voxel grid construction unit 175 constructs a proxy voxel grid using the generated transformation matrix, occupied cell index information, UVW coordinate information, and a set of active flags.

[0194] When the process of step S165 ends, the proxy voxel grid decoding process ends, and the process returns to FIG.

[0195] <Flow of UVW Coordinate Decoding Process> Next, an example of the flow of the UVW coordinate decoding process executed in step S163 of FIG. 18 will be described with reference to the flowchart of FIG.

[0196] When the UVW coordinate decoding process starts, in step S171, the prediction parameter initialization unit 181 initializes the prediction parameters.

[0197] In step S172, the XYZ coordinate derivation unit 182 processes the XYZ coordinates of the next vertex.

[0198] In step S173, the predicted value derivation unit 183 derives a predicted value using the XYZ coordinates and the prediction parameters.

[0199] In step S174, the differential decoding unit 184 decodes the UVW coordinate bitstream and derives the difference value between the UVW coordinate and its predicted value.

[0200] In step S175, the predicted value adding unit 185 adds the derived predicted value to the derived difference value to derive UVW coordinates.

[0201] In step S176, the prediction parameter update unit 186 updates the prediction parameters using the difference value and the XYZ coordinates.

[0202] In step S177, the predicted value adding unit 185 determines whether all vertices have been processed. If it is determined that an unprocessed vertex exists, the process returns to step S172. The processes of steps S172 to S177 are performed for each vertex, and if it is determined in step S177 that all vertices have been processed, the process proceeds to step S178.

[0203] In step S178, the predicted value adding unit 185 outputs the derived UVW coordinate information.

[0204] When the process of step S178 ends, the UVW coordinate decoding process ends, and the process returns to FIG.

[0205] <Flow of Rendering Process> Next, an example of the flow of the rendering process executed in step S113 of FIG. 13 will be described with reference to the flowchart of FIG.

[0206] When the rendering process starts, the rendering unit 132 derives UVW coordinates of sample points in step S181.

[0207] In step S182, the rendering unit 132 derives the line of sight direction after the rotation.

[0208] In step S183, the rendering unit 132 inputs the UVW coordinates of the sample points and the line of sight direction after the store opens into a neural network (neural luminance field) and generates a rendering image.

[0209] When the process of step S183 ends, the rendering process ends and the process returns to FIG.

[0210] By performing each process as described above, the information processing system 100 (the terminal device 111 and the server 112) can apply the present technology described in <3. Application of Voxel Grid>, thereby suppressing an increase in the amount of proxy data of the neural luminance field. Therefore, an increase in the processing load of generating, editing, encoding, decoding, rendering, and the like of proxy data can be suppressed. Furthermore, when transmitting proxy data between a server and a client, for example, an increase in the transmission volume can be suppressed, and an increase in the transmission-related load can also be suppressed. Furthermore, by suppressing these increases in load, an increase in cost and an increase in processing time can be suppressed. Furthermore, by applying a proxy voxel grid, a decrease in coding efficiency can be suppressed compared to the case of a proxy tetrahedral mesh.

[0211] <5. Second Embodiment> <Information Processing System> The present technology can be applied to an information processing system with any configuration, and is not limited to the configuration example of FIG. 8 . For example, generation of a proxy voxel grid may be performed in a server. FIG. 21 is a block diagram showing another configuration example of an information processing system to which the present technology described above in <3. Application of Voxel Grid> is applied. The information processing system 300 shown in FIG. 21 is a system similar to the information processing system 100 of FIG. 8 . However, the configurations of the terminal device and the server (the processes executed by each) differ from those of the example of FIG. 8 .

[0212] 21 , the information processing system 300 includes a terminal device 311 and a server 312. The terminal device 311 and the server 312 are connected to each other via a network (not shown) so that they can communicate with each other. For example, the terminal device 311 operates as a client of the server 312.

[0213] The server 312 generates a proxy voxel grid using the voxel grid as proxy data corresponding to the neural luminance field (NeRF data), encodes the proxy voxel grid, and supplies the bit stream to the terminal device 311.

[0214] The terminal device 311 decodes the supplied bitstream to generate a proxy voxel grid, edits the decoded proxy voxel grid, encodes the edited proxy voxel grid, and supplies the bitstream to the server 312.

[0215] The server 112 decodes the supplied bitstream to generate a proxy voxel grid, and performs rendering using the decoded proxy voxel grid to generate a rendered image.

[0216] The configurations of the terminal device 311 and the server 312 are each arbitrary. For example, the information processing system 300 may be provided with a plurality of terminal devices 311 and a plurality of servers 312. The number of terminal devices 311 and the number of servers 312 do not have to be the same. The terminal device 311 may be configured with a plurality of information processing devices. The server 312 may be configured with a plurality of information processing devices. For example, the server 312 may be configured as a cloud server, the device configuration of which is not limited.

[0217] 21 , the terminal device 311 includes a proxy voxel grid decoding unit 331, a proxy voxel grid editing unit 332, and a proxy voxel grid encoding unit 333. The server 312 includes a proxy voxel grid generation unit 321, a proxy voxel grid encoding unit 322, a proxy voxel grid decoding unit 323, and a rendering unit 324.

[0218] The proxy voxel grid generation unit 321 has the same configuration and performs the same processing as the proxy voxel grid generation unit 121 in Fig. 8. The proxy voxel grid generation unit 321 may supply the generated proxy voxel grid to a proxy voxel grid encoding unit 322.

[0219] The proxy voxel grid encoding unit 322 has the same configuration as the proxy voxel grid encoding unit 123 in Fig. 8 and performs the same processing. However, the proxy voxel grid encoding unit 322 encodes the proxy voxel grid before editing. The proxy voxel grid encoding unit 322 may supply the bit stream of the proxy voxel grid to (the proxy voxel grid decoding unit 331 of) the terminal device 311.

[0220] The proxy voxel grid decoding unit 331 has the same configuration as the proxy voxel grid decoding unit 131 in Fig. 8 and performs the same processing. The proxy voxel grid decoding unit 331 may supply the decoded proxy voxel grid generated by decoding the bit stream to the proxy voxel grid editing unit 332.

[0221] The proxy voxel grid editing unit 332 has the same configuration and performs the same processing as the proxy voxel grid editing unit 122 in Fig. 8. The proxy voxel grid editing unit 332 may supply the edited proxy voxel grid to the proxy voxel grid encoding unit 333.

[0222] The proxy voxel grid encoding unit 333 has the same configuration and performs the same processing as the proxy voxel grid encoding unit 322. However, the proxy voxel grid encoding unit 333 encodes the edited proxy voxel grid. The proxy voxel grid encoding unit 333 may supply the bit stream of the generated proxy voxel grid to the server 312 (the proxy voxel grid decoding unit 323 thereof).

[0223] The proxy voxel grid decoding unit 323 has the same configuration and performs the same processing as the proxy voxel grid decoding unit 331. The proxy voxel grid decoding unit 323 may supply the decoded “edited proxy voxel grid” generated by decoding the bitstream to the rendering unit 324.

[0224] The rendering unit 324 has the same configuration and performs the same processing as the rendering unit 132 in Fig. 8. The rendering unit 324 may output a rendered image generated using the decoded "edited proxy voxel grid" and the neural luminance field (NeRF data).

[0225] Even when the information processing system 300 has such a configuration, the present technology described in <3. Application of Voxel Grid> can be applied. Therefore, the information processing system 300 (terminal device 311 and server 312) can suppress an increase in the amount of proxy data of the neural luminance field. Therefore, an increase in the processing load of generating, editing, encoding, decoding, rendering, etc. of the proxy data can be suppressed. Furthermore, when transmitting proxy data between a server and a client, for example, an increase in the transmission amount can be suppressed, and an increase in the transmission-related load can also be suppressed. Furthermore, by suppressing these increases in load, an increase in cost and an increase in processing time can be suppressed. Furthermore, by applying a proxy voxel grid, a decrease in coding efficiency can be suppressed more than in the case of a proxy tetrahedral mesh.

[0226] <Flow of Neural Luminance Field Editing Process> An example of the flow of neural luminance field editing process executed by the information processing system 300 will be described with reference to the flowchart of FIG.

[0227] When the process starts, the proxy voxel grid generator 321 of the server 312 generates a proxy voxel grid corresponding to the NeRF data in step S311.

[0228] In step S312, the proxy voxel grid encoding unit 322 encodes the proxy voxel grid to generate a bitstream.

[0229] In step S313, the proxy voxel grid encoding unit 322 supplies the generated proxy voxel grid bit stream to the terminal device 311.

[0230] In step S301, the proxy voxel grid decoding unit 331 of the terminal device 311 acquires the bitstream.

[0231] In step S302, the proxy voxel grid decoding unit 331 decodes the bitstream and generates a decoded proxy voxel grid.

[0232] In step S303, the proxy voxel grid editing unit 332 edits the proxy voxel grid.

[0233] In step S304, the proxy voxel grid encoding unit 333 encodes the edited proxy voxel grid to generate a bitstream.

[0234] In step S305, the proxy voxel grid encoding unit 333 supplies the generated proxy voxel grid bit stream to the server 312.

[0235] In step S314, the proxy voxel grid decoding unit 323 of the server 312 acquires the bitstream.

[0236] In step S315, the proxy voxel grid decoding unit 323 decodes the bit stream and generates a decoded proxy voxel grid.

[0237] In step S316, the rendering unit 324 performs rendering using the decoded proxy voxel grid to generate a rendered image.

[0238] When the process of step S316 is completed, the neural luminance field editing process is completed.

[0239] Each step of the process is executed in the same manner as in the first embodiment.

[0240] Even when the information processing system 300 performs each process in this manner, the present technology described in <3. Application of Voxel Grid> can be applied. Therefore, the information processing system 300 (the terminal device 311 and the server 312) can suppress an increase in the amount of proxy data of the neural luminance field. Therefore, an increase in the processing load of generating, editing, encoding, decoding, rendering, etc. of the proxy data can be suppressed. Furthermore, when transmitting proxy data between a server and a client, for example, an increase in the transmission amount can be suppressed, and an increase in the transmission-related load can also be suppressed. Furthermore, by suppressing these increases in load, an increase in cost and an increase in processing time can be suppressed. Furthermore, by applying a proxy voxel grid, a decrease in coding efficiency can be suppressed compared to the case of a proxy tetrahedral mesh.

[0241] 6. Third Embodiment Application of Attributes Although the geometry of the proxy data has been described above, the proxy data may have attributes. That is, attributes (also referred to as cell attributes) may be added to voxels (occupied cells) of the proxy voxel grid. This attribute may contain any information. For example, the average density value (floating-point value) within the cell may be included in this attribute. Furthermore, the average color within the cell (a three-dimensional vector in the case of RGB color) may be included in this attribute. Of course, information other than these examples may also be used.

[0242] In this case, the information processing system 100 basically has the same configuration as the example in Fig. 8. However, each processing unit has a function capable of processing this attribute.

[0243] <Proxy Voxel Grid Encoding Unit> An example of the main configuration of the proxy voxel grid encoding unit 123 in this case is shown in Fig. 23. As shown in Fig. 23, in this case, the proxy voxel grid encoding unit 123 has a cell attribute encoding unit 451 in addition to the configuration in Fig. 9.

[0244] The cell attribute encoding unit 451 encodes attributes (cell attributes) attached to each occupied cell constituting the proxy voxel grid and generates a bit stream (also called a cell attribute bit stream). The cell attribute encoding unit 451 receives the proxy voxel grid, encodes the attributes of all occupied cells using point cloud attribute encoding, and outputs the cell attribute bit stream. For point cloud attribute encoding, for example, RAHT (Region Adaptive Hierarchical Transform) encoding is used.

[0245] <Proxy Voxel Grid Decoding Unit> An example of the main configuration of the proxy voxel grid decoding unit 131 in this case is shown in Fig. 24. As shown in Fig. 24, in this case, the proxy voxel grid decoding unit 131 has a cell attribute decoding unit 471 in addition to the configuration in Fig. 11.

[0246] The cell attribute decoding unit 471 decodes the cell attribute bitstream and generates attributes (cell attributes) attached to each occupied cell constituting the proxy voxel grid. The cell attribute decoding unit 471 decodes a cell attribute set from the cell attribute bitstream using an occupied cell index set. A point cloud attribute decoding method is used for the decoding. For example, RAHT (Region Adaptive Hierarchical Transform) decoding is used as the point cloud attribute decoding method.

[0247] Even with the above configuration, the information processing system 100 (terminal device 111 and server 112) can suppress an increase in the amount of proxy data for the neural luminance field. Therefore, an increase in the processing load of generating, editing, encoding, decoding, rendering, and the like of proxy data can be suppressed. Furthermore, when transmitting proxy data between a server and a client, for example, an increase in the transmission volume can be suppressed, and an increase in the transmission-related load can also be suppressed. Furthermore, by suppressing these increases in load, an increase in cost and an increase in processing time can be suppressed. Furthermore, by applying a proxy voxel grid, a decrease in coding efficiency can be suppressed compared to the case of a proxy tetrahedral mesh.

[0248] The information processing system 300 may also be similarly expanded to have a function capable of processing cell attributes.

[0249] <Flow of Proxy Voxel Grid Generation Processing> An example of the flow of proxy voxel grid generation processing in this case will be described with reference to the flowchart of FIG.

[0250] The processes in steps S421 to S425 are executed in the same manner as the processes in steps S121 to S125 (FIG. 14).

[0251] In step S426, the proxy voxel grid generator 121 generates cell attributes.

[0252] When the processing in step S426 ends, the proxy voxel grid generation processing ends, and the processing returns to FIG.

[0253] <Flow of Proxy Voxel Grid Editing Process> An example of the flow of the proxy voxel grid editing process in this case will be described with reference to the flowchart in FIG.

[0254] The processes in steps S431 to S434 are executed in the same manner as the processes in steps S131 to S134 (FIG. 15).

[0255] In step S435, the proxy voxel grid editing unit 122 edits the cell attributes.

[0256] When the processing in step S435 ends, the proxy voxel grid editing processing ends, and the processing returns to FIG.

[0257] <Flow of Proxy Voxel Grid Encoding Process> An example of the flow of the proxy voxel grid encoding process in this case will be described with reference to the flowchart in FIG.

[0258] The processes in steps S441 to S444 are executed in the same manner as the processes in steps S141 to S144 (FIG. 16).

[0259] In step S445, the cell attribute encoding unit 451 encodes the cell attribute.

[0260] When the processing in step S445 is completed, the proxy voxel grid encoding processing is completed, and the processing returns to FIG.

[0261] <Flow of Proxy Voxel Grid Decoding Process> An example of the flow of the proxy voxel grid decoding process in this case will be described with reference to the flowchart in FIG.

[0262] The processes in steps S461 to S464 are executed in the same manner as the processes in steps S161 to S164 (FIG. 18).

[0263] In step S465, the cell attribute decoding unit 471 decodes the cell attribute bit stream to generate a cell attribute.

[0264] The process of step S466 is executed in the same manner as the process of step S165 (FIG. 18).

[0265] When the processing in step S466 ends, the proxy voxel grid decoding processing ends, and the processing returns to FIG.

[0266] Even when performing each process as described above, the information processing system 100 (the terminal device 111 and the server 112) can suppress an increase in the amount of proxy data for the neural luminance field. Therefore, an increase in the processing load of generating, editing, encoding, decoding, rendering, and the like of proxy data can be suppressed. Furthermore, when transmitting proxy data between a server and a client, for example, an increase in the transmission volume can be suppressed, and an increase in the transmission-related load can also be suppressed. Furthermore, by suppressing these increases in load, an increase in cost and an increase in processing time can be suppressed. Furthermore, by applying a proxy voxel grid, a decrease in coding efficiency can be suppressed compared to the case of a proxy tetrahedral mesh.

[0267] The neural luminance field editing process executed by the information processing system 300 may also be similarly extended to include a function capable of processing cell attributes.

[0268] <7. Supplementary Notes> <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0269] FIG. 29 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0270] In a computer 900 shown in FIG. 29, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0271] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0272] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0273] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0274] The program executed by the computer can be applied by recording it on, for example, a removable medium 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable medium 921 into the drive 915.

[0275] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0276] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0277] <Application of the Present Technology> The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0278] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).

[0279] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0280] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0281] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.

[0282] <Others> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in a bitstream, but also in a bitstream that includes differential information of the identification information relative to certain reference information. Therefore, in this specification, "flag" and "identification information" encompass not only the information itself, but also differential information relative to the reference information.

[0283] Furthermore, various types of information (e.g., metadata) related to the coded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, the associated pieces of data may be combined into one piece of data or may be separate pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Furthermore, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer not to the entire data, but to only part of the data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0284] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0285] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0286] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0287] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0288] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0289] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0290] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0291] The present technology can also be configured as follows: (1) An information processing device comprising: a proxy voxel grid generation unit that generates a proxy voxel grid using a voxel grid as proxy data for editing a neural luminance field that is a neural network that approximates a luminance field function that represents a spatial scene including an object. (2) The information processing device according to (1), wherein the proxy voxel grid generation unit generates, as the proxy voxel grid, occupied cell index information constituted by indexes of all occupied cells and UVW coordinate information constituted by UVW coordinates indicating initial values ​​of coordinates of vertices of all the occupied cells, and the occupied cells are voxels that constitute the voxel grid and have geometry of the object present therein. (3) The information processing device according to (2), wherein the proxy voxel grid generation unit generates the voxel grid by dividing the space in voxel units, determines for each of the voxels constituting the voxel grid whether it is an occupied cell or an unoccupied cell, and generates the occupied cell index information by octree-structuring the indexes of the occupied cells, and the unoccupied cells are voxels constituting the voxel grid that do not have geometry of the object inside. (4) The information processing device according to (3), wherein the proxy voxel grid generation unit derives a density of the center of the voxel based on the neural luminance field, and determines the voxel to be an occupied cell if the density is greater than a predetermined threshold, and determines the voxel to be an unoccupied cell if the density is not greater than the threshold. (5) The information processing device according to any one of (2) to (4), wherein the proxy voxel grid generation unit further generates active flag information as the proxy voxel grid, the active flag being configured by an active flag set for all the occupied cells, and the active flag is flag information indicating the presence or absence of geometry of the object after editing of the occupied cells.(6) The information processing device according to any one of (2) to (5), wherein the proxy voxel grid generation unit further generates, as the proxy voxel grid, cell attribute information configured by attributes to be added to each of the occupied cells. (7) An information processing method, wherein a proxy voxel grid is generated using a voxel grid as proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object.

[0292] (11) An information processing device comprising: a proxy voxel grid editing unit that edits a proxy voxel grid that is proxy data for editing a neural luminance field that is a neural network that approximates a luminance field function that represents a spatial scene including an object, wherein the proxy voxel grid is the proxy data using a voxel grid. (12) The information processing device according to (11), wherein the proxy voxel grid editing unit generates or edits a transformation matrix that indicates a correspondence between an index of an occupied cell and a position of the occupied cell after editing, wherein the occupied cell is a voxel that constitutes the voxel grid and has geometry of the object present inside. (13) The information processing device according to (11) or (12), wherein the proxy voxel grid editing unit edits coordinates of vertices of the occupied cells that constitute the voxel grid. (14) The information processing device according to any of (11) to (13), wherein the proxy voxel grid editing unit edits an active flag, which is flag information indicating the presence or absence of geometry of the object after editing of an occupied cell, and the occupied cell is a voxel constituting the voxel grid, inside which the geometry of the object exists. (15) The information processing device according to any of (11) to (14), wherein the proxy voxel grid editing unit edits an attribute to be added to an occupied cell, and the occupied cell is a voxel constituting the voxel grid, inside which the geometry of the object exists. (16) An information processing method, wherein a proxy voxel grid is edited, which is proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object, and the proxy voxel grid is the proxy data using a voxel grid.

[0293] (21) An information processing device comprising: a proxy voxel grid encoding unit that encodes a proxy voxel grid that is proxy data for editing a neural luminance field that is a neural network that approximates a luminance field function that represents a spatial scene including an object, wherein the proxy voxel grid is the proxy data using a voxel grid. (22) The information processing device according to (21), wherein the proxy voxel grid encoding unit comprises: an occupied cell index encoding unit that encodes occupied cell index information constituted by indices of all occupied cells included in the proxy voxel grid; and a UVW coordinate encoding unit that encodes UVW coordinate information constituted by UVW coordinates indicating initial values ​​of coordinates of vertices of all the occupied cells included in the proxy voxel grid, wherein the occupied cells are voxels that constitute the voxel grid and inside which the geometry of the object exists. (23) The information processing device according to (22), wherein the occupied cell index encoding unit losslessly encodes the occupied cell index information in which the indexes of the occupied cells are organized into an octree. (24) The information processing device according to (22) or (23), wherein the UVW coordinate encoding unit encodes a difference value between a predicted value and an actual value of the UVW coordinate. (25) The information processing device according to (24), wherein the UVW coordinate encoding unit derives the predicted value using a post-edit coordinate of the vertex of the occupied cell and a prediction parameter. (26) The information processing device according to (25), wherein the UVW coordinate encoding unit updates the prediction parameter for each vertex using the difference value. (27) The information processing device according to any of (22) to (26), wherein the UVW coordinate encoding unit encodes the UVW coordinates of each vertex in Morton order. (28) The information processing device according to any of (22) to (27), wherein the proxy voxel grid encoding unit further includes an active flag encoding unit that encodes active flag information configured by an active flag that is set for all the occupied cells included in the proxy voxel grid, and the active flag is flag information that indicates whether or not geometry of the object exists after editing of the occupied cell.(29) The information processing device according to any of (22) to (28), wherein the proxy voxel grid encoding unit further comprises a transformation matrix encoding unit that encodes a transformation matrix indicating a correspondence between the index of the occupied cell and a position of the occupied cell after editing. (30) The information processing device according to any of (22) to (29), wherein the proxy voxel grid encoding unit further comprises a cell attribute encoding unit that encodes cell attribute information constituted by attributes added to each of the occupied cells included in the proxy voxel grid. (31) An information processing method, comprising: encoding a proxy voxel grid that is proxy data for editing a neural luminance field that is a neural network that approximates a luminance field function that represents a spatial scene including an object, wherein the proxy voxel grid is the proxy data using a voxel grid.

[0294] (41) An information processing device comprising: a proxy voxel grid decoding unit that decodes a bit stream and generates a proxy voxel grid that is proxy data for editing a neural luminance field that is a neural network that approximates a luminance field function that represents a spatial scene including an object, wherein the proxy voxel grid is the proxy data using a voxel grid. (42) The proxy voxel grid decoding unit comprises: an occupied cell index decoding unit that decodes the bit stream and generates occupied cell index information composed of indexes of all occupied cells included in the proxy voxel grid; and a UVW coordinate decoding unit that decodes the bit stream and generates UVW coordinate information composed of UVW coordinates that indicate initial values ​​of coordinates of vertices of all the occupied cells included in the proxy voxel grid, wherein the occupied cells are voxels that constitute the voxel grid and inside which the geometry of the object exists. (43) The information processing device according to (42), wherein the occupied cell index decoding unit losslessly decodes the bitstream and generates the occupied cell index information in which the indexes of the occupied cells are organized into an octree. (44) The information processing device according to (42) or (43), wherein the UVW coordinate decoding unit decodes the bitstream to generate a difference value between a predicted value and an actual value of the UVW coordinate, derives the predicted value, and derives the UVW coordinate by adding the derived predicted value to the difference value. (45) The information processing device according to (44), wherein the UVW coordinate decoding unit derives the predicted value using post-edit coordinates of the vertices of the occupied cell and prediction parameters. (46) The information processing device according to (45), wherein the UVW coordinate decoding unit derives post-edit coordinates of the vertices of the occupied cell using the occupied cell index information and a transformation matrix, wherein the transformation matrix indicates a correspondence relationship between the index of the occupied cell and the post-edit position of the occupied cell. (47) The information processing device according to (45) or (46), wherein the UVW coordinate decoding unit updates the prediction parameters for each vertex by using the edited coordinates of the vertex of the occupied cell and the difference value.(48) The information processing device according to any of (42) to (47), wherein the proxy voxel grid decoding unit further comprises an active flag decoding unit that decodes the bit stream and generates active flag information constituted by active flags that are set for all of the occupied cells included in the proxy voxel grid, the active flag being flag information indicating the presence or absence of geometry of the object after editing of the occupied cells. (49) The information processing device according to any of (42) to (48), wherein the proxy voxel grid decoding unit further comprises a transformation matrix decoding unit that decodes the bit stream and generates a transformation matrix indicating a correspondence between the index of the occupied cell and a position of the occupied cell after editing. (50) The information processing device according to any of (42) to (49), wherein the proxy voxel grid decoding unit further comprises a cell attribute decoding unit that decodes the bit stream and generates cell attribute information constituted by attributes to be added to each of the occupied cells included in the proxy voxel grid. (51) An information processing method, comprising: decoding a bitstream; and generating a proxy voxel grid, which is proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object; wherein the proxy voxel grid is the proxy data using a voxel grid.

[0295] (61) An information processing device comprising: a rendering unit that performs rendering using a proxy voxel grid that is proxy data for editing a neural luminance field that is a neural network that approximates a luminance field function that represents a spatial scene including an object, wherein the proxy voxel grid is the proxy data using the voxel grid. (62) The information processing device according to (61), wherein the rendering unit generates a rendered image using UVW coordinates of sample points, a gaze direction after rotation, and the neural luminance field, wherein the UVW coordinates of the sample points indicate initial values ​​of coordinates of the sample points. (63) The information processing device according to (62), wherein the rendering unit searches for an occupied cell that contains the sample point, derives the UVW coordinates of the sample point using coordinates of the vertices of the occupied cell, and derives the gaze direction after rotation using a rotation matrix of the sample point, wherein the occupied cell is a voxel that contains the geometry of the object among voxels that make up the voxel grid. (64) An information processing method, comprising: rendering using a proxy voxel grid that is proxy data for editing a neural luminance field that is a neural network that approximates a luminance field function that represents a spatial scene including an object; and the proxy voxel grid is the proxy data using a voxel grid.

[0296] (71) Proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object, comprising: occupied cell index information composed of indices of all occupied cells; and UVW coordinate information composed of UVW coordinates indicating initial values ​​of coordinates of vertices of all the occupied cells, wherein the occupied cells are voxels that constitute a voxel grid and within which geometry of the object exists. (72) The proxy voxel grid according to (71), further comprising active flag information composed of active flags that are set for all the occupied cells, wherein the active flags are flag information indicating whether geometry of the object exists after editing of the occupied cells. (73) The proxy voxel grid according to (71) or (72), further comprising a transformation matrix indicating a correspondence between the index of the occupied cell and a position of the occupied cell after editing. (74) The proxy voxel grid according to any one of (71) to (73), further comprising cell attribute information configured by attributes added to the occupied cells.

[0297] 100 Information processing system, 111 Terminal device, 112 Server, 121 Proxy voxel grid generation unit, 122 Proxy voxel grid editing unit, 123 Proxy voxel grid encoding unit, 131 Proxy voxel grid decoding unit, 132 Rendering unit, 151 Transformation matrix encoding unit, 152 Occupied cell index encoding unit, 153 UVW coordinate encoding unit, 154 Active flag encoding unit, 161 Prediction parameter initialization unit, 162 Coordinate supply unit, 163 Prediction value derivation unit, 164 Difference encoding unit, 165 Prediction parameter update unit, 171 Transformation matrix decoding unit, 172 Occupied cell index decoding unit, 173 UVW coordinate decoding unit, 174 Active flag decoding unit, 175 Proxy voxel grid construction unit, 181 Prediction parameter initialization unit, 182 XYZ coordinate derivation unit, 183 predicted value derivation unit, 184 difference decoding unit, 185 predicted value addition unit, 186 predicted parameter update unit, 300 information processing system, 311 terminal device, 312 server, 321 proxy voxel grid generation unit, 322 proxy voxel grid encoding unit, 323 proxy voxel grid decoding unit, 324 rendering unit, 331 proxy voxel grid decoding unit, 332 proxy voxel grid editing unit, 333 proxy voxel grid encoding unit, 451 cell attribute encoding unit, 471 cell attribute decoding unit, 900 computer

Claims

1. An information processing device having a proxy voxel grid generation unit that generates a proxy voxel grid using a voxel grid as proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object.

2. The information processing device according to claim 1, wherein the proxy voxel grid generation unit generates, as the proxy voxel grid, occupied cell index information consisting of indices of all occupied cells and UVW coordinate information consisting of UVW coordinates indicating initial values ​​of coordinates of vertices of all the occupied cells, and the occupied cells are those voxels that make up the voxel grid and inside which geometry of the object exists.

3. The information processing device according to claim 2, wherein the proxy voxel grid generation unit generates the voxel grid by dividing the space into voxels, determines whether each of the voxels constituting the voxel grid is an occupied cell or an unoccupied cell, and generates the occupied cell index information by organizing the indexes of the occupied cells into an octree, and the unoccupied cells are those voxels constituting the voxel grid that do not have the geometry of the object inside.

4. The information processing device according to claim 3, wherein the proxy voxel grid generation unit derives a density at the center of the voxel based on the neural luminance field, and determines the voxel to be an occupied cell if the density is greater than a predetermined threshold, and determines the voxel to be an unoccupied cell if the density is not greater than the threshold.

5. The information processing device according to claim 2, wherein the proxy voxel grid generation unit further generates active flag information as the proxy voxel grid, the active flag being configured by an active flag set for all the occupied cells, the active flag being flag information indicating the presence or absence of geometry of the object after editing of the occupied cells.

6. The information processing device according to claim 1, further comprising a proxy voxel grid encoding unit that encodes the proxy voxel grid.

7. The information processing device according to claim 6, wherein the proxy voxel grid encoding unit comprises: an occupied cell index encoding unit that encodes occupied cell index information composed of indices of all occupied cells included in the proxy voxel grid; and a UVW coordinate encoding unit that encodes UVW coordinate information composed of UVW coordinates indicating initial values ​​of coordinates of vertices of all occupied cells included in the proxy voxel grid, and the occupied cells are those voxels that constitute the voxel grid and inside which geometry of the object exists.

8. An information processing method for generating a proxy voxel grid using a voxel grid as proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object.

9. An information processing device comprising: a rendering unit that performs rendering using a proxy voxel grid, which is proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object; and the proxy voxel grid is the proxy data using a voxel grid.

10. The information processing device according to claim 9, wherein the rendering unit generates a rendering image using the UVW coordinates of the sample points, the gaze direction after rotation, and the neural luminance field, and the UVW coordinates of the sample points indicate the initial values ​​of the coordinates of the sample points.

11. The information processing device according to claim 10, wherein the rendering unit searches for an occupied cell that contains the sample point, derives the UVW coordinates of the sample point using the coordinates of the vertices of the occupied cell, and derives the viewing direction after the rotation using a rotation matrix of the sample point, and the occupied cell is a voxel that constitutes the voxel grid and has the geometry of the object inside.

12. The information processing device according to claim 9, further comprising a proxy voxel grid decoding unit that decodes a bitstream and generates the proxy voxel grid, wherein the rendering unit is configured to perform rendering using the generated proxy voxel grid.

13. The information processing device according to claim 12, wherein the proxy voxel grid decoding unit comprises: an occupied cell index decoding unit that decodes the bit stream and generates occupied cell index information consisting of indices of all occupied cells included in the proxy voxel grid; and a UVW coordinate decoding unit that decodes the bit stream and generates UVW coordinate information consisting of UVW coordinates indicating initial values ​​of coordinates of vertices of all occupied cells included in the proxy voxel grid; and the occupied cells are those voxels that make up the voxel grid and inside which geometry of the object exists.

14. The information processing device according to claim 13, wherein the occupied cell index decoding unit losslessly decodes the bit stream and generates the occupied cell index information in which the indexes of the occupied cells are organized into an octree.

15. The information processing device according to claim 13, wherein the UVW coordinate decoding unit decodes the bitstream to generate a difference value between the predicted value and the actual value of the UVW coordinate, derives the predicted value, and derives the UVW coordinate by adding the derived predicted value to the difference value.

16. An information processing method, comprising: rendering using a proxy voxel grid, which is proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object; and the proxy voxel grid, which is proxy data using a voxel grid.

17. Proxy data for editing a neural luminance field, which is a neural network that approximates a luminance field function that represents a spatial scene including an object, comprising: occupied cell index information composed of indices of all occupied cells; and UVW coordinate information composed of UVW coordinates indicating initial values ​​of the coordinates of the vertices of all the occupied cells, wherein the occupied cells are voxels that make up the voxel grid and within which the geometry of the object exists.

18. The proxy voxel grid according to claim 17, further comprising active flag information configured by active flags set for all of the occupied cells, the active flags being flag information indicating whether or not the geometry of the object exists after editing of the occupied cells.

19. The proxy voxel grid of claim 17, further comprising a transformation matrix indicating a correspondence between the index of the occupied cell and the position of the occupied cell after editing.

20. The proxy voxel grid of claim 17, further comprising cell attribute information consisting of attributes attached to the occupied cells.

Images