Information processing apparatus and method and proxy voxel grid

Abstract

The present disclosure relates to an information processing apparatus and method capable of suppressing an increase in the amount of proxy data of a neural radiance field, and a proxy voxel grid. The proxy voxel grid is applied as proxy data for editing a neural radiance field, which is a neural network approximating a radiance field function representing a scene including a space of an object, and includes: occupancy cell index information including indices of all occupancy cells; and UVW coordinate information including UVW coordinates indicating initial values of vertex coordinates of all occupancy cells. A proxy voxel grid thereof is generated. A proxy voxel grid thereof is edited. A proxy voxel grid thereof is encoded. A bitstream is decoded, and a proxy voxel grid thereof is generated. A proxy voxel grid thereof is then used to perform rendering. The present disclosure is applicable to, for example, an information processing apparatus, an electronic device, an information processing method, a program, proxy data, and the like.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus and methods, and surrogate voxel grids, and more specifically to information processing apparatus and methods, and surrogate voxel grids, capable of suppressing the increase in the amount of surrogate data used for neural radiation fields. Background Technology

[0002] Typically, a neural radiance field (NeRF) exists, in which a neural network approximates the radiance field function. This function is obtained by modeling a complex 3D scene where the color of objects changes depending on their position and viewing direction due to reflection and other phenomena. Because NeRF is an implicit function representation, the scene information (the radiance field function information) is stored as parameters of the neural network in a black-box manner. Therefore, it is not possible to directly perform editing operations such as deleting or moving objects in the scene.

[0003] As a method for editing neural radiation fields, NeRFShop exists (see, for example, Non-Patent Document 1). NeRFShop edits NeRFs using proxy data for editing. In the case of NeRFShop, a tetrahedral mesh is applied as proxy data. By moving or deleting vertices of the tetrahedral mesh, a rendered image reflecting the editing is obtained. That is, the regions corresponding to the moved or deleted vertices are moved or deleted in the rendered image.

[0004] Reference List

[0005] Non-patent literature

[0006] Non-patent literature 1: CLEMENT JAMBON, BERNHARD KERBL, GEORGIOS KOPANAS, STAVROSDIOLATZIS, THOMAS LEIMKUHLER, GEORGE DRETTAKIS, "NeRFshop: InteractiveEditing of Neural Radiance Fields", Proceedings of the ACM on ComputerGraphics and Interactive Techniques, Volume 6, Number 1, https: / / www-sop.inria.fr / reves / basilic / 2023 / JKKDLD23 / nerfshop.pdf, May 2023. Summary of the Invention

[0007] The problem to be solved by the present invention

[0008] However, to perform editing operations with higher accuracy, it is necessary to have higher resolution proxy data. That is, a higher resolution tetrahedral mesh is required. By increasing the resolution of the tetrahedral mesh, the number of tetrahedra in the proxy data increases. Generally, the data volume of a tetrahedral mesh is roughly proportional to the number of tetrahedra. In other words, when using a tetrahedral mesh with sufficiently high resolution as proxy data for editing operations, there is a possibility of an increase in data volume.

[0009] This disclosure is made in view of the circumstances and its purpose is to suppress the increase in the amount of data used for proxy data of neural radiation fields.

[0010] Solution to the problem

[0011] According to one aspect of the present technology, an information processing apparatus includes a proxy voxel grid generation unit that generates a proxy voxel grid using a voxel grid as proxy data for editing a neural radiation field, the neural radiation field being a neural network that approximates a radiation field function representing a scene including an object in a space.

[0012] According to one aspect of the present technology, the information processing method is as follows: it includes generating a proxy voxel grid using a voxel grid as proxy data for editing a neural radiation field, which is a neural network that approximates a radiation field function representing a scene in a space including objects.

[0013] According to another aspect of the present technology, an information processing apparatus includes a rendering unit that performs rendering by using a proxy voxel grid, the proxy voxel grid being proxy data for editing a neural radiation field, the neural radiation field being a neural network that approximates a radiation field function representing a scene including an object space, wherein the proxy voxel grid is proxy data using a voxel grid.

[0014] According to another aspect of the present technology, the information processing method is as follows: it includes performing rendering by using a proxy voxel grid, the proxy voxel grid being proxy data for editing a neural radiation field, the neural radiation field being a neural network that approximates a radiation field function representing a scene including a space of objects, wherein the proxy voxel grid is proxy data using a voxel grid.

[0015] According to another aspect of the present technology, a proxy voxel grid is a proxy data for editing a neural radiation field, which is a neural network that approximates a radiation field function representing a scene including the space of objects. The proxy voxel grid includes: occupancy cell index information, which includes the index of all occupancy cells; and UVW coordinate information, which includes UVW coordinates indicating the initial values ​​of the vertex coordinates of all occupancy cells, wherein an occupancy cell is a voxel among the voxels constituting the voxel grid that contains the geometry of an object.

[0016] In an information processing apparatus and method according to one aspect of the present technology, a proxy voxel grid using a voxel grid is generated as proxy data for editing a neural radiation field, which is a neural network that approximates a radiation field function representing a scene in a space including objects.

[0017] In another aspect of the information processing apparatus and method according to the present technology, rendering is performed by using a proxy voxel mesh, which is proxy data for editing a neural radiation field, the neural radiation field being a neural network that approximates a radiation field function representing a scene including objects in a space.

[0018] According to another aspect of the present technology, the surrogate voxel mesh is surrogate data for editing a neural radiation field, the neural radiation field being a neural network that approximates a radiation field function representing a scene including the space of objects, and the surrogate voxel mesh includes: occupancy cell index information, which includes the indexes of all occupancy cells; and UVW coordinate information, which includes UVW coordinates indicating the initial values ​​of the vertex coordinates of all occupancy cells. Attached Figure Description

[0019] Figure 1 It is a diagram used to illustrate the nerve radiation field.

[0020] Figure 2 This is a diagram used to illustrate NeRFShop.

[0021] Figure 3 This is a diagram used to illustrate the edited rendering.

[0022] Figure 4 This is a diagram illustrating an example of proxy data used for neural radiation fields.

[0023] Figure 5 This is a diagram used to illustrate the proxy voxel mesh.

[0024] Figure 6 This is a diagram used to illustrate the proxy voxel mesh.

[0025] Figure 7 This is a diagram used to illustrate the proxy voxel mesh.

[0026] Figure 8 This is a block diagram illustrating a primary configuration example of an information processing system.

[0027] Figure 9 This is a block diagram showing a main configuration example of the proxy voxel grid encoding unit.

[0028] Figure 10 This is a block diagram showing a main configuration example of the UVW coordinate encoding section.

[0029] Figure 11 This is a block diagram showing a main configuration example of the proxy voxel grid decoding unit.

[0030] Figure 12 This is a block diagram showing a main configuration example of the UVW coordinate decoding unit.

[0031] Figure 13 This is a flowchart illustrating an example of the process for editing neural radiation fields.

[0032] Figure 14 This is a flowchart illustrating an example of the process for generating proxy voxel meshes.

[0033] Figure 15 This is a flowchart illustrating an example of the process for editing proxy voxel meshes.

[0034] Figure 16 This is a flowchart illustrating an example of the process for proxy voxel mesh encoding.

[0035] Figure 17 This is a flowchart illustrating an example of the UVW coordinate encoding process.

[0036] Figure 18 This is a flowchart illustrating an example of the proxy voxel mesh decoding process.

[0037] Figure 19 This is a flowchart illustrating an example of the UVW coordinate decoding process.

[0038] Figure 20 This is a flowchart illustrating an example of the rendering process.

[0039] Figure 21 This is a block diagram illustrating a primary configuration example of an information processing system.

[0040] Figure 22 This is a flowchart illustrating an example of the process for editing neural radiation fields.

[0041] Figure 23 This is a block diagram showing a main configuration example of the proxy voxel grid encoding unit.

[0042] Figure 24 This is a block diagram showing a main configuration example of the proxy voxel grid decoding unit.

[0043] Figure 25 This is a flowchart illustrating an example of the process for generating proxy voxel meshes.

[0044] Figure 26 This is a flowchart illustrating an example of the process for editing proxy voxel meshes.

[0045] Figure 27 This is a flowchart illustrating an example of the process for proxy voxel mesh encoding.

[0046] Figure 28 This is a flowchart illustrating an example of the proxy voxel mesh decoding process.

[0047] Figure 29 This is a block diagram illustrating a typical configuration example of a computer. Detailed Implementation

[0048] In the following text, a mode for implementing the present disclosure (hereinafter referred to as an implementation) will be described. Note that the description will be given in the following order.

[0049] 1. Supporting documents containing technical content and terminology, etc.

[0050] 2. Proximity data for neural radiation fields

[0051] 3. Application of Voxel Mesh

[0052] 4. First Implementation Method (Information Processing System)

[0053] 5. Second Implementation Method (Information Processing System)

[0054] 6. Third Implementation Method (Application of Attributes)

[0055] 7. Supplementary Explanation

[0056] <1. Supporting documents containing technical content and terminology, etc.>

[0057] In addition to the contents disclosed in the embodiments, the scope of disclosure in this technology also includes the contents disclosed in the following non-patent documents known at the time of submission, the contents of other documents cited in the following non-patent documents, etc.

[0058] Non-patent literature 1: (as described above)

[0059] Non-patent literature 2: S. Gumhold, S. Guthe and W. Strasser, "Tetrahedralmesh compression with the cut-border machine", Proceedings Visualization '99 (Cat. No.99CB37067), San Francisco, CA, USA, 1999, pp. 51-509, doi: 10.1109 / VISUAL.1999.809868.https: / / ieeexplore.ieee.org / document / 809868, 22 March 2018

[0060] In other words, the content described in the aforementioned non-patent documents, as well as the content of other documents cited in the aforementioned non-patent documents, are also used as the basis for determining the supporting requirements.

[0061] <2. Proxy data used for neural radiation fields>

[0062] <Neural Radiation Field>

[0063] The function F: (x, y, z, θ, φ) → (r, g, b, σ) that outputs density and color (r, g, b, σ) when given 3D coordinates (x, y, z) and a viewing direction (θ, φ) is called the radiation field function or radiation field. However, the density σ does not depend on the viewing direction (θ, φ), but is determined by the coordinates (x, y, z). This is because geometry depends only on position, not on the viewing direction. Complex 3D scenes where the color of an object's surface changes according to position and viewing direction due to reflection and other phenomena can be modeled as a radiation field function.

[0064] Since the radiation field function F is a high-dimensional function, it can be approximated by using a function Fθ with some compact parameters θ. Neural Radiation Field (NeRF) is a method that uses a neural network Fθ to approximate the radiation field function F. In this case, θ are the parameters of the neural network. Figure 1 As shown in A, the neural network Fθ takes three-dimensional coordinates and a gaze direction vector (x, y, z, θ, φ) as input and outputs a density and color vector (r, g, b, σ).

[0065] <Volume rendering method>

[0066] To render NeRF (or general radiation field) as an image, a rendering algorithm called volume rendering is used. An overview of how the color of a pixel in the rendered image is calculated using volume rendering is as follows.

[0067] 1: Generate rays from the viewpoint to the pixel position, such as Figure 1 As shown in B.

[0068] 2: Sampling points on the ray. As an example of a sampling method, there exists a method that selects points on the ray at equal intervals. Figure 1 In B, the black dots on the light rays indicate the sampling points.

[0069] 3: Input the three-dimensional coordinates (x, y, z) of each sampling point and the line-of-sight direction (θ, φ) of the light ray into the neural network Fθ to obtain the density and color (r, g, b, σ) of the sampling point.

[0070] 4: The weighted sum of the colors of all sampled points on the light source becomes the color of the final pixel. Figure 1 (The white point on the light ray in B). The weight of the sampling point is the visibility ratio. The visibility ratio indicates the proportion of how well a sampling point is visible from the viewpoint. For example, in the presence of an obstacle, the visibility ratio of the sampling point is close to 0. The visibility ratio is obtained by integrating the value obtained through a nonlinear transformation of density.

[0071] <nerfshop>

[0072] Because NeRF uses implicit function representation, scene information (radiation field information) is stored as parameters of the neural network in a black-box manner. That is, it is not obvious how a particular region of the scene relates to any parameter of the neural network. Therefore, it is impossible to directly perform editing operations such as deleting or moving objects in the scene.

[0073] Therefore, for example, as described in Non-Patent Document 1, a method for editing NeRF (NeRFShop) using proxy data for editing has been considered. In NeRFShop, a tetrahedral mesh (also known as a proxy tetrahedral mesh) is applied as proxy data. By moving or deleting vertices of the proxy tetrahedral mesh, the rendered image changes to reflect the editing. For example, when some vertices of the proxy tetrahedral mesh are deleted, the corresponding region in the rendered image is deleted.

[0074] A tetrahedral mesh is a tetrahedral set. For example... Figure 2 As shown in Figure A, each tetrahedron includes four vertices and connection information between them. In the proxy tetrahedron mesh, XYZ coordinates and UVW coordinates are added as attributes to each vertex. The XYZ coordinates represent the vertex position (x, y, z) in world space. The XYZ coordinates change when a vertex is moved due to editing, etc. The UVW coordinates represent the vertex position (u, v, w) before editing. The vertex position before editing is also called the initial vertex position. That is, in the initial state (the state before editing), for all vertices, (x, y, z) = (u, v, w). Even if the XYZ coordinates are moved through editing, the UVW coordinates do not change. The connection information represents the indices (i1, i2, i3, i4) of the tetrahedron's four vertices.

[0075] For example, such as Figure 2 As shown in B, a proxy tetrahedral mesh reproduces objects in a scene by using such a set of tetrahedra, and can be deformed, added to, or deleted based on editing operations. For example, when the number of vertices is V and the number of tetrahedra is T, the tetrahedral mesh includes V XYZ coordinates, V UVW coordinates, and T connection information.

[0076] <Rendering of Editing Results>

[0077] In rendering the results of the editing, a portion of the volume rendering method described above is modified to obtain a rendered image that reflects the editing operations on the proxy tetrahedral mesh. Specifically, the processing for calculating density and color based on the coordinates of the sampling points and the line-of-sight direction of the light rays is modified as follows.

[0078] [Before modification] The neural network Fθ inputs the XYZ coordinates (x, y, z) of the sampling points and the line-of-sight direction (θ, φ) of the light rays.

[0079] [After modification] The neural network Fθ inputs the UVW coordinates (u, v, w) of the sampling points and the rotated line of sight (θ', φ').

[0080] The UVW coordinates (u, v, w) of the sampling point and the rotated line of sight (θ', φ') are obtained as follows.

[0081] 1: Search for a tetrahedron containing the sampling point from the coordinates (x, y, z). If no tetrahedron is found, the process ends.

[0082] 2: Interpolate the UVW coordinates of the four vertices of the tetrahedron to obtain the UVW coordinates (u, v, w) of the sampling point.

[0083] 3: Obtain the rotation matrix in the tetrahedron to determine the line-of-sight direction (θ', φ') after rotation.

[0084] In other words, such as Figure 3 As shown, density and color are derived by inputting the initial state of the sampled points relative to the proxy tetrahedral mesh, corresponding to the position and viewing direction of the edited sampled points relative to the proxy tetrahedral mesh, into the neural network Fθ. Figure 3 In the diagram, the left side indicates objects in the XYZ coordinate space, and the right side indicates objects in the UVW coordinate space. By using UVW coordinates instead of XYZ coordinates during rendering, a rendering result that reflects the edits can be obtained. Note that in the initial state, since (u, v, w, θ', φ') = (x, y, z, θ, φ), a normal rendered image can be obtained.

[0085] Editing operations on tetrahedral meshes include, for example, deleting, copying, and moving the tetrahedron. By performing rendering as described above, when these editing operations are performed on the tetrahedron, the operations are reflected in the rendered image. For example, when deleting a tetrahedron, the region included within the tetrahedron is deleted in the rendered image. Furthermore, when copying a tetrahedron, the region included within the tetrahedron is copied in the rendered image. Additionally, when moving a tetrahedron, the region included within the tetrahedron is moved in the rendered image.

[0086] <Data volume of proxy tetrahedral mesh>

[0087] In recent years, there has been a demand for performing such editing operations with higher accuracy. However, to perform editing operations with higher accuracy, it is necessary to reduce the editable portion. That is, it is necessary to make the proxy data more clear. For example, in the case of applying a proxy tetrahedral mesh as proxy data, such as NeRFShop, the editable portion corresponds to one tetrahedron. In other words, to perform editing operations with higher accuracy, it is necessary to increase the number of tetrahedrons in the proxy tetrahedral mesh.

[0088] Typically, the data volume of a tetrahedral mesh includes a portion proportional to the number of vertices and the number of tetrahedra, as shown below.

[0089] Data volume of a tetrahedral mesh = (bit depth of XYZ coordinates) × 3 × (number of vertices) + (bit depth of UVW coordinates) × 3 × (number of vertices) + (bit depth of vertex indices) × 4 × (number of tetrahedra)

[0090] Therefore, when using tetrahedral meshes with sufficiently high resolution as proxy data for editing operations, there is a possibility that the amount of proxy data will increase as editing operations become more accurate. When the amount of proxy data increases, there is a possibility that the processing load for proxy data, such as generation, editing, encoding, decoding, and rendering, will increase. Furthermore, for example, when transferring proxy data between the server and client, there is a possibility that the amount of data transferred will increase, and the load associated with that transfer will also increase. With these increased loads, there is a possibility that costs or processing time will increase.

[0091] Furthermore, as a method for encoding tetrahedral meshes, there exists, for example, the method described in Non-Patent Document 2. In this method, all tetrahedra in the target tetrahedral mesh are searched, and all connection information, vertex coordinates, and vertex attributes are sequentially encoded. The encoding of vertex coordinates is performed using relatively simple differential encoding, for example, encoding the difference value with the centroid position of a nearby triangle. Furthermore, the encoding of vertex attributes is performed using differential encoding of the following values: these values ​​are obtained by linear interpolation of the attribute values ​​of four vertexes in a neighboring tetrahedron.

[0092] Therefore, it is necessary to encode connectivity information and vertex coordinates separately. Furthermore, in the encoding of vertex coordinates, since there are usually no vertices to be encoded on the selected triangle, the difference value to be encoded will not become 0. Additionally, in the encoding of vertex attributes, since prediction is performed using only four vertex attribute values, there is a possibility of numerical instability or an increase in the difference value to be encoded in the case of noisy or collapsed tetrahedrons. Therefore, it is difficult to achieve sufficiently high encoding efficiency when encoding surrogate tetrahedron meshes. In other words, even when encoding surrogate tetrahedron meshes, it is difficult to suppress the increase in data volume.

[0093] <3. Application of Voxel Meshes>

[0094] Method 1

[0095] Therefore, as Figure 4 As shown at the top of the table, the voxel grid is used as proxy data for editing the neural radiation field (Method 1).

[0096] For example, as proxy data for editing neural radiation fields (which are neural networks that approximate the radiation field function of a scene representing a space including objects), a proxy voxel mesh is applied. This proxy voxel mesh includes occupancy cell index information and UVW coordinate information. The occupancy cell index information includes the index of all occupancy cells, and the UVW coordinate information includes UVW coordinates indicating the initial values ​​of the vertex coordinates of all occupancy cells. Note that an occupancy cell is a voxel within the voxels that constitute the voxel mesh, containing the geometry of an object.

[0097] A voxel grid is a data structure that arranges voxels (cubic units) into a three-dimensional lattice. When the resolution of the voxel grid is N^3 (e.g., 512^3), each voxel corresponds one-to-one with an integer index (i, j, k) (i, j, k = 0, ..., N-1). In the case of a proxy voxel grid, the voxel corresponding to the maximum resolution of the voxel grid (that is, the voxel with the smallest size) is the editable part. Note that in this specification, superscripts, such as exponents, are indicated by "^". For example, in the case of "A^B", "B" indicates the superscript (e.g., exponent) appended to "A".

[0098] All voxels constituting the voxel mesh are divided into occupied cells and unoccupied cells. An occupied cell is a voxel whose internal geometry exists. In other words, an occupied cell includes coordinates within the voxel whose NeRF output density is equal to or greater than a threshold. An unoccupied cell is any cell other than an occupied cell. That is, an unoccupied cell is a voxel whose internal geometry does not exist, and which is not included in the voxel's internal coordinates where the NeRF output density is equal to or greater than a threshold.

[0099] Occupied cell index information includes information such as the index (also known as the occupied cell index) corresponding to each occupied cell. Empirically, occupied cells are far fewer than unoccupied cells. Therefore, as... Figure 5 As shown, the surrogate voxel mesh only requires a set of indices for occupied cells (excluding indices for unoccupied cells). The indices are then information that corresponds one-to-one with the occupied cells. Therefore, the occupied cell index information can reduce the amount of data compared to the connection information representing the connections between the vertices of a tetrahedral mesh. In other words, by applying the surrogate voxel mesh, the increase in the amount of surrogate data used for neural radiation fields can be suppressed compared to applying the surrogate tetrahedral mesh.

[0100] In addition, such as Figure 5 As shown in the example, since the distribution of occupancy cells empirically favors a portion of the space, the increase in data volume can be suppressed by converting the index set of occupancy cells into an octree. In this octree, voxels are obtained by recursively dividing the space into eight (two in each of the mutually perpendicular axes), and the index of each occupancy cell is represented by a tree structure according to the hierarchical structure of the voxels. By converting to an octree in this way, the indices of regions where all voxels are occupancy cells can be combined (represented by a single index). Therefore, the data volume of the index set can be reduced compared to representing all voxels in the space with the minimum voxel (the voxel with the highest resolution). In other words, by applying such a conversion to an octree, the increase in the amount of data for surrogate data used in neural radiation fields can be further suppressed. On the other hand, such a conversion to a tree structure has always been difficult in the case of surrogate tetrahedral meshes, and suppressing the increase in data volume has always been challenging.

[0101] Because voxels are rectangular, such as Figure 6 As shown, a occupancy cell therefore has eight vertices. Each vertex has UVW coordinates. The UVW coordinate information includes the initial values ​​(UVW coordinates) of the individual vertex coordinates of all occupancy cells in space. Note that vertices are managed independently of voxels in the UVW coordinate information. That is, for example, in the case of adjacent occupancy cells, they share common vertices. That is, in the case of overlapping vertices between occupancy cells, the vertex is treated as a single vertex. For example, in the case of A occupancy cells and B vertices in space, the number of UVW coordinates included in the UVW coordinate information is B instead of A × 8. Similar to the UVW coordinates of NeRFShop, the UVW coordinates are initialized at the initial positions of their vertices. Note that the values ​​of the vertex coordinates can be changed by editing the proxy voxel mesh (e.g., adding noise, etc.).

[0102] As described above, the surrogate voxel mesh includes occupied cell index information instead of the connection information of the surrogate tetrahedral mesh. Therefore, by applying the surrogate voxel mesh, the increase in the amount of surrogate data used for the neural radiation field can be suppressed compared to applying the surrogate tetrahedral mesh.

[0103] Therefore, the increased workload of processing proxy data (such as generation, editing, encoding, decoding, and rendering) can be suppressed. Furthermore, for example, in cases where proxy data is transmitted between the server and client, the increase in transmission volume can be suppressed, as can the increase in transmission-related workload. Additionally, by suppressing these increases in workload, the increase in cost and processing time can be suppressed.

[0104] Furthermore, by applying a proxy voxel grid, the reduction in coding efficiency can be suppressed compared to the case of a proxy tetrahedral grid.

[0105] Note that a proxy voxel mesh can include any information other than that described above. For example, a proxy voxel mesh can also include activation flag information, which includes activation flags set for all occupied cells. Activation flags are flags that indicate the presence or absence of the object's geometry after editing occupied cells, such as... Figure 6 As shown, an activation flag is set for each occupied cell. For example, an activation flag of ON (true) indicates that the occupied cell is occupied (containing geometry). An activation flag of OFF (false) indicates that the occupied cell is unoccupied (containing no geometry). The value of the activation flag can be updated by editing the proxy voxel mesh. For example, if an occupied cell in a region is deleted during editing of the proxy voxel mesh, the activation flag of the occupied cell can be set to OFF instead of deleting the corresponding index from the octet of the occupied cell index information. Note that in rendering, occupied cells with an activation flag of OFF are treated similarly to unoccupied cells. By doing so, the editing results can be reflected in the proxy voxel mesh without updating the occupied cell index information. Therefore, editing operations can be sped up.

[0106] Additionally, the proxy voxel mesh may include a transformation matrix that indicates the correspondence between the index of an occupied cell and its position after editing. The transformation matrix H is a 4 × 4 homogeneous transformation matrix that stores transformations (e.g., translation, rotation, scaling, shearing, etc.) from the index space (i, j, k) to the world space (x, y, z). For example, the index (i, j, k) of an occupied cell and the XYZ coordinates (x, y, z) of the vertex closer to the origin of that cell have the following relationship. Note that the superscript T denotes the transpose of a vector or matrix.

[0107] (x, y, z, 1)^T = H × (i, j, k, 1)^T

[0108] For example, assuming the number of occupied cells is O and the number of vertices is V, the proxy voxel mesh includes O indices (stored as an octree), O activation flags, V UVW coordinates and a transformation matrix.

[0109] Additionally, the surrogate voxel mesh can include cell attribute information, which includes attributes added to the occupying cell. For example, the average density value (a floating-point value) within the cell, the average color within the cell (a 3D vector in the case of RGB color), etc., can be added as attributes to the occupying cell. The surrogate voxel mesh can include cell attribute information indicating these attributes. By doing so, the surrogate voxel mesh can correspond to the editing of cell attributes.

[0110] <Method 1-1>

[0111] When applying method 1 described above, a proxy voxel mesh can be generated, such as... Figure 4 The second row from the top of the table is shown in Method 1-1.

[0112] For example, a first information processing device (e.g., Figure 8 Terminal device 111 and Figure 21 Server 312 in the middle may include a proxy voxel mesh generation unit (e.g., Figure 8 The proxy voxel mesh generation unit 121 and Figure 21 In the proxy voxel mesh generation unit 321, the proxy voxel mesh generation unit uses voxel meshes to generate proxy voxel meshes as proxy data for editing neural radiation fields. These neural radiation fields are neural networks that approximate radiation field functions, which represent a scene including a space containing objects. Furthermore, in the first information processing method executed by the first information processing device (e.g., ...), ... Figure 13 Editing and processing of neural radiation fields in Figure 22 In neural radiation field editing, voxel meshes can be used to generate proxy voxel meshes as proxy data for editing neural radiation fields, which are neural networks that approximate radiation field functions that represent a scene in space including objects.

[0113] By generating proxy voxel grids as proxy data in this manner, the first information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0114] For example, such as Figure 14 As shown in the example, the surrogate voxel mesh generation unit can generate occupied cell index information and UVW coordinate information. As a surrogate voxel mesh, the occupied cell index information includes the indices of all occupied cells, and the UVW coordinate information includes the UVW coordinates indicating the initial values ​​of the vertex coordinates of all occupied cells. By doing so, the first information processing device can suppress the increase in the amount of surrogate data used for the neural radiation field, as described above in <Method 1>.

[0115] In addition, such as Figure 14 As shown in the example, the proxy voxel grid generation unit can generate a voxel grid by voxel-by-voxel partitioning of the space, determine whether each voxel constituting the voxel grid is an occupied cell or an unoccupied cell, and convert the index of the occupied cells into an octree to generate occupied cell index information. By doing so, the first information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0116] Furthermore, the proxy voxel grid generation unit can determine the density of voxel centers based on the neural radiation field, identifying voxels as occupied units when the density is greater than a predetermined threshold, and identifying voxels as unoccupied units when the density is not greater than the threshold. In doing so, the proxy voxel grid generation unit can easily determine whether each voxel constituting the voxel grid is an occupied or unoccupied unit. Therefore, as described above in <Method 1>, the first information processing device can suppress the increase in the amount of proxy data used for the neural radiation field.

[0117] In addition, such as Figure 14 As shown in the example, the proxy voxel mesh generation unit can also generate activation flag information, including activation flags set for all occupied cells, as the proxy voxel mesh. By doing so, the first information processing device can speed up the editing operation, as described above in <Method 1>.

[0118] In addition, such as Figure 25 As shown in the example, the proxy voxel mesh generation unit can also generate cell attribute information, including attributes added to each of the occupied cells, as the proxy voxel mesh. By doing so, the first information processing device can make the proxy voxel mesh correspond to the editing of cell attributes, as described above in <Method 1>.

[0119] <Method 1-2>

[0120] When applying method 1 described above, the proxy voxel mesh can be edited, such as... Figure 4 The third row from the top of the table is shown (Method 1-2).

[0121] For example, a second information processing device (e.g., Figure 8 Terminal device 111 and Figure 21 The terminal device 311 in the middle) may include a proxy voxel mesh editing unit (e.g., Figure 8 Agent voxel mesh editing unit 122 and Figure 21 In the proxy voxel mesh editing unit 332, the proxy voxel mesh is edited. This proxy voxel mesh is proxy data used to edit neural radiation fields, which are neural networks that approximate radiation field functions representing a scene including objects in space. Furthermore, in the second information processing method executed by the second information processing device (e.g., ...), ... Figure 13 Editing and processing of neural radiation fields in Figure 22 In the neural radiation field editing process, a proxy voxel mesh can be edited. This proxy voxel mesh is surrogate data used to edit the neural radiation field, which is a neural network that approximates a radiation field function representing a scene in space including objects. Here, the proxy voxel mesh is surrogate data using a voxel mesh.

[0122] By editing the proxy voxel grid in this way, the second information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0123] For example, such as Figure 15 As shown in the example, the proxy voxel mesh editing unit can generate or edit a transformation matrix that indicates the correspondence between the index of the occupied cell and the position of the edited occupied cell. By doing so, the second information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0124] In addition, such as Figure 15 As shown in the example, the proxy voxel mesh editing unit can edit the vertex coordinates of the occupying cells that constitute the voxel mesh. By doing so, the second information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0125] In addition, such as Figure 15 As shown in the example, the proxy voxel mesh editing unit can edit activation flags, which are flags indicating the presence or absence of the object's geometry after editing the occupied cells. By doing so, the second information processing device can speed up the editing operation, as described above in <Method 1>.

[0126] In addition, such as Figure 26 As shown in the example, the proxy voxel mesh editing unit can edit the attributes added to the occupying cells. By doing so, the second information processing device can make the proxy voxel mesh correspond to the edited cell attributes, as described above in <Method 1>.

[0127] <Methods 1-3>

[0128] When applying method 1 described above, the proxy voxel mesh can be encoded, such as... Figure 4 As shown in the fourth row from the top of the table (Method 1-3).

[0129] For example, a third information processing device (e.g., Figure 8 Terminal equipment 111 in Figure 21 Terminal equipment 311 and Figure 21 Server 312 in the middle) may include a proxy voxel grid encoding unit (e.g., Figure 8 The proxy voxel grid encoding section 123 in Figure 21 The proxy voxel grid encoding section 322 and Figure 21 The proxy voxel grid encoding unit 333 encodes a proxy voxel grid, which is proxy data for editing a neural radiation field. This neural radiation field is a neural network that approximates a radiation field function representing a scene including an object in space. Furthermore, in a third information processing method executed by a third information processing device (e.g., ...), ... Figure 13 Editing and processing of neural radiation fields in Figure 22 In the neural radiation field editing process, a proxy voxel mesh can be encoded. This proxy voxel mesh is the proxy data used to edit the neural radiation field, which is a neural network that approximates a radiation field function representing a scene in space, including objects. Here, the proxy voxel mesh is the proxy data using a voxel mesh.

[0130] By encoding the proxy voxel grid in this manner, the third information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0131] For example, the proxy voxel grid encoding section may include an occupied cell index encoding section (e.g., Figure 9 The occupied unit index encoding section 152) and the UVW coordinate encoding section (e.g., Figure 9 The UVW coordinate encoding unit 153 encodes occupancy cell index information, which is included in the surrogate voxel grid and includes the indexes of all occupancy cells. The UVW coordinate encoding unit encodes UVW coordinate information, which is included in the surrogate voxel grid and includes the initial values ​​of the vertex coordinates of all occupancy cells. By doing so, the third information processing device can suppress the increase in the amount of surrogate data used for the neural radiation field, as described above in <Method 1>.

[0132] Furthermore, the occupancy unit index encoding unit can perform lossless encoding of the occupancy unit index information, in which the index of the occupancy unit is converted into an octree. By doing so, the third information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0133] In addition, such as Figure 10 As shown in the example, the UVW coordinate encoding unit can encode the difference between the predicted and actual UVW coordinates. By doing so, the third information processing device can suppress the increase in the amount of data used for proxy data of the neural radiation field, as described above in <Method 1>.

[0134] In addition, such as Figure 10 As shown in the example, the UVW coordinate encoding unit can derive the predicted UVW coordinates using the vertex coordinates of the edited occupant cell and the prediction parameters. By doing so, the third information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0135] In addition, such as Figure 10 As shown in the example, the UVW coordinate encoding unit can update the prediction parameters for each vertex by using the difference between the predicted and actual UVW coordinates. In doing so, the third information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0136] Furthermore, the UVW coordinate encoding unit can encode the UVW coordinates of each vertex in Morton order. By doing so, the third information processing device can suppress the increase in the amount of data used for the proxy data of the neural radiation field, as described above in <Method 1>.

[0137] Furthermore, the proxy voxel grid encoding unit may also include an activation flag encoding unit (e.g., Figure 9 The activation flag encoding unit 154 encodes activation flag information included in the proxy voxel grid and including activation flags set for all occupied units. By doing so, the third information processing device can speed up the editing operation, as described above in <Method 1>.

[0138] Furthermore, the proxy voxel grid encoding unit may also include a transformation matrix encoding unit (e.g., Figure 9 The transformation matrix encoding unit 151 encodes a transformation matrix that indicates the correspondence between the index of the occupied cell and the position of the edited occupied cell. By doing so, the third information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0139] Furthermore, the proxy voxel mesh encoding unit may also include a cell attribute encoding unit (e.g., Figure 23 The cell attribute encoding unit 451 encodes cell attribute information included in the proxy voxel mesh and including attributes added to each occupied cell. By doing so, the third information processing device can make the proxy voxel mesh correspond to the editing of cell attributes, as described above in <Method 1>.

[0140] <Methods 1-4>

[0141] When applying method 1 described above, the bitstream of the proxy voxel grid can be decoded, such as... Figure 4 The fifth row from the top of the table is shown (methods 1-4).

[0142] For example, a fourth information processing device (e.g., Figure 8 Server 112 in Figure 21 Terminal equipment 311 and Figure 21 Server 312 in the middle may include a proxy voxel grid decoding unit (e.g., Figure 8 The proxy voxel grid decoding unit 131 in Figure 21 The proxy voxel grid decoding unit 323 and Figure 21 The proxy voxel grid decoding unit 331 decodes the bitstream and generates a proxy voxel grid, which is proxy data for editing a neural radiation field. This neural radiation field is a neural network that approximates a radiation field function representing a scene including objects in a space. Furthermore, in a fourth information processing method executed by a fourth information processing device (e.g., ...), ... Figure 13 Editing and processing of neural radiation fields in Figure 22 In the neural radiation field editing process, bitstreams can be decoded, and proxy voxel grids can be generated. These proxy voxel grids are proxy data used to edit neural radiation fields, which are neural networks that approximate radiation field functions, representing a scene in space including objects. Here, the proxy voxel grid is proxy data using voxel grids.

[0143] By encoding the proxy voxel grid in this manner, the fourth information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0144] For example, the proxy voxel mesh decoding unit may include an occupied cell index decoding unit (e.g., Figure 11 The occupied unit index decoding unit 172) and the UVW coordinate decoding unit (e.g., Figure 11 The UVW coordinate decoding unit 173 decodes the bitstream and generates occupancy cell index information, which includes the indices of all occupancy cells in the proxy voxel grid. The UVW coordinate decoding unit also decodes the bitstream and generates UVW coordinate information, which includes the initial values ​​of the vertex coordinates of all occupancy cells in the proxy voxel grid. By doing so, the fourth information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0145] Furthermore, the occupancy unit index decoding unit can perform lossless decoding of the bitstream and generate occupancy unit index information, in which the index of the occupancy unit is converted into an octree. By doing so, the fourth information processing device can suppress the increase in the amount of data used for the proxy data of the neural radiation field, as described above in <Method 1>.

[0146] In addition, such as Figure 12 As shown in the example, the UVW coordinate decoding unit can decode the bitstream to generate a difference between the predicted and actual UVW coordinates, obtain a predicted value, and add the predicted value to the difference to obtain the UVW coordinates. By doing so, the fourth information processing device can suppress the increase in the amount of data used for proxy data of the neural radiation field, as described above in <Method 1>.

[0147] In addition, such as Figure 12 As shown in the example, the UVW coordinate decoding unit can derive the predicted UVW coordinates using the edited vertex coordinates of the occupied unit and the prediction parameters. By doing so, the fourth information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0148] In addition, such as Figure 12 As shown in the example, the UVW coordinate decoding unit can derive the vertex coordinates of the edited occupant cell using the occupant cell index information and the transformation matrix. Note that the transformation matrix indicates the correspondence between the index of the occupant cell and the position of the edited occupant cell. By doing so, the fourth information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0149] In addition, such as Figure 12 As shown in the example, the UVW coordinate decoding unit can update the prediction parameters for each vertex by using the edited vertex coordinates of the occupied unit and the difference between the predicted and actual UVW coordinates. By doing so, the fourth information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0150] Furthermore, the proxy voxel grid decoding unit may also include an activation flag decoding unit (e.g., Figure 11 The activation flag decoding unit 174 decodes the bitstream and generates activation flag information that is included in the proxy voxel grid and includes activation flags set for all occupied units. By doing so, the fourth information processing device can speed up the editing operation, as described above in <Method 1>.

[0151] Additionally, the proxy voxel grid decoding unit may also include a transformation matrix decoding unit (e.g., Figure 11 The transformation matrix decoding unit 171 decodes the bitstream and generates a transformation matrix indicating the correspondence between the index of the occupied cell and the position of the edited occupied cell. By doing so, the fourth information processing device can suppress the increase in the amount of data used for the proxy data of the neural radiation field, as described above in <Method 1>.

[0152] Furthermore, the proxy voxel mesh encoding unit may also include a cell attribute decoding unit (e.g., Figure 24 The cell attribute decoding unit 471 decodes the bitstream and generates cell attribute information, which is included in the proxy voxel grid and includes attributes added to each occupied cell. By doing so, the fourth information processing apparatus can make the proxy voxel grid correspond to the editing of cell attributes, as described above in <Method 1>.

[0153] <Methods 1-5>

[0154] When applying method 1 described above, a proxy voxel mesh can be used to perform rendering, such as Figure 4 The bottom row of the table is shown (methods 1-5).

[0155] For example, a fifth information processing device (e.g., Figure 8 Server 112 and Figure 21 Server 312 in the middle) may include a rendering unit (e.g., Figure 8 The rendering unit 132 and Figure 21 The rendering unit 324 performs rendering using a proxy voxel mesh, which is proxy data for editing a neural radiation field. This neural radiation field is a neural network that approximates a radiation field function representing a scene including objects in space. Furthermore, in the fifth information processing method executed by the fifth information processing device (e.g., ...), ... Figure 13 Editing and processing of neural radiation fields in Figure 22 In neural radiation field editing (in the context of rendering), a proxy voxel mesh can be used to perform rendering. This proxy voxel mesh is proxy data used to edit the neural radiation field, which is a neural network that approximates a radiation field function representing a scene including the space of objects. Here, the proxy voxel mesh is proxy data using a voxel mesh.

[0156] By using proxy voxel meshes to perform rendering in this manner, the fifth information processing device can suppress the increase in the amount of data used for proxy data of neural radiation fields, as described above in <Method 1>.

[0157] For example, the rendering unit can generate a rendered image using the UVW coordinates of the sampling point, the rotated viewing direction, and the neural radiation field. Here, the UVW coordinates of the sampling point indicate the initial values ​​of the sampling point's coordinates. By doing so, the fifth information processing device can suppress the increase in the amount of proxy data used for the neural radiation field, as described above in <Method 1>.

[0158] Furthermore, the rendering unit can search for occupied cells including the sampling point, derive the UVW coordinates of the sampling point using the vertex coordinates of the occupied cell, and derive the rotated viewing direction using the rotation matrix of the sampling point, such as... Figure 7 As shown in the example. By doing so, the fifth information processing device can suppress the increase in the amount of data used for proxy data of the neural radiation field, as described above in <Method 1>.

[0159] <Combination>

[0160] Each method described above can be applied in combination with any other method, as long as inconsistency does not occur. Three or more methods can be applied in combination. For example, methods 1-1 through 1-5 can be appropriately combined and applied. Furthermore, composable methods can include not only... Figure 4 The methods shown as "Methods" in the table above can include all the elements described in <3. Application of Voxel Meshes>. Furthermore, each method described above can be applied in combination with other methods besides those described above.

[0161] Note that the descriptions of higher-level methods in this specification also apply to lower-level methods that are subordinate methods, provided there is no inconsistency. For example, if it is described that "method 1 can be applied", then each of the methods from method 1-1 to method 1-5 can be applied.

[0162] <4. First Implementation Method>

[0163] Information Processing System

[0164] Figure 8 This is a block diagram illustrating a main configuration example of an information processing system that applies the technique described above in <3. Application of Voxel Mesh>. Figure 8 The information processing system 100 shown is a system for editing proxy data for neural radiation fields and generating a rendered image corresponding to the editing result.

[0165] like Figure 8 As shown, the information processing system 100 includes a terminal device 111 and a server 112. The terminal device 111 and the server 112 are connected to each other via a network (not shown). For example, the terminal device 111 operates as a client of the server 112.

[0166] Terminal device 111 uses a voxel grid to generate a proxy voxel grid as proxy data corresponding to neural radiation field (NeRF data); edits the proxy voxel grid; encodes the edited proxy voxel grid; and provides its bitstream to server 112. On the other hand, server 112 decodes the provided bitstream to generate a proxy voxel grid, performs rendering using the decoded proxy voxel grid, and generates a rendered image.

[0167] The configuration of terminal devices 111 and servers 112 is arbitrary. For example, in the information processing system 100, multiple terminal devices 111 or multiple servers 112 can be configured. Furthermore, the number of terminal devices 111 and the number of servers 112 may be different. Additionally, terminal devices 111 may include multiple information processing devices. Furthermore, servers 112 may include multiple information processing devices. For example, server 112 can be configured as a cloud server, regardless of its device configuration.

[0168] like Figure 8 As shown, the terminal device 111 includes a proxy voxel mesh generation unit 121, a proxy voxel mesh editing unit 122, and a proxy voxel mesh encoding unit 123. Furthermore, the server 112 includes a proxy voxel mesh decoding unit 131 and a rendering unit 132.

[0169] The proxy voxel mesh generation unit 121 of the terminal device 111 performs processing related to the generation of the proxy voxel mesh. For example, the proxy voxel mesh generation unit 121 can acquire NeRF data. NeRF data is data from a neural network (neural radiation field) that takes three-dimensional coordinates and a view direction vector (x, y, z, θ, φ) as input and outputs density and color vectors (r, g, b, σ). For example, NeRF data can have parameters such as the weight matrix of a neural network (neural radiation field). For neural networks, only the input and output are specified, and the specific processing is not limited. That is, any NeRF can be used as long as it includes a neural network with the input and output described above. Note that in this specification, it is assumed that NeRF data is shared in advance between the cloud environment and the local environment. That is, it is assumed that the terminal device 111 and the server 112 have NeRF data in advance, and that NeRF data is not transmitted between the terminal device 111 and the server 112.

[0170] Furthermore, the proxy voxel mesh generation unit 121 can use NeRF data to generate a proxy voxel mesh. In generating the proxy voxel mesh, for example, the proxy voxel mesh generation unit 121 can initialize an N × N × N voxel mesh (N is the resolution of the voxel mesh). Additionally, the proxy voxel mesh generation unit 121 can determine whether a voxel in the voxel mesh is an occupied cell or an unoccupied cell for all voxels. In determining this, for example, the proxy voxel mesh generation unit 121 can input the XYZ coordinates (x, y, z) of the voxel's center and the dummy viewing direction (θ, φ) = (0, 0) into the neural network to obtain the density σ of the voxel's center. Then, if the density σ is greater than a threshold, the proxy voxel mesh generation unit 121 can determine the voxel as an occupied cell. Otherwise, the proxy voxel mesh generation unit 121 can determine the voxel as an unoccupied cell and delete the voxel. That is, the proxy voxel mesh generation unit 121 can... Figure 5 As shown in the example, only occupied cells are left from the initialized voxel mesh. Note that the threshold can be predetermined. In this way, the proxy voxel mesh generation unit 121 can generate occupied cell index information.

[0171] Furthermore, the proxy voxel mesh generation unit 121 can set an activation flag with a value of "ON" for all occupied cells that have been determined as described above.

[0172] In addition, such as Figure 6 As shown in the example, the proxy voxel mesh generation unit 121 can initialize the UVW coordinates of all vertices of all occupied cells using the XYZ coordinates of the vertices. In other words, the proxy voxel mesh generation unit 121 can generate UVW coordinate information. The proxy voxel mesh generation unit 121 can then provide the proxy voxel mesh generated as described above to the proxy voxel mesh editing unit 122.

[0173] The proxy voxel mesh editing unit 122 performs processing related to editing the proxy voxel mesh. For example, the proxy voxel mesh editing unit 122 can acquire a proxy voxel mesh provided by the proxy voxel mesh generation unit 121. Furthermore, the proxy voxel mesh editing unit 122 can perform editing operations on the proxy voxel mesh. Editing operations on the proxy voxel mesh may include, for example, editing the transformation matrix, editing the UVW coordinates, editing the activation flags, etc.

[0174] The proxy voxel mesh editing unit 122 can perform movements, rotations, scaling, and shearing of the proxy voxel mesh in world space by editing the transformation matrix added to the proxy voxel mesh. As a result of this editing, the position, orientation, scale, and shearing of the entire scene can also change in the rendered image.

[0175] The proxy voxel mesh editor 122 can add noise to the UVW coordinates of some vertices of the occupying cells by editing them. This editing may also cause localized deformation of a portion of the scene in the rendered image.

[0176] The proxy voxel mesh editing unit 122 can, for example, set the activation flag to OFF by editing the activation flag of the desired region. As a result of such editing, even in the rendered image, a portion of the scene can be deleted.

[0177] The "edited proxy voxel mesh" generated by the editing operation described above is used to generate a rendered image reflecting the editing. The proxy voxel mesh editing unit 122 can provide the "edited proxy voxel mesh" to the proxy voxel mesh encoding unit 123.

[0178] The proxy voxel grid encoding unit 123 performs processing related to the encoding of the proxy voxel grid. For example, the proxy voxel grid encoding unit 123 can obtain the "edited proxy voxel grid" provided by the proxy voxel grid editing unit 122. Furthermore, the proxy voxel grid encoding unit 123 can encode the "edited proxy voxel grid" and generate its bitstream. The proxy voxel grid encoding unit 123 can provide (transmit) the generated bitstream (the bitstream of the "edited proxy voxel grid") to the server 112.

[0179] The proxy voxel grid decoding unit 131 of server 112 performs processing related to decoding the bitstream of the proxy voxel grid. For example, the proxy voxel grid decoding unit 131 can acquire the bitstream provided from the proxy voxel grid encoding unit 123 of terminal device 111. Furthermore, the proxy voxel grid decoding unit 131 can decode the bitstream and generate a "decoded proxy voxel grid". The "decoded proxy voxel grid" is obtained by restoring the "edited proxy voxel grid" described above. That is, the "decoded proxy voxel grid" and the "edited proxy voxel grid" correspond to each other and indicate the same scene. Note that the "decoded proxy voxel grid" may include encoding distortion. That is, the "decoded proxy voxel grid" and the "edited proxy voxel grid" may be completely identical to each other, or they may not be completely identical to each other. The proxy voxel grid decoding unit 131 can provide (transmit) the generated "decoded proxy voxel grid" to the rendering unit 132.

[0180] The rendering unit 132 performs rendering-related processing. For example, the rendering unit 132 can acquire a "decoded proxy voxel mesh" provided by the proxy voxel mesh decoding unit 131. Furthermore, the rendering unit 132 can perform rendering and generate a rendered image by using the "decoded proxy voxel mesh" and NeRF data.

[0181] In this case, the rendering unit 132 changes the processing of density and color derived from the coordinates of the sampling point and the line of sight of the light in the case of NeRFShop, as shown below.

[0182] [Before modification] The neural network Fθ inputs the XYZ coordinates (x, y, z) of the sampling points and the line-of-sight direction (θ, φ) of the light rays.

[0183] [After modification] The neural network Fθ inputs the UVW coordinates (u, v, w) of the sampling points and the rotated line of sight (θ', φ').

[0184] Specifically, the UVW coordinates (u, v, w) of the sampling point and the rotated line of sight (θ', φ') are obtained as follows.

[0185] 1: Search for occupied cells containing the sampling point from the coordinates (x, y, z). If not found, the process ends.

[0186] 2: If the activation flag of the cell is OFF, the process ends.

[0187] 3: For example Figure 7 As shown in the example, the UVW coordinates of the eight vertices of the voxel are interpolated using a trilinear method (also known as trilinear interpolation) to calculate the UVW coordinates (u, v, w) of the sample point.

[0188] 4: Using the rotation matrix R at the sampling point, rotate the line of sight (θ, φ) to obtain the rotated line of sight (θ', φ'), as shown in the following formula.

[0189] (θ', φ')^T = R × (θ, φ)^T

[0190] The rotation matrix R at the sampling point is obtained by the singular value decomposition of the following Jacobian matrix J.

[0191] [Mathematical expression. 1]

[0192]

[0193] Here, (u, v, w) are the UVW coordinates after trilinear interpolation, and (x, y, z) are the XYZ coordinates of the sampling point. As a method for calculating the Jacobian matrix J, for example, an automatic differentiation algorithm can be used.

[0194] When editing of the proxy voxel mesh is performed in the proxy voxel mesh editing unit 122, the editing result is reflected in the rendered image. The rendering unit 132 can output the generated rendered image. The output destination is arbitrary. For example, the rendered image can be displayed as visual information, or the rendered image can be provided to other devices.

[0195] <Proxy Voxel Mesh Encoding Department>

[0196] Figure 9 It is shown Figure 8 A block diagram illustrating the main configuration example of the proxy voxel grid encoding unit 123 in the diagram. (See diagram below.) Figure 9 As shown, the proxy voxel grid encoding unit 123 may include a transformation matrix encoding unit 151, an occupied cell index encoding unit 152, a UVW coordinate encoding unit 153, and an activation flag encoding unit 154.

[0197] The transformation matrix encoding unit 151 performs processing related to the encoding of the transformation matrix. For example, the transformation matrix encoding unit 151 can encode the transformation matrix included in the proxy voxel grid and generate its bitstream (also referred to as the transformation matrix bitstream). The encoding method of the transformation matrix is ​​arbitrary. Furthermore, since the data volume of the transformation matrix is ​​small, this encoding can be omitted. The transformation matrix encoding unit 151 can provide the generated transformation matrix bitstream as the "edited proxy voxel grid bitstream" to the proxy voxel grid decoding unit 131 of the server 112.

[0198] The occupancy unit index encoding unit 152 executes processing related to the encoding of occupancy unit index information. For example, the occupancy unit index encoding unit 152 may encode the occupancy unit index information included in the proxy voxel grid and generate a bitstream thereof (also referred to as an occupancy unit index bitstream). At this time, the occupancy unit index encoding unit 152 may perform lossless encoding on the octree of the index set of the occupancy units of the proxy voxel grid. That is, the occupancy unit index encoding unit 152 may perform lossless encoding on the index sets (i, j, k) of all occupancy units. Note that this octree encoding method is arbitrary. For example, Geometry-based Point Cloud Compression (G-PCC) encoding may be applied. The occupancy unit index encoding unit 152 may provide the generated occupancy unit index bitstream to (the proxy voxel grid decoding unit 131 of) the server 112 as the "bitstream of the edited proxy voxel grid".

[0199] The UVW coordinate encoding unit 153 executes processing related to the encoding of UVW coordinate information. For example, the UVW coordinate encoding unit 153 may encode the UVW coordinate information included in the proxy voxel grid and generate a bitstream thereof (also referred to as a UVW coordinate bitstream). At this time, the UVW coordinate encoding unit 153 may encode the UVW coordinate sets of the vertices of all occupancy units. At this time, the UVW coordinate encoding unit 153 may encode each UVW coordinate through predictive encoding that encodes the difference value between the predicted value and the actual value. The UVW coordinate encoding unit 153 may provide the generated UVW coordinate bitstream to (the proxy voxel grid decoding unit 131 of) the server 112 as the "bitstream of the edited proxy voxel grid".

[0200] The activation flag encoding unit 154 executes processing related to the encoding of activation flags. For example, the activation flag encoding unit 154 may encode the activation flags included in the proxy voxel grid and generate a bitstream thereof (also referred to as an activation flag bitstream). At this time, the activation flag encoding unit 154 may perform lossless encoding on the activation flag set of all occupancy units and generate the activation flag bitstream. At this time, the activation flag encoding unit 154 may arrange the occupancy units into a line according to the Morton order of the indices, and then sequentially perform entropy encoding on the activation flags from the front.

[0201] <UVW coordinate encoding unit>

[0202] Figure 10 is a diagram showing Figure 9 a block diagram of a main configuration example of the UVW coordinate encoding unit 153 in Figure 10 as shown in the figure, the UVW coordinate encoding unit 153 may include a prediction parameter initialization unit 161, a coordinate providing unit 162, a predicted value derivation unit 163, a differential encoding unit 164, and a prediction parameter update unit 165.

[0203] The prediction parameter initialization unit 161 performs processing related to the initialization of the prediction parameters. For example, the prediction parameter initialization unit 161 can initialize the prediction parameters (A, b) used in the prediction value output unit 163. For example, the prediction parameter initialization unit 161 can initialize the prediction parameter A as a 3 × 3 identity matrix. Furthermore, the prediction parameter initialization unit 161 can initialize the prediction parameter b as a three-dimensional zero vector. In other words, the prediction parameter initialization unit 161 can provide the initial value of each of the prediction parameters (A, b) to the prediction value output unit 163.

[0204] The coordinate providing unit 162 performs processing related to the provision of vertex coordinates. For example, the coordinate providing unit 162 may arrange all vertices of all occupied cells in the proxy voxel grid into rows according to the Morton order of the index, and then sequentially provide the XYZ coordinates of each vertex to the prediction value output unit 163 starting from the front. In addition, the coordinate providing unit 162 may provide the UVW coordinates of each vertex to the differential encoding unit 164 in the above order.

[0205] The prediction value output unit 163 performs processing related to obtaining the predicted UVW coordinates of the vertex. For example, the prediction value output unit 163 can obtain the predicted UVW coordinates (u', v', w') from the XYZ coordinates (x, y, z) and the prediction parameters (A, b), and provide the obtained predicted values ​​to the differential coding unit 164. Note that the prediction parameter A can be a 3 × 3 matrix. The prediction parameter b can be a three-dimensional vector. The prediction value output unit 163 can obtain the predicted values ​​by using the following linear prediction.

[0206] (u', ​​v', w')^T ← A × (x, y, z)^T + b

[0207] The differential coding unit 164 performs processing related to differential coding of UVW coordinates. For example, the differential coding unit 164 can derive the difference value (du, dv, dw) from the predicted value (u', v', w') and the actual value (u, v, w), as shown below.

[0208] (du, dv, dw) ← (u', v', w') - (u, v, w)

[0209] Then, the differential coding unit 164 can entropy-encode the obtained difference value and generate a bitstream (UVW coordinate bitstream). When the predicted value is close to the actual value, the difference value is close to 0, and the coding efficiency is improved. The differential coding unit 164 can provide the generated UVW coordinate bitstream to the proxy voxel grid decoding unit 131 of the server 112. In addition, the differential coding unit 164 can provide the obtained difference value to the prediction parameter update unit 165.

[0210] The prediction parameter update unit 165 performs processing related to updating the prediction parameters. For example, the prediction parameter update unit 165 can update the prediction parameters (A, b) used in the prediction value output unit 163 by using the difference values ​​(du, dv, dw). In other words, the prediction parameter update unit 165 can update the prediction parameters (A, b) by using the difference values ​​(du, dv, dw) and provide the updated prediction parameters (A, b) to the prediction value output unit 163.

[0211] The purpose of this update is to reduce the prediction error of the UVW coordinates to be encoded next. For example, the prediction parameter update unit 165 can update the prediction parameters (A, b) in the direction that minimizes the prediction error function L, as shown below. η is a pre-set coefficient.

[0212]

[0213]

[0214] Here, It is the differential value of the prediction parameter A, and can be a 3 × 3 matrix. Furthermore, This is the differential value of the prediction parameter b, and can be a three-dimensional vector. Note that the squared error function can be applied as the prediction error function L. In this case, L, and As shown below.

[0215]

[0216]

[0217]

[0218] <Proxy Voxel Mesh Decoding Department>

[0219] Figure 11 It is shown Figure 8 A block diagram illustrating the main configuration example of the proxy voxel grid decoding unit 131 in the diagram. Figure 11 As shown, the proxy voxel mesh decoding unit 131 may include a transformation matrix decoding unit 171, an occupied cell index decoding unit 172, a UVW coordinate decoding unit 173, an activation flag decoding unit 174, and a proxy voxel mesh construction unit 175.

[0220] The transformation matrix decoding unit 171 performs processing related to the decoding of the transformation matrix bitstream. For example, the transformation matrix decoding unit 171 can decode the transformation matrix bitstream and generate a transformation matrix. The decoding method for this transformation matrix bitstream is arbitrary. Note that if the transformation matrix is ​​transmitted without being encoded, this decoding can be omitted. The transformation matrix decoding unit 171 can provide the generated transformation matrix to the proxy voxel mesh construction unit 175.

[0221] Occupancy cell index decoding unit 172 performs processing related to decoding the occupancy cell index bitstream. For example, occupancy cell index decoding unit 172 can decode the occupancy cell index bitstream and generate occupancy cell index information. At this time, occupancy cell index decoding unit 172 can perform lossless decoding on the occupancy cell index bitstream and generate an octree of the index set of occupancy cells of the proxy voxel mesh. That is, occupancy cell index decoding unit 172 can generate the index set (i, j, k) of all occupancy cells through lossless decoding. Note that the decoding method of this octree is arbitrary. For example, geometry-based point cloud compression (G-PCC) decoding can be applied. Occupancy cell index decoding unit 172 can provide the generated occupancy cell index information to proxy voxel mesh construction unit 175.

[0222] The UVW coordinate decoding unit 173 performs processing related to the decoding of the UVW coordinate bitstream. For example, the UVW coordinate decoding unit 173 can decode the UVW coordinate bitstream and generate UVW coordinate information. At this time, the UVW coordinate decoding unit 173 can decode the UVW coordinate bitstream and generate a set of UVW coordinates for all vertices occupying cells. At this time, the UVW coordinate decoding unit 173 can generate each UVW coordinate by predictive decoding to generate the difference between the predicted value and the actual value by decoding the bitstream. The UVW coordinate decoding unit 173 can provide the generated UVW coordinate information to the proxy voxel mesh construction unit 175.

[0223] The activation flag decoding unit 174 performs processing related to the decoding of the activation flag bitstream. For example, the activation flag decoding unit 174 can decode the activation flag bitstream and generate an activation flag set. At this time, the activation flag decoding unit 174 can generate an activation flag set for all occupied units by lossless decoding of the activation flag bitstream. At this time, the activity flag decoding unit 174 can decode the bitstream and generate activity flags according to the order in which the occupied units are arranged in Morton's index.

[0224] The proxy voxel grid construction unit 175 executes processing related to the construction of the proxy voxel grid. For example, the proxy voxel grid construction unit 175 may construct the proxy voxel grid by using the decoding transformation matrix, occupied unit index information, UVW coordinate information, and active flag set. In the UVW coordinate information, the UVW coordinates are arranged according to the Morton order of the vertices. Additionally, in the active flag set, the active flags are arranged according to the Morton order of the units. Therefore, the proxy voxel grid construction unit 175 may sequentially copy these values to the attributes of the vertices or units starting from the front. The proxy voxel grid construction unit 175 may provide the constructed proxy voxel grid to the rendering unit 132.

[0225] <UVW coordinate decoding unit>

[0226] Figure 12 is a block diagram showing a main configuration example Figure 11 of the UVW coordinate decoding unit 173 in Figure 12 As shown in the figure, the UVW coordinate decoding unit 173 may include a prediction parameter initialization unit 181, an XYZ coordinate derivation unit 182, a prediction value derivation unit 183, a differential decoding unit 184, a prediction value addition unit 185, and a prediction parameter update unit 186.

[0227] The prediction parameter initialization unit 181 executes processing related to initialization of prediction parameters. For example, the prediction parameter initialization unit 181 may initialize the prediction parameters (A, b) used in the prediction value derivation unit 183. For example, the prediction parameter initialization unit 181 may initialize the prediction parameter A as a 3×3 identity matrix. Furthermore, the prediction parameter initialization unit 181 may initialize the prediction parameter b as a three-dimensional zero vector. In other words, the prediction parameter initialization unit 181 may provide the initial value of each of the prediction parameters (A, b) to the prediction value derivation unit 183.

[0228] The XYZ coordinate derivation unit 182 executes processing related to deriving the XYZ coordinates of the vertices. For example, the XYZ coordinate derivation unit 182 may arrange all vertices of the occupied units in rows according to the Morton order of the indices, and then sequentially derive the XYZ coordinates of each vertex starting from the front. The XYZ coordinate derivation unit 182 may derive the XYZ coordinates of each vertex by using the index of the occupied unit corresponding to the XYZ coordinates and the transformation matrix. The XYZ coordinate derivation unit 182 may provide the derived XYZ coordinates of each vertex to the prediction value derivation unit 183 and the prediction parameter update unit 186.

[0229] The prediction value output unit 183 performs processing related to obtaining the predicted values ​​of the UVW coordinates of the vertex. For example, the prediction value output unit 183 can obtain the predicted values ​​(u', v', w') of the UVW coordinates based on the XYZ coordinates (x, y, z) and the prediction parameters (A, b), and provide the obtained predicted values ​​to the prediction value addition unit 185. Note that the prediction parameter A can be a 3 × 3 matrix. The prediction parameter b can be a three-dimensional vector. The prediction value output unit 183 can obtain the predicted values ​​by using the following linear prediction.

[0230] (u', ​​v', w')^T ← A × (x, y, z)^T + b

[0231] The differential decoding unit 184 performs processing related to differential decoding of the UVW coordinate bitstream. For example, the differential decoding unit 184 can perform entropy decoding on the UVW coordinate bitstream to obtain the difference (du, dv, dw) between the predicted value (u', v', w') and the actual value (u, v, w). The differential decoding unit 184 can provide the obtained difference (du, dv, dw) to the prediction value addition unit 185 and the prediction parameter update unit 186.

[0232] The prediction value addition unit 185 performs processing related to the addition of prediction values. For example, the prediction value addition unit 185 adds the difference value (du, dv, dw) to the prediction value (u', v', w') to obtain the decoded UVW coordinates (u, v, w).

[0233] (u, v, w) ← (u', v', w') + (du, dv, dw)

[0234] The prediction addition unit 185 provides the obtained "decoded UVW coordinates (u, v, w)" to the proxy voxel mesh construction unit 175.

[0235] The prediction parameter update unit 186 performs processing related to updating the prediction parameters. For example, the prediction parameter update unit 186 can update the prediction parameters (A, b) used in the prediction value output unit 183 by using the difference values ​​(du, dv, dw) and XYZ coordinates (x, y, z). In other words, the prediction parameter update unit 186 can update the prediction parameters (A, b) by using the difference values ​​(du, dv, dw) and XYZ coordinates (x, y, z), and provide the updated prediction parameters (A, b) to the prediction value output unit 183.

[0236] The purpose of this update is to reduce the prediction error of the UVW coordinates to be added to the next difference value. For example, the prediction parameter update unit 186 can update the prediction parameters (A, b) in the direction that minimizes the prediction error function L, in a manner similar to that of the prediction parameter update unit 165.

[0237] With the configuration described above, the information processing system 100 (terminal device 111 and server 112) can apply the technology described above in <3. Application of Voxel Meshes>, and can suppress the increase in the amount of proxy data used for neural radiation fields. Therefore, the increase in the load of proxy data processing (such as generation, editing, encoding, decoding, and rendering) can be suppressed. Furthermore, for example, in the case of transmitting proxy data between the server and the client, the increase in transmission volume can be suppressed, and the increase in transmission-related load can also be suppressed. In addition, by suppressing these increases in load, the increase in cost and processing time can be suppressed. Furthermore, by applying proxy voxel meshes, the reduction in encoding efficiency can be suppressed compared to the case of proxy tetrahedral meshes.

[0238] <Flowchart for Editing and Processing Neural Radiation Fields>

[0239] Reference Figure 13 The flowchart in the document is an example of the process of neural radiation field editing processing performed by the information processing system 100.

[0240] When processing begins, in step S101, the proxy voxel grid generation unit 121 of the terminal device 111 generates a proxy voxel grid corresponding to the NeRF data.

[0241] In step S102, the proxy voxel mesh editing unit 122 edits the proxy voxel mesh.

[0242] In step S103, the proxy voxel grid encoding unit 123 encodes the edited proxy voxel grid and generates a bit stream.

[0243] In step S104, the proxy voxel grid encoding unit 123 provides the bit stream of the generated proxy voxel grid to the server 112.

[0244] In step S111, the proxy voxel grid decoding unit 131 of server 112 acquires the bit stream.

[0245] In step S112, the proxy voxel grid decoding unit 131 decodes the bit stream and generates a decoded proxy voxel grid.

[0246] In step S113, the rendering unit 132 performs rendering by using a decoded proxy voxel mesh and generates a rendered image.

[0247] When the processing in step S113 is completed, the neural radiation field editing process ends.

[0248] <Flowchart of proxy voxel mesh generation and processing>

[0249] Next, we will refer to Figure 14 The flowchart description in Figure 13 An example of the process of performing proxy voxel mesh generation in step S101.

[0250] When the proxy voxel mesh generation process begins, in step S121, the proxy voxel mesh generation unit 121 divides the space and generates a voxel mesh.

[0251] In step S122, the proxy voxel grid generation unit 121 determines whether each voxel of the voxel grid is occupied or unoccupied.

[0252] In step S123, the proxy voxel mesh generation unit 121 generates occupied cell indexes based on the determination result, and generates an octree of occupied cell indexes.

[0253] In step S124, the proxy voxel mesh generation unit 121 generates initial vertex coordinates (UVW coordinates) for all vertices of all occupied cells.

[0254] In step S125, the proxy voxel mesh generation unit 121 generates an activation flag for each occupied cell.

[0255] When the processing in step S125 is completed, the proxy voxel mesh generation process ends, and the process returns to... Figure 13 .

[0256] <Process of Agent Voxel Mesh Editing>

[0257] Next, we will refer to Figure 15 The flowchart description in Figure 13 An example of the process of performing proxy voxel mesh editing in step S102.

[0258] When the proxy voxel mesh editing process begins, in step S131, the proxy voxel mesh editing unit 122 receives editing input.

[0259] In step S132, the proxy voxel mesh editing unit 122 generates or edits the transformation matrix according to the editing input.

[0260] In step S133, the proxy voxel mesh editing unit 122 edits the vertex coordinates according to the editing input.

[0261] In step S134, the proxy voxel mesh editing unit 122 edits the activation flag according to the editing input.

[0262] When the processing in step S134 is completed, the proxy voxel mesh editing process ends, and the process returns to... Figure 13 .

[0263] <Procedure of proxy voxel grid encoding processing>

[0264] Next, referring to Figure 16 the flowchart in, an example of the procedure of the proxy voxel grid encoding processing executed in step S103 in Figure 13 is described.

[0265] When the proxy voxel grid encoding processing starts, in step S141, the transform matrix encoding unit 151 of the proxy voxel grid encoding unit 123 encodes a transform matrix and generates a transform matrix bitstream.

[0266] In step S142, the occupied unit index encoding unit 152 encodes occupied unit index information and generates an occupied unit index bitstream.

[0267] In step S143, the UVW coordinate encoding unit 153 encodes UVW coordinate information and generates a UVW coordinate bitstream.

[0268] In step S144, the activation flag encoding unit 154 encodes an activation flag and generates an activation flag bitstream.

[0269] When the processing in step S144 ends, the proxy voxel grid encoding processing ends, and the processing returns to Figure 13 .

[0270] Note that the processing order of the respective processes of steps S141 to S144 is arbitrary, and is not limited to the above example.

[0271] <Procedure of UVW coordinate encoding processing>

[0272] Next, referring to Figure 17 the flowchart in, an example of the procedure of the UVW coordinate encoding processing executed in step S143 in Figure 16 is described.

[0273] When the UVW coordinate encoding processing starts, in step S151, the differential encoding unit 164 initializes a UVW coordinate bitstream.

[0274] In step S152, the prediction parameter initialization unit 161 initializes a prediction parameter.

[0275] In step S153, the coordinate providing unit 162 sets the XYZ coordinates and UVW coordinates of the next vertex as a processing target.

[0276] In step S154, the prediction value derivation unit 163 derives a prediction value by using the XYZ coordinates and the prediction parameter.

[0277] In step S155, the differential encoding unit 164 obtains the differential value by using the UVW coordinates and the predicted value.

[0278] In step S156, the differential encoding unit 164 encodes the differential value and adds the differential value to the UVW coordinate bit stream.

[0279] In step S157, the prediction parameter update unit 165 updates the prediction parameters by using the difference value.

[0280] In step S158, the differential coding unit 164 determines whether all vertices have been processed. If it is determined that there are unprocessed vertices, the process returns to step S153. The processing steps S153 to S158 are performed for each vertex, and if it is determined in step S158 that all vertices have been processed, the process proceeds to step S159.

[0281] In step S159, the differential coding unit 164 outputs the UVW coordinate bit stream.

[0282] When the processing in step S159 is completed, the UVW coordinate encoding process ends, and the process returns to... Figure 16 .

[0283] <Flowchart of proxy voxel mesh decoding>

[0284] Next, we will refer to Figure 18 The flowchart description in Figure 13 An example of the process of performing proxy voxel mesh decoding in step S112.

[0285] When the proxy voxel grid decoding process begins, in step S161, the transformation matrix decoding unit 171 of the proxy voxel grid decoding unit 131 decodes the transformation matrix bitstream and generates a transformation matrix.

[0286] In step S162, the occupancy unit index decoding unit 172 decodes the occupancy unit index bitstream and generates occupancy unit index information.

[0287] In step S163, the UVW coordinate decoding unit 173 encodes the UVW coordinate bitstream using the generated transformation matrix and occupied cell index information, and generates UVW coordinate information.

[0288] In step S164, the activation flag decoding unit 174 decodes the activation flag bit stream and generates an activation flag set.

[0289] In step S165, the proxy voxel mesh construction unit 175 constructs a proxy voxel mesh by using the generated transformation matrix, occupied cell index information, UVW coordinate information and activation flag set.

[0290] When the processing in step S165 ends, the proxy voxel grid decoding processing ends, and the processing returns to Figure 13 .

[0291] <Flow of UVW coordinate decoding processing>

[0292] Next, with reference to Figure 19 the flowchart in, an example of the flow of the UVW coordinate decoding processing executed in step S163 in Figure 18 will be described.

[0293] When the UVW coordinate decoding processing starts, in step S171, a prediction parameter initialization unit 181 initializes prediction parameters.

[0294] In step S172, an XYZ coordinate deriving unit 182 sets the XYZ coordinates of the next vertex as a processing target.

[0295] In step S173, a prediction value deriving unit 183 derives a prediction value by using the XYZ coordinates and the prediction parameters.

[0296] In step S174, a difference decoding unit 184 decodes a UVW coordinate bitstream, and derives a difference value between the UVW coordinates and the prediction value.

[0297] In step S175, a prediction value addition unit 185 adds the derived prediction value and the derived difference value to derive the UVW coordinates.

[0298] In step S176, a prediction parameter update unit 186 updates the prediction parameters by using the difference value and the XYZ coordinates.

[0299] In step S177, the prediction value addition unit 185 determines whether all vertices have been processed. In a case where an unprocessed vertex is present, the processing returns to step S172. The respective processing of step S172 to step S177 is executed for each vertex, and in a case where it is determined in step S177 that all vertices have been processed, the processing proceeds to step S178.

[0300] In step S178, the prediction value addition unit 185 outputs the derived UVW coordinate information.

[0301] In a case where the processing in step S178 ends, the UVW coordinate decoding processing ends, and the processing returns to Figure 18 .

[0302] <Flow of rendering processing>

[0303] Next, with reference to Figure 20 the flowchart in, the processing executed in Figure 13 An example of the rendering process performed in step S113.

[0304] When the rendering process begins, in step S181, the rendering unit 132 obtains the UVW coordinates of the sampling point.

[0305] In step S182, the rendering unit 132 obtains the rotated viewing direction.

[0306] In step S183, the rendering unit 132 inputs the UVW coordinates of the sampling points and the viewing direction after opening into the neural network (neural radiation field) and generates a rendered image.

[0307] When the processing in step S183 is completed, the rendering process ends, and the process returns to... Figure 13 .

[0308] By performing the processes described above, the information processing system 100 (terminal device 111 and server 112) can apply the technology described above in <3. Application of Voxel Meshes>, and can suppress the increase in the amount of proxy data used for neural radiation fields. Therefore, the increased load on proxy data processing (such as generation, editing, encoding, decoding, and rendering) can be suppressed. Furthermore, for example, in the case of transmitting proxy data between the server and the client, the increase in transmission volume can be suppressed, and the increase in transmission-related load can also be suppressed. In addition, by suppressing these increases in load, the increase in cost and processing time can be suppressed. Furthermore, by applying proxy voxel meshes, the decrease in encoding efficiency can be suppressed compared to the case of proxy tetrahedral meshes.

[0309] <5. Second Implementation Method>

[0310] Information Processing System

[0311] This technology can be applied to information processing systems with arbitrary configurations, and is not limited to... Figure 8 The configuration example is shown below. For instance, the generation of proxy voxel meshes can be performed on the server. Figure 21 This is a block diagram illustrating another configuration example of an information processing system that applies the technique described above in <3. Application of Voxel Mesh>. Figure 21 The information processing system 300 shown is with Figure 8 The information processing system 100 is similar to this system. However, the configuration of each of the terminal devices and servers (the processing performed by each of them) is different from... Figure 8 The examples in the text are different.

[0312] like Figure 21 As shown, the information processing system 300 includes a terminal device 311 and a server 312. The terminal device 311 and the server 312 are connected to each other via a network (not shown). For example, the terminal device 311 operates as a client of the server 312.

[0313] Server 312 uses voxel grids to generate proxy voxel grids as proxy data corresponding to neural radiation fields (NeRF data), encodes the proxy voxel grids, and provides their bitstreams to terminal device 311.

[0314] Terminal device 311 decodes the provided bitstream to generate a proxy voxel grid, edits the decoded proxy voxel grid, encodes the edited proxy voxel grid, and provides its bitstream to server 312.

[0315] Server 112 decodes the provided bitstream to generate a proxy voxel mesh, performs rendering using the decoded proxy voxel mesh, and generates a rendered image.

[0316] The configuration of terminal devices 311 and servers 312 is arbitrary. For example, in the information processing system 300, multiple terminal devices 311 or multiple servers 312 can be configured. Furthermore, the number of terminal devices 311 and the number of servers 312 may be different. Additionally, terminal devices 311 may include multiple information processing devices. Furthermore, servers 312 may include multiple information processing devices. For example, server 312 can be configured as a cloud server, regardless of its device configuration.

[0317] like Figure 21 As shown, the terminal device 311 includes a proxy voxel mesh decoding unit 331, a proxy voxel mesh editing unit 332, and a proxy voxel mesh encoding unit 333. Additionally, the server 312 includes a proxy voxel mesh generation unit 321, a proxy voxel mesh encoding unit 322, a proxy voxel mesh decoding unit 323, and a rendering unit 324.

[0318] The proxy voxel mesh generation unit 321 has the same as Figure 8 The proxy voxel mesh generation unit 121 is configured similarly to the proxy voxel mesh generation unit 321 and performs similar processing. The proxy voxel mesh generation unit 321 can provide the generated proxy voxel mesh to the proxy voxel mesh encoding unit 322.

[0319] The proxy voxel grid encoding unit 322 has a similar function to... Figure 8 The proxy voxel grid encoding unit 123 in the terminal device 311 has a similar configuration and performs similar processing. However, the proxy voxel grid encoding unit 322 encodes the proxy voxel grid before editing. The proxy voxel grid encoding unit 322 can provide the bit stream of the proxy voxel grid to the proxy voxel grid decoding unit 331 in the terminal device 311.

[0320] The proxy voxel grid decoding unit 331 has a similar function to... Figure 8 The proxy voxel grid decoding unit 331 is configured similarly to the proxy voxel grid decoding unit 131 and performs similar processing. The proxy voxel grid decoding unit 331 can provide the decoded proxy voxel grid generated by decoding the bitstream to the proxy voxel grid editing unit 332.

[0321] Agent voxel mesh editing unit 332 has with Figure 8 The proxy voxel mesh editing unit 122 is configured similarly and performs similar processing. The proxy voxel mesh editing unit 332 can provide the edited proxy voxel mesh to the proxy voxel mesh encoding unit 333.

[0322] The proxy voxel grid encoding unit 333 has a similar configuration to the proxy voxel grid encoding unit 322 and performs similar processing. However, the proxy voxel grid encoding unit 333 encodes the edited proxy voxel grid. The proxy voxel grid encoding unit 333 can provide the bitstream of the generated proxy voxel grid to the proxy voxel grid decoding unit 323 of the server 312.

[0323] The proxy voxel grid decoding unit 323 has a similar configuration to the proxy voxel grid decoding unit 331 and performs similar processing. The proxy voxel grid decoding unit 323 can provide the decoded "edited proxy voxel grid" generated by decoding the bitstream to the rendering unit 324.

[0324] Rendering unit 324 has the same as Figure 8 The rendering unit 132 is configured similarly and performs similar processing. The rendering unit 324 can output a rendered image generated by using a decoded "edited proxy voxel mesh" and neural radiation field (NeRF data).

[0325] Even with such a configuration in the information processing system 300, the technique described above in <3. Application of Voxel Meshes> can still be applied. Therefore, the information processing system 300 (terminal device 311 and server 312) can suppress the increase in the amount of proxy data used for neural radiation fields. Thus, the increased workload of proxy data processing (such as generation, editing, encoding, decoding, and rendering) can be suppressed. Furthermore, for example, in the case of transmitting proxy data between the server and the client, the increase in transmission volume can be suppressed, and the increase in transmission-related workload can also be suppressed. Additionally, by suppressing these increases in workload, the increase in cost and processing time can be suppressed. Moreover, by applying proxy voxel meshes, the reduction in encoding efficiency can be suppressed compared to the case of proxy tetrahedral meshes.

[0326] <Flowchart for Editing and Processing Neural Radiation Fields>

[0327] Reference Figure 22 The flowchart in the document is an example of the process of neural radiation field editing processing performed by the information processing system 300.

[0328] When processing begins, in step S311, the proxy voxel grid generation unit 321 of server 312 generates a proxy voxel grid corresponding to the NeRF data.

[0329] In step S312, the proxy voxel grid encoding unit 322 encodes the proxy voxel grid and generates a bit stream.

[0330] In step S313, the proxy voxel grid encoding unit 322 provides the bit stream of the generated proxy voxel grid to the terminal device 311.

[0331] In step S301, the proxy voxel grid decoding unit 331 of the terminal device 311 acquires the bit stream.

[0332] In step S302, the proxy voxel grid decoding unit 331 decodes the bit stream and generates a decoded proxy voxel grid.

[0333] In step S303, the proxy voxel mesh editing unit 332 edits the proxy voxel mesh.

[0334] In step S304, the proxy voxel grid encoding unit 333 encodes the edited proxy voxel grid and generates a bit stream.

[0335] In step S305, the proxy voxel grid encoding unit 333 provides the bit stream of the generated proxy voxel grid to the server 312.

[0336] In step S314, the proxy voxel grid decoding unit 323 of the server 312 acquires the bit stream.

[0337] In step S315, the proxy voxel grid decoding unit 323 decodes the bit stream and generates a decoded proxy voxel grid.

[0338] In step S316, the rendering unit 324 performs rendering by using a decoded proxy voxel mesh and generates a rendered image.

[0339] When the processing in step S316 is completed, the neural radiation field editing process ends.

[0340] Note that each step is performed in a similar manner to the case of the first embodiment.

[0341] Even when the information processing system 300 performs various processes in this manner, the technique described above in <3. Application of Voxel Meshes> can still be applied. Therefore, the information processing system 300 (terminal device 311 and server 312) can suppress the increase in the amount of proxy data used for neural radiation fields. Thus, the increased load on proxy data processing (such as generation, editing, encoding, decoding, and rendering) can be suppressed. Furthermore, for example, in the case of transmitting proxy data between the server and the client, the increase in transmission volume can be suppressed, and the increase in transmission-related load can also be suppressed. In addition, by suppressing these increases in load, the increase in cost and processing time can be suppressed. Furthermore, by applying proxy voxel meshes, the reduction in encoding efficiency can be suppressed compared to the case of proxy tetrahedral meshes.

[0342] <6. Third Implementation Method>

[0343] <Application of Attributes>

[0344] Although the geometry of the proxy data has been described above, the proxy data can have attributes. That is, attributes (also known as cell attributes) can be added to the voxels (occupying cells) of the proxy voxel mesh. This attribute can be any information. For example, the average density value (a floating-point value) in the cell can be included in this attribute. Additionally, the average color within the cell (a 3D vector in the case of RGB color) can be included in this attribute. Of course, information other than these examples can be used.

[0345] In the same case, the information processing system 100 basically has the same characteristics as... Figure 8 The configuration in the example is similar to that in the example. However, each processing unit has the capability to process that attribute.

[0346] <Proxy Voxel Mesh Encoding Department>

[0347] Figure 23 A primary configuration example of the proxy voxel grid encoding unit 123 in this scenario is shown. For example... Figure 23 As shown, in this case, except Figure 9 In addition to the configuration in the proxy voxel grid encoding unit 123, the unit attribute encoding unit 451 is also included.

[0348] The cell attribute encoding unit 451 encodes the attributes (cell attributes) added to each occupied cell constituting the surrogate voxel mesh and generates its bitstream (also referred to as the cell attribute bitstream). The cell attribute encoding unit 451 receives the surrogate voxel mesh, encodes the attributes of all occupied cells using point cloud attribute encoding, and outputs the cell attribute bitstream. For point cloud attribute encoding, for example, Region Adaptive Hierarchical Transformation (RAHT) encoding is used.

[0349] <Proxy Voxel Mesh Decoding Department>

[0350] Figure 24 A primary configuration example of the proxy voxel grid decoding unit 131 in this scenario is shown. Figure 24 As shown, in this case, except Figure 11 In addition to the configuration in the proxy voxel grid decoding unit 131, the unit attribute decoding unit 471 is also included.

[0351] The cell attribute decoding unit 471 decodes the cell attribute bitstream and generates attributes (cell attributes) added to each occupied cell constituting the proxy voxel mesh. The cell attribute decoding unit 471 decodes the cell attribute set from the cell attribute bitstream using an occupied cell index set. Decoding is performed using a point cloud attribute decoding method. For example, Region Adaptive Hierarchical Transformation (RAHT) decoding is used as the point cloud attribute decoding method.

[0352] Even with the above configuration, the information processing system 100 (terminal device 111 and server 112) can suppress the increase in the amount of proxy data used for neural radiation fields. Therefore, the increased load on proxy data processing (such as generation, editing, encoding, decoding, and rendering) can be suppressed. Furthermore, for example, in the case of transmitting proxy data between the server and the client, the increase in transmission volume can be suppressed, and the increase in transmission-related load can also be suppressed. In addition, by suppressing these increases in load, the increase in cost and processing time can be suppressed. Furthermore, by applying proxy voxel meshes, the reduction in encoding efficiency can be suppressed compared to the case of proxy tetrahedral meshes.

[0353] Note that the information processing system 300 can be similarly extended to have the function of processing unit attributes.

[0354] <Flowchart of proxy voxel mesh generation and processing>

[0355] Reference Figure 25 The flowchart in the document describes an example of the proxy voxel mesh generation process in this case.

[0356] The processes in steps S421 to S425 are consistent with the processes in steps S121 to S125. Figure 14 A similar process is executed.

[0357] In step S426, the proxy voxel mesh generation unit 121 generates cell properties.

[0358] When the processing in step S426 is completed, the proxy voxel mesh generation process ends, and the process returns to... Figure 13 .

[0359] <Process of Agent Voxel Mesh Editing>

[0360] Reference Figure 26 The flowchart in the document describes an example of the proxy voxel mesh editing process in this case.

[0361] The processes in steps S431 to S434 are consistent with the processes in steps S131 to S134. Figure 15 A similar process is executed.

[0362] In step S435, the proxy voxel mesh editing unit 122 edits the cell attributes.

[0363] When the processing in step S435 is completed, the proxy voxel mesh editing process ends, and the process returns to... Figure 13 .

[0364] <Flowchart of proxy voxel mesh encoding processing>

[0365] Reference Figure 27 The flowchart in the document describes an example of the proxy voxel mesh encoding process in this case.

[0366] The processes in steps S441 to S444 are consistent with the processes in steps S141 to S144. Figure 16 A similar process is executed.

[0367] In step S445, the cell attribute encoding unit 451 encodes the cell attributes.

[0368] When the processing in step S445 is completed, the proxy voxel mesh encoding process ends, and the process returns to... Figure 13 .

[0369] <Flowchart of proxy voxel mesh decoding>

[0370] Reference Figure 28 The flowchart in the document describes an example of the proxy voxel mesh decoding process in this case.

[0371] The processes in steps S461 to S464 are consistent with the processes in steps S161 to S164. Figure 18 A similar process is executed.

[0372] In step S465, the cell attribute decoding unit 471 decodes the cell attribute bit stream and generates cell attributes.

[0373] The processing in step S466 is consistent with that in step S165. Figure 18 The process is similar to that in the execution of the process.

[0374] When the processing in step S466 is completed, the proxy voxel mesh decoding process ends, and the process returns to... Figure 13 .

[0375] Even when performing the processes described above, the information processing system 100 (terminal device 111 and server 112) can suppress the increase in the amount of proxy data used for the neural radiation field. Therefore, the increased load on proxy data processing (such as generation, editing, encoding, decoding, and rendering) can be suppressed. Furthermore, for example, in the case of transmitting proxy data between the server and the client, the increase in transmission volume can be suppressed, and the increase in transmission-related load can also be suppressed. In addition, by suppressing these increases in load, the increase in cost and processing time can be suppressed. Furthermore, by applying a proxy voxel mesh, the decrease in encoding efficiency can be suppressed compared to the case of a proxy tetrahedral mesh.

[0376] Note that the neural radiation field editing process performed by the information processing system 300 can be similarly extended to have the function of processing unit attributes.

[0377] <7. Supplementary Explanation>

[0378] Computer

[0379] The above series of processes can be performed using hardware or software. In the case of performing the series of processes using software, a program forming the software is installed in the computer. Here, the computer includes, for example, a computer incorporating dedicated hardware, or a general-purpose personal computer capable of performing various functions by installing various programs.

[0380] Figure 29 This is a block diagram illustrating an example of the hardware structure of a computer that performs the above series of processes through a program.

[0381] exist Figure 29 In the computer 900 shown, the central processing unit (CPU) 901, read-only memory (ROM) 902 and random access memory (RAM) 903 are interconnected via bus 904.

[0382] In addition, the input / output interface 910 is also connected to the bus 904. The input / output interface 910 is connected to the input section 911, the output section 912, the storage section 913, the communication section 914, and the driver 915.

[0383] Input unit 911 includes, for example, a keyboard, mouse, microphone, touch panel, input terminals, etc. Output unit 912 includes, for example, a display, speaker, output terminals, etc. Storage unit 913 includes, for example, a hard disk, RAM disk, non-volatile memory, etc. Communication unit 914 includes, for example, a network interface. Driver 915 drives removable media 921 such as a hard disk, optical disk, magneto-optical disk, or semiconductor memory.

[0384] In the computer configured as described above, for example, the CPU 901 loads a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes the program, thereby performing the series of processes described above. The RAM 903 also appropriately stores data required by the CPU 901 to perform various types of processing.

[0385] A computer-executed program can be applied, for example, by recording it on a removable medium 921, which serves as the encapsulation medium. In this case, by attaching the removable medium 921 to the drive 915, the program can be installed in the storage unit 913 via the input / output interface 910.

[0386] Furthermore, the program can be provided via wired or wireless transmission media (such as a local area network, the Internet, or digital satellite broadcasting). In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0387] Alternatively, the program can be pre-installed in ROM 902 or storage unit 913.

[0388] <Target audience for this technology>

[0389] This technology can be applied to any configuration. For example, it can be applied to various electronic devices.

[0390] Furthermore, this technology can be implemented as a partial configuration of a device, such as a processor used as a system large-scale integration (LSI) (e.g., a video processor), a module using multiple processors (e.g., a video module), a part using multiple modules (e.g., a video part), or an entire unit to which additional functions are added (e.g., a video unit).

[0391] Furthermore, this technology can also be applied to network systems comprising multiple devices. For example, it can be implemented as cloud computing, where multiple devices perform processing collaboratively via a network. For example, it can be implemented in cloud services that provide image (moving image) related services to any terminal such as computers, audiovisual (AV) equipment, portable information processing terminals, and Internet of Things (IoT) devices.

[0392] Note that in this specification, "system" refers to a collection of multiple components (devices, modules (parts), etc.) and does not limit whether all components are contained in the same housing. Therefore, both multiple devices housed in separate housings and connected via a network, and a single device containing multiple modules within a single housing, constitute a system.

[0393] <Fields and Applications of This Technology>

[0394] Systems, equipment, and processing units utilizing this technology can be used in any field, such as transportation, healthcare, crime prevention, agriculture, animal husbandry, mining, beauty, factories, home appliances, weather, and natural environment monitoring. Furthermore, its applications are virtually limitless.

[0395] <Other>

[0396] Note that in this specification, "flag" refers to information used to identify multiple states, including not only information used to identify the states of true (1) and false (0), but also information capable of identifying three or more states. Therefore, the value that a "flag" can take can be, for example, binary or ternary (or more bases) of 1 / 0. That is, the number of bits forming the "flag" is arbitrary and can be one or more bits. Furthermore, identification information (including flags) is assumed to include not only the identification information in the bitstream, but also the differential information of the identification information relative to a reference information in the bitstream. Therefore, in this specification, "flag" and "identification information" include not only this information, but also the differential information relative to the reference information.

[0397] Furthermore, various information associated with the encoded data (bitstream) (e.g., metadata) can be transmitted or recorded in any form, as long as it is associated with the encoded data. Here, the term "associated" means, for example, allowing the use (linking) of other data while processing one piece of data. That is, related data items can be combined into one piece of data, or they can be processed as separate pieces of data. For example, information associated with the encoded data (image) can be transmitted on a different transmission path than the encoded data (image). Additionally, for example, information associated with the encoded data (image) can be recorded in a different storage medium (or another recording area of ​​the same storage medium) than the encoded data (image). Note that this "association" may not be the entirety of the data, but rather a portion of it. For example, an image and the information corresponding to that image can be associated with each other in any unit, such as multiple frames, a single frame, or a portion within a frame.

[0398] Note that in this specification, terms such as “combine,” “reuse,” “add,” “integrate,” “include,” “store,” “place,” “embed,” and “insert” refer to combining multiple items into one item, such as combining coded data and metadata into a single data item, and indicate a method of “association” as described above.

[0399] Furthermore, the implementation of this technology is not limited to the above-described implementation, and various modifications can be made without departing from the scope of this technology.

[0400] For example, a configuration described as a single device (or processing unit) can be divided and configured into multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) can be combined and configured into a single device (or processing unit). Furthermore, it goes without saying that configurations other than those described above can be added to the configuration of each device (or each processing unit). Additionally, a portion of the configuration of one device (or processing unit) can be included in the configuration of another device (or another processing unit), provided that the overall system configuration and operation remain substantially the same.

[0401] Furthermore, for example, the above program can be executed on any device. In this case, the device only needs to have the required functions (function blocks, etc.) and be able to obtain the necessary information.

[0402] Furthermore, for example, each step of a flowchart can be executed by a single device or by multiple devices in a distributed manner. Additionally, if a step includes multiple processes, these processes can be executed by a single device or by multiple devices in a distributed manner. In other words, multiple processes included in a step can be executed as processes of multiple steps. Conversely, processes described as multiple steps can also be executed as a single step.

[0403] Furthermore, for example, in a program executed by a computer, the processes describing the steps of the program can be executed sequentially in the order described in this specification, or they can be executed in parallel, or individually at the required time, such as when a call is made. That is, as long as no inconsistency occurs, the processes can be executed in an order different from the above-described order. In addition, the processes used to describe the steps of the program can be executed in parallel with the processes of other programs, or they can be executed in combination with the processes of other programs.

[0404] Furthermore, for example, multiple technologies related to this technology can be implemented independently and separately, provided there is no inconsistency. It goes without saying that any multiple technologies can also be implemented in combination. For example, part or all of the technologies described in any of the embodiments can also be implemented in combination with part or all of the technologies described in other embodiments. Furthermore, part or all of any of the technologies described above can also be implemented together with other technologies not described above.

[0405] Note that this technology can also have the following configurations.

[0406] (1) An information processing device, comprising:

[0407] A proxy voxel mesh generation unit generates a proxy voxel mesh that uses a voxel mesh as proxy data for editing a neural radiation field, which is a neural network that approximates a radiation field function, representing a scene including the space of objects.

[0408] (2) The information processing apparatus according to (1), wherein,

[0409] The proxy voxel mesh generation unit generates occupied cell index information and UVW coordinate information as the proxy voxel mesh. The occupied cell index information includes the indices of all occupied cells, and the UVW coordinate information includes UVW coordinates indicating the initial values ​​of the vertex coordinates of all occupied cells.

[0410] The occupancy unit is a voxel that constitutes the voxel grid and contains the geometry of the object.

[0411] (3) The information processing apparatus according to (2), wherein,

[0412] The proxy voxel mesh generation unit generates the voxel mesh by dividing the space voxel by voxel, determines whether each voxel constituting the voxel mesh is an occupied cell or an unoccupied cell, and generates the occupied cell index information by converting the index of the occupied cell into an octree.

[0413] The unoccupied cell is a voxel that constitutes the voxel grid and whose interior does not contain the geometry of the object.

[0414] (4) The information processing apparatus according to (3), wherein,

[0415] The proxy voxel grid generation unit determines the density of the voxel center based on the neural radiation field, and identifies the voxel as the occupied unit if the density is greater than a predetermined threshold, and identifies the voxel as the unoccupied unit if the density is not greater than the threshold.

[0416] (5) The information processing apparatus according to any one of (2) to (4), wherein,

[0417] The proxy voxel mesh generation unit also generates activation flag information, including activation flags set for all occupied cells, as the proxy voxel mesh, and

[0418] The activation flag is a flag indicating the presence or absence of the object's geometry after the occupant unit has been edited.

[0419] (6) The information processing apparatus according to any one of (2) to (5), wherein,

[0420] The proxy voxel mesh generation unit also generates cell attribute information, including attributes added to each occupied cell, as the proxy voxel mesh.

[0421] (7) An information processing method, comprising:

[0422] A proxy voxel grid using a voxel grid is generated as proxy data for editing a neural radiation field, which is a neural network that approximates a radiation field function representing a scene including the space of objects.

[0423] (11) An information processing apparatus, comprising:

[0424] A proxy voxel mesh editing unit edits proxy voxel meshes, which are proxy data used to edit neural radiation fields. The neural radiation fields are neural networks that approximate radiation field functions, which represent a scene including the space of objects.

[0425] The proxy voxel grid is proxy data that uses a voxel grid.

[0426] (12) The information processing apparatus according to (11), wherein,

[0427] The proxy voxel mesh editing unit generates or edits a transformation matrix, which indicates the correspondence between the index of the occupied cell and the position of the occupied cell after editing.

[0428] The occupancy unit is a voxel that constitutes the voxel grid and contains the geometry of the object.

[0429] (13) The information processing apparatus according to (11) or (12), wherein,

[0430] The proxy voxel mesh editing unit edits the vertex coordinates of the occupying cells that constitute the voxel mesh.

[0431] (14) The information processing apparatus according to any one of (11) to (13), wherein,

[0432] The proxy voxel mesh editing unit edits an activation flag, which is flag information indicating the presence or absence of the object's geometry after editing the occupied cell.

[0433] The occupancy unit is a voxel that constitutes the voxel grid and contains the geometry of the object.

[0434] (15) The information processing apparatus according to any one of (11) to (14), wherein,

[0435] The proxy voxel mesh editing unit edits the attributes added to the occupied cells, and

[0436] The occupancy unit is a voxel that constitutes the voxel grid and contains the geometry of the object.

[0437] (16) An information processing method, comprising:

[0438] Editing a proxy voxel mesh, wherein the proxy voxel mesh is proxy data used to edit a neural radiation field, the neural radiation field being a neural network that approximates a radiation field function, the radiation field function representing a scene including objects in space, wherein...

[0439] The proxy voxel grid is proxy data that uses a voxel grid.

[0440] (21) An information processing apparatus, comprising:

[0441] A proxy voxel grid encoding unit encodes proxy voxel grids, which are proxy data used to edit neural radiation fields. The neural radiation fields are neural networks that approximate radiation field functions, which represent a scene including objects in space.

[0442] The proxy voxel grid is proxy data that uses a voxel grid.

[0443] (22) The information processing apparatus according to (21), wherein,

[0444] The proxy voxel grid encoding unit includes:

[0445] Occupied cell index encoding unit, which encodes occupied cell index information including the indexes of all occupied cells in the proxy voxel grid; and

[0446] The UVW coordinate encoding unit encodes UVW coordinate information, which includes UVW coordinates in the proxy voxel mesh and includes initial values ​​indicating the vertex coordinates of all occupied cells.

[0447] The occupancy unit is a voxel that constitutes the voxel grid and contains the geometry of the object.

[0448] (23) The information processing apparatus according to (22), wherein,

[0449] The occupancy unit index encoding unit performs lossless encoding on the occupancy unit index information, and the index of the occupancy unit in the occupancy unit index information is converted into an octree.

[0450] (24) The information processing apparatus according to (22) or (23), wherein,

[0451] The UVW coordinate encoding unit encodes the difference between the predicted and actual values ​​of the UVW coordinates.

[0452] (25) The information processing apparatus according to (24), wherein,

[0453] The UVW coordinate encoding unit derives the predicted value by using the edited vertex coordinates of the occupied unit and the prediction parameters.

[0454] (26) The information processing apparatus according to (25), wherein,

[0455] The UVW coordinate encoding unit uses the difference value to update the prediction parameters for each of the vertices.

[0456] (27) The information processing apparatus according to any one of (22) to (26), wherein,

[0457] The UVW coordinate encoding unit encodes the UVW coordinates of each vertex in Morton order.

[0458] (28) The information processing apparatus according to any one of (22) to (27), wherein,

[0459] The proxy voxel grid encoding unit further includes an activation flag encoding unit, which encodes activation flag information included in the proxy voxel grid and including activation flags set for all occupied cells.

[0460] The activation flag is a flag indicating the presence or absence of the object's geometry after the occupant unit has been edited.

[0461] (29) The information processing apparatus according to any one of (22) to (28), wherein,

[0462] The proxy voxel grid encoding unit further includes a transformation matrix encoding unit, which encodes a transformation matrix that indicates the correspondence between the index of the occupied cell and the position of the occupied cell after editing.

[0463] (30) The information processing apparatus according to any one of (22) to (29), wherein,

[0464] The proxy voxel grid encoding unit further includes a cell attribute encoding unit, which encodes cell attribute information included in the proxy voxel grid and including attributes added to each occupied cell.

[0465] (31) An information processing method, comprising:

[0466] Encoding a proxy voxel mesh, wherein the proxy voxel mesh is proxy data used to edit a neural radiation field, the neural radiation field being a neural network that approximates a radiation field function, the radiation field function representing a scene including objects in space, wherein...

[0467] The proxy voxel grid is proxy data that uses a voxel grid.

[0468] (41) An information processing apparatus, comprising:

[0469] A proxy voxel grid decoding unit decodes the bitstream and generates a proxy voxel grid. This proxy voxel grid is proxy data used to edit neural radiation fields. The neural radiation field is a neural network that approximates a radiation field function, which represents a scene including objects in space.

[0470] The proxy voxel grid is proxy data that uses a voxel grid.

[0471] (42) The information processing apparatus according to (41), wherein,

[0472] The proxy voxel grid decoding unit includes:

[0473] Occupancy cell index decoding unit decodes the bitstream and generates occupancy cell index information that includes the indexes of all occupancy cells in the proxy voxel grid.

[0474] The UVW coordinate decoding unit decodes the bitstream and generates UVW coordinate information, which includes the UVW coordinates in the proxy voxel grid and includes initial values ​​indicating the vertex coordinates of all occupied cells.

[0475] The occupancy unit is a voxel that constitutes the voxel grid and contains the geometry of the object.

[0476] (43) The information processing apparatus according to (42), wherein,

[0477] The occupancy unit index decoding unit performs lossless decoding on the bitstream and generates the occupancy unit index information, in which the index of the occupancy unit is converted into an octree.

[0478] (44) The information processing apparatus according to (42) or (43), wherein,

[0479] The UVW coordinate decoding unit decodes the bitstream to generate a difference value between the predicted value and the actual value of the UVW coordinates, obtains the predicted value, and obtains the UVW coordinates by adding the obtained predicted value to the difference value.

[0480] (45) The information processing apparatus according to (44), wherein,

[0481] The UVW coordinate decoding unit derives the predicted value by using the edited vertex coordinates of the occupied unit and the prediction parameters.

[0482] (46) The information processing apparatus according to (45), wherein,

[0483] The UVW coordinate decoding unit uses the occupancy cell index information and transformation matrix to derive the vertex coordinates of the edited occupancy cell, and

[0484] The transformation matrix indicates the correspondence between the index of the occupied cell and the position of the occupied cell after editing.

[0485] (47) The information processing apparatus according to (45) or (46), wherein,

[0486] The UVW coordinate decoding unit updates the prediction parameters for each of the vertices by using the edited vertex coordinates of the occupied unit and the difference value.

[0487] (48) The information processing apparatus according to any one of (42) to (47), wherein,

[0488] The proxy voxel grid decoding unit further includes an activation flag decoding unit, which decodes the bitstream and generates activation flag information included in the proxy voxel grid and including activation flags set for all occupied cells.

[0489] The activation flag is a flag indicating the presence or absence of the object's geometry after the occupant unit has been edited.

[0490] (49) The information processing apparatus according to any one of (42) to (48), wherein,

[0491] The proxy voxel grid decoding unit further includes a transformation matrix decoding unit, which decodes the bitstream and generates a transformation matrix indicating the correspondence between the index of the occupied cell and the position of the occupied cell after editing.

[0492] (50) The information processing apparatus according to any one of (42) to (49), wherein,

[0493] The proxy voxel grid decoding unit further includes a cell attribute decoding unit, which decodes the bitstream and generates cell attribute information including attributes added to each occupied cell.

[0494] (51) An information processing method, comprising:

[0495] The bitstream is decoded, and a proxy voxel grid is generated. This proxy voxel grid is surrogate data used to edit the neural radiation field, which is a neural network approximating a radiation field function representing a scene including objects in space.

[0496] The proxy voxel grid is proxy data that uses a voxel grid.

[0497] (61) An information processing apparatus, comprising:

[0498] The rendering unit performs rendering using proxy voxel meshes, which are proxy data for editing neural radiation fields. These neural radiation fields are neural networks that approximate a radiation field function, which represents a scene including the space of objects.

[0499] The proxy voxel grid is proxy data that uses a voxel grid.

[0500] (62) The information processing apparatus according to (61), wherein,

[0501] The rendering unit generates a rendered image using the UVW coordinates of the sampling points, the rotated viewing direction, and the neural radiation field.

[0502] The UVW coordinates of the sampling point indicate the initial value of the sampling point's coordinates.

[0503] (63) The information processing apparatus according to (62), wherein,

[0504] The rendering unit search includes the occupancy unit of the sampling point, deriving the UVW coordinates of the sampling point using the vertex coordinates of the occupancy unit, and deriving the rotated viewing direction using the rotation matrix of the sampling point.

[0505] The occupancy unit is a voxel that constitutes the voxel grid and contains the geometry of the object.

[0506] (64) An information processing method, comprising:

[0507] Rendering is performed using proxy voxel meshes, which are proxy data for editing neural radiation fields. These neural radiation fields are neural networks that approximate a radiation field function, which represents a scene including the space of objects.

[0508] The proxy voxel grid is the proxy data that uses a voxel grid.

[0509] (71) A proxy voxel grid, the proxy voxel grid being proxy data for editing a neural radiation field, the neural radiation field being a neural network approximating a radiation field function, the radiation field function representing a scene including objects in a space, the proxy voxel grid comprising:

[0510] Occupied unit index information, which includes the indexes of all occupied units; and

[0511] UVW coordinate information, which includes UVW coordinates indicating the initial values ​​of the vertex coordinates of all occupied units, wherein...

[0512] The occupancy unit is a voxel that constitutes a voxel mesh and contains the geometry of the object.

[0513] (72) The proxy voxel mesh according to (71) further includes:

[0514] Activation flag information, which includes activation flags set for all occupied units, wherein...

[0515] The activation flag is a flag indicating the presence or absence of the object's geometry after the occupant unit has been edited.

[0516] (73) The proxy voxel mesh according to (71) or (72) further includes:

[0517] A transformation matrix that indicates the correspondence between the index of the occupied cell and the position of the occupied cell after editing.

[0518] (74) The proxy voxel mesh according to any one of (71) to (73) further includes:

[0519] Unit attribute information, which includes attributes added to the occupied unit.

[0520] Reference tag list

[0521] 100 Information Processing System

[0522] 111 Terminal Equipment

[0523] 112 server

[0524] 121 Agent Voxel Mesh Generation Unit

[0525] 122 Agent Voxel Mesh Editing Department

[0526] 123 Agent Voxel Mesh Encoding Department

[0527] 131 Agent Voxel Mesh Decoding Unit

[0528] 132 Rendering Department

[0529] 151 Transform Matrix Encoding Unit

[0530] 152 Occupied Unit Index Encoding Section

[0531] 153 UVW Coordinate Encoding Department

[0532] 154 Activation Flag Encoding Section

[0533] 161 Prediction Parameter Initialization Section

[0534] 162 Coordinate Providing Department

[0535] 163 Predictions are worth publishing.

[0536] 164 Differential Coding Unit

[0537] 165 Prediction Parameter Update Department

[0538] 171 Transform Matrix Decoding Unit

[0539] 172 Occupied Unit Index Decoding Section

[0540] 173 UVW Coordinate Decoding Department

[0541] 174 Activation Flag Decoding Unit

[0542] 175. Agent Voxel Mesh Construction Department

[0543] 181 Prediction Parameter Initialization Section

[0544] 182 XYZ coordinates derived part

[0545] 183 Predictions are worth publishing.

[0546] 184 Differential Decoding Unit

[0547] 185 Predicted Value Addition Section

[0548] 186 Prediction Parameter Update Department

[0549] 300 Information Processing System

[0550] 311 Terminal Equipment

[0551] 312 server

[0552] 321 Agent Voxel Mesh Generation Unit

[0553] 322 Agent Voxel Mesh Encoding Unit

[0554] 323 Proxy Voxel Mesh Decoding Unit

[0555] 324 Rendering Department

[0556] 331 Agent Voxel Mesh Decoding Unit

[0557] 332 Agent Voxel Mesh Editorial Department

[0558] 333 Agent Voxel Mesh Encoding Department

[0559] 451 Unit Attribute Encoding Section

[0560] 471 Unit Attribute Decoding Unit

[0561] 900 Computers< / nerfshop>

Claims

1. An information processing apparatus, comprising: A proxy voxel mesh generation unit generates a proxy voxel mesh that uses a voxel mesh as proxy data for editing a neural radiation field, which is a neural network that approximates a radiation field function, representing a scene including the space of objects.

2. The information processing apparatus according to claim 1, wherein, The proxy voxel mesh generation unit generates occupied cell index information and UVW coordinate information as the proxy voxel mesh. The occupied cell index information includes the indices of all occupied cells, and the UVW coordinate information includes UVW coordinates indicating the initial values ​​of the vertex coordinates of all occupied cells. The occupancy unit is a voxel that constitutes the voxel grid and contains the geometry of the object.

3. The information processing apparatus according to claim 2, wherein, The proxy voxel mesh generation unit generates the voxel mesh by dividing the space voxel by voxel, determines whether each voxel constituting the voxel mesh is an occupied cell or an unoccupied cell, and generates the occupied cell index information by converting the index of the occupied cell into an octree. The unoccupied cell is a voxel that constitutes the voxel grid and whose interior does not contain the geometry of the object.

4. The information processing apparatus according to claim 3, wherein, The proxy voxel grid generation unit determines the density of the voxel center based on the neural radiation field, and identifies the voxel as the occupied unit if the density is greater than a predetermined threshold, and identifies the voxel as the unoccupied unit if the density is not greater than the threshold.

5. The information processing apparatus according to claim 2, wherein, The proxy voxel mesh generation unit also generates activation flag information, including activation flags set for all occupied cells, as the proxy voxel mesh, and The activation flag is a flag indicating the presence or absence of the object's geometry after the occupant unit has been edited.

6. The information processing apparatus according to claim 1, further comprising: A proxy voxel grid encoding unit encodes the proxy voxel grid.

7. The information processing apparatus according to claim 6, wherein, The proxy voxel grid encoding unit includes: Occupancy cell index encoding unit, which encodes occupancy cell index information including the indexes of all occupancy cells in the proxy voxel grid; and The UVW coordinate encoding unit encodes UVW coordinate information, which includes UVW coordinates in the proxy voxel mesh and includes initial values ​​indicating the vertex coordinates of all occupied cells. The occupancy unit is a voxel that constitutes the voxel grid and contains the geometry of the object.

8. An information processing method, comprising: A proxy voxel grid using a voxel grid is generated as proxy data for editing a neural radiation field, which is a neural network that approximates a radiation field function representing a scene including the space of objects.

9. An information processing apparatus, comprising: The rendering unit performs rendering using proxy voxel meshes, which are proxy data for editing neural radiation fields. These neural radiation fields are neural networks that approximate a radiation field function, which represents a scene including the space of objects. The proxy voxel grid is the proxy data that uses a voxel grid.

10. The information processing apparatus according to claim 9, wherein, The rendering unit generates a rendered image using the UVW coordinates of the sampling points, the rotated viewing direction, and the neural radiation field. The UVW coordinates of the sampling point indicate the initial value of the sampling point's coordinates.

11. The information processing apparatus according to claim 10, wherein, The rendering unit search includes the occupancy unit of the sampling point, deriving the UVW coordinates of the sampling point using the vertex coordinates of the occupancy unit, and deriving the rotated viewing direction using the rotation matrix of the sampling point. The occupancy unit is a voxel that constitutes the voxel grid and contains the geometry of the object.

12. The information processing apparatus according to claim 9, further comprising: A proxy voxel grid decoding unit decodes the bitstream and generates the proxy voxel grid, wherein... The rendering unit is configured to perform rendering by using a generated proxy voxel mesh.

13. The information processing apparatus according to claim 12, wherein, The proxy voxel grid decoding unit includes: Occupancy cell index decoding unit decodes the bitstream and generates occupancy cell index information that includes the indices of all occupancy cells in the proxy voxel grid; and The UVW coordinate decoding unit decodes the bitstream and generates UVW coordinate information, which includes the UVW coordinates in the proxy voxel grid and includes initial values ​​indicating the vertex coordinates of all occupied cells. The occupancy unit is a voxel that constitutes the voxel grid and contains the geometry of the object.

14. The information processing apparatus according to claim 13, wherein, The occupancy unit index decoding unit performs lossless decoding on the bitstream and generates the occupancy unit index information, in which the index of the occupancy unit is converted into an octree.

15. The information processing apparatus according to claim 13, wherein, The UVW coordinate decoding unit decodes the bitstream to generate a difference between the predicted and actual values ​​of the UVW coordinates, obtains the predicted value, and obtains the UVW coordinates by adding the obtained predicted value to the difference value.

16. An information processing method, comprising: Rendering is performed using proxy voxel meshes, which are proxy data for editing neural radiation fields. These neural radiation fields are neural networks that approximate a radiation field function, which represents a scene including the space of objects. The proxy voxel grid is the proxy data that uses a voxel grid.

17. A proxy voxel mesh, the proxy voxel mesh being proxy data for editing a neural radiation field, the neural radiation field being a neural network approximating a radiation field function representing a scene including objects in a space, the proxy voxel mesh comprising: Occupied unit index information, which includes the indexes of all occupied units; as well as UVW coordinate information, which includes UVW coordinates indicating the initial values ​​of the vertex coordinates of all occupied units, wherein... The occupancy unit is a voxel that constitutes a voxel mesh and contains the geometry of the object.

18. The proxy voxel mesh of claim 17, further comprising: Activation flag information, which includes activation flags set for all occupied units, wherein... The activation flag is a flag indicating the presence or absence of the object's geometry after the occupant unit has been edited.

19. The proxy voxel mesh according to claim 17, further comprising: A transformation matrix that indicates the correspondence between the index of the occupied cell and the position of the occupied cell after editing.

20. The proxy voxel mesh of claim 17, further comprising: Unit attribute information, which includes attributes added to the occupied unit.

Images