Information processing apparatus and information processing method

Abstract

The present disclosure pertains to an information processing device and method capable of suppressing an increase in processing time for decoding a mesh. The present invention recursively repeats, until a desired layer: a first generation process for decoding a bitstream, generating a base mesh, and independently generating, for each edge in a lower layer of fineness, an edge and a vertex of a processing target layer of fineness; a second generation process for generating edges and vertexes of a processing target layer for a center triangle in the lower layer; and a third generation process for independently generating triangles of a processing target layer for each triangle in the lower layer. Accordingly, the base mesh is subdivided into a desired fineness, and the displacement vector is applied to a division point, which is a vertex of the base mesh subdivided into the desired fineness. The present disclosure can be applied, for example, to an information processing device, an electronic device, an information processing method, or a program.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus and information processing method, and in particular, to information processing apparatus and information processing method capable of suppressing the increase in grid decoding processing time. Background Technology

[0002] V-DMC (Video-based Dynamic Mesh Coding) has existed as a method for encoding meshes, which are 3D data representing the three-dimensional structure of an object through vertices and connections (see, for example, Non-Patent Literature 1). In V-DMC, the mesh to be encoded (the original mesh) is represented by a coarse (i.e., low-resolution) base mesh and displacement vectors of the segmented points obtained by subdividing the base mesh, and both the base mesh and the displacement vectors are encoded. The displacement vectors are stored (packed) in a two-dimensional image and encoded as an image. Since the object can change in the temporal direction (is dynamic), the mesh (i.e., the base mesh and the displacement vectors) is also dynamic. Therefore, the displacement vectors are encoded as a motion picture (displacement video), in which the two-dimensional image becomes a frame.

[0003] In decoding, the bitstream is decoded using a decoding method corresponding to the encoding method to recover (generate) the base grid and displacement vectors. Then, the base grid is subdivided, and the displacement vectors are applied to the subdivision points to recover (generate) the grid corresponding to the original grid.

[0004] Citation List

[0005] Patent documents

[0006] Non-Patent Literature 1: Khaled Mammou, Jungsun Kim, Alexis Tourapis, DimitriPodborski, Krasimir Kolarov, “[V-CG] Apple's Dynamic Mesh Coding CfP Response”, ISO / IEC JTC 1 / SC 29 / WG 7m59281, April 2022 Summary of the Invention

[0007] Technical issues

[0008] However, past base mesh subdivision methods involve sequential processing, which makes parallelization of the processing difficult and therefore raises concerns about increased processing time.

[0009] In view of the above, this disclosure is intended to enable the suppression of increases in grid decoding processing time.

[0010] Solution to the problem

[0011] According to one aspect of the present technology, an information processing apparatus includes: a decoding unit that decodes a bitstream to generate a base grid; and a subdivision unit that subdivides the base grid until a desired level of refinement is obtained, the subdivision being performed by recursively repeating a first generation process, a second generation process, and a third generation process until the desired level is reached, wherein the first generation process is for independently generating edges and vertices of a processing target level for each edge at a lower level of refinement, and the second generation process is for generating edges and vertices of the processing target level relative to a central triangle at the lower level of refinement. The third generation process is for independently generating triangles at the processing target level for each triangle at the lower level; and the displacement vector application unit applies displacement vectors to segmentation points, which are vertices of the base mesh subdivided until the desired level of refinement is achieved, wherein the base mesh is a mesh with a lower level of refinement than the original mesh for encoding the target, the original mesh including vertices and connections representing the three-dimensional structure of the object, the base mesh being generated by dividing the vertices from the original mesh; and the displacement vectors indicate the positional difference between the segmentation points and the vertices of the original mesh.

[0012] According to one aspect of the present technology, an information processing method includes: decoding a bitstream to generate a base grid; subdividing the base grid until a desired level of refinement is obtained, the subdivision being performed by recursively repeating a first generation process, a second generation process, and a third generation process until the desired level is reached, wherein the first generation process is for independently generating edges and vertices of a processing target level for each edge at a lower level of refinement, the second generation process is for generating edges and vertices of the processing target level relative to a central triangle at the lower level, and the third generation process is for independently generating triangles of the processing target level for each triangle at the lower level; and applying a displacement vector to a segmentation point, the segmentation point being a vertex of the base grid subdivided until the desired level of refinement is obtained, wherein the base grid is a grid having a lower level of refinement than an original grid representing a three-dimensional structure of an object, the original grid comprising vertices and connections representing the three-dimensional structure of an object, the base grid being generated by dividing the vertices from the original grid, and the displacement vector indicating the positional difference between the segmentation point and a vertex of the original grid.

[0013] In the information processing apparatus and method according to various aspects of the present technology, a bitstream is decoded to generate a base grid; the base grid is subdivided until a desired level of fineness is obtained, the subdivision is performed by recursively repeating a first generation process, a second generation process, and a third generation process until the desired level is reached, the first generation process being a process for independently generating edges and vertices of a processing target level for each edge of a lower level of fineness, the second generation process being a process for generating edges and vertices of the processing target level relative to the central triangle of the lower level, and the third generation process being a process for independently generating triangles of the processing target level for each triangle of the lower level; and a displacement vector is applied to the dividing points, the dividing points being vertices of the base grid subdivided until the desired level of fineness is obtained. Attached Figure Description

[0014] Figure 1 It is a diagram used to illustrate the grid.

[0015] Figure 2 This is a diagram used to illustrate V-DMC.

[0016] Figure 3 This is a diagram used to illustrate the subdivision process.

[0017] Figure 4 This is a diagram used to illustrate the subdivision process.

[0018] Figure 5 This is a diagram used to illustrate the subdivision process.

[0019] Figure 6 This is a diagram used to illustrate the subdivision process.

[0020] Figure 7 This is a diagram used to illustrate the subdivision process.

[0021] Figure 8 This is a diagram used to illustrate the subdivision process.

[0022] Figure 9 This is a diagram used to illustrate the subdivision process.

[0023] Figure 10 This is a diagram used to illustrate the subdivision process.

[0024] Figure 11 This is a diagram used to illustrate the subdivision process.

[0025] Figure 12 This is a block diagram illustrating a primary configuration example of a decoding device.

[0026] Figure 13 It is a graph used to illustrate the updates of edges and vertices.

[0027] Figure 14 This is a flowchart illustrating an example of the decoding process.

[0028] Figure 15 This is a flowchart illustrating an example of the process of subdividing the processing.

[0029] Figure 16 This is a diagram used to illustrate the first generation process.

[0030] Figure 17 This is a diagram used to illustrate the second generation process.

[0031] Figure 18 This is a diagram used to illustrate the third generation process.

[0032] Figure 19 This is a diagram used to illustrate the parallelization of the processing.

[0033] Figure 20 This is a block diagram illustrating a typical configuration example of a computer. Detailed Implementation

[0034] In the following text, a mode for implementing the present disclosure (hereinafter referred to as an implementation) will be described. Note that the description will be given in the following order.

[0035] 1. Supporting documents containing technical content and terminology, etc.

[0036] 2. Subdivision Processing

[0037] 3. Independence of element processing in subdivision processing

[0038] 4. Note

[0039] <1. Supporting documents on technical content and terminology, etc.>

[0040] The scope of disclosure in this technology includes not only the content described in the embodiments, but also the content described in the following non-patent documents known at the time of application, and the content of other documents cited in the following non-patent documents.

[0041] Non-patent literature 1: (as described above)

[0042] In other words, the content described in the aforementioned non-patent documents, as well as the content of other documents cited in the aforementioned non-patent documents, also serve as the basis for determining the supporting conditions.

[0043] <2. Subdivision Processing>

[0044] <v-dmc>

[0045] Historically, 3D data representing the three-dimensional structure (an object with a three-dimensional shape) has existed in the form of meshes, which represent the three-dimensional shape of an object's surface by using vertices and connections (also known as edges) to form polygons.

[0046] like Figure 1 As shown on the upper left side of the image, in the mesh, vertex 11 and the connection 12 connecting vertex 11 form polygonal planes (polygons). These polygons (also called faces) represent the surfaces of objects with a three-dimensional structure, that is, the three-dimensional shape of the object. Note that texture 13 can be applied to each of the faces in this mesh.

[0047] Grid data includes, for example, Figure 1 The information shown below. Figure 1 The first vertex information shown from the left in the downward row, 14, is information indicating the three-dimensional position (three-dimensional coordinates (X, Y, Z)) of each of the vertices 11 that form the mesh. Figure 1 The second connection information shown from the left in the next row, 15, is information indicating each of the connections (edges) 12 that form the grid. Figure 1 The third texture image shown from the left in the next row, 16, is map information about the texture 13 attached to each face. Figure 1 The fourth UV from the left in the downward direction is shown. Figure 17 This information indicates the correspondence between vertex 11 and texture 13. UV Figure 17 The coordinates (UV coordinates) of each of the vertices 11 in the texture image 16 are shown.

[0048] An example of such a grid coding method is V-DMC (Video-based Dynamic Grid Coding) disclosed in Non-Patent Document 1.

[0049] In V-DMC, the mesh to be encoded (referred to as the original mesh in this specification) is represented by a base mesh with a lower fineness (i.e., coarser) than the original mesh and displacement vectors of the subdivision points obtained by subdividing the base mesh, and the base mesh and displacement vectors are encoded.

[0050] For example, suppose there exists such as Figure 2 The original grid is shown in the top row. Figure 2 In this diagram, black dots indicate vertices, and lines connecting the black dots indicate connections (edges). As mentioned above, meshes are originally formed using vertices and edges to create polygons, but for ease of explanation, this paper will describe a mesh as a linearly (continuously) connected set of vertices.

[0051] By decimate (decimate) some of the vertices of the original mesh, such as Figure 2 The second row from the top shows a coarse (low-resolution) grid. This is called the base grid.

[0052] By subdividing each polygon of the base mesh, such as... Figure 2 The third row from the top shows the addition of vertices and edges. In this document, it is assumed that this subdivision adds vertices equal to the number of vertices divided from the original mesh. Therefore, a mesh with the same number of vertices as the original mesh is obtained. In this specification, these added vertices are also referred to as subdivision points.

[0053] However, because the connections are updated when the vertices of the original mesh are removed, and split points are formed on the updated connections (edges), the shape of the subdivided base mesh differs from the shape of the original mesh. More specifically, as... Figure 2 As shown in the bottom row, the positions of the split points (on the dashed lines) differ from those of the original mesh. In this specification, the difference between the positions of the split points and the vertices of the original mesh will be referred to as the displacement vector.

[0054] By representing the original mesh as a base mesh and displacement vectors in this way, the number of polygons (i.e., the number of vertices and edges) is reduced. Therefore, by encoding the base mesh and displacement vectors instead of the original mesh, the reduction in encoding efficiency (the increase in encoding amount) can be suppressed.

[0055] In decoding, the bitstream is decoded using a decoding method corresponding to the encoding method to recover (generate) the base grid and displacement vectors. Then, the base grid is subdivided, and the displacement vectors are applied to each subdivision point to recover (generate) the original grid. Note that in reality, the recovered grid may contain encoding distortions, etc., so there is a possibility that the recovered original grid will not perfectly match the original grid obtained before encoding. However, in the following description, for ease of explanation, it is assumed that the original grid is recovered through decoding without considering encoding distortions, etc. Furthermore, in the following explanation, it is assumed that each polygon has a triangular shape.

[0056] <Subdivision of the base mesh>

[0057] Next, a description of the subdivision of the base mesh in this decoding will be given. As mentioned above, the base mesh has a lower level of detail (fewer polygons) than the original mesh, so the polygons are subdivided until their detail becomes equivalent to that of the original mesh (i.e., vertices and edges are added to subdivide the polygons). Such a method for subdivision is arbitrary, but generally a mesh with the desired level of detail is recovered (generated) by recursively repeating the process of adding vertices and edges to the base mesh to subdivide each of the polygons (triangles) into a predetermined number (i.e., subdividing the base mesh layer by layer).

[0058] For example, in the case of a method called midpoint subdivision, each edge is divided into two by adding dividing points (vertices), thus dividing the polygon (triangle) into four. By recursively repeating this process, a mesh with the desired level of detail is restored (generated). The following description will be given with midpoint subdivision as an example.

[0059] In this detailed processing, three types of information are managed: vertex information, edge information (connection information), and triangle information (polygon information). Vertex information (vertex(float3[])) includes the identifier information (also known as vertex ID) and coordinates of each vertex. Edge information (edges(int2[])) includes the identifier information (also known as edge ID) of each edge and the identifier information of the vertices located at the two ends of the edge. Triangle information (triangles(int3[])) includes the identifier information (also known as triangle ID) of each polygon and the identifier information (vertex ID) of each vertex of the polygon.

[0060] As described above, in the subdivision of the base mesh, the level of detail is updated layer by layer through recursive processing. That is, vertex information, edge information, and triangle information are also updated layer by layer. Note that in this specification, the level of detail (also referred to as LoD (Level of Detail)) is described assuming that the base mesh (LoD0) is the lowest level and that this level becomes the next higher level each time a polygon is subdivided. In other words, the higher the level (i.e., the smaller the polygons) becomes, the higher the level becomes.

[0061] For example, suppose the base mesh has a fineness level of LoD0. In LoD0, such as... Figure 3 As shown on the left, the vertices of the base mesh are set as the vertices of LoD0, and information associated with these vertices is stored in the vertex information. Furthermore, the polygons (triangles) formed by connecting these vertices are set as triangles of LoD0, and information associated with these triangles is stored in the triangle information. Note that since no edges are formed at LoD0, information associated with the edges of LoD0 is not stored in the edge information.

[0062] A higher-level mesh (LoD1) is formed by dividing the base mesh into four polygons. For example, as... Figure 3 As shown in the center, the edges connecting the vertices of LoD0 are set as edges of LoD1, and information related to these edges is stored in the edge information. Furthermore, the midpoints of these edges of LoD1 are set as vertices of LoD1, and information related to these vertices is stored (added) in the vertex information. Additionally, the polygon (triangle) formed by connecting the vertices of LoD0 and LoD1 is set as a triangle of LoD1, and information related to this triangle is stored in the triangle information (updating the triangle information).

[0063] Similarly, a higher-level (LoD2) mesh is formed by dividing the polygons of the LoD1 mesh once (dividing it into four). For example, as... Figure 3 As shown on the right, the edges connecting the vertices of LoD1 are set as edges of LoD2, and the information associated with these edges is stored (added) in the edge information. Furthermore, the midpoints of these edges of LoD2 are set as vertices of LoD2, and the information associated with these vertices is stored (added) in the vertex information. Additionally, the polygon (triangle) formed by connecting the vertices of LoD0 to LoD2 is set as a triangle of LoD2, and the information associated with this triangle is stored in the triangle information (updating the triangle information).

[0064] <Side information update>

[0065] A more detailed description will be given. For example, such as Figure 4 As shown, assume vertices v0 to v4 and triangles t0 and t1 are formed at the base mesh (LoD0). Vertex information stores information about each vertex (v0 to v3). In fact, it also stores the coordinates of each vertex, but in... Figure 4 The diagram is omitted. Edge information stores information about edges e0 to e3. However, since edges e0 to e3 do not actually exist, dummy information (-1, -1) is stored. In addition, triangle information stores information about triangles t0 and t1 (e.g., the vertex IDs of the triangle's vertices ((v0, v1, v2), (v1, v3, v2)) etc.).

[0066] When updating edge information in the subdivision, `vertexToTriangle` and `vertexToEdge` are generated, and edge information is derived from `vertexToTriangle` and `vertexToEdge`. `vertexToTriangle` links a vertex ID to the triangle ID of the vertex to which that vertex belongs (a triangle with the vertex indicated by that vertex ID as its vertex). `vertexToEdge` links a vertex ID to the vertex ID of another vertex connected to that vertex via an edge. Both `vertexToTriangle` and `vertexToEdge` need to be generated by exhaustively searching the relationships between vertices and triangles.

[0067] First, such as Figure 5 As shown, each triangle is searched to examine the relationship between vertices and triangles, and `vertexToTriangles` is generated. Using this information, vertices are linked to triangles that touch that vertex (have that vertex as a vertex). In other words, `vertexToTriangles` allows us to derive the triangle IDs of triangles touching a vertex from its vertex ID. For example, in... Figure 5 In this case, by using vertexToTriangles, t0 is derived from v0. Furthermore, t0 and t1 are derived from v1. Furthermore, t0 and t1 are derived from v2. Furthermore, t1 is derived from v3. In other words, vertexToTriangles indicates that vertex v0 belongs to triangle t0, vertices v1 and v2 belong to triangles t0 and t1, and vertex v3 belongs to triangle t1.

[0068] Next, as Figure 6 The diagram shows the generation of `vertexToEdge`. This information links each vertex to its adjacent vertices connected via edges. In other words, `vertexToEdge` allows us to derive the vertex IDs of other vertices connected to that vertex via edges from its vertex ID. When generating `vertexToEdge`, the aforementioned `vertexToTriangles` method is used to perform a search for other vertices connected to the target vertex via edges for each triangle.

[0069] In other words, such as Figure 6 As shown in the example, vertices v1 and v2 are detected as other vertices connected to vertex v0 via edges and belonging to triangle t0. Furthermore, vertices v0 and v2 are detected as other vertices connected to vertex v1 via edges and belonging to triangle t0. Furthermore, vertices v2 and v3 are detected as other vertices connected to vertex v1 via edges and belonging to triangle t1. Furthermore, vertices v0 and v1 are detected as other vertices connected to vertex v2 via edges and belonging to triangle t0. Furthermore, vertices v1 and v3 are detected as other vertices connected to vertex v2 via edges and belonging to triangle t1. Furthermore, vertices v1 and v2 are detected as other vertices connected to vertex v3 via edges and belonging to triangle t1.

[0070] Vertices detected in this way are added to vertexToEdge. However, vertices already detected in other triangles are not added to vertexToEdge to avoid overlap. For example, vertex v2, which has been detected as being connected to vertex v1 via an edge and belongs to another vertex of triangle t1, is not added to vertexToEdge because vertex v2 is also detected as belonging to triangle t0. Similarly, vertex v1, which has been detected as being connected to vertex v2 via an edge and belongs to another vertex of triangle t1, is not added to vertexToEdge because vertex v1 is also detected as belonging to triangle t0. This process needs to be performed sequentially.

[0071] Next, use `vertexToEdge` to update the vertex information. That is, as... Figure 7 As shown in the example, the connections between vertices indicated by vertexToEdge are treated as edges, and their information is added to the edge information. However, information about edges that overlap with those already stored in the edge information is not stored. That is, overlapping edges are detected. For example, in Figure 7 In the case of vertex v0, by using vertexToEdge, we can obtain the edges from vertex v0 to vertex v1, and also the edges from vertex v1 to vertex v0. That is, vertexToEdge includes information about edges from vertex v0 to vertex v1 and from vertex v1 to vertex v0. Since these edges are identical (overlapping), they are stored as a single edge e4(v0, v1) in the edge information. Similarly, vertexToEdge includes information about edges from vertex v0 to vertex v2 and from vertex v2 to vertex v0. Since these edges are identical (overlapping), they are stored as a single edge e5(v0, v2) in the edge information. Furthermore, vertexToEdge includes information about edges from vertex v1 to vertex v2 and from vertex v2 to vertex v1. Since these edges are identical (overlapping), they are stored as a single edge e6(v1, v2) in the edge information. Furthermore, `vertexToEdge` includes information about edges from vertex v1 to vertex v3 and from vertex v3 to vertex v1. Since these edges are identical (overlapping), they are stored as a single edge `e7(v1, v3)` in the edge information. Similarly, `vertexToEdge` includes information about edges from vertex v2 to vertex v3 and from vertex v3 to vertex v2. Since these edges are identical (overlapping), they are stored as a single edge `e8(v2, v3)` in the edge information.

[0072] To avoid overlap as described above, edges e4 to e8 are added to the edge information as the processing target edges. That is, vertexToTriangle and vertexToEdge are generated, and edge information can be generated from vertexToTriangle and vertexToEdge.

[0073] <Vertex Information Update>

[0074] When updating vertex information in subdivision, the edge information updated as described above is used to update the vertex information. That is, as... Figure 8 As shown, the midpoints of the edges of the target LoD being processed, generated (added) as described above, are added to the vertex information as vertices of the target LoD. That is, in Figure 8 In the diagram, the midpoint of edge e4 is added as vertex v4. Furthermore, the midpoint of edge e5 is added as vertex v5. Additionally, the midpoint of edge e6 is added as vertex v6. Furthermore, the midpoint of edge e7 is added as vertex v7. Furthermore, the midpoint of edge e8 is added as vertex v8. In this way, the edge ID and the vertex ID of the vertex formed on that edge correspond to each other.

[0075] <Update of Triangle Information>

[0076] When updating triangle information during subdivision, the edge and vertex information updated as described above is used to update the triangle information. That is, triangles for processing the target LoD are generated such that any three vertices generated so far are used as vertices, the lower-level triangles are subdivided into four, and their information is stored in the triangle information. For example, as... Figure 9 As shown, triangles t0 to t7 are formed, each having any three vertices v0 to v8 as vertices, and their information is stored in triangle information.

[0077] To generate a triangle, use edgeToVertex, which returns the index of the midpoint of the edge ab from vertex pair a and b. edgeToVertex can be represented as follows.

[0078] edgeToVertex[a][b]=id

[0079] edge[id] = (a, b)

[0080] vertex[id] = the midpoint between a and b

[0081] For example, such as Figure 10 As shown, in triangles LoD0 When [t] = (a, b, c), the indices (a to f) of the vertices of the dividing triangle can be obtained according to the following three equations.

[0082] edgeToVertex[a][b]=d

[0083] edgeToVertex[b][c]=e

[0084] edgeToVertex[a][c]=f

[0085] Therefore, the triangle information can be updated as shown in the following four equations. In other words, the triangle information can be updated to divide the triangle into four.

[0086] triangles LoD1 [t]←(d,f,e)

[0087] triangles LoD1 [u]←(a,d,e)

[0088] triangles LoD1 [v]←(d,b,f)

[0089] triangles LoD1 [w]←(e,f,c)

[0090] The generation of edgeToVertex involves sequential processing.

[0091] As mentioned above, in the case of midpoint subdivision, when updating triangle information, the lower-level triangles are divided into four (id) now :0 to Triangle count -1). The triangle obtained after segmentation, located at the center of the triangle obtained before segmentation, is assigned the same ID as the triangle obtained before segmentation. In other words, information about the triangle obtained after segmentation, located at the center of the triangle obtained before segmentation, is written into the triangle information at the same position (id) as the triangle obtained before segmentation. center =id now The information about the other three triangles is stored after the information about the central triangle. In other words, (Id) is obtained. other =3+id now +Triangle count ) and write (Id) other +0,id other +1,id other +2).

[0092] For example, in Figure 11 In this case, triangle t0 shown on the left side of the figure is divided into four parts to generate triangles t0, t4, t5, and t6 shown on the right side of the figure. In this case, information about the triangle t0 obtained after the division is stored at the position of the triangle t0 obtained before the division, and information about triangles t4 to t6 is stored after the information about the triangles t0 to t3 obtained after the division.

[0093] Increased processing time due to sequential processing

[0094] As mentioned above, updating edge and triangle information requires sequential processing, such as exhaustive search and overlap detection, where processing is performed based on the results of other processes. Therefore, it is difficult to parallelize element processing as part of the subdivision process, and thus processing must be performed sequentially. Consequently, there is a concern that the processing time for subdivision will increase.

[0095] <3. Independence of element processing in subdivision>

[0096] In this respect, the individual elements of the subdivision process can be executed independently. For example, the information processing apparatus may include: a decoding unit that decodes a bitstream to generate a base grid; a subdivision unit that subdivides the base grid until a desired level of refinement is achieved, the subdivision being performed by recursively repeating a first generation process, a second generation process, and a third generation process until the desired level is reached, the first generation process being for independently generating edges and vertices of the target level of refinement for each edge of a lower level of refinement, the second generation process being for generating edges and vertices of the target level of refinement relative to a central triangle of a lower level of refinement, and the third generation process being for independently generating triangles of the target level of refinement for each triangle of a lower level of refinement; and a displacement vector application unit that applies displacement vectors to the subdivision points, the subdivision points being vertices of the base grid subdivided until the desired level of refinement is achieved.

[0097] Furthermore, in the information processing device, the bitstream can be decoded to generate a base grid; the base grid can be subdivided until a desired level of fineness is achieved. Subdivision is performed by recursively repeating a first generation process, a second generation process, and a third generation process until the desired level is reached. The first generation process is used to independently generate edges and vertices of the target level of fineness for each edge of a lower level of fineness. The second generation process is used to generate edges and vertices of the target level of fineness relative to the center triangle of a lower level of fineness. The third generation process is used to independently generate triangles of the target level of fineness for each triangle of a lower level of fineness. And a displacement vector can be applied to the subdivision points, which are the vertices of the base grid subdivided until the desired level of fineness is achieved.

[0098] Note that the base mesh is a mesh with a lower level of refinement than the original mesh used to encode the object, which includes vertices and connections representing the object's 3D structure. The base mesh is generated by removing vertices from the original mesh. Furthermore, the displacement vector indicates the positional difference between the split points and the vertices of the original mesh.

[0099] Decoding device

[0100] Figure 12 This is a block diagram illustrating a configuration example of a decoding device as an information processing apparatus that applies the present technology. Figure 12 The decoding device 100 shown is a device for decoding the encoded data of a grid. The decoding device 100 decodes the encoded data of a grid encoded by a method substantially similar to that described in Non-Patent Document 1 (V-DMC).

[0101] Figure 12 It shows the main processing unit, data flow, etc., and Figure 12 The examples shown are not necessarily all of them. In other words, there may be some that are not shown in the decoding device 100. Figure 12 The processing units are shown in boxes, and there may be some that are not shown in boxes. Figure 12 The processing and data flow are indicated by arrows, etc.

[0102] like Figure 12 As shown, the decoding device 100 includes: a demultiplexing unit 111, a base grid decoding unit 112, a displacement video decoding unit 113, an atlas data decoding unit 114, an attribute video decoding unit 115, a subdivision unit 116, and a displacement vector application unit 117. Note that the base grid decoding unit 112, the displacement video decoding unit 113, the atlas data decoding unit 114, and the attribute video decoding unit 115 can be combined to form a decoding unit 121 as a single processing unit.

[0103] A bitstream generated by an encoding device that encodes a mesh using the V-DMC method is provided to the decoding device 100. That is, the bitstream stores encoded data of the base mesh, encoded data of the displacement video, encoded data of the atlas data, encoded data of the attribute video, etc. The demultiplexing unit 111 demultiplexes the bitstream and extracts each piece of encoded data included in the bitstream. For example, the demultiplexing unit 111 extracts the encoded data of the base mesh from the bitstream and supplies the encoded data to the base mesh decoding unit 112. The demultiplexing unit 111 also extracts the encoded data of the displacement video from the bitstream and supplies the encoded data to the displacement video decoding unit 113. The demultiplexing unit 111 also extracts the encoded data of the atlas data from the bitstream and supplies the encoded data to the atlas data decoding unit 114. The demultiplexing unit 111 also extracts the encoded data of the attribute video from the bitstream and supplies the encoded data to the attribute video decoding unit 115.

[0104] The base grid decoding unit 112 decodes the encoded data of the base grid supplied from the demultiplexing unit 111 using a decoding method corresponding to the encoding method to generate (recover) the base grid. The base grid decoding unit 112 supplies the generated base grid to the subdivision unit 116.

[0105] The displacement video decoding unit 113 decodes the encoded data of the displacement video supplied from the demultiplexing unit 111 using a decoding method corresponding to the encoding method to generate (restore) the displacement video. The displacement video is a motion picture on which a plane mapped with displacement vectors is used as a frame. For example, in an encoding device, the displacement vectors are stored in frames at each clock time, and the displacement vectors are encoded into displacement video using a motion picture encoding method. The displacement video decoding unit 113 decodes the encoded data using a motion picture decoding method corresponding to the motion picture encoding method and generates a displacement video storing the displacement vectors. Note that the displacement video decoding unit 113 can perform decoding using atlas data supplied from the atlas data decoding unit 114. The displacement video decoding unit 113 supplies the generated displacement video (i.e., displacement vectors) to the displacement vector application unit 117.

[0106] Atlas data decoding unit 114 decodes the encoded data of the atlas data supplied from demultiplexing unit 111 using a decoding method corresponding to the encoding method to generate (restore) atlas data. The atlas data may include various types of information related to the base grid and displacement vectors. For example, the atlas data may include information for subdividing the base grid. The atlas data may also include information for decoding the encoded data of the displacement video. The atlas data may also include information for extracting displacement vectors. Atlas data decoding unit 114 supplies the generated atlas data to displacement video decoding unit 113, subdivision unit 116, and displacement vector application unit 117.

[0107] The attribute video decoding unit 115 decodes the encoded data of the attribute video supplied from the demultiplexing unit 111 using a decoding method corresponding to the encoding method to generate (restore) the attribute video. The attribute video is a motion picture of a plane on which attributes mapped to a grid are used as frames. The attribute video decoding unit 115 outputs the frame image of the generated attribute video (i.e., the attribute map corresponding to the decoded grid) to the outside of the decoding device 100. This attribute map is used, for example, for rendering in subsequent devices.

[0108] Subdivision unit 116 subdivides the base grid supplied from base grid decoding unit 112 and supplies the subdivided base grid to displacement vector application unit 117. Note that subdivision unit 116 can subdivide the base grid using atlas data supplied from atlas data decoding unit 114.

[0109] The displacement vector application unit 117 extracts (unpacks) displacement vectors from each frame of the displacement video supplied by the displacement video decoding unit 113. Furthermore, the displacement vector application unit 117 applies the extracted displacement vectors to the vertices (segmentation points) of the subdivision base mesh supplied by the subdivision unit 116 to reconstruct the mesh. In this specification, this reconstructed mesh will also be referred to as the decoded mesh. That is, it can be said that the displacement vector application unit 117 generates the decoded mesh. Note that the displacement vector application unit 117 can apply displacement vectors using atlas data supplied by the atlas data decoding unit 114. The displacement vector application unit 117 outputs the generated decoded mesh to the outside of the decoding device 100. This decoded mesh has the properties described above. Figure 1 It is used for purposes such as rendering on subsequent devices.

[0110] In the decoding device 100 with this configuration, as described above, the subdivision unit 116 independently performs the processing of each element of the subdivision process. For example, as Figure 13 As shown, the subdivision unit 116 performs vertex and edge updates through the following two generation processes.

[0111] In the first generation process, subdivision unit 116 generates vertices and edges of the target LoD located on each edge of the lower-level LoD (update for each edge). That is, subdivision unit 116 equally divides each edge of the lower-level LoD into two, sets each of the two as an edge of the target LoD, and sets the midpoint of each edge of the target LoD as a vertex of the target LoD. Vertex and edge generation is performed independently for each edge of the lower-level LoD. In other words, subdivision unit 116 can perform the generation of vertices and edges on the target edges of the lower-level LoD without depending on the processing or the result of the generation of vertices and edges on other edges of the lower-level LoD. Therefore, subdivision unit 116 generates these vertices and edges independently for each edge of the lower-level LoD.

[0112] In the second generation process, subdivision unit 116 generates vertices and edges of the target LoD located on each side of the central triangle of the target LoD (updated for each triangle). That is, subdivision unit 116 sets each edge of the central triangle as an edge of the target LoD and sets the midpoint of each edge of the target LoD as a vertex of the target LoD. Vertex and edge generation is performed independently for each triangle. In other words, subdivision unit 116 can perform the generation of vertices and edges on each side of the target triangle without depending on the processing or the result of the generation of vertices and edges on the edges of other triangles. Therefore, subdivision unit 116 generates these vertices and edges independently for each central triangle of the target LoD.

[0113] Furthermore, the subdivision unit 116 uses the vertices and edges generated by the first and second generation processes to perform a third generation process for independently generating triangles for each triangle of the target LoD for the lower-level LoD.

[0114] With this configuration, the subdivision unit 116 can independently perform element processing for each subdivision step. For example, the subdivision unit 116 can independently perform element processing for each edge of the lower-level LoD and element processing for each triangle. Therefore, for example, the subdivision unit 116 can perform element processing in multiple steps in parallel. Therefore, the subdivision unit 116 can suppress the increase in processing time of the subdivision step. Therefore, the decoding device 100 can suppress the increase in processing time for decoding the mesh.

[0115] <Decoding Process>

[0116] Reference Figure 14 The flowchart shown illustrates an example of the decoding process performed by the decoding device 100.

[0117] When the decoding process begins, in step S101, the demultiplexing unit 111 demultiplexes the bit stream and extracts various types of encoded data.

[0118] In step S102, the atlas data decoding unit 114 decodes the encoded data of the atlas data extracted in step S101 to generate (restore) the atlas data.

[0119] In step S103, the displacement video decoding unit 113 decodes the encoded data of the displacement video extracted in step S101 to generate (restore) the displacement video. At this time, the displacement video decoding unit 113 can use the atlas data generated in step S102.

[0120] In step S104, the base grid decoding unit 112 decodes the encoded data of the base grid extracted in step S101 to generate (recover) the base grid.

[0121] In step S105, subdivision unit 116 performs subdivision processing to subdivide the base mesh generated in step S104. At this time, subdivision unit 116 can use the atlas data generated in step S102.

[0122] In step S106, the displacement vector application unit 117 extracts (unpacks) displacement vectors from the displacement video generated in step S103. At this time, the displacement vector application unit 117 can use the atlas data generated in step S102. Furthermore, the displacement vector application unit 117 applies the extracted displacement vectors to the vertices (segmentation points) of the base mesh subdivided by the subdivision processing in step S105 to generate (restore) the decoded mesh. The displacement vector application unit 117 outputs the generated decoded mesh to the outside of the decoding device 100.

[0123] In step S107, the attribute video decoding unit 115 decodes the encoded data of the attribute video extracted in step S101 to generate the attribute video. The attribute video decoding unit 115 outputs the frame image (attribute map) of the attribute video to the outside of the decoding device 100.

[0124] The decoded mesh and attribute map output to the external of the decoding device 100 are used for rendering, for example, in other subsequent devices.

[0125] <Detailed Processing Flow>

[0126] Next, we will refer to Figure 15 The flowchart shown describes the process in Figure 14 An example of the subdivision processing flow performed in step S105. Refer to [reference needed] as required. Figures 16 to 18 Provide a description.

[0127] When the subdivision process begins, in step S131, subdivision unit 116 generates edge information about the edges of LoD1 based on the base mesh and updates the triangle information. Furthermore, in step S132, subdivision unit 116 generates vertex information about the vertices of LoD1 based on the base mesh. The processes in steps S131 and S132 are also referred to as preprocessing. Any method can be used for this preprocessing. For example, a reference method can be applied. Figures 3 to 11 The method described.

[0128] In other words, subdivision unit 116 can generate edges, vertices, and triangles at the penultimate level of fineness based on the base mesh corresponding to the lowest level of fineness, and perform the first to third generation processes described below for the third penultimate level of fineness and subsequent levels of fineness.

[0129] In step S133, the subdivision unit 116 updates the processing target LoD to a higher level.

[0130] In step S134, the subdivision unit 116 generates edge information about the edge of the target LoD and vertex information about the vertex of the target LoD for each edge of the lower-level LoD (first generation process). That is, in the first generation process, the subdivision unit 116 generates the vertices and edges of the target LoD located on each edge of the lower-level LoD (update for each edge). For example, the subdivision unit 116 equally divides each edge of the lower-level LoD into two, sets each of the two as an edge of the target LoD, and sets the midpoint of each edge of the target LoD as a vertex of the target LoD. The subdivision unit 116 generates vertices and edges independently for each edge of the lower-level LoD.

[0131] For example, suppose subdivision unit 116 performs as follows Figure 16 The first generation process shown generates vertices and edges of LoD2 located on the edges of LoD1. For example, when the first generation process is performed on the edge (0, 1) of LoD1 corresponding to the vertex at index 3 of LoD1, the edge (0, 3) at index 6 and the edge (1, 3) at index 7 of LoD2 are generated. Additionally, the vertex at index 6, which is the midpoint of the edge at index 6, and the vertex at index 7, which is the midpoint of the edge at index 7, are generated. These vertices are the vertices of LoD2.

[0132] When Edge Count Indicates the number of edges in the lower-level LoD, Edge Offset The offset value of the storage location of the indicator edge information and the Triangle Count When indicating the number of triangles to process the target LoD, in Figure 16 In the example shown, these values ​​are as follows.

[0133] Edge Count =3

[0134] Edge Offset =3

[0135] Triangle Count =4

[0136] Then, in the first generation process, each time two edges are added to the existing edge index id. edge .

[0137] id_edge:0 to (Edge) Count -1)

[0138] id' = 2 * id edge +Edge Offset +Edge count

[0139] The indexes for writing to the destination become id'+0 and id'+1. Therefore, the update of the edge information can be represented as follows.

[0140] edges[id'+0]=(edges[id edge ][0],id edges )

[0141] edges[id'+1]=(edges[id edge [1],id edges )

[0142] In step S135, the subdivision unit 116 generates edge information about the edges of the target LoD and vertex information about the vertices of the target LoD relative to the central triangle (second generation process). That is, in the second generation process, the subdivision unit 116 generates the vertices and edges of the target LoD located on each edge of the central triangle of the target LoD (update for each triangle). For example, the subdivision unit 116 sets each edge of the central triangle as an edge of the target LoD and sets the midpoint of each edge of the target LoD as a vertex of the target LoD. The subdivision unit 116 generates vertices and edges independently for each central triangle of the target LoD.

[0143] For example, suppose subdivision unit 116 performs as follows Figure 17 The second generation process shown is used to generate the vertices and edges of LoD2 located on the sides of the central triangle of LoD1. In this case, the edges (3,5), (4,5), and (3,4) of the gray central triangle 0 (3,5,4) of LoD1 are set as the edges of LoD2 (indices 12 to 14), and the midpoints of these edges are set as the vertices of LoD2 (indices 12 to 14).

[0144] In this case, Edge Count Edge Offset and Triangle Count The values ​​are as follows.

[0145] Edge Count =3

[0146] Edge Offset =3

[0147] Triangle Count =4

[0148] Then, in the second generation process, the three edges are added to the existing triangle index id each time. tri .

[0149] id tri :0 to (Triangle) Count / 4-1)

[0150] id' = 3 * id tri +Edge Offset +3*Edge count

[0151] The indices for writing to the destination become id'+0, id'+1, and id'+2. Therefore, the update of the edge information can be represented as follows.

[0152] edges[id'+0]=(triangles[0],triangles[1])

[0153] edges[id'+1]=(triangles[1],triangles[2])

[0154] edges[id'+2]=(triangles[2],triangles[0])

[0155] In step S136, subdivision unit 116 updates the triangle information (third generation process). That is, in the third generation process, subdivision unit 116 uses the vertices and edges generated through the first and second generation processes to independently generate triangles for the target LoD for each triangle of the lower-level LoD. Subdivision unit 116 independently generates triangles for the target LoD for each triangle.

[0156] The method for the third generation process is arbitrary. For example, information that can be processed, similar to egdeToVertex, can be prepared before the recursive repetition of the process in the subdivision process, and this information can be updated during the recursive repetition of the process.

[0157] For example, in the third generation process, subdivision unit 116 can update edgeToVertex using edges and vertices at the processing target level, where edgeToVertex represents other vertices linked to the desired vertex via edges, and subdivision unit 116 can use edgeToVertex to generate triangles at the processing target level after the update. For example, subdivision unit 116 can generate edgeToVertex of LoD1 in preprocessing and update edgeToVertex in a similar manner to edge information when the process is repeated recursively.

[0158] The triangle information can be updated in a manner similar to conventional methods. In this case, subdivision unit 116 further updates edgeToVertex. Subdivision unit 116 performs the update of edgeToVertex together with the updates of edge information and vertex information.

[0159] For example, in Figure 18 In this case, subdivision unit 116 can add vertices g and edges g as follows.

[0160] edges_LoD2[g]←(a,d)

[0161] edgeToVertex[a][d]←g

[0162] With this configuration, the subdivision unit 116 can apply a method for updating triangle information according to conventional techniques.

[0163] Furthermore, in the third generation process, the subdivision unit 116 can derive processing target-level edges corresponding to the edges of the lower-level triangles, and the subdivision unit 116 can use the derived edges to generate processing target-level triangles. For example, the subdivision unit 116 can pre-construct triangle information such that three edges are obtained from the triangle instead of three vertices, and these edges are derived from the edge list.

[0164] Since the triangle information indicates the edge information, the subdivision unit 116 can obtain the vertex indices a to f of the triangle obtained after subdivision based on this information as follows.

[0165] triangles LoD0 [t] = (d, f, e)

[0166] edges[d] = (a, b)

[0167] edges[f] = (b, c)

[0168] edges[e] = (a, c)

[0169] Subdivision unit 116 converts these into edges. Therefore, the indices of edges(d,f), (e,f), and (d,e) can be derived as follows.

[0170] edges[3t+3edge Count +edge Offset ]=(d,f)

[0171] edges[3t+3edge Count +edge Offset +1]=(e,f)

[0172] edges[3t+3edge Count +edge Offset +2]=(d,e)

[0173] In addition, the indices of edges(a,d), (b,d), etc. can be derived as follows.

[0174] edges[2d+edge Count +edge Offset ]=(a,d)

[0175] edges[2d+edge Count +edge Offset +1]=(b,d)

[0176] This allows us to obtain the indices of all the edges of the triangle formed after the segmentation, thus enabling us to update the triangle.

[0177] With this configuration, since the data to be pre-generated is only the edges of LoD1, the subdivision unit 116 can suppress the increase in required resources. The subdivision unit 116 can also suppress the increase in data transmission load.

[0178] Alternatively, triangle information about the triangles at LoD1 can be generated in advance. For example, in the third generation process, the subdivision unit 116 can generate the remaining triangles at the processing target level and the center triangles at a higher level of refinement based on the lower-level triangles and the center triangles at the processing target level.

[0179] In 0≤t <triangleCount Lod0 triangles of time LoD0 [t] = (a, b, c) and triangles LoD1 Given [t] = (d, f, e), edges[d], edges[f], and edges[e] can be obtained according to the triangle update rules of the conventional method. Therefore, the triangle can be updated in a similar way to the method described above. For example, when repeating the i-th time, the triangle at LoD I+1 is updated.

[0180] edges[d] = (a, b)

[0181] edges[f] = (b, c)

[0182] edges[e] = (a, c)

[0183] With this configuration, edgeToVertex is not required, thus suppressing the increase in required resources compared to the methods described above. Furthermore, the subdivision unit 116 can suppress the increase in data transmission load.

[0184] In step S137, subdivision unit 116 determines whether subdivision has been performed until the desired LoD is achieved. That is, it determines whether subdivision has been performed until a level of detail equivalent to that of the original mesh is obtained.

[0185] When it is determined that subdivision has not been performed until the desired LoD is reached, that is, when it is determined that the fineness is still lower than the fineness of the original mesh (coarse), subdivision unit 116 moves the processing back to step S133 and repeats the processing thereafter. In other words, the processing of steps S133 to S137 is recursively repeated until the base mesh is subdivided until the desired LoD is reached.

[0186] Then, when it is determined in step S137 that subdivision has been performed until the desired LoD is reached (subdivision has been performed until a level of detail equivalent to the original mesh is obtained), the subdivision process ends, and the process returns to... Figure 14 .

[0187] By performing the subdivision processing as described above, the subdivision unit 116 can independently perform the individual element processing of the subdivision process. For example, the subdivision unit 116 can independently perform element processing for each edge of the lower-level LoD and element processing for each triangle. Therefore, for example, the subdivision unit 116 can perform element processing in parallel. Therefore, the subdivision unit 116 can suppress the increase in processing time of the subdivision processing. Therefore, the decoding device 100 can suppress the increase in processing time for decoding the mesh.

[0188] <Parallel Processing>

[0189] As described above, the subdivision unit 116 can independently perform the processing of each element of the subdivision process. Therefore, the subdivision unit 116 can perform the processing of each element in parallel, for example, as... Figure 19 As shown in Figure 202.

[0190] exist Figure 19 In the diagram, each square represents an element processing (thread). For example... Figure 19 As shown in Figure 201, subdivision is performed for each level of fineness (LoD). Within the processing of each level, as illustrated in Figure 201, the individual element processing cannot be successfully performed in parallel using conventional methods. Therefore, there is a concern that processing time will increase.

[0191] Conversely, in the case of the aforementioned technique, element processing can be performed independently at each level. Therefore, as... Figure 19 As shown in Figure 202, element processing can be performed in parallel.

[0192] For example, in Figure 202, squares with diagonal lines from the top left to the bottom right indicate threads of the first generation process, and numbers indicate the indices of each thread. As shown in Figure 202, in the processing of each LoD, subdivision unit 116 can execute threads of the first process in parallel. In other words, subdivision unit 116 can execute the first generation process, which is performed independently for each edge of the lower level, in parallel.

[0193] Furthermore, the shaded squares indicate the threads of the second generation process, and the numbers indicate the indices of each thread. As shown in Figure 202, in the processing of each LoD, the subdivision unit 116 can execute the threads of the second process in parallel.

[0194] Furthermore, as shown in Figure 202, in the processing of each LoD, the subdivision unit 116 can perform the first processing and the second processing in parallel.

[0195] Furthermore, squares with diagonal lines from the upper right to the lower left indicate threads of the third generation process, and numbers indicate the indices of each thread. As shown in Figure 202, in the processing of each LoD, subdivision unit 116 can execute threads of the third processing in parallel. In other words, subdivision unit 116 can execute the third generation process, which is performed independently for each triangle at a lower level, in parallel.

[0196] Furthermore, the subdivision unit 116 can perform the first generation process to the third generation process in parallel for each triangle at the two lower levels of refinement.

[0197] In this way, by performing the processing in parallel, the subdivision unit 116 can suppress the increase in processing time for subdivision processing. Therefore, the decoding device 100 can suppress the increase in processing time for decoding the grid.

[0198] <CPU、GPU>

[0199] Note the above detailed processing ( Figure 15 The steps shown can be performed by any hardware. For example, the subdivision processing can be performed by a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). Typically, GPUs have higher parallel processing performance than CPUs and can process subdivision processing using this technique at a higher speed. In other words, it is possible to suppress increases in processing time. Furthermore, since the CPU's resource utilization can be reduced by using a GPU, it is possible to suppress increases in processing time for processes other than subdivision processing performed by the CPU.

[0200] <Other>

[0201] In the description above, midpoint subdivision has been used as an example of a subdivision method, but the subdivision method is arbitrary. For example, vertices can also be formed at locations other than the midpoint of an edge. Furthermore, the number of vertices, edges, and triangles added in a single subdivision (within the processing of a single LoD) can differ from the number in the example above.

[0202] Furthermore, in the description above, the polygon's shape is a triangle. However, this shape is merely an example, and the polygon can be any polygonal shape.

[0203] <4. Note>

[0204] <Application and Control of This Technology>

[0205] Note that another device, such as an encoder, can control whether the decoder (e.g., decoding device 100) applies this technique. For example, flag information indicating whether to apply this technique to decode the mesh can be associated with the bitstream as metadata and supplied to the decoder. In this case, the decoder can determine whether to apply this technique to decode the mesh based on the flag.

[0206] Computer

[0207] The above series of processes can be performed either by hardware or by software. When the series of processes are performed by software, a program configuring the software is installed in the computer. In this document, "computer" includes computers incorporated into dedicated hardware, such as general-purpose personal computers, for example, in which various programs can be installed to perform various functions.

[0208] Figure 20 This is a block diagram illustrating an example of the hardware configuration of a computer that performs the above series of processes through a program.

[0209] exist Figure 20 In the computer 900 shown, the CPU (Central Processing Unit) 901, ROM (Read Only Memory) 902 and RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0210] The input / output interface 910 is also connected to the bus 904. The input unit 911, output unit 912, storage unit 913, communication unit 914 and driver 915 are connected to the input / output interface 910.

[0211] Input unit 911 includes, for example, a keyboard, mouse, microphone, touch panel, input terminals, etc. Output unit 912 includes, for example, a display, speaker, output terminals, etc. Storage unit 913 includes, for example, a hard disk, RAM disk, non-volatile memory, etc. Communication unit 914 includes, for example, a network interface. Driver 915 drives a removable recording medium 921 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0212] In the computer configured as described above, for example, the CPU 901 loads a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes the program to perform the series of processes described above. Data required by the CPU 901 to perform various types of processing is also appropriately stored in the RAM 903.

[0213] For example, a program executed by a computer can be applied by being recorded on a removable recording medium 921, which serves as the encapsulation medium. In this case, by loading the removable recording medium 921 into the drive 915, the program can be installed in the storage unit 913 via the input / output interface 910.

[0214] Furthermore, the program can also be provided via wired or wireless transmission media such as local area networks, the Internet, and digital satellite broadcasting. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0215] Alternatively, the program can be pre-installed in ROM 902 or storage unit 913.

[0216] <Application Objectives of This Technology>

[0217] This technology can be applied to any configuration. For example, it can be applied to various electronic devices.

[0218] Furthermore, for example, this technology can be implemented as a partial configuration of an apparatus such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a device (e.g., a video device) obtained by adding other functions to the unit.

[0219] Furthermore, this technology can also be applied to network systems comprising multiple devices. For example, this technology can be implemented as cloud computing, where multiple devices share and collaborate on processing via a network. For example, this technology can be implemented in cloud services that provide image (moving image) related services to any terminal such as computers, AV (Audio Visual) devices, mobile information processing terminals, and IoT (Internet of Things) devices.

[0220] Note that in this specification, a system refers to a collection of multiple constituent elements (devices, modules (components), etc.), regardless of whether all constituent elements are housed in the same housing. Therefore, multiple devices housed in different housings and connected via a network, as well as a single device in which multiple modules are housed in a single housing, are all considered systems.

[0221] <Areas / Applications where this technology can be applied>

[0222] Systems, devices, and processing units utilizing this technology can be used in any field, such as transportation, healthcare, crime prevention, agriculture, animal husbandry, mining, beauty and personal care, factories, home appliances, weather, and nature monitoring. Furthermore, their applications are also arbitrary.

[0223] <Other>

[0224] Note that the term "flag" as used in this specification refers to information used to identify multiple states, including not only information used to identify two states, namely true (1) and false (0), but also information that can identify three or more states. Therefore, the value of the "flag" can be, for example, a binary value such as 1 / 0 or a ternary or more base value. In other words, the number of bits configured for the "flag" is arbitrary and can be one bit or more. Furthermore, since the identification information (including the flag) can take the form not only of identification information incorporated into the bitstream, but also of difference information regarding the difference between the identification information and some information used as a reference incorporated into the bitstream, in this specification, "flag" or "identification information" includes not only that information, but also the difference information relative to the information used as a reference.

[0225] Furthermore, various types of information (metadata, etc.) associated with encoded data (bitstream) can be sent or recorded in any form, as long as the information is associated with the encoded data. In this document, the term "association" means that when processing one piece of data, it enables the use (linking) of other data. In other words, related data can be compiled into a single piece of data, or they can each be treated as separate pieces of data. For example, information associated with encoded data (image) can be transmitted on a different transmission path than the transmission path of the encoded data (image). Additionally, for example, information associated with encoded data (image) can be recorded on a different recording medium than the recording medium of the encoded data (image) (or in different recording areas of the same recording medium). Note that this "association" can be applied not only to the entire data but also to a portion of the data. For example, an image and its corresponding information can be associated with each other on any unit basis, such as for each of multiple frames, for each frame, or for each portion of a frame.

[0226] Note that in this specification, for example, the terms "composite", "reuse", "add", "integrate", "merge", "store", "put in", "introduce" and "insert" mean compiling multiple objects into one, such as compiling encoded data and metadata into a single data item, and refer to one method of the aforementioned "association".

[0227] Furthermore, the implementation of this technology is not limited to the above-described implementation, and various modifications can be made without departing from the spirit of this technology.

[0228] For example, a configuration described as a single device (or processing unit) can be divided into multiple devices (or processing units). Conversely, a configuration described above as multiple devices (or processing units) can be collectively configured as a single device (or processing unit). Furthermore, configurations other than those described above can, of course, be added to the configurations of the individual devices (or processing units). Additionally, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) can be incorporated into the configuration of another device (or processing unit).

[0229] Furthermore, for example, the above procedure can be executed in any device. In this case, the device only needs to have the necessary functions (function blocks, etc.) to be able to acquire the necessary information.

[0230] Furthermore, for example, the individual steps of a single flowchart can be executed by a single device, or they can be shared by multiple devices for execution. Additionally, when multiple processes are included in a single step, these multiple processes can be executed by a single device, or they can be shared by multiple devices for execution. In other words, multiple processes included in a single step can be executed as processes of multiple steps. Conversely, processes described as multiple steps can be executed as a single step overall.

[0231] Furthermore, for example, regarding a program executed by a computer, the processing of the steps describing the program can be performed sequentially in the order described in this specification, or it can be performed in parallel and individually at necessary timings, such as when it is called. In other words, the processing of the individual steps can be performed in a different order than described above, provided that it does not cause contradiction. Additionally, the processing of the steps describing the program can be performed in parallel with the processing of other programs, or it can be performed in combination with the processing of other programs.

[0232] Furthermore, for example, multiple technologies related to this technology can be implemented independently and separately, provided that no contradiction arises. Of course, multiple arbitrary technologies can be combined and implemented. For example, part or all of the technologies described in any embodiment can be combined and implemented in combination with part or all of the technologies described in other embodiments. Furthermore, any part or all of the aforementioned technologies can be combined and implemented in combination with other technologies not described above.

[0233] Note that this technology can also be configured as follows.

[0234] (1) An information processing device, comprising:

[0235] A decoding unit that decodes the bitstream to generate a base grid;

[0236] A subdivision unit subdivides the base mesh until a desired level of refinement is achieved. This subdivision is performed by recursively repeating a first generation process, a second generation process, and a third generation process until the desired level is reached. The first generation process independently generates edges and vertices of the target refinement level for each edge at a lower level of refinement. The second generation process generates the target refinement level edges and vertices relative to the center triangle at the lower level of refinement. The third generation process independently generates the target refinement level triangles for each triangle at the lower level of refinement.

[0237] A displacement vector application unit applies a displacement vector to a segmentation point, which is a vertex of the base mesh that is subdivided until the desired level of refinement is achieved.

[0238] The base mesh is a mesh with a lower level of detail than the original mesh used to encode the target object. The original mesh includes vertices and connections representing the three-dimensional structure of the object. The base mesh is generated by removing vertices from the original mesh.

[0239] The displacement vector indicates the positional difference between the segmentation point and the vertices of the original mesh.

[0240] (2) The information processing apparatus according to (1), wherein,

[0241] The subdivision units execute the first generation process, which is performed independently for each edge of the lower level, in parallel with each other.

[0242] (3) The information processing apparatus according to (1) or (2), wherein,

[0243] The subdivision units execute the third generation process, which is performed independently for each triangle at the lower level, in parallel with each other.

[0244] (4) The information processing apparatus according to any one of (1) to (3), wherein,

[0245] The subdivided units execute the first generation process and the second generation process in parallel.

[0246] (5) The information processing apparatus according to any one of (1) to (4), wherein,

[0247] The subdivision unit performs the first generation process to the third generation process in parallel for each triangle at the two lower levels of the fineness.

[0248] (6) The information processing apparatus according to any one of (1) to (5), wherein,

[0249] In the third generation process, the subdivision unit updates edgeToVertex using the edges and vertices at the processing target level, where edgeToVertex represents other vertices linked to the desired vertex via edges, and the subdivision unit uses the updated edgeToVertex to generate triangles at the processing target level.

[0250] (7) The information processing apparatus according to any one of (1) to (5), wherein,

[0251] In the third generation process, the subdivision unit derives the processing target level edges corresponding to each edge of the lower-level triangle, and the subdivision unit uses the derived edges to generate the processing target level triangle.

[0252] (8) The information processing apparatus according to any one of (1) to (5), wherein,

[0253] In the third generation process, the subdivision unit generates the remaining triangles at the processing target level and the center triangles at the higher level of refinement based on the lower-level triangles and the center triangles at the processing target level.

[0254] (9) The information processing apparatus according to any one of (1) to (8), wherein,

[0255] The subdivision unit

[0256] Edges, vertices, and triangles of the second-to-last lowest level of refinement are generated based on the base mesh corresponding to the lowest level of refinement.

[0257] The first generation process to the third generation process is performed relative to the third lowest level of refinement and subsequent levels.

[0258] (10) An information processing method, comprising:

[0259] Decode the bitstream to generate the base grid;

[0260] The base mesh is subdivided until a desired level of refinement is achieved. This subdivision is performed by recursively repeating a first generation process, a second generation process, and a third generation process until the desired level is reached. The first generation process is used to independently generate edges and vertices of the target refinement level for each edge at a lower level of refinement. The second generation process is used to generate the target refinement level edges and vertices relative to the central triangle at the lower level of refinement. The third generation process is used to independently generate the target refinement level triangles for each triangle at the lower level of refinement.

[0261] The displacement vector is applied to the dividing points, which are the vertices of the base mesh that are subdivided until the desired level of refinement is achieved.

[0262] The base mesh is a mesh with a lower level of detail than the original mesh used to encode the target object. The original mesh includes vertices and connections representing the three-dimensional structure of the object. The base mesh is generated by removing vertices from the original mesh.

[0263] The displacement vector indicates the positional difference between the segmentation point and the vertices of the original mesh.

[0264] List of reference numerals

[0265] 100 Decoding Devices

[0266] 111 Demultiplexing Unit

[0267] 112-base grid decoding unit

[0268] 113 Displacement Video Decoding Unit

[0269] 114 Atlas Data Decoding Unit

[0270] 115 Attribute Video Decoding Unit

[0271] 116 subdivision units

[0272] 117 Displacement Vector Application Unit

[0273] 121 Decoding Unit

[0274] 900 Computers

Claims

1. An information processing apparatus, comprising: A decoding unit that decodes the bitstream to generate a base grid; A subdivision unit subdivides the base mesh until a desired level of refinement is achieved. The subdivision is performed by recursively repeating a first generation process, a second generation process, and a third generation process until the desired level is reached. The first generation process is for independently generating edges and vertices of the target level of refinement for each edge at a lower level of refinement. The second generation process is for generating edges and vertices of the target level of refinement relative to the central triangle at the lower level of refinement. The third generation process is for independently generating triangles of the target level of refinement for each triangle at the lower level of refinement. as well as A displacement vector application unit applies a displacement vector to a segmentation point, which is a vertex of the base mesh that is subdivided until the desired level of refinement is achieved. The base mesh is a mesh with a lower level of detail than the original mesh used to encode the target object. The original mesh includes vertices and connections representing the three-dimensional structure of the object. The base mesh is generated by removing vertices from the original mesh. The displacement vector indicates the positional difference between the segmentation point and the vertices of the original mesh.

2. The information processing apparatus according to claim 1, wherein, The subdivision units execute the first generation process, which is performed independently for each edge of the lower level, in parallel with each other.

3. The information processing apparatus according to claim 1, wherein, The subdivision units execute the third generation process, which is performed independently for each triangle at the lower level, in parallel with each other.

4. The information processing apparatus according to claim 1, wherein, The subdivided units execute the first generation process and the second generation process in parallel.

5. The information processing apparatus according to claim 1, wherein, The subdivision unit performs the first generation process to the third generation process in parallel for each triangle at the two lower levels of the fineness.

6. The information processing apparatus according to claim 1, wherein, In the third generation process, the subdivision unit updates edgeToVertex using the edges and vertices at the processing target level, where edgeToVertex represents other vertices linked to the desired vertex via edges, and the subdivision unit uses the updated edgeToVertex to generate triangles at the processing target level.

7. The information processing apparatus according to claim 1, wherein, In the third generation process, the subdivision unit derives the processing target level edges corresponding to each edge of the lower-level triangle, and the subdivision unit uses the derived edges to generate the processing target level triangle.

8. The information processing apparatus according to claim 1, wherein, In the third generation process, the subdivision unit generates the remaining triangles at the processing target level and the center triangles at the higher level of refinement based on the lower-level triangles and the center triangles at the processing target level.

9. The information processing apparatus according to claim 1, wherein, The subdivision unit Edges, vertices, and triangles of the second-to-last lowest level of refinement are generated based on the base mesh corresponding to the lowest level of refinement. The first generation process to the third generation process is performed relative to the third lowest level of refinement and subsequent levels.

10. An information processing method, comprising: Decode the bitstream to generate the base grid; The base mesh is subdivided until a desired level of refinement is achieved. This subdivision is performed by recursively repeating a first generation process, a second generation process, and a third generation process until the desired level is reached. The first generation process is used to independently generate edges and vertices of the target level of refinement for each edge at a lower level of refinement. The second generation process is used to generate the target level of edges and vertices relative to the central triangle at the lower level of refinement. The third generation process is used to independently generate the target level of triangles for each triangle at the lower level of refinement. as well as The displacement vector is applied to the dividing points, which are the vertices of the base mesh that are subdivided until the desired level of refinement is achieved. The base mesh is a mesh with a lower level of detail than the original mesh used to encode the target object. The original mesh includes vertices and connections representing the three-dimensional structure of the object. The base mesh is generated by removing vertices from the original mesh. The displacement vector indicates the positional difference between the segmentation point and the vertices of the original mesh.

Images