Information processing device and method

Abstract

The present disclosure relates to an information processing device and a method configured so that it is possible to suppress an increase in the data length of a vertex normal vector. The method includes deriving a maximum scalar value of a face normal vector of an adjacent face around a target vertex of a subdivided base mesh, using the maximum scalar value to scale the face normal vector of the adjacent face, using the scaled face normal vector to derive the vertex normal vector of the target vertex, and deriving a local-coordinate displacement vector based on the vertex normal vector. The present disclosure can, for example, be applied to information processing devices, electronic equipment, information processing methods, or programs, etc.

Description

Information processing device and method

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that can suppress an increase in the data length of vertex normal vectors.

[0002] Conventionally, V-DMC (Video-based Dynamic Mesh Coding) has been used as a method for encoding meshes, which are 3D data that represent the three-dimensional structure of an object using vertices and connections (see, for example, Non-Patent Document 1). In V-DMC, a base mesh, displacement vectors for vertex correction, attributes, and atlas information are encoded and transmitted as a bitstream. This displacement vector is expressed in a local coordinate system that is set based on the vertex normal vector. One method proposed for deriving this vertex normal vector is to add the face normal vectors of adjacent faces (see, for example, Non-Patent Document 2).

[0003] Generally, the scalar values ​​of vertex normal vectors and face normal vectors are expressed in floating-point format. The derivation and application of displacement vectors for each vertex are executed as parallel processing. However, the face normal vector addition process described in Non-Patent Document 2 requires mutual exclusion. Therefore, an atomic function is used for this addition. This atomic function only supports integer types (int / uint) as data types. Therefore, during this addition, the scalar value of the face normal vector is converted to integer type (int / uint).

[0004] Khaled Mammou, Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Krasimir Kolarov, "[V-CG] Apple's Dynamic Mesh Coding CfP Response", ISO / IEC JTC 1 / SC 29 / WG 7 m59281, April 2022Satoru Kuma, Kao Hayashi, Danillo Graziosi, Ohji Nakagami, "[V-DMC][New Proposal] Improvement on local coordinate system for displacement coding", ISO / IEC JTC 1 / SC 29 / WG 7 m61020, October 2022

[0005] However, because the face normal vector of an adjacent face is derived using the coordinates of each vertex of that face, the magnitude of its absolute value depends on the area of ​​the face. In other words, the magnitude of the vertex normal vector derived by adding the face normal vectors of the adjacent faces depends on the area of ​​the adjacent faces. This can increase the data length of the vertex normal vector.

[0006] The present disclosure has been made in view of such circumstances, and makes it possible to suppress an increase in the data length of vertex normal vectors.

[0007] An information processing device according to one aspect of the present technology includes a maximum scalar value derivation unit that derives a maximum scalar value of face normal vectors of adjacent faces in the vicinity of a target vertex of a subdivided base mesh; a vertex normal vector derivation unit that scales the face normal vectors of the adjacent faces using the maximum scalar value and derives a vertex normal vector of the target vertex using the scaled face normal vector; a displacement vector derivation unit that derives a displacement vector of local coordinates based on the vertex normal vector; and an encoding unit that encodes the displacement vector to generate a bitstream, wherein the base mesh is a mesh with lower resolution than the original mesh to be encoded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning vertices from the original mesh, and the displacement vector is vector information that indicates the displacement of the vertices of the subdivided base mesh.

[0008] An information processing method according to one aspect of the present technology includes deriving a maximum scalar value of face normal vectors of adjacent faces in the vicinity of a target vertex of a subdivided base mesh, scaling the face normal vectors of the adjacent faces using the maximum scalar value, deriving a vertex normal vector of the target vertex using the scaled face normal vector, deriving a displacement vector in local coordinates based on the vertex normal vector, and encoding the displacement vector to generate a bitstream, wherein the base mesh is a mesh with lower resolution than the original mesh to be encoded, which is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning vertices from the original mesh, and the displacement vector is vector information indicating the displacement of the vertices of the subdivided base mesh.

[0009] and a vertex normal vector derivation unit that uses the maximum scalar value to scale the face normal vectors of the adjacent faces and derive a vertex normal vector of the target vertex using the scaled face normal vector. A displacement vector application unit that uses the derived vertex normal vector to apply the displacement vector in local coordinates based on the vertex normal vector to the subdivided base mesh, wherein the base mesh is a mesh with lower resolution than the original mesh to be encoded, the original mesh being composed of vertices and connections that represent a three-dimensional structure of an object, and the displacement vector is vector information indicating the displacement of a vertex of the subdivided base mesh.

[0010] An information processing method according to another aspect of the present technology includes: decoding a bit stream of displacement vectors to generate the displacement vectors; deriving a maximum scalar value of face normal vectors of adjacent faces in the vicinity of a target vertex of a subdivided base mesh; scaling the face normal vectors of the adjacent faces using the maximum scalar value; deriving a vertex normal vector of the target vertex using the scaled face normal vectors; and applying the displacement vector in local coordinates based on the vertex normal vectors to the subdivided base mesh using the derived vertex normal vectors, wherein the base mesh is a mesh with lower resolution than the original mesh to be encoded, the original mesh being composed of vertices and connections that represent a three-dimensional structure of an object, and the displacement vector is vector information indicating the displacement of the vertices of the subdivided base mesh.

[0011] An information processing device and method according to one aspect of the present technology includes deriving the maximum scalar value of the face normal vectors of adjacent faces in the vicinity of a target vertex of a subdivided base mesh, scaling the face normal vectors of the adjacent faces using the maximum scalar value, and using the scaled face normal vectors to derive a vertex normal vector of the target vertex, deriving a displacement vector in local coordinates based on the vertex normal vector, and encoding the displacement vector to generate a bitstream.

[0012] Another aspect of the information processing device and method of the present technology includes decoding a bit stream of a displacement vector to generate the displacement vector, deriving a maximum scalar value of the face normal vectors of adjacent faces in the vicinity of a target vertex of the subdivided base mesh, using the maximum scalar value to scale the face normal vectors of the adjacent faces, and using the scaled face normal vectors to derive a vertex normal vector of the target vertex, and using the derived vertex normal vector to apply a displacement vector in local coordinates based on the vertex normal vector to the subdivided base mesh.

[0013] FIG. 1 is a diagram illustrating a mesh. FIG. 1 is a diagram illustrating V-DMC. FIG. 1 is a diagram illustrating vertex normal vectors. FIG. 1 is a diagram illustrating an example of a method for deriving a vertex normal vector. FIG. 2 is a diagram illustrating an example of how a maximum scalar value is derived. FIG. 3 is a diagram illustrating an example of how a vertex normal vector is derived. FIG. 4 is a diagram illustrating an example of how a vertex normal vector is applied. A block diagram showing an example of the main configuration of an encoding device. A block diagram showing an example of the main configuration of a displacement vector generation unit. A block diagram showing an example of the main configuration of a V-DMC encoding unit. A flowchart illustrating an example of the flow of an encoding process. A flowchart illustrating an example of the flow of a displacement vector generation process. A flowchart illustrating an example of the flow of a V-DMC encoding process. A block diagram showing an example of the main configuration of a decoding device. A block diagram showing an example of the main configuration of a displacement vector generation unit. A flowchart illustrating an example of the flow of a decoding process. A flowchart illustrating an example of the flow of a displacement vector application process. A block diagram showing an example of the main configuration of a computer.

[0014] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. Literature etc. supporting technical content and technical terminology 2. Deriving displacement vector 3. Scaling normal vector 4. First embodiment (encoding device) 5. Second embodiment (decoding device) 6. Supplementary notes

[0015] <1. Literature, etc. supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the content described in the embodiments, but also the content described in the following non-patent documents, etc. that were publicly known at the time of filing, and the content of other documents referenced in the following non-patent documents.

[0016] Non-patent document 1: (described above) Non-patent document 2: (described above)

[0017] In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.

[0018] In this specification, superscripts such as exponents are indicated by "^". For example, in the case of "A^B", "B" indicates a superscript (e.g., an exponent) attached to "A". Subscripts are indicated by "_". For example, in the case of "A_B", "B" indicates a subscript attached to "A".

[0019] <2. Derivation of Displacement Vector> <V-DMC> Conventionally, 3D data representing the three-dimensional structure of a three-dimensional structure (an object with a three-dimensional shape) has been available as a mesh, which represents the three-dimensional shape of the object surface by forming polygons with vertices and connections (also called edges).

[0020] As shown in the upper left of Figure 1, in a mesh, vertices 11 and connections 12 connecting these vertices 11 form polygonal planes (polygons). These polygons (also called faces) represent the surface of a three-dimensional object, i.e., the three-dimensional shape of the object. A texture 13 can be applied to each face of this mesh.

[0021] Mesh data is composed of information such as that shown in the lower part of Figure 1. Vertex information 14, shown first from the left in the lower part of Figure 1, is information indicating the three-dimensional position (three-dimensional coordinates (X, Y, Z)) of each vertex 11 that constitutes the mesh. Connection information 15, shown second from the left in the lower part of Figure 1, is information indicating each connection (edge) 12 that constitutes the mesh. A texture image 16, shown third from the left in the lower part of Figure 1, is map information for the texture 13 that is applied to each face. A UV map 17, shown fourth from the left in the lower part of Figure 1, is information indicating the correspondence between the vertices 11 and the texture 13. The UV map 17 indicates the coordinates (UV coordinates) of each vertex 11 in the texture image 16.

[0022] As an example of such a mesh coding method, there is V-DMC (Video-based Dynamic Mesh Coding) as disclosed in Non-Patent Document 1.

[0023] In V-DMC, the mesh to be encoded (referred to in this specification as the original mesh) is represented as a base mesh that is less fine (i.e., coarser) than the original mesh, and displacement vectors of the division points obtained by subdividing the base mesh, and the base mesh and displacement vectors are then encoded.

[0024] For example, assume that there is an original mesh as shown in the top row of Figure 2. The original mesh is a mesh composed of vertices and connections that represent the three-dimensional structure of an object, and is the target of encoding. For example, the original mesh is generated from a captured image of an object in real space (by camera capture). In Figure 2, black dots represent vertices, and lines connecting the black dots represent connections (edges). As described above, a mesh essentially forms polygons using vertices and edges, but for convenience of explanation, it is described here as a group of vertices connected linearly (in series).

[0025] By simplifying the original mesh, a coarse (low-resolution) mesh like the one shown in the second row from the top of Figure 2 is formed. This is called the base mesh. One simplification method is to thin out some of the vertices (decimate). In other words, the base mesh is a mesh with lower resolution than the original mesh, generated by thinning out vertices from the original mesh (i.e., simplifying the original mesh).

[0026] By subdividing each polygon of this base mesh, vertices and edges are added, as shown in the third row from the top of Figure 2. The degree of subdivision is arbitrary. That is, the number of vertices and edges added is arbitrary. For example, this subdivision can add vertices equal to the number of vertices thinned out from the original mesh. That is, subdivision can be used to maintain the same number of vertices as the original mesh. In this specification, these added vertices are also referred to as division points. This subdivision can also be repeated recursively. For example, in a technique called midpoint, the process of adding vertices to the midpoints of edges (subdivision) is repeated recursively. In other words, recursive subdivision increases the number of vertices and improves the resolution of the mesh. In this way, it is possible to perform subdivision up to any desired level of resolution (i.e., control the resolution of the subdivided mesh). In other words, the subdivided mesh can be layered according to its level of resolution. In other words, this can be considered a layering of the subdivision process and the vertices (division points) and edges obtained by the subdivision process.

[0027] However, the connections of the base mesh are updated when the vertices of the original mesh are thinned out. Therefore, the division points obtained by subdivision are formed on these updated connections (edges). As a result, the shape of the subdivided base mesh differs from the shape of the original mesh. More specifically, as shown in the bottom part of Figure 2, the positions of the division points (on the dotted line) differ from those of the original mesh. In addition, the positions of the vertices of the base mesh may differ from those of the original mesh.

[0028] In other words, by moving the positions of the vertices of the subdivided base mesh (the vertices or division points of the base mesh) closer to the vertex positions of the original mesh, the difference in shape between the subdivided base mesh and the original mesh can be reduced. In this specification, such movement of the vertices of the subdivided base mesh (the vertices or division points of the base mesh) is also referred to as displacement. Furthermore, the amount and direction of this displacement, expressed as a vector, is also referred to as a displacement vector. Ideally, by displacing each vertex of the subdivided base mesh, the shape of the subdivided base mesh can be made to match the shape of the original mesh. In other words, the original mesh can be expressed as a base mesh and a displacement vector.

[0029] In V-DMC, such base meshes and displacement vectors are coded instead of the original mesh (geometry). By coding the base meshes and displacement vectors in this way, it is possible to code with a reduced number of polygons (i.e., the number of vertices and edges) compared to coding the original mesh, which generally reduces the amount of code for the same quality. In other words, it is possible to improve coding efficiency.

[0030] During decoding, as described above, a mesh is restored (generated) by subdividing a base mesh and applying a displacement vector to each vertex of the subdivided base mesh to displace it. In this specification, this mesh is also referred to as a decoded mesh. Attributes are then applied to the decoded mesh, and the decoded mesh with the applied attributes is rendered. While the shape of the polygon (face) may be any polygonal shape, the following description will be given assuming that the polygon is triangular. Therefore, the polygon (face) will also be referred to as a triangle in the following description.

[0031] <Encoding and Decoding of V-DMC Data> In the case of V-DMC, mesh data consists of a base mesh, displacement vectors, attributes, and atlas information. This data group is also referred to as V-DMC data. The base mesh consists of information indicating vertices and connections, and is coded using an existing mesh coding method such as Draco. Note that the base mesh can be coded not only intra-coded but also inter-coded. When inter-coding, the base mesh is coded as a motion vector between frames.

[0032] Displacement vectors are arithmetically coded or packed into a two-dimensional plane and encoded as video data using a coding method for 2D video. For example, when encoded as video data, displacement vectors are converted into displacement coefficients using a predetermined method. The displacement coefficients are arranged as pixel values ​​in a two-dimensional area (also called a displacement map). This arrangement (mapping) of displacement coefficients is also called packing. Video (also called displacement video) using the displacement map as frame images is encoded using a coding method for 2D video. In other words, displacement coefficients are scalar values ​​corresponding to the displacement vectors. A displacement map is map information (also called image data) that stores displacement coefficients as pixel values. Displacement video is video image data using the displacement map as frame images.

[0033] An attribute is non-geometry information applied to a mesh (geometry), which is 3D data. For example, an attribute may include a texture applied to a face of the mesh (geometry). The attribute (e.g., texture) is divided into multiple subregions, each of which is projected in a predetermined projection direction, and the projected images (patches) are arranged in a two-dimensional region (also called an attribute map). In other words, attribute patches are packed into the attribute map. A video (also called attribute video) using the attribute map as frame images is encoded using a 2D video encoding method. In other words, the attribute map is map information (also called image data) that stores the patches (projected textures) as pixel values. Attribute video is video data using the attribute map as frame images.

[0034] Atlas information is information used when reconstructing a mesh. For example, atlas information may include correspondence between the base mesh and a displacement map or attribute map (such as a UV map), quantized values ​​of displacement vectors, etc. This atlas information is encoded using a predetermined encoding method.

[0035] The coded data (bitstream) of each data is decoded by a decoding method corresponding to the coding method. In other words, by decoding the coded data (bitstream), various information such as base meshes, displacement vectors, attributes, and atlas information is restored (generated).

[0036] <Setting a Local Coordinate System> In this specification, the magnitude (length) of a vector is also referred to as a scalar value. A vector existing in a space can be expressed by a scalar value in each axial direction (i.e., the components of length in each axial direction) in any coordinate system that represents that space. For example, a vector located in a space expressed in an XYZ Cartesian coordinate system can be expressed by a scalar value for the X-axis component, a scalar value for the Y-axis component, and a scalar value for the Z-axis component. Similarly, a displacement vector can be expressed as a scalar value in each axial direction in any coordinate system. Therefore, a method of expressing a displacement vector in a separate coordinate system (a coordinate system for that displacement vector) has been devised. In this specification, this coordinate system for each displacement vector is also referred to as a local coordinate system. Furthermore, the coordinate values ​​in that local coordinate system are also referred to as local coordinates. As this local coordinate system, for example, a coordinate system based on the normal vector of the vertex to which the displacement vector corresponds has been devised. That is, a method has been devised in which each displacement vector is expressed using local coordinates in a local coordinate system based on the normal vector of the vertex to which the displacement vector corresponds. In this specification, the normal vector of a mesh vertex is also referred to as a vertex normal vector. The normal vector of a mesh face is also referred to as a face normal vector. In this case, the local coordinate system may be defined in any way as long as it is uniquely determined based on the vertex normal vector. For example, the local coordinate system may be a Cartesian coordinate system with the direction of the vertex normal vector as one axis.

[0037] When a displacement vector is expressed using a local coordinate system based on a vertex normal vector in this way, a method of adding the face normal vector of an adjacent face has been proposed as a method of deriving the vertex normal vector in, for example, Non-Patent Document 2. In this specification, a face adjacent to a vertex (a face including the vertex) is also referred to as an adjacent face.

[0038] For example, as shown in Figure 3, suppose there are faces 31, 32, and 33, with vertex 41 at the boundary between faces 31 and 32 and vertex 42 at the boundary between faces 32 and 33. Furthermore, face 32 has division points 43, 44, and 45 derived from the midpoint. Division point 43 is a division point of LoD0 located at the midpoint of the edge between vertex 41 and vertex 42. Division point 44 is a division point of LoD1 located at the midpoint of the edge between vertex 41 and division point 43. Division point 45 is a division point of LoD1 located at the midpoint of the edge between vertex 42 and division point 43.

[0039] In this case, the vertex normal vector 61 of vertex 41, the vertex normal vector 62 of vertex 42, the vertex normal vector 63 of division point 43, the vertex normal vector 64 of division point 44, and the vertex normal vector 65 of division point 45 are each derived by adding the face normal vectors of the adjacent faces. When deriving the vertex normal vectors, for example, the following process is executed.

[0040] Step 1: The face normal vector of each adjacent face around the target vertex (the target vertex for which the vertex normal vector is derived) is calculated from the coordinates of each vertex of that face. Step 2: The face normal vectors of each face adjacent to the target vertex are added. Step 3: The result of Step 2 is normalized.

[0041] In the example of FIG. 4 , in step 1, a face normal vector 71A of face 71 is derived using the coordinates of each vertex of face 71. Similarly, a face normal vector 72A of face 72 is derived using the coordinates of each vertex of face 72. A face normal vector 73A of face 73 is derived using the coordinates of each vertex of face 73. A face normal vector 74A of face 74 is derived using the coordinates of each vertex of face 74. A face normal vector 75A of face 75 is derived using the coordinates of each vertex of face 75. A face normal vector 76A of face 76 is derived using the coordinates of each vertex of face 76.

[0042] In step 2, the face normal vector 76A is added to the face normal vector 71A. In step 3, the result of this addition is normalized to derive the vertex normal vector 81A.

[0043] 3, the vertex normal vector 61 of vertex 41 is derived by adding the face normal vector 51 of face 31 and the face normal vector 52 of face 32. Similarly, the vertex normal vector 62 of vertex 42 is derived by adding the face normal vector 52 of face 32 and the face normal vector 53 of face 33. Similarly, the vertex normal vector 63 of division point 43, the vertex normal vector 64 of division point 44, and the vertex normal vector 65 of division point 45 are derived by adding the face normal vectors of the adjacent faces. However, division points 43, 44, and 45 are vertices on face 32, and their adjacent face is face 32. Therefore, the vertex normal vector 61, vertex normal vector 62, and vertex normal vector 63 are all perpendicular to face 32. The "addition of face normal vectors" in step 2 can be performed using a scalar value. That is, the scalar values ​​representing each face normal vector may be added together to generate a scalar value representing the sum of the face normal vectors. For example, by adding the "scalar values ​​in each axis direction of the Cartesian coordinate system" corresponding to each face normal vector for each coaxial component, it is possible to derive the "scalar values ​​in each axis direction of the Cartesian coordinate system" corresponding to the sum of the face normal vectors.

[0044] Generally, the scalar values ​​of vertex normal vectors and face normal vectors are expressed in floating-point format. The derivation and application of the displacement vectors for each vertex are performed as parallel processes. However, the face normal vector addition process (step 2 above) described in Non-Patent Document 2 requires mutual exclusion. Therefore, an atomic function is used for this addition. This atomic function only supports integer types (int or uint) as data types. Therefore, during this addition, the scalar value of the face normal vector is converted to an integer type (int or uint).

[0045] However, the face normal vector of an adjacent face is derived using the coordinates of each vertex of that face. For example, the face normal vector of each adjacent face is derived by the cross product of the vectors between the vertices of that face. In other words, the larger the area of ​​a face, the longer the length of the face normal vector, and the smaller the area of ​​the face, the shorter the length of the face normal vector. Therefore, the length of a vertex normal vector derived by adding the face normal vectors of adjacent faces also depends on the area of ​​the adjacent faces. In other words, the larger the area of ​​a face, the longer the length of the vertex normal vector, and the smaller the area of ​​the face, the shorter the length of the vertex normal vector. As such, the range of the length of the vertex normal vector is not limited, and there is a risk of its range increasing. Therefore, to prevent overflow and a loss of precision, it is necessary to sufficiently lengthen the data length of the scalar value of the vertex normal vector. In other words, the above-mentioned derivation method has the risk of increasing the data length of the scalar value of the vertex normal vector.

[0046] For example, errors can occur if the processable data length (buffer size) is insufficient for the data length of the scalar value of a vertex normal vector. In other words, an increase in the data length of a vertex normal vector can reduce the accuracy of the vertex normal vector. And, a reduction in the accuracy of a vertex normal vector can reduce the accuracy of a local coordinate system based on that vertex normal vector. And, a reduction in the accuracy of the local coordinate system can reduce the accuracy of a displacement vector.

[0047] For example, if a mismatch occurs between the vertex normal vectors generated in the encoder and the vertex normal vectors generated in the decoder, a mismatch occurs between the local coordinate system in the encoder and the local coordinate system in the decoder. Therefore, a mismatch occurs between the displacement vectors generated in the encoder and the displacement vectors restored in the decoder. This increases the degradation of the restored mesh (the difference from the original mesh), and may increase the subjective quality of the decoded mesh.

[0048] To prevent a reduction in the accuracy of vertex normal vectors, it is necessary to ensure that the processable data length (buffer size) is long enough relative to the data length of the scalar value of the vertex normal vectors. Therefore, an increase in the data length of the vertex normal vectors could increase the performance requirements of the encoder and decoder (i.e., make the requirements stricter). For example, an increase in the performance requirements could increase costs. In other words, an increase in the data length of the vertex normal vectors can be said to increase the load of processing using the vertex normal vectors.

[0049] For example, graphics processing units (GPUs) on mobile devices generally only support atomic operations using 32-bit integers. However, when calculating the vertex normal vectors mentioned above, the data length of 32-bit integers can be insufficient. This can result in the decoding process not working properly or a significant reduction in the subjective quality of the decoded mesh.

[0050] <3. Scaling of normal vectors> <Method 1> Therefore, as shown in the top row of the table in FIG. 5, in deriving the vertex normal vector, the maximum scalar value of the normal vectors of adjacent faces is found, the normal vector (scalar value) is scaled based on this maximum scalar value, and the scaled normal vector (scalar value) is added (Method 1).

[0051] Hereinafter, an information processing device (encoding device) that encodes 3D data including a base mesh and a displacement vector and generates a bitstream is also referred to as a first information processing device. For example, the first information processing device includes a maximum scalar value derivation unit that derives a maximum scalar value of face normal vectors of adjacent faces around a target vertex of a subdivided base mesh, a vertex normal vector derivation unit that scales the face normal vectors of the adjacent faces using the maximum scalar value and derives a vertex normal vector of the target vertex using the scaled face normal vector, a displacement vector derivation unit that derives a displacement vector in local coordinates based on the vertex normal vector, and an encoding unit that encodes the displacement vector to generate a bitstream.

[0052] In addition, the first information processing device derives the maximum scalar value of the face normal vectors of adjacent faces in the vicinity of a target vertex of the subdivided base mesh, scales the face normal vectors of the adjacent faces using the maximum scalar value, derives a vertex normal vector of the target vertex using the scaled face normal vector, derives a displacement vector in local coordinates based on the vertex normal vector, and encodes the displacement vector to generate a bitstream.

[0053] The base mesh is a mesh with lower resolution than the original mesh to be coded, which is composed of vertices and connections that represent the three-dimensional structure of the object, and is generated by thinning out vertices from the original mesh. The displacement vector is vector information indicating the displacement of the vertices of the subdivided base mesh. The maximum scalar value indicates the maximum value among all scalar values ​​of the face normal vectors of all adjacent faces of the target vertex. Here, the scalar value may be any scalar value that can represent a face normal vector. For example, it may be the scalar value of the axial component of a coordinate system representing space.

[0054] By scaling the face normal vectors of adjacent faces based on the maximum scalar value in this manner, the range of the scalar values ​​of the vertex normal vectors can be limited within the range of that scale. That is, the first information processing device can derive a displacement vector while suppressing an increase in the data length of the vertex normal vectors. Therefore, the first information processing device can suppress a decrease in the accuracy of the vertex normal vectors. Therefore, the first information processing device can suppress a decrease in the accuracy of the local coordinate system. Therefore, the first information processing device can suppress a decrease in the accuracy of the displacement vectors. Therefore, the first information processing device can suppress a decrease in the accuracy of the decoded mesh. That is, the first information processing device can suppress an increase in degradation of the decoded mesh (the difference from the original mesh) and suppress a decrease in the subjective quality of the decoded mesh.

[0055] Furthermore, the first information processing device can suppress an increase in the processing load related to generating displacement vectors by suppressing an increase in the data length of vertex normal vectors. In other words, the first information processing device can suppress an increase in the level of performance requirements and suppress an increase in costs.

[0056] Note that the method for deriving the maximum scalar value may be any method. For example, the maximum scalar value derivation unit of the first information processing device may store integer parts of the scalar values ​​of the face normal vectors of the adjacent faces, and may select the maximum value of the stored integer parts as the maximum scalar value.

[0057] For example, as shown in Figure 6, the scalar values ​​of each axial component of the face normal vector (Normal[v]) of each adjacent face corresponding to the target vertex are extracted as integer data by rounding off the decimal values, and stored. Then, the maximum value among all the scalar values ​​of all the adjacent faces stored in this way is identified (Max(Normal[v])). By storing and comparing the integer parts of the scalar values ​​in this way, the first information processing device can easily derive the maximum scalar value.

[0058] Furthermore, the method of scaling the face normal vector using the maximum scalar value may be any method, for example, the vertex normal vector derivation unit of the first information processing device may divide the scalar value of the face normal vector of the adjacent face by the maximum scalar value.

[0059] For example, as shown in FIG. 7, the scalar value of the face normal vector of an adjacent face (round(cross(v1-v0, v2-v0))) may be divided (÷) by the maximum scalar value (MaxNormal[v]). Note that cross(A,B) indicates the cross product of vector A and vector B. In other words, cross(A,B) indicates the normal vector of the plane containing vectors A and B. v0, v1, and v2 indicate the vertices of the adjacent faces, as shown in the upper part of FIG. 7. In other words, (v1-v0) and (v2-v0) indicate the vectors between the vertices (the edges of the face), respectively. Furthermore, round(A) indicates a function that rounds A up or down. In other words, "round(cross(v1-v0, v2-v0))÷MaxNormal[v]" indicates the value obtained by dividing the scalar value of the face normal vector of an adjacent face by the maximum scalar value, i.e., the value obtained by scaling the scalar value of the face normal vector of an adjacent face by the maximum scalar value.

[0060] By performing scaling in this manner, the first information processing apparatus can easily perform scaling of the face normal vector using the maximum scalar value.

[0061] In the following description, an information processing device (decoding device) that decodes a bit stream of 3D data including a base mesh and a displacement vector and reconstructs a decoded mesh is also referred to as a second information processing device. For example, the second information processing device includes a displacement vector decoding unit that decodes a bit stream of the displacement vector to generate the displacement vector, a maximum scalar value derivation unit that derives a maximum scalar value of face normal vectors of adjacent faces in the vicinity of a target vertex of the subdivided base mesh, a vertex normal vector derivation unit that scales the face normal vectors of the adjacent faces using the maximum scalar value and derives a vertex normal vector of the target vertex using the scaled face normal vector, and a displacement vector application unit that applies a displacement vector in local coordinates based on the vertex normal vector to the subdivided base mesh using the derived vertex normal vector.

[0062] In addition, in a second information processing device, the following steps are performed: decoding the bit stream of the displacement vector to generate the displacement vector; deriving the maximum scalar value of the face normal vectors of adjacent faces in the vicinity of the target vertex of the subdivided base mesh; scaling the face normal vectors of the adjacent faces using the maximum scalar value; deriving the vertex normal vector of the target vertex using the scaled face normal vector; and applying the displacement vector in local coordinates based on the vertex normal vector to the subdivided base mesh using the derived vertex normal vector.

[0063] The base mesh is a mesh with lower resolution than the original mesh to be coded, which is composed of vertices and connections that represent the three-dimensional structure of the object, and is generated by thinning out vertices from the original mesh. The displacement vector is vector information indicating the displacement of the vertices of the subdivided base mesh. The maximum scalar value indicates the maximum value among all scalar values ​​of the face normal vectors of all adjacent faces corresponding to the target vertex. Here, the scalar value may be any scalar value that can represent a face normal vector. For example, it may be the scalar value of the axial component of a coordinate system representing space.

[0064] The displacement vector application unit of the second information processing device converts the displacement vector (dst[v]) in the local coordinate system into an expression (vertex[v]) using the world coordinate system, as shown in Figure 8, and applies it to the subdivided base mesh.

[0065] By scaling the face normal vectors of adjacent faces based on the maximum scalar value in this manner, the range of the scalar values ​​of the vertex normal vectors can be limited within the range of that scale. That is, the second information processing device can apply displacement vectors to the subdivided base mesh while suppressing an increase in the data length of the vertex normal vectors. Therefore, the second information processing device can suppress a decrease in the accuracy of the vertex normal vectors. Therefore, the second information processing device can suppress a decrease in the accuracy of the local coordinate system. Therefore, the second information processing device can suppress a decrease in the accuracy of the displacement vectors. Therefore, the second information processing device can suppress a decrease in the accuracy of the decoded mesh. That is, the second information processing device can suppress an increase in degradation of the decoded mesh (the difference from the original mesh) and suppress a decrease in the subjective quality of the decoded mesh.

[0066] Furthermore, the second information processing device can suppress an increase in the data length of the vertex normal vectors, thereby suppressing an increase in the processing load related to the application of the displacement vectors. In other words, the second information processing device can suppress an increase in the level of performance requirements and suppress an increase in costs.

[0067] Note that any method for deriving the maximum scalar value may be used. For example, as in the case of the first information processing device described above, the maximum scalar value derivation unit of the second information processing device may store the integer parts of the scalar values ​​of the face normal vectors of the adjacent faces and select the maximum value of the stored integer parts as the maximum scalar value. In this way, by storing and comparing the integer parts of the scalar values, the second information processing device can easily derive the maximum scalar value.

[0068] Furthermore, the method of scaling the face normal vector using the maximum scalar value may be any method. For example, as in the case of the first information processing device described above, the vertex normal vector derivation unit of the second information processing device may divide the scalar value of the face normal vector of the adjacent face by the maximum scalar value as scaling. By performing scaling in this manner, the second information processing device can easily perform scaling of the face normal vector using the maximum scalar value.

[0069] As described above, in deriving the vertex normal vector, the length of the face normal vector to be added can be considered as the weight value of the normalized face normal vector (face normal vector with a length of "1"). In other words, the addition of the face normal vectors of each adjacent face described above can be considered as adding the values ​​obtained by multiplying the normalized face normal vector of each adjacent face by its respective weight value. Therefore, scaling the face normal vector using the maximum scalar value described above can be considered as scaling using the maximum weight value of the (normalized) face normal vector. Therefore, in the first information processing device and the second information processing device, the maximum scalar value derivation unit can be considered as deriving the maximum value of this weight value (also referred to as the maximum weight value). In other words, the maximum scalar value derivation unit can be considered as a maximum weight value derivation unit. Furthermore, the vertex normal vector derivation unit can be considered as scaling the face normal vector of the adjacent face using the maximum weight value and deriving the vertex normal vector of the target vertex using the scaled face normal vector.

[0070] <Method 1-1> When Method 1 is applied, the scalar value of the face normal vector may be quantized (Method 1-1) as shown in the second row from the top of the table in FIG. 5 . For example, when deriving the vertex normal vector (adding the face normal vector), the scalar value of the face normal vector may be quantized with the quantized value (F2I_DIGIT) as shown in FIG. 6 . For example, in the first and second information processing devices, the vertex normal vector derivation unit may quantize the scalar value of the face normal vector with the quantized value (F2I_DIGIT). This allows the first and second information processing devices to control the precision of the scalar value when converting to integer data. This allows the first and second information processing devices to more appropriately control the data length of the vertex normal vector. In other words, the first and second information processing devices can prevent unnecessary precision reduction of the vertex normal vector while suppressing errors such as overflow.

[0071] <Method 1-1-1> The quantization value may be any value. For example, when Method 1-1 is applied, quantization may be performed using a fixed quantization value, as shown in the third row from the top of the table in FIG. 5 (Method 1-1-1). In other words, the quantization value (F2I_DIGIT) may be predetermined by a standard or the like, and the quantization may be performed using the predetermined quantization value. For example, the vertex normal vector derivation units of the first information processing device and the second information processing device may quantize the face normal vector using a predetermined quantization value, and derive the vertex normal vector of the target vertex using the scaled and quantized face normal vector.

[0072] That is, in this case, the first information processing device and the second information processing device each perform this quantization using a predetermined quantization value. That is, by applying this method, it is possible to omit the transmission of information about this quantization value from the encoding side to the decoding side. Therefore, the first information processing device and the second information processing device can suppress an increase in the amount of code. Furthermore, since the quantization value is predetermined, there is no need for processing such as deriving the quantization value. Therefore, the first information processing device and the second information processing device can suppress an increase in the processing load related to quantization.

[0073] The fixed quantized value (F2I_DIGIT) may be any value, for example, 2^16 as shown in FIG.

[0074] <Method 1-1-2> When Method 1-1 is applied, a quantized value may be transmitted (Method 1-1-2), as shown in the fourth row from the top of the table in FIG. 5 . That is, the encoding device may set this quantized value (F2I_DIGIT) and use it for the above-mentioned quantization, and may transmit the quantized value to the decoding device. For example, the vertex normal vector derivation unit of the first information processing device may set a quantized value, quantize a face normal vector using the set quantized value, and derive a vertex normal vector of the target vertex using the scaled and quantized face normal vector. Then, the encoding unit may store the set quantized value in the bitstream.

[0075] In response to this, the decoding device may obtain the transmitted quantized value and use it for the above-mentioned quantization. For example, the displacement vector decoding unit of the second information processing device may further obtain the quantized value from the bitstream. Then, the vertex normal vector derivation unit may quantize the face normal vector using the quantized value, and derive the vertex normal vector of the target vertex using the scaled and quantized face normal vector.

[0076] The quantization value may be stored anywhere in the bitstream. For example, the quantization value may be stored in a high-level syntax. The quantization value may or may not be coded. In this way, the encoding device and the decoding device transmit and share the quantization value, so that the quantization value can be changed. Therefore, the first information processing device and the second information processing device can perform quantization by applying a more appropriate quantization value. In other words, the first information processing device and the second information processing device can more appropriately control the data length of the vertex normal vector than when the quantization value is a fixed value.

[0077] The control unit of the quantization value is arbitrary. For example, the quantization value may be set (updated) for each model. Alternatively, the quantization value may be set (updated) for each scene. Alternatively, the quantization value may be set (updated) for each frame. Alternatively, the quantization value may be set (updated) at predetermined time intervals. Alternatively, the quantization value may be set (updated) for each sequence. The method for setting the quantization value is arbitrary. For example, the setting method (Method 1-1-4) described below may be applied.

[0078] <Method 1-1-3> When Method 1-1 is applied, an index corresponding to the quantization value may be transmitted (Method 1-1-3), as shown in the fifth row from the top of the table in FIG. 5 . That is, the encoding device may set this quantization value (F2I_DIGIT) and use it for the quantization described above, and may also obtain an index corresponding to the quantization value in the specified table information and transmit the index to the decoding device. For example, the vertex normal vector derivation unit of the first information processing device may set a quantization value, quantize a face normal vector using the set quantization value, and derive a vertex normal vector of the target vertex using the scaled and quantized face normal vector. The vertex normal vector derivation unit may then obtain an index corresponding to the quantization value based on the specified table information. The encoding unit may then store the index in the bitstream.

[0079] In response to this, the decoding device may obtain the transmitted index, calculate a quantization value corresponding to the index using the same table information as the encoding device, and use the quantization value for the above-mentioned quantization. For example, the displacement vector decoding unit of the second information processing device may further obtain the index from the bitstream. Then, the vertex normal vector derivation unit may quantize the face normal vector using the quantization value corresponding to the index in the predetermined table information, and derive the vertex normal vector of the target vertex using the scaled and quantized face normal vector.

[0080] The index may be stored at any location in the bitstream. For example, the index may be stored in a high-level syntax. The index may or may not be coded. In this way, the coding device and the decoding device transmit and share the index corresponding to the quantization value, so that the quantization value can be changed. Therefore, the first information processing device and the second information processing device can perform quantization by applying a more appropriate quantization value. In other words, the first information processing device and the second information processing device can more appropriately control the data length of the vertex normal vector than when the quantization value is a fixed value.

[0081] The control unit of the quantization value (index) is arbitrary. For example, the quantization value (index) may be set (updated) for each model. The quantization value (index) may be set (updated) for each scene. The quantization value (index) may be set (updated) for each frame. The quantization value (index) may be set (updated) at predetermined time intervals. The quantization value (index) may be set (updated) for each sequence. The method for setting the quantization value is arbitrary. For example, the setting method (Method 1-1-4) described below may be applied.

[0082] <Method 1-1-4> When Method 1-1 is applied, the encoding device and decoding device may derive the quantized value (Method 1-1-4), as shown in the bottom row of the table in Figure 5. That is, the encoding device may derive this quantized value (F2I_DIGIT) and use it for the above-mentioned quantization. For example, the vertex normal vector derivation unit of the first information processing device may derive the quantized value, quantize the face normal vector using the derived quantized value, and derive the vertex normal vector of the target vertex using the scaled and quantized face normal vector.

[0083] In contrast, the decoding device may also derive this quantized value (F2I_DIGIT) in the same manner as the encoding device and use it for the above-mentioned quantization. For example, the vertex normal vector derivation unit of the second information processing device may derive a quantized value, quantize a face normal vector using the derived quantized value, and derive a vertex normal vector of the target vertex using the scaled and quantized face normal vector.

[0084] By applying this method, it is possible to omit the transmission of information regarding this quantization value from the encoding side to the decoding side. Therefore, the first information processing device and the second information processing device can suppress an increase in the amount of code. Furthermore, with this method, the encoding device and the decoding device can change the quantization value. Therefore, the first information processing device and the second information processing device can perform quantization by applying a more appropriate quantization value. In other words, the first information processing device and the second information processing device can more appropriately control the data length of the vertex normal vector than when the quantization value is a fixed value.

[0085] The control unit of the quantization value (index) is arbitrary. For example, the quantization value (index) may be set (updated) for each model. Alternatively, the quantization value (index) may be set (updated) for each scene. Alternatively, the quantization value (index) may be set (updated) for each frame. Alternatively, the quantization value (index) may be set (updated) at predetermined time intervals. Alternatively, the quantization value (index) may be set (updated) for each sequence.

[0086] The quantization value may be derived by any method. For example, the encoding device and the decoding device may each prepare table information of certain parameters and quantization values ​​and derive the quantization value based on the table information and the parameters. The encoding device and the decoding device may also each derive the quantization value based on the number (k) of adjacent faces around the target vertex. For example, the vertex normal vector derivation units of the first information processing device and the second information processing device may derive the quantization value based on the number of adjacent faces.

[0087] For example, the vertex normal vector derivation unit may use the following formula (1) to derive the quantized value (F2I_DIGIT) from the number of adjacent faces (k) around the target vertex.

[0088] F2I_DIGIT = 2^(32-bitSizeOf(k)) ...(1)

[0089] In this formula (1), bitSizeOf(k) indicates the bit size (bit length) required to express the number of adjacent faces (k). This function may be converted into table information, and the vertex normal vector derivation unit may derive the quantized value (F2I_DIGIT) from the number of adjacent faces (k) around the target vertex based on the table information.

[0090] <Mode Control> Two or more of the above-described methods 1-1-1 to 1-1-4 may be prepared as operation modes, and the encoding device and the decoding device may select and apply a desired mode from the options. In this case, the encoding device and the decoding device may transmit flag information that controls the mode so that the same mode is applied to each other. In this way, the first information processing device and the second information processing device can select a more appropriate mode and more appropriately control the data length of the vertex normal vectors.

[0091] <Scope of application of explanation> In this specification, an explanation given for a higher-level method also applies to lower-level methods belonging to that method, unless a contradiction arises. For example, when it is explained that "Method 1 may be applied," it means that Method 1-1 may be applied. Of course, it also means that an even lower-level method (e.g., Method 1-1-1) may be applied.

[0092] <Combination> Each of the above-described methods may be applied in combination with any other method as long as no contradiction occurs. Three or more methods may be applied in combination. Furthermore, techniques that can be combined may include not only those shown in the table of FIG. 5 as "methods," but also all elements described in this specification. Furthermore, each of the above-described methods may be applied in combination with methods other than those described above.

[0093] <4. First embodiment> <Encoding device> The present technology can be applied to any device. For example, the present technology can be applied to an encoding device that encodes a mesh and generates a bitstream. FIG. 9 is a block diagram showing an example of the configuration of an encoding device, which is one aspect of an information processing device to which the present technology is applied. The encoding device 300 (first information processing device) shown in FIG. 9 is a device that encodes a mesh and generates a bitstream thereof. Therefore, the encoding device 300 can also be said to be a bitstream generation device that generates a bitstream.

[0094] Fig. 9 shows the main processing units, data flows, etc., but is not limited to all that is shown in Fig. 9. In other words, in encoding device 300, there may be processing units that are not shown as blocks in Fig. 9, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 9.

[0095] The encoding device 300 encodes a mesh using a method essentially similar to the V-DMC method described in the aforementioned non-patent document, except that the present technology is applied. For example, the encoding device 300 acquires an original mesh to be encoded and an attribute map including a texture corresponding to the original mesh. Note that the original mesh includes not only information about the mesh's geometry but also information indicating the correspondence with the attribute map (e.g., a UV list). The encoding device 300 encodes the original mesh and attribute map using the V-DMC method, generates a V-DMC bitstream, and outputs it.

[0096] 9 , the encoding device 300 (first information processing device) has a preprocessing unit 311 and a V-DMC encoding unit 312. The preprocessing unit 311 performs preprocessing before encoding. As shown in FIG. 9 , the preprocessing unit 311 has a base mesh generation unit 321, an atlas information generation unit 322, and a displacement vector generation unit 323.

[0097] The base mesh generation unit 321 performs processing related to the generation of a base mesh. For example, the base mesh generation unit 321 may obtain an original mesh to be input to the encoding device 300. The base mesh generation unit 321 may also perform decimation processing (thinning out vertices) on the original mesh to generate a base mesh. The base mesh generation unit 321 may supply the generated base mesh together with the original mesh to the atlas information generation unit 322.

[0098] The atlas information generation unit 322 performs processing related to the generation of atlas information corresponding to the base mesh. For example, the atlas information generation unit 322 may acquire the base mesh or original mesh supplied from the base mesh generation unit 321. The atlas information generation unit 322 may generate the atlas information by UV unwrapping the base mesh, etc. The atlas information generation unit 322 may supply the generated atlas information to the displacement vector generation unit 323 together with the base mesh, etc.

[0099] The displacement vector generation unit 323 performs processing related to the generation of displacement vectors. For example, the displacement vector generation unit 323 may acquire a base mesh, atlas information, etc. supplied from the atlas information generation unit 322. The displacement vector generation unit 323 may also acquire an original mesh input to the encoding device 300. The displacement vector generation unit 323 may also use this information to generate displacement vectors that displace vertices of the subdivided base mesh. The displacement vector generation unit 323 may supply the generated displacement vectors to the V-DMC encoding unit 312 together with the base mesh, atlas information, etc.

[0100] The V-DMC encoder 312 performs processing related to encoding of V-DMC data. For example, the V-DMC encoder 312 may acquire an original mesh input to the encoding device 300. The V-DMC encoder 312 may also acquire a base mesh, a displacement vector, atlas information, and the like supplied from the displacement vector generator 323. The V-DMC encoder 312 may also acquire an attribute map input to the encoding device 300. The V-DMC encoder 312 may apply the above-described method 1 and use this information to encode the atlas information, the base mesh, the displacement vector, and the attribute map, respectively, to generate the respective encoded data. Therefore, the V-DMC encoder 312 can also be referred to as an encoder. The V-DMC encoder 312 may also multiplex these encoded data as substreams to generate a single bitstream. This bitstream is also referred to as a V-DMC bitstream. Therefore, the V-DMC encoder 312 can also be called a bitstream generator (or a V-DMC bitstream generator). The V-DMC encoder 312 may output the generated V-DMC bitstream to the outside of the encoding device 300.

[0101] <Displacement Vector Generator> Fig. 10 is a block diagram showing an example of the main configuration of the displacement vector generator 323. Note that Fig. 10 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, the displacement vector generator 323 may include processing units that are not shown as blocks in Fig. 10, or processes or data flows that are not shown as arrows, etc. in Fig. 10.

[0102] As shown in FIG. 10, the displacement vector generation unit 323 includes a subdivision unit 331 , a maximum scalar value derivation unit 332 , a vertex normal vector derivation unit 333 , and a displacement vector derivation unit 334 .

[0103] The subdivision unit 331 executes processing related to subdivision of the faces of the base mesh. For example, the subdivision unit 331 may acquire information such as the base mesh supplied from the atlas information generation unit 322. The subdivision unit 331 may subdivide (the faces of) the base mesh to generate division points. In other words, the subdivision unit 331 may subdivide the base mesh generated by the base mesh generation unit 321. The subdivision unit 331 may supply the "subdivided base mesh" to the maximum scalar value derivation unit 332 together with information such as the base mesh.

[0104] The maximum scalar value derivation unit 332 performs processing related to the derivation of the maximum scalar value. For example, the maximum scalar value derivation unit 332 may acquire information such as the "subdivided base mesh" and the base mesh supplied from the subdivision unit 331. The maximum scalar value derivation unit 332 may apply the above-described method 1 to derive the maximum scalar value of the face normal vectors of adjacent faces around the target vertex of the subdivided base mesh. For example, as shown in the example of FIG. 6 , the maximum scalar value derivation unit 332 may retain the integer parts of the scalar values ​​of the face normal vectors of the adjacent faces and select the maximum value of the retained integer parts as the maximum scalar value. The maximum scalar value derivation unit 332 may supply the derived maximum scalar value to the vertex normal vector derivation unit 333 together with information such as the "subdivided base mesh" and the base mesh.

[0105] The vertex normal vector derivation unit 333 performs processing related to the derivation of vertex normal vectors. For example, the vertex normal vector derivation unit 333 may acquire information such as the "subdivided base mesh," the base mesh, and the maximum scalar value supplied from the maximum scalar value derivation unit 332. The vertex normal vector derivation unit 333 may apply the above-described method 1, scale the face normal vectors of adjacent faces around the target vertex of the "subdivided base mesh" using the maximum scalar value, and derive the vertex normal vector of the target vertex using the scaled face normal vector. For example, as shown in the example of FIG. 7 , the vertex normal vector derivation unit 333 may scale the face normal vectors of adjacent faces around the target vertex by dividing the scalar value of the face normal vector of the adjacent face by the maximum scalar value. The vertex normal vector derivation unit 333 may supply the derived vertex normal vectors to the displacement vector derivation unit 334, along with information such as the "subdivided base mesh" and the base mesh.

[0106] The displacement vector derivation unit 334 performs processing related to the derivation of a displacement vector. For example, the displacement vector derivation unit 334 may acquire information such as a "subdivided base mesh," a base mesh, and vertex normal vectors supplied from the vertex normal vector derivation unit 333. Alternatively, the displacement vector derivation unit 334 may acquire an original mesh input to the encoding device 300. Alternatively, the displacement vector derivation unit 334 may acquire atlas information supplied from the atlas information generation unit 322. The displacement vector derivation unit 334 may derive, as a displacement vector, the difference in position between the vertices of the acquired "subdivided base mesh" and the original mesh. In this case, the displacement vector derivation unit 334 may apply the above-described method 1 and use a vertex normal vector to derive a displacement vector in local coordinates based on the vertex normal vector (a displacement vector expressed in a local coordinate system). The displacement vector derivation unit 334 may supply the derived displacement vector (local coordinates) to the V-DMC encoding unit 312 together with information such as the base mesh and atlas information.

[0107] <V-DMC Encoder> Fig. 11 is a block diagram showing an example of the main configuration of the V-DMC encoder 312. Note that Fig. 11 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, the V-DMC encoder 312 may include processing units that are not shown as blocks in Fig. 11, and may include processing and data flows that are not shown as arrows or the like in Fig. 11.

[0108] As shown in FIG. 11, the V-DMC encoding unit 312 has an atlas information encoding unit 351, a base mesh encoding unit 352, a displacement vector correction unit 353, a displacement vector encoding unit 354, a mesh reconstruction unit 355, an attribute map conversion unit 356, an attribute encoding unit 357, and a multiplexing unit 358.

[0109] The atlas information encoding unit 351 performs processing related to encoding of the atlas information. For example, the atlas information encoding unit 351 may acquire atlas information supplied from the displacement vector generation unit 323. The atlas information encoding unit 351 may also encode the acquired atlas information using a predetermined encoding method to generate encoded data of the atlas information. The atlas information encoding unit 351 may also supply the generated encoded data of the atlas information to the multiplexing unit 358.

[0110] The base mesh encoding unit 352 performs processing related to encoding of the base mesh. For example, the base mesh encoding unit 352 may acquire a base mesh supplied from the displacement vector generation unit 323. The base mesh encoding unit 352 may acquire atlas information supplied from the displacement vector generation unit 323. The base mesh encoding unit 352 may quantize the acquired base mesh and encode it using a predetermined encoding method (e.g., Draco) to generate encoded data of the base mesh. The base mesh encoding unit 352 may supply the generated encoded data of the base mesh to the displacement vector correction unit 353. The base mesh encoding unit 352 may also supply the generated encoded data of the base mesh to the multiplexing unit 358.

[0111] The displacement vector correction unit 353 performs processing related to the correction of the displacement vector. For example, the displacement vector correction unit 353 may acquire a base mesh and a displacement vector supplied from the displacement vector generation unit 323. Alternatively, the displacement vector correction unit 353 may acquire encoded data of the base mesh supplied from the base mesh encoding unit 352. The displacement vector correction unit 353 may correct the displacement vector based on this information. For example, the displacement vector correction unit 353 may decode the acquired encoded data of the base mesh, compare the base mesh before and after encoding to determine encoding distortion of the base mesh, and correct the displacement vector in accordance with the encoding distortion. The displacement vector correction unit 353 may supply the corrected displacement vector to the displacement vector encoding unit 354. Alternatively, the displacement vector correction unit 353 may dequantize the decoded base mesh and supply it to the mesh reconstruction unit 355.

[0112] The displacement vector encoding unit 354 performs processing related to encoding of displacement vectors. For example, the displacement vector encoding unit 354 may acquire displacement vectors supplied from the displacement vector correction unit 353. The displacement vector encoding unit 354 may also generate a displacement map by wavelet transforming the displacement vectors, quantizing them, and packing them into a two-dimensional region. The displacement vector encoding unit 354 may also generate a displacement video in which the displacement map is used as frame images. In other words, the displacement video is a moving image in which the frame images are a displacement map, which is a two-dimensional region in which displacement vectors are packed. The displacement vector encoding unit 354 may also encode the generated displacement video using a predetermined encoding method for 2D moving images to generate encoded data of displacement vectors (displacement video). The displacement vector encoding unit 354 may also supply the encoded data of the displacement vectors generated in this manner to the multiplexing unit 358. The displacement vector encoding unit 354 may also decode the generated encoded data, unpack the displacement vectors from the displacement map, and dequantize the displacement vectors. The displacement vector encoding unit 354 may supply the dequantized displacement vector to the mesh reconstruction unit 355 .

[0113] The displacement vector encoding unit 354 may arithmetically encode the displacement vector to generate encoded data of the displacement vector. In this case, the displacement vector encoding unit 354 may arithmetically decode the encoded data to generate a displacement vector and supply the generated displacement vector to the mesh reconstruction unit 355.

[0114] The mesh reconstruction unit 355 performs processing related to mesh reconstruction. For example, the mesh reconstruction unit 355 may acquire a base mesh supplied from the displacement vector correction unit 353. The mesh reconstruction unit 355 may also acquire a displacement vector supplied from the displacement vector encoding unit 354. The mesh reconstruction unit 355 may use these to reconstruct a mesh. The mesh reconstruction unit 355 may supply the reconstructed mesh to the attribute map conversion unit 356.

[0115] The attribute map conversion unit 356 performs processing related to attribute map conversion. For example, the attribute map conversion unit 356 may acquire a reconstructed mesh supplied from the mesh reconstruction unit 355. The attribute map conversion unit 356 may also acquire atlas information supplied from the displacement vector generation unit 323. The attribute map conversion unit 356 may also acquire an original mesh and attribute map input to the encoding device 300. The attribute map conversion unit 356 may convert the acquired attribute map based on other acquired information. For example, the attribute map conversion unit 356 may convert the attribute map based on the atlas information, the original mesh, etc., so that it corresponds to the reconstructed mesh. In other words, the attribute map conversion unit 356 can be said to generate a converted attribute map. Therefore, the attribute map conversion unit 356 can also be said to be an attribute map generation unit. The attribute map conversion unit 356 may supply the converted attribute map to the attribute encoding unit 357 .

[0116] The attribute encoding unit 357 performs processing related to encoding of attributes. For example, the attribute encoding unit 357 may acquire an attribute map supplied from the attribute map conversion unit 356. The attribute encoding unit 357 may also generate attribute video using the acquired attribute map as frame images. The attribute encoding unit 357 may also encode the generated attribute video using a predetermined encoding method for 2D video to generate encoded data of attributes. The attribute encoding unit 357 may also supply the generated encoded data of attributes to the multiplexing unit 358.

[0117] The multiplexing unit 358 performs processing related to multiplexing of encoded data (substreams). For example, the multiplexing unit 358 may acquire encoded data of atlas information supplied from the atlas information encoding unit 351. Alternatively, the multiplexing unit 358 may acquire encoded data of base meshes supplied from the base mesh encoding unit 352. Alternatively, the multiplexing unit 358 may acquire encoded data of displacement vectors supplied from the displacement vector encoding unit 354. Alternatively, the multiplexing unit 358 may acquire encoded data of attributes supplied from the attribute encoding unit 357. The multiplexing unit 358 may multiplex these pieces of encoded data as substreams to generate a V-DMC bitstream. Therefore, the multiplexing unit 358 can also be referred to as a bitstream generation unit (or V-DMC bitstream generation unit). The multiplexing unit 358 may output the generated V-DMC bitstream to the outside of the encoding device 300. For example, the multiplexing unit 358 may supply the V-DMC bitstream to a decoding device 400 (described later). Therefore, the multiplexing unit 358 can also be said to be a supply unit (providing unit) for the V-DMC bitstream.

[0118] With this configuration, the encoding device 300 can scale the face normal vectors of adjacent faces based on the maximum scalar value. This allows the encoding device 300 to limit the range of the scalar values ​​of the vertex normal vectors within the scale range. That is, the encoding device 300 can derive a displacement vector while suppressing an increase in the data length of the vertex normal vectors. Therefore, the encoding device 300 can suppress a decrease in the accuracy of the vertex normal vectors. Therefore, the encoding device 300 can suppress a decrease in the accuracy of the local coordinate system. Therefore, the encoding device 300 can suppress a decrease in the accuracy of the displacement vectors. Therefore, the encoding device 300 can suppress a decrease in the accuracy of the decoded mesh. In other words, the encoding device 300 can suppress an increase in the degradation of the decoded mesh (the difference from the original mesh) and a decrease in the subjective quality of the decoded mesh.

[0119] Furthermore, the encoding device 300 can suppress an increase in the data length of the vertex normal vectors, thereby suppressing an increase in the processing load related to generating the displacement vectors. In other words, the encoding device 300 can suppress an increase in the level of performance requirements and suppress an increase in costs.

[0120] <Flow of Encoding Process> An example of the flow of the encoding process executed by the encoding device 300 will be described with reference to the flowchart of FIG.

[0121] When the encoding process starts, in step S301, the base mesh generation unit 321 of the encoding device 300 decimates the original mesh to be encoded and generates a base mesh.

[0122] In step S302, the atlas information generating unit 322 generates atlas information for the base mesh.

[0123] In step S303, the displacement vector generation unit 323 executes the displacement vector generation process and generates a displacement vector by applying the above-described method 1.

[0124] In step S304, the V-DMC encoding unit 312 performs a V-DMC encoding process, applying the above-described method 1 to encode the V-DMC data and generate a V-DMC bitstream.

[0125] The encoding process ends when the process of step S304 is completed. The encoding device 300 executes such encoding process for each frame of the original mesh.

[0126] <Flow of Displacement Vector Generation Processing> Next, an example of the flow of the displacement vector generation processing executed in step S303 of FIG. 12 will be described with reference to the flowchart of FIG.

[0127] When the displacement vector generation process starts, the vertex normal vector derivation unit 333 initializes a buffer in step S321.

[0128] In step S322, the subdivision unit 331 subdivides the base mesh.

[0129] In step S323, the maximum scalar value derivation unit 332 applies the above-described method 1 to derive the maximum scalar value of the face normal vectors of the adjacent faces in the vicinity of the target vertex of the subdivided base mesh.

[0130] In step S324, the vertex normal vector derivation unit 333 applies the above-mentioned method 1 to scale the face normal vectors (scalar values) of adjacent faces around the target vertex of the subdivided base mesh by the maximum scalar value, and derives the vertex normal vector using the scaled face normal vectors.

[0131] In step S325, the displacement vector derivation unit 334 applies the above-described method 1 to derive a displacement vector in a local coordinate system based on the vertex normal vector.

[0132] When the process of step S325 ends, the displacement vector generation process ends, and the process returns to FIG.

[0133] <Flow of V-DMC Encoding Process> Next, an example of the flow of the V-DMC encoding process executed in step S304 of FIG. 12 will be described with reference to the flowchart of FIG.

[0134] When the V-DMC encoding process starts, the atlas information encoding unit 351 encodes the atlas information including the priority control information in step S341.

[0135] In step S342, the base mesh encoding unit 352 encodes the base mesh.

[0136] In step S343, the displacement vector correction unit 353 corrects the displacement vector.

[0137] In step S344, the displacement vector encoding unit 354 encodes the corrected displacement vector. For example, the displacement vector encoding unit 354 may pack the displacement vector into a displacement video and encode it using a 2D encoding method. Alternatively, the displacement vector encoding unit 354 may arithmetically encode the displacement vector.

[0138] In step S345, the mesh reconstructing unit 355 reconstructs the mesh.

[0139] In step S346, the attribute map conversion unit 356 converts the attribute map.

[0140] In step S347, the attribute encoding unit 357 encodes the attribute video using the attribute map as a frame image.

[0141] In step S348, the multiplexing unit 358 multiplexes the coded data of the atlas information, the coded data of the base mesh, the coded data of the displacement vector, and the coded data of the attribute to generate a V-DMC bitstream.

[0142] When the process of step S348 ends, the V-DMC encoding process ends, and the process returns to FIG.

[0143] By performing the above-described processes, the encoding device 300 can scale the face normal vectors of adjacent faces based on the maximum scalar value. This allows the encoding device 300 to limit the range of the scalar values ​​of the vertex normal vectors to within the scale range. That is, the encoding device 300 can derive a displacement vector while suppressing an increase in the data length of the vertex normal vectors. Therefore, the encoding device 300 can suppress a decrease in the accuracy of the vertex normal vectors. Therefore, the encoding device 300 can suppress a decrease in the accuracy of the local coordinate system. Therefore, the encoding device 300 can suppress a decrease in the accuracy of the displacement vectors. Therefore, the encoding device 300 can suppress a decrease in the accuracy of the decoded mesh. In other words, the encoding device 300 can suppress an increase in the degradation of the decoded mesh (the difference from the original mesh) and a decrease in the subjective quality of the decoded mesh.

[0144] Furthermore, the encoding device 300 can suppress an increase in the data length of the vertex normal vectors, thereby suppressing an increase in the processing load related to generating the displacement vectors. In other words, the encoding device 300 can suppress an increase in the level of performance requirements and suppress an increase in costs.

[0145] 5. Second embodiment Decoding device The present technology can be applied to a decoding device that decodes encoded data of a mesh. Fig. 15 is a block diagram showing an example of the configuration of a decoding device that is one aspect of an information processing device to which the present technology is applied. The decoding device 400 (second information processing device) shown in Fig. 15 is a device that decodes, for example, encoded data of a mesh generated in the encoding device 300 (Fig. 9) (a V-DMC bitstream generated by the multiplexing unit 358 (Fig. 11)) and reconstructs a decoded mesh.

[0146] Fig. 15 shows the main processing units, data flows, etc., but does not necessarily include all of them. In other words, in the decoding device 400, there may be processing units that are not shown as blocks in Fig. 15, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 15.

[0147] The decoding device 400 decodes coded data of a mesh that has been coded using a method essentially similar to the V-DMC method described in the aforementioned non-patent document, except that the present technology is applied, and reconstructs the decoded mesh. For example, the decoding device 400 obtains a V-DMC bitstream. This V-DMC bitstream may be generated by, for example, the coding device 300. As part of the reconstruction process, the decoding device 400 decodes the V-DMC bitstream and reconstructs a mesh (also referred to as a decoded mesh). The decoding device 400 also applies a texture to the decoded mesh, generates a display image for displaying the decoded mesh, and outputs the display image to an external device. For example, the decoding device 400 supplies the display image to an external display device for display.

[0148] As shown in Figure 15, the decoding device 400 (second information processing device) has a demultiplexing unit 411, an atlas information decoding unit 412, a base mesh decoding unit 413, a subdivision unit 414, a displacement vector decoding unit 415, a displacement vector application unit 416, an attribute decoding unit 417, an attribute application unit 418, and a display processing unit 419.

[0149] The demultiplexing unit 411 performs demultiplexing processing. For example, the demultiplexing unit 411 may acquire a V-DMC bitstream to be decoded and supplied to the decoding device 400. The demultiplexing unit 411 may also demultiplex the acquired V-DMC bitstream to extract coded data of atlas information, coded data of base meshes, coded data of displacement vectors, and coded data of attributes. Therefore, the demultiplexing unit 411 can also be considered an acquirer of a V-DMC bitstream or various information contained in the V-DMC bitstream. The demultiplexing unit 411 may supply the coded data of the extracted atlas information to the atlas information decoding unit 412. The demultiplexing unit 411 may also supply the coded data of the extracted base meshes to the base mesh decoding unit 413. The demultiplexing unit 411 may also supply the coded data of the extracted displacement vectors to the displacement vector decoding unit 415. Furthermore, the demultiplexing unit 411 may supply the coded data of the extracted attributes to the attribute decoding unit 417 .

[0150] The atlas information decoding unit 412 performs processing related to decoding of the atlas information. For example, the atlas information decoding unit 412 may acquire encoded data of the atlas information supplied from the demultiplexing unit 411. The atlas information decoding unit 412 may also decode the acquired encoded data of the atlas information to generate (restore) the atlas information. The atlas information decoding unit 412 may supply the generated atlas information to the base mesh decoding unit 413. The atlas information decoding unit 412 may supply the generated atlas information to the subdivision unit 414. The atlas information decoding unit 412 may supply the generated atlas information to the displacement vector decoding unit 415. The atlas information decoding unit 412 may supply the generated atlas information to the displacement vector application unit 416. The atlas information decoding unit 412 may supply the generated atlas information to the attribute decoding unit 417. The atlas information decoder 412 may provide the generated atlas information to the attribute application unit 418 .

[0151] The base mesh decoding unit 413 performs processing related to decoding of the base mesh. For example, the base mesh decoding unit 413 may acquire coded data of the base mesh supplied from the demultiplexing unit 411. The base mesh decoding unit 413 may also decode the acquired coded data (bit stream) of the base mesh using a predetermined decoding method (e.g., Draco) to generate (restore) a base mesh (e.g., a vertex list, a triangle list, etc.). In this case, the base mesh decoding unit 413 may acquire atlas information supplied from the atlas information decoding unit 412 and decode the coded data of the base mesh based on the atlas information. The base mesh decoding unit 413 may also supply the generated base mesh to the subdivision unit 414.

[0152] The subdivision unit 414 performs processing related to subdivision of the faces of the base mesh. For example, the subdivision unit 414 may obtain a base mesh provided from the base mesh decoding unit 413. The subdivision unit 414 may subdivide (the faces of) the base mesh to generate division points. The subdivision unit 414 may provide the subdivided base mesh to the displacement vector application unit 416.

[0153] The displacement vector decoding unit 415 performs processing related to the decoding of displacement vectors. For example, the displacement vector decoding unit 415 may obtain encoded data of displacement vectors (i.e., a displacement bitstream) supplied from the demultiplexing unit 411. The displacement vector decoding unit 415 may apply the above-described method 1 to decode the encoded data (bitstream) of the displacement vectors and generate (restore) displacement vectors. For example, if the displacement vectors are encoded as displacement video, the displacement vector decoding unit 415 may decode the encoded data of the displacement vectors using a predetermined decoding method for 2D video, generate (restore) the displacement video, and unpack the displacement vectors from a displacement map, which is a frame image of the displacement video. Furthermore, if the displacement video is arithmetically coded, the displacement vector decoding unit 415 may arithmetically decode the encoded data of the displacement vectors to generate displacement vectors. The displacement vector decoding unit 415 may supply the displacement vectors obtained in this manner to the displacement vector application unit 416.

[0154] The displacement vector application unit 416 performs processing related to application of a displacement vector to a subdivided base mesh. For example, the displacement vector application unit 416 may obtain a subdivided base mesh supplied from the subdivision unit 414. The displacement vector application unit 416 may obtain a displacement vector supplied from the displacement vector decoding unit 415. The displacement vector application unit 416 may apply the displacement vector to the vertices of the subdivided base mesh. In other words, the displacement vector application unit 416 may generate (reconstruct) a decoded mesh. The displacement vector application unit 416 may supply the decoded mesh thus generated to the attribute application unit 418.

[0155] The attribute decoding unit 417 executes processing related to attribute decoding. For example, the attribute decoding unit 417 may acquire coded attribute data supplied from the demultiplexing unit 411. The attribute decoding unit 417 may also decode the acquired coded attribute data using a predetermined decoding method for 2D video to generate (restore) attribute video. In this case, the attribute decoding unit 417 may acquire atlas information supplied from the atlas information decoding unit 412 and decode attributes based on the atlas information. The attribute decoding unit 417 may also supply an attribute map, which is a frame image of the generated attribute video, to the attribute application unit 418.

[0156] The attribute application unit 418 performs processing related to the application of attributes to the decoded mesh. For example, the attribute application unit 418 may acquire the decoded mesh supplied from the displacement vector application unit 416. The attribute application unit 418 may acquire an attribute map supplied from the attribute decoding unit 417. The attribute application unit 418 may apply attributes of the attribute map to the decoded mesh. In this case, the attribute application unit 418 may acquire atlas information supplied from the atlas information decoding unit 412 and apply attributes to the decoded mesh based on the atlas information. The attribute application unit 418 may supply the decoded mesh to which the attributes have been applied in this manner to the display processing unit 419.

[0157] The display processing unit 419 performs processing related to mesh display. For example, the display processing unit 419 may acquire a decoded mesh to which attributes have been applied, supplied from the attribute application unit 418. The display processing unit 419 may render the acquired decoded mesh and generate a display image for displaying the decoded mesh. The display processing unit 419 may then supply the generated display image to an external device outside the decoding device 400, and display the display image on another device or the like.

[0158] <Displacement Vector Application Unit> Fig. 16 is a block diagram showing an example of the main configuration of the displacement vector application unit 416. Note that Fig. 16 shows the main processing units, data flows, etc., and is not limited to all of the components shown in Fig. 16. In other words, the displacement vector application unit 416 may include processing units that are not shown as blocks in Fig. 16, or processing and data flows that are not shown as arrows, etc. in Fig. 16.

[0159] As shown in FIG. 16, the displacement vector application unit 416 includes a maximum scalar value derivation unit 431 , a vertex normal vector derivation unit 432 , and a displacement vector application unit 433 .

[0160] The maximum scalar value derivation unit 431 performs processing related to the derivation of the maximum scalar value. For example, the maximum scalar value derivation unit 431 may acquire information such as the "subdivided base mesh" supplied from the subdivision unit 414. The maximum scalar value derivation unit 431 may apply the above-described method 1 to derive the maximum scalar value of the face normal vectors of adjacent faces in the vicinity of the target vertex of the subdivided base mesh. For example, as shown in the example of FIG. 6 , the maximum scalar value derivation unit 431 may retain the integer parts of the scalar values ​​of the face normal vectors of the adjacent faces and select the maximum value of the retained integer parts as the maximum scalar value. The maximum scalar value derivation unit 431 may supply the derived maximum scalar value to the vertex normal vector derivation unit 432 together with information such as the "subdivided base mesh."

[0161] The vertex normal vector derivation unit 432 performs processing related to the derivation of vertex normal vectors. For example, the vertex normal vector derivation unit 432 may acquire information such as the "subdivided base mesh" and the maximum scalar value supplied from the maximum scalar value derivation unit 431. The vertex normal vector derivation unit 432 may apply the above-described method 1, scale the face normal vectors of adjacent faces around the target vertex of the "subdivided base mesh" using the maximum scalar value, and derive the vertex normal vector of the target vertex using the scaled face normal vector. For example, as shown in the example of FIG. 7 , the vertex normal vector derivation unit 432 may scale the face normal vectors of adjacent faces around the target vertex by dividing the scalar value of the face normal vectors of the adjacent faces by the maximum scalar value. The vertex normal vector derivation unit 432 may supply the derived vertex normal vectors, along with information such as the "subdivided base mesh," to the displacement vector application unit 433.

[0162] The displacement vector application unit 433 executes processing related to the derivation of a displacement vector. For example, the displacement vector application unit 433 may acquire information such as a "subdivided base mesh" and vertex normal vectors supplied from the vertex normal vector derivation unit 432. The displacement vector application unit 433 may also acquire a displacement vector supplied from the displacement vector decoding unit 415. This displacement vector is expressed in a local coordinate system based on the vertex normal vectors. The displacement vector application unit 433 may apply the displacement vector to the "subdivided base mesh" using the acquired vertex normal vectors by applying the above-described method 1. For example, as shown in the example of FIG. 8 , the displacement vector application unit 433 may convert a position vector expressed in the local coordinate system into a world coordinate system representation using the vertex normal vectors and apply the converted position vector to the "subdivided base mesh." In other words, the displacement vector application unit 433 may generate a decoded mesh. The displacement vector application unit 433 may supply the decoded mesh generated by applying the displacement vector in this manner to the attribute application unit 418.

[0163] With this configuration, the decoding device 400 can scale the face normal vectors of adjacent faces based on the maximum scalar value. This allows the decoding device 400 to limit the range of the scalar values ​​of the vertex normal vectors within the scale range. That is, the decoding device 400 can apply the displacement vectors to the subdivided base mesh while suppressing an increase in the data length of the vertex normal vectors. Therefore, the decoding device 400 can suppress a decrease in the accuracy of the vertex normal vectors. Therefore, the decoding device 400 can suppress a decrease in the accuracy of the local coordinate system. Therefore, the decoding device 400 can suppress a decrease in the accuracy of the displacement vectors. Therefore, the decoding device 400 can suppress a decrease in the accuracy of the decoded mesh. That is, the decoding device 400 can suppress an increase in the degradation of the decoded mesh (the difference from the original mesh) and a decrease in the subjective quality of the decoded mesh.

[0164] Furthermore, the decoding device 400 can suppress an increase in the data length of the vertex normal vectors, thereby suppressing an increase in the processing load related to the application of the displacement vectors. In other words, the decoding device 400 can suppress an increase in the level of performance requirements and suppress an increase in costs.

[0165] <Flow of Decoding Process> An example of the flow of the decoding process executed by the decoding device 400 will be described with reference to the flowchart of FIG.

[0166] When the decoding process starts, the demultiplexing unit 411 of the decoding device 400 demultiplexes the V-DMC bitstream in step S401.

[0167] In step S402, the atlas information decoding unit 412 decodes the coded data of the atlas information to generate (restore) the atlas information.

[0168] In step S403, the base mesh decoding unit 413 decodes the coded data of the base mesh to generate (restore) the base mesh.

[0169] In step S404, the subdivision unit 414 subdivides the base mesh to generate division points.

[0170] In step S405, the disparity vector decoding unit 415 decodes the coded data (bit stream) of the disparity vector to generate (restore) the disparity vector.

[0171] In step S406, the displacement vector application unit 416 executes a displacement vector application process to apply the displacement vectors to the vertices of the subdivided base mesh, thereby generating a decoded mesh.

[0172] In step S407, the attribute decoding unit 417 decodes the coded data of the attribute and generates (restores) an attribute map.

[0173] In step S408, the attribute application unit 418 applies the attributes contained in the attribute map to the decoded mesh.

[0174] In step S409, the display processing unit 419 renders the decoded mesh to generate a display image.

[0175] When the process of step S409 is completed, the decoding process is completed. The decoding device 400 executes such a decoding process for each frame of the original mesh.

[0176] <Flow of Displacement Vector Application Process> Next, an example of the flow of the displacement vector application process executed in step S406 of FIG. 17 will be described with reference to the flowchart of FIG.

[0177] When the displacement vector application process starts, the vertex normal vector derivation unit 432 initializes a buffer in step S421.

[0178] In step S422, the maximum scalar value derivation unit 431 applies the above-described method 1 to derive the maximum scalar value of the face normal vectors of the adjacent faces in the vicinity of the target vertex of the subdivided base mesh.

[0179] In step S423, the vertex normal vector derivation unit 432 applies the above-described method 1 to scale the face normal vectors (scalar values) of adjacent faces around the target vertex of the subdivided base mesh by the maximum scalar value, and derives the vertex normal vector using the scaled face normal vectors.

[0180] In step S424, the displacement vector application unit 433 applies the above-described method 1 and applies the displacement vector in the local coordinate system to the "subdivided base mesh" using the vertex normal vector.

[0181] When the process of step S424 ends, the displacement vector application process ends, and the process returns to FIG.

[0182] By performing each process as described above, the decoding device 400 can scale the face normal vectors of adjacent faces based on the maximum scalar value. This allows the decoding device 400 to limit the range of the scalar values ​​of the vertex normal vectors within the scale range. That is, the decoding device 400 can apply the displacement vectors to the subdivided base mesh while suppressing an increase in the data length of the vertex normal vectors. Therefore, the decoding device 400 can suppress a decrease in the accuracy of the vertex normal vectors. Therefore, the decoding device 400 can suppress a decrease in the accuracy of the local coordinate system. Therefore, the decoding device 400 can suppress a decrease in the accuracy of the displacement vectors. Therefore, the decoding device 400 can suppress a decrease in the accuracy of the decoded mesh. In other words, the decoding device 400 can suppress an increase in the degradation of the decoded mesh (the difference from the original mesh) and a decrease in the subjective quality of the decoded mesh.

[0183] Furthermore, the decoding device 400 can suppress an increase in the data length of the vertex normal vectors, thereby suppressing an increase in the processing load related to the application of the displacement vectors. In other words, the decoding device 400 can suppress an increase in the level of performance requirements and suppress an increase in costs.

[0184] 6. Supplementary Notes Polygon Shape In the above description, the polygon shape is a triangle, but this shape is just an example. The polygon shape may be any polygonal shape.

[0185] <Encoding Method> In the above, V-DMC has been used as an example of an encoding method to which the present technology can be applied, but the present technology is not limited to this example, and can be applied to any encoding method that encodes a base mesh, a displacement vector, an attribute map including texture, atlas information, or information equivalent thereto.

[0186] <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, etc., that can execute various functions by installing various programs.

[0187] FIG. 19 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0188] In a computer 900 shown in FIG. 19, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0189] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0190] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0191] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0192] The program executed by the computer can be applied by recording it on, for example, a removable medium 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable medium 921 into the drive 915.

[0193] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0194] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0195] <Application of the Present Technology> The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0196] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).

[0197] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0198] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0199] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.

[0200] <Others> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in a bitstream, but also in a bitstream that includes differential information of the identification information relative to certain reference information. Therefore, in this specification, "flag" and "identification information" encompass not only the information itself, but also differential information relative to the reference information.

[0201] Furthermore, various types of information (e.g., metadata) related to the coded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, the associated pieces of data may be combined into one piece of data or may be separate pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Furthermore, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer not to the entire data, but to only part of the data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0202] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0203] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0204] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0205] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0206] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0207] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0208] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0209] Note that the present technology can also be configured as follows: (1) An information processing device comprising: a maximum scalar value derivation unit that derives a maximum scalar value of face normal vectors of adjacent faces in the vicinity of a target vertex of a subdivided base mesh; a vertex normal vector derivation unit that scales the face normal vectors of the adjacent faces using the maximum scalar value and derives a vertex normal vector of the target vertex using the scaled face normal vector; a displacement vector derivation unit that derives a displacement vector of a local coordinate system based on the vertex normal vector; and an encoding unit that encodes the displacement vector to generate a bitstream, wherein the base mesh is a mesh with lower resolution than an original mesh to be encoded, the original mesh being composed of vertices and connections that represent a three-dimensional structure of an object, and the displacement vector is vector information indicating the displacement of the vertices of the subdivided base mesh. (2) The information processing device according to (1), wherein the maximum scalar value derivation unit holds integer parts of scalar values ​​for the face normal vectors of the adjacent faces and selects a maximum value from the held integer parts as the maximum scalar value. (3) The information processing device according to (1) or (2), wherein the vertex normal vector derivation unit divides the scalar value of the face normal vector of the adjacent faces by the maximum scalar value as the scaling. (4) The information processing device according to any of (1) to (3), wherein the vertex normal vector derivation unit quantizes the face normal vector using a predetermined quantization value and derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector. (5) The information processing device according to any one of (1) to (4), wherein the vertex normal vector derivation unit sets a quantization value, quantizes the face normal vector using the set quantization value, and derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector; and the encoding unit stores the set quantization value in the bitstream.(6) The information processing device according to any one of (1) to (5), wherein the vertex normal vector derivation unit sets a quantization value, quantizes the face normal vector using the set quantization value, derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector, and finds an index corresponding to the quantization value based on predetermined table information, and the encoding unit stores the index in the bitstream. (7) The information processing device according to any one of (1) to (6), wherein the vertex normal vector derivation unit derives a quantization value, quantizes the face normal vector using the derived quantization value, and derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector. (8) The information processing device according to (7), wherein the vertex normal vector derivation unit derives the quantization value based on the number of adjacent faces. (9) The information processing device according to any one of (1) to (8), further comprising a subdivision unit that subdivides the base mesh. (10) The information processing device according to any one of (1) to (10), further comprising a base mesh generating unit that generates the base mesh, wherein the subdivision unit is configured to subdivide the generated base mesh. (11) The information processing device according to any one of (1) to (10), wherein the encoding unit further encodes the base mesh and stores the encoded base mesh in the bitstream.(12) An information processing method comprising: deriving a maximum scalar value of face normal vectors of adjacent faces around a target vertex of a subdivided base mesh; scaling the face normal vectors of the adjacent faces using the maximum scalar value and deriving a vertex normal vector of the target vertex using the scaled face normal vector; deriving a displacement vector in local coordinates based on the vertex normal vector; and encoding the displacement vector to generate a bitstream, wherein the base mesh is a mesh with lower resolution than the original mesh to be encoded, which is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh, and the displacement vector is vector information indicating the displacement of the vertices of the subdivided base mesh.

[0210] (21) An information processing device comprising: a displacement vector decoding unit that decodes a bit stream of displacement vectors to generate the displacement vectors; a maximum scalar value derivation unit that derives a maximum scalar value of face normal vectors of adjacent faces in the vicinity of a target vertex of a subdivided base mesh; a vertex normal vector derivation unit that scales the face normal vectors of the adjacent faces using the maximum scalar value and derives a vertex normal vector of the target vertex using the scaled face normal vectors; and a displacement vector application unit that uses the derived vertex normal vectors to apply the displacement vectors in local coordinates based on the vertex normal vectors to the subdivided base mesh, wherein the base mesh is a mesh with lower resolution than the original mesh to be coded, which is composed of vertices and connections that represent a three-dimensional structure of an object, by thinning out vertices from the original mesh; and the displacement vectors are vector information that indicate the displacements of the vertices of the subdivided base mesh. (22) The information processing device according to (21), wherein the maximum scalar value derivation unit holds integer parts of scalar values ​​for the face normal vectors of the adjacent faces and selects a maximum value from the held integer parts as the maximum scalar value. (23) The information processing device according to (21) or (22), wherein the vertex normal vector derivation unit divides the scalar value of the face normal vectors of the adjacent faces by the maximum scalar value as the scaling. (24) The information processing device according to any of (21) to (23), wherein the vertex normal vector derivation unit quantizes the face normal vectors using a predetermined quantization value and derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector.(25) The information processing device according to any one of (21) to (24), wherein the displacement vector decoding unit further acquires a quantized value from the bit stream, and the vertex normal vector derivation unit quantizes the face normal vector using the quantized value, and derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector. (26) The information processing device according to any one of (21) to (25), wherein the displacement vector decoding unit further acquires an index from the bit stream, and the vertex normal vector derivation unit quantizes the face normal vector using a quantized value corresponding to the index in predetermined table information, and derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector. (27) The information processing device according to any one of (21) to (26), wherein the vertex normal vector derivation unit derives a quantized value, quantizes the face normal vector using the derived quantized value, and derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector. (28) The information processing device according to (27), wherein the vertex normal vector derivation unit derives the quantized value based on the number of adjacent faces. (29) The information processing device according to any one of (21) to (28), further comprising a subdivision unit that subdivides the base mesh. (30) The information processing device according to (29), further comprising a base mesh decoding unit that decodes a bitstream of the base mesh, wherein the subdivision unit is configured to subdivide the base mesh obtained by decoding.(31) An information processing method comprising: decoding a bit stream of displacement vectors to generate the displacement vectors; deriving a maximum scalar value of face normal vectors of adjacent faces in the vicinity of a target vertex of a subdivided base mesh; scaling the face normal vectors of the adjacent faces using the maximum scalar value, and deriving a vertex normal vector of the target vertex using the scaled face normal vectors; and applying the displacement vector in local coordinates based on the vertex normal vectors to the subdivided base mesh using the derived vertex normal vectors, wherein the base mesh is a mesh with lower resolution than the original mesh to be coded, which is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh, and the displacement vector is vector information indicating the displacement of the vertices of the subdivided base mesh.

[0211] 300 Encoding device, 311 Preprocessing unit, 312 V-DMC encoding unit, 321 Base mesh generation unit, 322 Atlas information generation unit, 323 Displacement vector generation unit, 331 Subdivision unit, 332 Maximum scalar value derivation unit, 333 Vertex normal vector derivation unit, 334 Displacement vector derivation unit, 351 Atlas information encoding unit, 352 Base mesh encoding unit, 353 Displacement vector correction unit, 354 Displacement vector encoding unit, 355 Mesh reconstruction unit, 356 Attribute map conversion unit, 357 Attribute encoding unit, 358 Multiplexing unit, 400 Decoding device, 411 Demultiplexing unit, 412 Atlas information decoding unit, 413 Base mesh decoding unit, 414 Subdivision unit, 415 Displacement vector decoding unit 416 Displacement vector application unit, 417 Attribute decoding unit, 418 Attribute application unit, 419 Display processing unit, 431 Maximum scalar value derivation unit, 432 Vertex normal vector derivation unit, 433 Displacement vector application unit, 900 Computer

Claims

1. An information processing device comprising: a maximum scalar value derivation unit that derives the maximum scalar value of the face normal vectors of adjacent faces around a target vertex of a subdivided base mesh; a vertex normal vector derivation unit that scales the face normal vectors of the adjacent faces using the maximum scalar value and derives the vertex normal vector of the target vertex using the scaled face normal vector; a displacement vector derivation unit that derives a displacement vector of local coordinates based on the vertex normal vector; and an encoding unit that encodes the displacement vector to generate a bitstream, wherein the base mesh is a mesh with lower resolution than the original mesh to be encoded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh, and the displacement vector is vector information that indicates the displacement of the vertices of the subdivided base mesh.

2. The information processing device according to claim 1, wherein the maximum scalar value derivation unit holds integer parts of the scalar values ​​of the face normal vectors of the adjacent faces, and selects the maximum value of the held integer parts as the maximum scalar value.

3. The information processing device according to claim 1, wherein the vertex normal vector derivation unit performs the scaling by dividing the scalar value of the face normal vector of the adjacent face by the maximum scalar value.

4. The information processing device according to claim 1, wherein the vertex normal vector derivation unit quantizes the face normal vector using a predetermined quantization value, and derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector.

5. The information processing device according to claim 1, wherein the vertex normal vector derivation unit sets a quantization value, quantizes the face normal vector using the set quantization value, and derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector, and the encoding unit stores the set quantization value in the bitstream.

6. The information processing device according to claim 1, wherein the vertex normal vector derivation unit sets a quantization value, quantizes the face normal vector using the set quantization value, derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector, and obtains an index corresponding to the quantization value based on predetermined table information; and the encoding unit stores the index in the bitstream.

7. The information processing device according to claim 1, wherein the vertex normal vector derivation unit derives a quantized value, quantizes the face normal vector using the derived quantized value, and derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector.

8. The information processing device according to claim 7, wherein the vertex normal vector derivation unit derives the quantized value based on the number of adjacent faces.

9. The information processing device according to claim 1, further comprising a subdivision unit that subdivides the base mesh.

10. An information processing method comprising: deriving a maximum scalar value of face normal vectors of adjacent faces around a target vertex of a subdivided base mesh; scaling the face normal vectors of the adjacent faces using the maximum scalar value and deriving a vertex normal vector of the target vertex using the scaled face normal vector; deriving a displacement vector in local coordinates based on the vertex normal vector; and encoding the displacement vector to generate a bitstream, wherein the base mesh is a mesh with lower resolution than an original mesh to be encoded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh, and the displacement vector is vector information indicating the displacement of the vertices of the subdivided base mesh.

11. An information processing device comprising: a displacement vector decoding unit that decodes a bit stream of displacement vectors to generate the displacement vectors; a maximum scalar value derivation unit that derives a maximum scalar value of face normal vectors of adjacent faces in the vicinity of a target vertex of a subdivided base mesh; a vertex normal vector derivation unit that scales the face normal vectors of the adjacent faces using the maximum scalar value and derives a vertex normal vector of the target vertex using the scaled face normal vectors; and a displacement vector application unit that uses the derived vertex normal vectors to apply the displacement vectors in local coordinates based on the vertex normal vectors to the subdivided base mesh, wherein the base mesh is a mesh with lower resolution than the original mesh to be coded, which is composed of vertices and connections that represent the three-dimensional structure of an object, by thinning out vertices from the original mesh, and the displacement vectors are vector information that indicate the displacements of the vertices of the subdivided base mesh.

12. The information processing device according to claim 11, wherein the maximum scalar value derivation unit holds integer parts of the scalar values ​​of the face normal vectors of the adjacent faces, and selects the maximum value of the held integer parts as the maximum scalar value.

13. The information processing device according to claim 11, wherein the vertex normal vector derivation unit performs the scaling by dividing the scalar value of the face normal vector of the adjacent face by the maximum scalar value.

14. The information processing device according to claim 11, wherein the vertex normal vector derivation unit quantizes the face normal vector using a predetermined quantization value, and derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector.

15. The information processing device described in claim 11, wherein the displacement vector decoding unit further obtains a quantized value from the bitstream, and the vertex normal vector derivation unit quantizes the face normal vector using the quantized value, and derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector.

16. The information processing device described in claim 11, wherein the displacement vector decoding unit further obtains an index from the bitstream, and the vertex normal vector derivation unit quantizes the face normal vector using a quantization value corresponding to the index in predetermined table information, and derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector.

17. The information processing device according to claim 11, wherein the vertex normal vector derivation unit derives a quantized value, quantizes the face normal vector using the derived quantized value, and derives the vertex normal vector of the target vertex using the scaling and the quantized face normal vector.

18. The information processing device according to claim 17, wherein the vertex normal vector derivation unit derives the quantized value based on the number of adjacent faces.

19. The information processing device according to claim 11, further comprising a subdivision unit that subdivides the base mesh.

20. An information processing method comprising: decoding a bit stream of displacement vectors to generate the displacement vectors; deriving a maximum scalar value of face normal vectors of adjacent faces in the vicinity of a target vertex of a subdivided base mesh; scaling the face normal vectors of the adjacent faces using the maximum scalar value and deriving a vertex normal vector of the target vertex using the scaled face normal vectors; and applying the displacement vector in local coordinates based on the vertex normal vectors to the subdivided base mesh using the derived vertex normal vectors, wherein the base mesh is a mesh with lower resolution than the original mesh to be coded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh; and the displacement vector is vector information indicating the displacement of the vertices of the subdivided base mesh.

Images