Information processing device and method
Patent Information
- Application Number
- MYPI2025002256
- Authority / Receiving Office
- MY · MY
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-10-18
- Filing Date
- 2023-10-02
- Publication Date
- 2024-04-18
AI Technical Summary
The existing Video-based Dynamic Mesh Coding (V-DMC) method for encoding 3D meshes faces a reduction in encoding efficiency due to the suboptimal selection of local coordinate systems, which affects the quality and quantity of displacement vectors, leading to increased information and decreased encoding efficiency.
An information processing device and method that decodes base meshes and displacement videos by deriving vertex normal vectors and applying local coordinates based on a vertex normal vector-derived local coordinate system, allowing for independent setting of the local coordinate system without relying on rendering methods, thereby improving encoding efficiency.
This approach suppresses the reduction in encoding efficiency by optimizing the local coordinate system selection, maintaining rendering quality, and reducing the amount of information in displacement videos, thus enhancing the overall encoding process.
Smart Images

Figure PI2025002256_DRAWING_001 
Figure PI2025002256_DRAWING_002 
Figure PI2025002256_DRAWING_003
Abstract
Description
Information processing device and method
[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that can suppress a decrease in coding efficiency.
[0002] Conventionally, V-DMC (Video-based Dynamic Mesh Coding) has been used as a method for encoding meshes, which are 3D data that represent the three-dimensional structure of an object using vertices and connections (see, for example, Non-Patent Document 1). In V-DMC, the mesh to be encoded is represented by a coarse base mesh and displacement vectors at division points obtained by subdividing the base mesh, and the base mesh and displacement vectors are encoded. The displacement vectors are stored (packed) in two-dimensional images, and the two-dimensional images are used as frames to encode a moving image (displacement video).
[0003] As a coordinate system for representing a displacement vector when storing the displacement vector in a two-dimensional image, for example, a local coordinate system is set for each vertex (for each displacement vector). Although a common coordinate system needs to be applied to the encoder and decoder, the local coordinate system is not explicitly transmitted. Conventionally, the encoder and decoder have set the local coordinate system in the same manner based on the normal vector of each vertex used for shadow processing during rendering.
[0004] These normal vectors were derived using an "interpolated method" to make them suitable for shadow processing during rendering. In the interpolated method, the normal vectors of the base mesh vertices are first derived, and then the normal vectors of the division points are derived using these normal vectors. The normal vectors of the base mesh vertices are weighted according to the positions of the division points (number of divisions), and the normal vectors of the division points are derived by weighted averaging. This method allows the direction of the normal vectors of each division point to be changed even when multiple division points are formed between the vertices of the base mesh. Therefore, by applying these normal vectors to shadow processing and other processes, independent processing results can be derived for each vertex, resulting in more precise processing results.
[0005] Khaled Mammou, Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Krasimir Kolarov, "[V-CG] Apple's Dynamic Mesh Coding CfP Response", ISO / IEC JTC 1 / SC 29 / WG 7 m59281, April 2022
[0006] The quantization value of the displacement vector for each axis is controlled by this local coordinate system. That is, the type of local coordinate system used affects the quality of the displacement vector (i.e., the reconstructed mesh). Furthermore, determining the local coordinate system to be applied determines the displacement vector (local coordinate) expressed in that local coordinate system. That is, the amount of information (i.e., the amount of code) in the displacement video (frame image) may change depending on the type of local coordinate system applied. That is, the type of local coordinate system applied affects the coding efficiency.
[0007] However, the local coordinate system to be applied to improve encoding efficiency depends on various factors, such as the shape of the object. Therefore, the normal vector derived by the "method with interpolation" as described above is not always optimal for deriving a local coordinate system to represent a displacement vector during encoding. In other words, using a normal vector for rendering to derive a local coordinate system may reduce the quality of the displacement vector or increase the amount of information in the displacement video, thereby reducing encoding efficiency.
[0008] The present disclosure has been made in light of such circumstances, and makes it possible to suppress a decrease in coding efficiency.
[0009] An information processing device according to one aspect of the present technology includes: a base mesh decoding unit that decodes encoded data of a base mesh; a displacement video decoding unit that decodes encoded data of a displacement video having frames of 2D images in which local coordinates are stored as displacement vectors; a vertex normal vector derivation unit that derives vertex normal vectors that are normal vectors of vertices of the subdivided base mesh; and a displacement vector application unit that applies the local coordinates of the vertices as the displacement vectors to vertices of the subdivided base mesh using a local coordinate system corresponding to the vertex normal vectors. An information processing device that generates a mesh that is coarser than an original mesh to be encoded, which is composed of points and connections, by thinning out vertices from the original mesh, wherein the displacement vector is the difference in position between a vertex of the subdivided base mesh and the vertex of the original mesh, the local coordinate system is a coordinate system for each vertex of the subdivided base mesh that is set based on the vertex normal vector, and the local coordinates are coordinates that represent the displacement vector of a vertex of the subdivided base mesh in the local coordinate system corresponding to the vertex normal vector of the vertex.
[0010] An information processing method according to one aspect of the present technology includes: decoding encoded data of a base mesh; decoding encoded data of a displacement video, the frames of which are 2D images in which local coordinates are stored as displacement vectors; deriving vertex normal vectors, which are normal vectors of vertices of the subdivided base mesh; and applying the local coordinates of the vertices as the displacement vectors to the vertices of the subdivided base mesh using a local coordinate system corresponding to the vertex normal vectors; the base mesh is a mesh that is coarser than an original mesh to be encoded, the original mesh being composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out the vertices from the original mesh; the displacement vectors are differences in position between the vertices of the subdivided base mesh and the vertices of the original mesh; the local coordinate system is a coordinate system for each vertex of the subdivided base mesh that is set based on the vertex normal vectors; and the local coordinates are coordinates representing the displacement vectors of the vertices of the subdivided base mesh in the local coordinate system corresponding to the vertex normal vectors of the vertices.
[0011] a vertex normal vector derivation unit that derives the vertex normal vector using the set derivation method; a local coordinate derivation unit that derives local coordinates that represent a displacement vector in a local coordinate system that corresponds to the vertex normal vector; a displacement video encoding unit that encodes a displacement video in which 2D images in which the local coordinates are stored as the displacement vectors are used as frames; a method designation flag generation unit that generates a method designation flag that designates the set derivation method; and a method designation flag encoding unit that encodes the method designation flag, wherein the base mesh is a mesh that is coarser than an original mesh to be encoded and is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out the vertices from the original mesh, the original mesh being composed of vertices and connections that represent a three-dimensional structure of the object; the displacement vectors are positional differences between the vertices of the subdivided base mesh and the vertices of the original mesh; and the local coordinate system is a coordinate system for each vertex of the subdivided base mesh that is set based on the vertex normal vectors.
[0012] Another aspect of the present technology is an information processing method that sets a derivation method for vertex normal vectors, which are normal vectors of vertices of a subdivided base mesh, derives the vertex normal vectors using the set derivation method, derives local coordinates that represent a displacement vector in a local coordinate system that corresponds to the vertex normal vectors, encodes a displacement video whose frames are 2D images in which the local coordinates are stored as the displacement vectors, generates a method designation flag that designates the set derivation method, and encodes the method designation flag, the base mesh is a mesh that is coarser than an original mesh to be encoded and is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out the vertices from the original mesh, the base mesh being an original mesh to be encoded, the original mesh being composed of vertices and connections that represent a three-dimensional structure of the object, the displacement vectors are positional differences between the vertices of the subdivided base mesh and the vertices of the original mesh, and the local coordinate system is a coordinate system for each vertex of the subdivided base mesh that is set based on the vertex normal vectors.
[0013] In an information processing device and method according to one aspect of the present technology, encoded data of a base mesh is decoded, encoded data of a displacement video having frames of 2D images in which local coordinates are stored as displacement vectors is decoded, vertex normal vectors, which are the normal vectors of the vertices of the subdivided base mesh, are derived, and the local coordinates of the vertices are applied as displacement vectors to the vertices of the subdivided base mesh using a local coordinate system corresponding to the vertex normal vectors.
[0014] In an information processing device and method according to another aspect of the present technology, a method for deriving a vertex normal vector, which is the normal vector of the vertices of a subdivided base mesh, is set, the vertex normal vector is derived using the set derivation method, local coordinates representing a displacement vector in a local coordinate system corresponding to the vertex normal vector are derived, a displacement video is encoded having 2D images in which the local coordinates are stored as the displacement vector as frames, a method designation flag that designates the set derivation method is generated, and the method designation flag is encoded.
[0015] 1 is a diagram illustrating V-DMC; 2 is a diagram illustrating a displacement vector; 3 is a diagram illustrating a displacement video; 4 is a diagram illustrating an example of a method for deriving a normal vector; 5 is a diagram illustrating an example of a method for deriving a normal vector; 6 is a diagram illustrating an example of an encoding method; 7 is a diagram illustrating an example of an interpolation-free method; 8 is a diagram illustrating an example of an interpolation-free method; 9 is a diagram illustrating an example of a method for selecting a derivation method; 10 is a diagram illustrating an example of a weight value; 11 is a diagram illustrating an example of clustering; 12 is a diagram illustrating an example of quantization; 13 is a block diagram illustrating an example of the main configuration of an encoding device; 14 is a block diagram illustrating an example of the main configuration of a local coordinate system setting unit; 15 is a flowchart illustrating an example of the flow of an encoding process; 16 is a flowchart illustrating an example of the flow of a local coordinate system setting process; 17 is a flowchart illustrating an example of the flow of a local coordinate system setting process; 18 is a block diagram illustrating an example of the main configuration of a decoding device; 19 is a block diagram illustrating an example of the main configuration of a local coordinate system setting unit; 19 is a flowchart illustrating an example of the flow of a decoding process; 20 is a flowchart illustrating an example of the flow of a local coordinate system setting process; 21 is a flowchart illustrating an example of the flow of a local coordinate system setting process; 22 is a block diagram illustrating an example of the main configuration of a local coordinate system setting unit; 1 is a flowchart illustrating an example of the flow of a local coordinate system setting process. FIG. 1 is a block diagram illustrating an example of the main configuration of a local coordinate system setting unit. FIG. 2 is a flowchart illustrating an example of the flow of a local coordinate system setting process. FIG. 3 is a block diagram illustrating an example of the main configuration of a local coordinate system setting unit. FIG. 4 is a flowchart illustrating an example of the flow of a local coordinate system setting process. FIG. 5 is a block diagram illustrating an example of the main configuration of a local coordinate system setting unit. FIG. 6 is a flowchart illustrating an example of the flow of a local coordinate system setting process. FIG. 7 is a block diagram illustrating an example of the main configuration of a local coordinate system setting unit.FIG. 1 is a block diagram illustrating an example of the main configuration of a computer.
[0016] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. Literature supporting technical content and technical terms 2. Local coordinate system 3. Local coordinate system for encoding 4. Embodiment 5. Supplementary notes
[0017] <1. Literature, etc. supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the content described in the embodiments, but also the content described in the following non-patent documents, etc. that were publicly known at the time of filing, and the content of other documents referenced in the following non-patent documents.
[0018] Non-patent document 1: (mentioned above)
[0019] In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.
[0020] <2. Local Coordinate System> <V-DMC> Conventionally, 3D data representing the three-dimensional structure of a three-dimensional structure (an object with a three-dimensional shape) has been provided as a mesh, which represents the three-dimensional shape of the object surface by forming polygons using vertices and connections. For example, V-DMC (Video-based Dynamic Mesh Coding) is disclosed in Non-Patent Document 1 as a method for encoding this mesh.
[0021] In V-DMC, a mesh to be coded is represented by a coarse base mesh and displacement vectors at division points obtained by subdividing the base mesh, and the base mesh and displacement vectors are coded.
[0022] For example, suppose there is an original mesh as shown in the top row of Figure 1. In Figure 1, black dots represent vertices, and lines connecting the black dots represent connections. As mentioned above, a mesh essentially forms polygons by vertices and connections, but for the sake of convenience, it is explained here as a group of vertices connected linearly (in series).
[0023] By decimating some of the vertices of the original mesh, a coarse mesh is formed as shown in the second row from the top of Figure 1. This is called the base mesh.
[0024] By subdividing each mesh of this base mesh, vertices are added, as shown in the third row from the top of Figure 1. Here, this subdivision adds vertices equal to the number of vertices thinned out from the original mesh. This results in a mesh with the same number of vertices as the original mesh. In this specification, these added vertices are also referred to as division points.
[0025] However, because the connections are updated when the vertices of the original mesh are thinned out, and the division points are formed on these updated connections, the shape of this subdivided base mesh differs from the shape of the original mesh. More specifically, as shown in the bottom part of Figure 1, the positions of the division points (on the dotted lines) differ from those of the original mesh. In this specification, the difference between the positions of the division points and the positions of the vertices of the original mesh is referred to as a displacement vector.
[0026] For example, suppose an original mesh 21 and a subdivided base mesh 22 exist in the positional relationship shown in Figure 2. Furthermore, suppose that the subdivided base mesh 22 has a vertex 23 and a vertex 24. In this case, as shown in Figure 2, the position of vertex 23 differs from the position of vertex 23' corresponding to vertex 23 in the original mesh 21. This difference is expressed as a displacement vector (displacement vector 25). Similarly, the difference between the position of vertex 24 and the position of vertex 24' corresponding to vertex 24 in the original mesh 21 is expressed as displacement vector 26. In this way, a displacement vector is set for each vertex of the subdivided base mesh.
[0027] Since the original mesh is known in the encoder, it can generate a base mesh and derive such a displacement vector. The decoder can generate (reconstruct) the original mesh (or a mesh equivalent to it) by subdividing the base mesh and applying the displacement vector to each vertex.
[0028] In this way, by reducing the number of vertices of the original mesh and encoding it as a base mesh, the amount of code can be reduced. Also, by storing the displacement vector in a two-dimensional image and encoding it using 2D encoding, the coding efficiency can be improved. In this specification, storing data in a two-dimensional image is also referred to as packing.
[0029] V-DMC supports scalable decoding. For example, as shown in FIG. 3 , displacement vectors are layered by level of resolution (the number of divisions of the base mesh) and packed into a two-dimensional image 31 as data for each layer. In the example of FIG. 3 , LoD0 packed into the two-dimensional image 31 indicates data on the displacement vector of a vertex in the highest layer (lowest resolution) of the displacement vectors of each vertex layered by level of resolution. Similarly, LoD1 indicates data on the displacement vector of a vertex in the layer immediately above LoD0. LoD2 indicates data on the displacement vector of a vertex in the layer immediately above LoD1. In this way, the displacement vectors are divided by layer (grouped into data for each layer) and packed.
[0030] The displacement vectors may be packed into a two-dimensional image as transform coefficients by coefficient transformation such as wavelet transform. The displacement vectors may also be quantized. For example, the displacement vectors may be transformed into transform coefficients by wavelet transform, the transform coefficients may be quantized, and the quantized transform coefficients (quantized coefficients) may be packed.
[0031] Furthermore, the three-dimensional shape of an object may change over time. In this specification, being variable over time is also referred to as being "dynamic." Therefore, the mesh (i.e., the base mesh and the displacement vector) is also dynamic. Therefore, the displacement vector is coded as a moving image with the two-dimensional image as a frame. In this specification, this moving image is also referred to as a displacement video.
[0032] <Setting of local coordinate system> A displacement vector is expressed as a coordinate in an arbitrary coordinate system, and the coordinate values are stored in a two-dimensional image. As a coordinate system for expressing such a displacement vector, for example, there is a local coordinate system set for each vertex (each displacement vector). In other words, each displacement vector is expressed in its own local coordinate system. In this specification, the coordinates that express a displacement vector in this local coordinate system are also referred to as local coordinates.
[0033] The encoder and decoder must apply the same local coordinate system to the same displacement vector, but the local coordinate system is not explicitly transmitted from the encoder to the decoder. Conventionally, the encoder and decoder have set their local coordinate systems in the same manner, based on the normal vector of each vertex used for shadow processing during rendering. For example, a Cartesian coordinate system with the normal vector as one axis has been set as the local coordinate system. In this specification, the normal vector of a mesh vertex is also referred to as a vertex normal vector.
[0034] These vertex normal vectors were derived using an "interpolated method" to make them suitable for shadow processing during rendering. In the interpolated method, the vertex normal vectors of the base mesh are first derived, and then the vertex normal vectors of the division points are derived using these vertex normal vectors. In this process, the vertex normal vectors of the base mesh are weighted according to the position of the division point (number of divisions), and the vertex normal vectors of the division points are derived by taking the weighted average. This method makes it possible to change the direction of the vertex normal vectors of each division point even when multiple division points are formed between the vertices of the base mesh.
[0035] For example, as shown in Figure 4, a face (polygon) with edges connecting vertices 42-1 to 42-3 of a base mesh is subdivided, and division points 44-1 to 44-3 are formed on the edge between vertices 42-1 and 42-2. The vertex normal vectors of these vertices are derived based on the normal vectors of the faces (also called surrounding faces) that include the vertices. In the "interpolation method," as described above, the normal vectors of the surrounding faces are weighted according to the position of each division point (the number of times the base mesh is divided), and the vertex normal vectors are derived by weighted averaging or the like.
[0036] 5, the vertex normal vector 51-1 of vertex 42-1, the vertex normal vector 52-1 of division point 44-1, the vertex normal vector 52-2 of division point 44-2, the vertex normal vector 52-3 of division point 44-3, and the vertex normal vector 51-2 of vertex 42-2 may point in different directions. Therefore, by applying the vertex normal vectors derived in this way to shadow processing, etc., it is possible to derive independent processing results for each vertex, and to obtain more precise processing results.
[0037] The quantization value of the displacement vector for each axis is controlled by this local coordinate system. Therefore, the quality of the displacement vector (i.e., the reconstructed mesh) may change depending on the orientation of the local coordinate system applied. Furthermore, the local coordinates representing the displacement vector may change depending on the orientation of the local coordinate system applied. In other words, the amount of information (i.e., the amount of code) in the displacement video (frame image) may also change depending on the orientation of the local coordinate system applied. In other words, the orientation of the local coordinate system applied affects the encoding efficiency.
[0038] However, the orientation of the local coordinate system that should be applied to improve encoding efficiency depends on various factors, such as the shape of the object. At the very least, applying a local coordinate system based on vertex normal vectors derived using the "interpolation method" as described above does not always result in optimal encoding efficiency. For example, in the "interpolation method," weighting is performed according to the position of the vertices, and the difference in orientation between the derived vertex normal vectors depends on the weighting method, as shown in the example of FIG. 5 . However, such a difference in orientation between the vertex normal vectors is not always optimal for representing each displacement vector.
[0039] In other words, by using the normal vector for rendering to derive the local coordinate system, the quality of the displacement vector may be reduced or the amount of information in the displacement video may increase, resulting in a decrease in encoding efficiency.
[0040] <3. Local Coordinate System for Encoding> <Method 1> Therefore, instead of reusing normal vectors for rendering, vertex normal vectors for setting local coordinate systems are derived in the encoder and decoder. The derivation method is the same for both the encoder and the decoder. That is, as shown in the top row of the table in Figure 6, vertex normal vectors are derived in the decoder, and a displacement vector is applied to the subdivided base mesh using the local coordinate system corresponding to the derived vertex normal vectors (Method 1).
[0041] For example, an information processing device (also referred to as a first information processing device) may include a base mesh decoding unit that decodes encoded data of a base mesh, a displacement video decoding unit that decodes encoded data of a displacement video whose frames are 2D images in which local coordinates are stored as displacement vectors, a vertex normal vector derivation unit that derives vertex normal vectors that are normal vectors of the vertices of a subdivided base mesh, and a displacement vector application unit that uses a local coordinate system corresponding to the vertex normal vectors to apply the local coordinates corresponding to the vertices as displacement vectors to the vertices of the subdivided base mesh.
[0042] For example, in an information processing method executed by a first information processing device, encoded data of a base mesh may be decoded, encoded data of a displacement video whose frames are 2D images in which local coordinates are stored as displacement vectors, vertex normal vectors which are normal vectors of the vertices of the subdivided base mesh may be derived, and using a local coordinate system corresponding to the vertex normal vectors, the local coordinates corresponding to those vertices may be applied as displacement vectors to the vertices of the subdivided base mesh.
[0043] The base mesh is a coarser mesh than the original mesh to be coded, which is composed of vertices and connections that represent the three-dimensional structure of the object, and is generated by thinning out vertices from the original mesh. The displacement vector is the difference in position between the vertices of the subdivided base mesh and the vertices of the original mesh. The local coordinate system is a coordinate system for each vertex of the subdivided base mesh, which is set based on the vertex normal vector. The local coordinate is the coordinate that represents the displacement vector of a vertex of the subdivided base mesh in the local coordinate system corresponding to the vertex normal vector of that vertex.
[0044] By applying local coordinates (displacement vectors) to the vertices of the base mesh, a mesh equivalent to the original mesh is reconstructed. In other words, by doing so, it is possible to set a local coordinate system independent of the vertex normal vectors used for rendering. Therefore, it is possible to suppress a decrease in coding efficiency. In addition, since there is no need to change the method for deriving the vertex normal vectors used for rendering, it is possible to suppress a decrease in coding efficiency without degrading the quality of rendering, such as shadow processing.
[0045] <Method 1-1> Any method may be used to derive the vertex normal vectors for setting this local coordinate system. For example, as shown in the second row from the top of the table in Fig. 6, the vertex normal vectors may be derived using the "non-interpolation method" (Method 1-1). For example, in the first information processing device, the vertex normal vector derivation unit may derive the vertex normal vector of a processing target point, which is a processing target vertex of a subdivided base mesh, using normal vectors of surrounding faces of the processing target point.
[0046] The non-interpolation method derives vertex normal vectors without weighting them according to the vertex positions, as in the interpolation method described above. In other words, the vertices of the base mesh and the division points are derived using the normal vectors of the surrounding faces in the same manner. For example, as in the virtual program shown in box 101 in Figure 7, the normal vector n of each face is derived from the positions of the three vertices of that face, and the average of the normal vectors n of the surrounding faces is derived as the vertex normal vector. In other words, the vertex normal vector derivation unit described above may derive the average of the normal vectors of the surrounding faces and use this value as the vertex normal vector of the processing point.
[0047] As shown in Figure 4, the orientation of each face obtained by subdividing a face of a base mesh is the same as the orientation of the face of the base mesh. Therefore, at division points located on one edge of the base mesh, such as division points 44-1 to 44-3 in Figure 4, the orientation of the surrounding faces is the same, and the vertex normal vectors also have the same orientation. For example, in the example of Figure 5, the orientations of vertex normal vectors 52-1 to 52-3 are the same.
[0048] In other words, by applying the "non-interpolation method," it is possible to obtain vertex normal vectors (i.e., local coordinate systems) that are oriented differently from those obtained by the "interpolation method." Therefore, by applying the "non-interpolation method," it is possible to at least suppress a decrease in coding efficiency due to the difference in the orientation of vertex normal vectors that occurs when the "interpolation method" is applied.
[0049] If the weighting method does not depend on the position of the division point, the weighted average of the normal vectors of the surrounding faces may be applied. In other words, the vertex normal vector derivation unit described above may derive the weighted average of the normal vectors of the surrounding faces and use this value as the vertex normal vector of the processing target point.
[0050] For example, the normal vectors of the surrounding faces may be weighted according to the area of the surrounding faces. In this case, for example, as in the virtual program shown in the box 102 in Figure 7, a weight value v is set for each face according to its size, and the vertex normal vector is derived by calculating the weighted average of the normal vectors n of the surrounding faces using the weight value v. In this case, the faces of the base mesh are also uniformly (evenly) subdivided, so that at division points located on one edge of the base mesh, such as division points 44-1 to 44-3 in Figure 4, the sizes of the surrounding faces are the same, and the vertex normal vectors are oriented in the same direction. In other words, the vertex normal vector derivation unit described above may derive the weighted average using a weight value based on the area of the surrounding faces.
[0051] Alternatively, the normal vectors may be weighted according to the angle (difference in orientation) between the surrounding faces. In this case, for example, as in the virtual program shown in box 103 in FIG. 8 , weight values ai, aj, and ak are set according to the angle (difference in orientation) between the faces, and the vertex normal vector is derived by calculating the weighted average of the normal vectors n of the surrounding faces using the weight values. In this case, too, the orientation of each face obtained by subdividing the faces of the base mesh is the same as the orientation of the face in the base mesh. Therefore, at division points located on one edge of the base mesh, such as division points 44-1 to 44-3 in FIG. 4, the differences in orientation between the surrounding faces are the same, and the vertex normal vectors are also oriented in the same direction. In other words, the vertex normal vector derivation unit described above may derive the weighted average using weight values based on the differences in orientation between the surrounding faces.
[0052] Alternatively, the normal vectors may be weighted based on both the area of the surrounding faces and the angle (difference in orientation) between the surrounding faces. In this case, for example, as in the virtual program shown in box 104 in FIG. 8, a weight value v is assigned to each face based on its size, and weight values ai, aj, and ak are assigned based on the angle (difference in orientation) between the faces. The vertex normal vector is derived by taking a weighted average of the normal vectors n of the surrounding faces using these weight values. In this case, too, the vertex normal vectors of division points located on one edge of the base mesh, such as division points 44-1 to 44-3 in FIG. 4, are oriented in the same direction, as in the example described above. In other words, the vertex normal vector derivation unit described above may derive a weighted average using weight values based on the area of the surrounding faces and the difference in orientation between the surrounding faces.
[0053] <Method 1-2> As shown in the third row from the top of the table in Figure 6, the decoder may also select either a "method with interpolation" or a "method without interpolation" and apply the selected method to derive vertex normal vectors (Method 1-2). For example, in a first information processing device, a vertex normal vector derivation unit may select either a first method (i.e., a method with interpolation) in which a processing target point, which is a vertex of a subdivided base mesh, is weighted according to its position and the vertex normal vector of the processing target point is derived using normal vectors of surrounding faces of the processing target point, or a second method (i.e., a method without interpolation) in which the vertex normal vector of the processing target point is derived using normal vectors of surrounding faces without weighting, and the vertex normal vector is derived using the selected method. Hereinafter, this derivation method will also be referred to as the "decoder selection method."
[0054] In this case, the decoder and the encoder simply select the "method with interpolation" or the "method without interpolation" using the same method (a predetermined method). In other words, in this case, the encoder does not need to explicitly notify the decoder of which method it has selected. This can suppress an increase in the amount of code. Furthermore, the encoder and the decoder can apply the more suitable one of the "method with interpolation" and the "method without interpolation," thereby further suppressing a decrease in coding efficiency.
[0055] This selection method (the method for determining whether to select the "method with interpolation" or the "method without interpolation") may be any method. For example, the selection of the "method with interpolation" or the "method without interpolation" may be determined based on the relationship (difference in orientation) between the orientations of the faces of the base mesh (i.e., the orientations of the normal vectors). For example, the vertex normal vector derivation unit described above may select either the first method or the second method for each face of the base mesh based on the difference in orientation between the target face and its surrounding faces (also referred to as surrounding faces).
[0056] For example, in general, where the object surface is flat, the shapes of the original mesh and the base mesh are also flat, and it is thought that the displacement vectors tend to be oriented in the same direction (there is little variance in the direction). Therefore, applying the "interpolation method" to make the directions of the vertex normal vectors (i.e., the directions of the local coordinates) different from each other is thought to reduce coding efficiency. Therefore, for example, if the difference in the orientation between faces of the base mesh is small, the "non-interpolation method" may be selected. For example, the vertex normal vector derivation unit may threshold-judge the difference in orientation between the target face and the surrounding faces, and select the "non-interpolation method" if the difference is smaller than (or equal to or smaller than) the threshold, and select the "interpolation method" otherwise.
[0057] The vertex normal vector of each vertex is derived based on the normal vectors of its surrounding faces. In other words, the normal vectors of the surrounding faces of the target face are represented by the vertex normal vectors of the three vertices of the target face. For example, in FIG. 9 , the target face 111, shown in gray, is the target face, and the surrounding face 114, shown in a dotted line frame, is the face located around it. Normal vector 112 is the normal vector of the target face 111. Normal vector 115 is the normal vector of the surrounding face 114. Note that in FIG. 9 , only one surrounding face and its normal vector are labeled, but all faces (triangles) in the dotted line frame are the surrounding faces 114, and their normal vectors are the normal vectors 115. Furthermore, vertex normal vectors 113-1, 113-2, and 113-3 are the vertex normal vectors of each vertex of the target face 111. In this case, the vertex normal vector 113-1 is derived (for example, by averaging) using the normal vectors 115 of the surrounding faces 114 that share the vertex. Similarly, the vertex normal vectors 113-2 and 113-3 are also derived (for example, by averaging) using the normal vectors 115 of the surrounding faces 114 that share the vertex. In other words, the vertex normal vectors 113-1 to 113-3 can be said to represent the normal vectors 115 of the surrounding faces 114 shown in FIG. 9.
[0058] Therefore, the difference in orientation between the target face and the surrounding faces may be calculated using the normal vector of the target face and the vertex normal vectors of its three vertices. For example, the dot product of the normal vector of the target face and the vertex normal vectors of each vertex of the target face is calculated, and the minimum value of the dot product is compared with a threshold value. If the minimum dot product is smaller (or if the minimum dot product is equal to the threshold value), the first method may be selected; otherwise (if the dot product is larger or if the minimum dot product is equal to the threshold value), the second method may be selected.
[0059] By doing so, it is expected that the increase in coding efficiency can be suppressed.
[0060] This threshold may be predetermined or variable. That is, the encoder and decoder may have a common threshold in advance, or the threshold applied by the encoder may be transmitted to the decoder (the decoder may apply that threshold). For example, the vertex normal vector derivation unit described above may compare the threshold transmitted from the encoder with the minimum value of the dot product. By transmitting the threshold in this manner, the decoder does not need to store the threshold. Furthermore, the threshold can be easily made variable.
[0061] <Method 1-3> Alternatively, as shown in the fourth row from the top of the table in Figure 6, the vertex normal vector derived using the "interpolation method" and the vertex normal vector derived using the "non-interpolation method" may be combined (Method 1-3). For example, in a first information processing device, the vertex normal vector derivation unit may weight a processing point, which is a vertex of a subdivided base mesh, according to its position, and derive a vertex normal vector (first vertex normal vector) for the processing point using normal vectors of the surrounding faces of the processing point. Alternatively, the vertex normal vector (second vertex normal vector) for the processing point may be derived using normal vectors of the surrounding faces without weighting. The derived first vertex normal vector and second vertex normal vector may then be combined, and the combined result (combined vector) may be used as the vertex normal vector for the processing point. Hereinafter, this derivation method will also be referred to as the "combining method."
[0062] By doing so, it is possible to obtain vertex normal vectors with different orientations whether applying the "method with interpolation" or the "method without interpolation." In other words, it is possible to set a local coordinate system with an orientation different from that in these cases. Therefore, by applying this "combining method," it is possible to at least suppress a decrease in coding efficiency caused by differences in the orientations of vertex normal vectors that occur when applying the "method with interpolation," and a decrease in coding efficiency caused by the orientations of vertex normal vectors becoming uniform that occurs when applying the "method without interpolation."
[0063] The first vertex normal vector and the second vertex normal vector may be combined by weighted averaging. For example, the vertex normal vector derivation unit may combine the derived first vertex normal vector and the second vertex normal vector by weighted averaging. For example, a weight value w may be set, and the weighted average of the first vertex normal vector (interpolated vector) and the second vertex normal vector (non-interpolated vector) may be derived as shown in the following equation (1), and used as a combined vector (updated normal vector).
[0064] updated normal vector = w*(no interpolated vector) + (1-w)*(interpolated vector) ...(1)
[0065] In this case, the weight value may be different for vertices located on the edge of the face (hereinafter also referred to as edge vertices) and vertices located on parts other than the edge of the face (hereinafter also referred to as non-edge vertices). For example, in face 121 of the base mesh shown in Figure 10, gray vertex 122 is an edge vertex and black vertex 123 is a non-edge vertex. For example, the vertex normal vector derivation unit may combine the first vertex normal vector and the second vertex normal vector corresponding to the vertices located on the edge of the face of the base mesh by weighted averaging using a first weight value, and combine the first vertex normal vector and the second vertex normal vector corresponding to the vertices located in positions other than the edge by weighted averaging using a second weight value different from the first weight value.
[0066] The weight value may be changed for each face of the base mesh, for each edge, or for each vertex.
[0067] The weight values may be predetermined or variable. That is, the encoder and decoder may have a common weight value in advance, or the weight value applied by the encoder may be transmitted to the decoder (the decoder may apply the weight value). For example, the vertex normal vector derivation unit may combine the derived first and second vertex normal vectors by weighted averaging using the weight value transmitted from the encoder. By transmitting the weight values in this manner, the decoder does not need to store the weight values. Furthermore, the weight values can be easily made variable.
[0068] <Method 1-4> Alternatively, as shown in the fifth row from the top of the table in Fig. 6, a method designation flag indicating the derivation method applied in the encoder may be transmitted from the encoder to the decoder (Method 1-4). For example, the first information processing device may further include a derivation method setting unit that sets the vertex normal vector derivation method based on the method designation flag that designates the vertex normal vector derivation method, and the vertex normal vector derivation unit may derive the vertex normal vector using the set derivation method. Hereinafter, this derivation method will also be referred to as the "flag method."
[0069] In this way, the encoder can select and apply a more suitable method from among a plurality of derivation methods, thereby further suppressing a decrease in encoding efficiency. Also, the decoder can apply a derivation method based on the flag as a method, so it can more easily apply the same derivation method as the encoder.
[0070] The encoder may select any method, and any method may be used to derive candidates for selection, and the number of candidates may be any number.
[0071] For example, the above-mentioned "interpolation method," "non-interpolation method," "decoder selection method," and "combining method" may be candidates. That is, the encoder may apply any of these derivation methods. In the decoder, for example, the derivation method setting unit may select one of the following methods as the vertex normal vector derivation method: a first method (interpolation method) that weights a processing point, which is a vertex of a subdivided base mesh, according to the position of the processing point and derives the vertex normal vector of the processing point using normal vectors of surrounding faces of the processing point; a second method (non-interpolation method) that derives the vertex normal vector of the processing point using normal vectors of surrounding faces without weighting; a third method (decoder selection method) that selects either the first method or the second method using a predetermined method; or a fourth method (combining method) that combines a first vertex normal vector derived using the first method and a second vertex normal vector derived using the second method, according to the method designation flag.
[0072] The method designation flag may be transmitted in any data unit. For example, the method designation flag may be transmitted for each original mesh sequence, for each frame, for each base mesh, for each face of the base mesh, or for each vertex of the base mesh. In other words, the method designation flag may designate the vertex normal vector derivation method for each original mesh sequence, for each frame, for each base mesh, for each face of the base mesh, or for each vertex of the base mesh. Furthermore, the derivation method setting unit may set the derivation method for each original mesh sequence, for each frame, for each base mesh, for each face of the base mesh, or for each vertex of the base mesh based on the method designation flag.
[0073] In this flag method, the encoder sets a vertex normal vector derivation method, applies the derivation method to derive the vertex normal vectors, sets a local coordinate system, and then generates a method specification flag that specifies the derivation method and transmits it to the decoder.
[0074] For example, an information processing device (also referred to as a second information processing device) may include a derivation method setting unit that sets a derivation method for vertex normal vectors, which are normal vectors of vertices of a subdivided base mesh, a vertex normal vector derivation unit that derives vertex normal vectors using the set derivation method, a local coordinate derivation unit that derives local coordinates that represent a displacement vector in a local coordinate system corresponding to the vertex normal vector, a displacement video encoding unit that encodes a displacement video in which 2D images in which the local coordinates are stored as displacement vectors are used as frames, a method designation flag generation unit that generates a method designation flag that designates the set derivation method, and a method designation flag encoding unit that encodes the method designation flag.
[0075] In addition, in an information processing method executed by a second information processing device, a method for deriving a vertex normal vector, which is the normal vector of the vertices of a subdivided base mesh, is set, the vertex normal vector is derived using the set derivation method, local coordinates representing a displacement vector in a local coordinate system corresponding to the vertex normal vector are derived, a displacement video is encoded in which 2D images in which the local coordinates are stored as displacement vectors are used as frames, a method designation flag that designates the set derivation method is generated, and the method designation flag is encoded.
[0076] The base mesh is a coarser mesh than the original mesh to be coded, which is composed of vertices and connections that represent the three-dimensional structure of the object, and is generated by thinning out vertices from the original mesh. The displacement vector is the difference in position between the vertices of the subdivided base mesh and the vertices of the original mesh. The local coordinate system is a coordinate system for each vertex of the subdivided base mesh, which is set based on the vertex normal vector. The local coordinate is the coordinate that represents the displacement vector of a vertex of the subdivided base mesh in the local coordinate system corresponding to the vertex normal vector of that vertex.
[0077] Furthermore, in this second information processing device, the derivation method setting unit may select one of the following methods as the method for derivation of the vertex normal vector: a first method (method with interpolation) in which a weighting is performed according to the position of a processing target point, which is a vertex of the subdivided base mesh to be processed, and the vertex normal vector of the processing target point is derived using the normal vectors of faces surrounding the processing target point; a second method (method without interpolation) in which the vertex normal vector of the processing target point is derived using the normal vectors of surrounding faces without performing weighting; a third method (decoder selection method) in which the decoder selects whether to apply the first method or the second method; and a fourth method (synthesis method) in which the first vertex normal vector derived by applying the first method and the second vertex normal vector derived by applying the second method are synthesized, and the synthesis result (synthesized vector) is used as the vertex normal vector of the processing target point.
[0078] If the third method (decoder selection method) is selected, the vertex normal vector derivation unit may select either the first method or the second method using the same method as the decoder and apply the selected method to derive vertex normal vectors. For example, the vertex normal vector derivation unit may select either the first method or the second method for each face of the base mesh based on the difference in orientation between the target face and surrounding faces. For example, the vertex normal vector derivation unit may calculate the dot product of the normal vector of the target face and the vertex normal vectors of each vertex of the target face, compare the minimum value of the dot product with a threshold, and select the first method if the minimum dot product is smaller (or if the minimum dot product is equal to the threshold). Otherwise (if the minimum dot product is larger or if the minimum dot product is equal to the threshold), select the second method. This threshold may be transmitted to the decoder. For example, in the second information processing device, the method specification flag encoding unit may further encode the threshold.
[0079] Furthermore, when the fourth method (combining method) is selected, the vertex normal vector derivation unit may apply the first method to derive a first vertex normal vector, apply the second method to derive a second vertex normal vector, and combine the derived first and second vertex normal vectors by weighted averaging. For example, the vertex normal vector derivation unit may combine the first and second vertex normal vectors corresponding to vertices located on edges of the face of the base mesh by weighted averaging using a first weight value. Then, the vertex normal vector derivation unit may combine the first and second vertex normal vectors corresponding to vertices located not on edges by weighted averaging using a second weight value different from the first weight value. Note that this threshold may be transmitted to the decoder. For example, in the second information processing device, the method specification flag encoding unit may further encode the weight value used in the weighted averaging.
[0080] As described above, the method designation flag may be transmitted in any data unit. For example, the method designation flag may be transmitted for each original mesh sequence, for each frame, for each base mesh, for each face of the base mesh, or for each vertex of the base mesh. In other words, in the second information processing device, the derivation method setting unit may set the vertex normal vector derivation method for each original mesh sequence, for each frame, for each base mesh, for each face of the base mesh, or for each vertex of the base mesh. In yet another way, in the second information processing device, the method designation flag generation unit may generate a method designation flag for each original mesh sequence, for each frame, for each base mesh, for each face of the base mesh, or for each vertex of the base mesh.
[0081] <Method 1-5> Furthermore, as shown in the sixth row from the top of the table in Fig. 6, in the encoder, the derived vertex normal vectors may be clustered using vertex normal vector candidates prepared in advance (Method 1-5). For example, the first information processing device may further include a clustering unit that clusters the derived vertex normal vectors with predetermined normal vector candidates prepared in advance. Furthermore, the second information processing device may further include a clustering unit that clusters the derived vertex normal vectors with predetermined normal vector candidates prepared in advance.
[0082] 11, six directions along each coordinate axis of a Cartesian coordinate system are set as candidates for the vertex normal vector, and when a vertex normal vector 131 is derived by some method, the clustering unit replaces the vertex normal vector 131 with the candidate (vertex normal vector 132) whose orientation is closest to the vertex normal vector 131. Then, a local coordinate system is set using the replaced vertex normal vector 132.
[0083] By clustering the vertex normal vectors in this way, the orientations of the vertex normal vectors are limited to the orientations of pre-prepared candidates. In other words, it is possible to suppress an increase in the variation in the orientations of the vertex normal vectors. In other words, by clustering in this way, the orientations of the local coordinate systems are corrected and limited to the orientations of pre-prepared candidates. In other words, it is possible to suppress an increase in the variation in the orientations of the local coordinate systems.
[0084] This method 1-5 can be applied in combination with any of the above-described methods 1-1 to 1-4. That is, in any of the above-described methods 1-1 to 1-4, the derived vertex normal vectors may be clustered.
[0085] 11 is merely an example, and the number of candidates may be any number. Furthermore, the orientation of the candidate vectors may be any orientation and is not limited to the example of FIG.
[0086] <Method 1-6> Furthermore, as shown in the bottom row of the table in Fig. 6, the derived vertex normal vectors (or their orientations) may be quantized in the encoder (Method 1-6). For example, the first information processing device may further include a quantization unit that quantizes the derived vertex normal vectors. Furthermore, the second information processing device may further include a quantization unit that quantizes the derived vertex normal vectors.
[0087] For example, as shown on the left in Fig. 12, when a vertex normal vector 141 is derived by some method, the quantization unit moves the vertex normal vector 131 onto a predetermined grid (quantizes the coordinates), and then sets a local coordinate system using the quantized vertex normal vector 142.
[0088] By quantizing the vertex normal vectors in this way, the orientation of the vertex normal vectors is limited. In other words, it is possible to suppress an increase in the variation in the orientation of each vertex normal vector. In other words, by such quantization, the orientation of the local coordinate system is corrected and limited to a predetermined orientation. In other words, it is possible to suppress an increase in the variation in the orientation of the local coordinate system.
[0089] Note that any quantization method may be used as long as it can correct the orientation of the vertex normal vectors (i.e., local coordinates). For example, the (orientation of) the vertex normal vectors may be quantized using a spherical coordinate system as shown on the right side of FIG. 12. For example, in this spherical coordinate system, θ, φ, r, etc. may be quantized. Of course, the quantization method is not limited to the example of FIG. 12.
[0090] This method 1-6 can be applied in combination with any of the above-described methods 1-1 to 1-4. That is, in any of the above-described methods 1-1 to 1-4, the derived vertex normal vectors may be quantized. Of course, methods 1-5 and 1-6 may also be applied in combination. That is, both quantization and clustering may be performed on the vertex normal vectors.
[0091] It should be noted that each of the above-described methods (Method 1-1 to Method 1-6) can be used in combination with other methods. For example, the applied method may be switched during a sequence. For example, it is possible to apply Method 1-1 to one frame, Method 1-2 to another frame, and Method 1-3 to yet another frame.
[0092] 4. Embodiments Encoding Device The present technology can be applied to an encoding device that encodes a mesh. Fig. 13 is a block diagram showing an example of the configuration of an encoding device, which is one aspect of an information processing device to which the present technology is applied. The encoding device 200 shown in Fig. 13 is a device that encodes a mesh. The encoding device 200 encodes a mesh using a method basically similar to V-DMC described in Non-Patent Document 1.
[0093] However, the encoding device 200 encodes the mesh by applying the method 1 described above in <3. Local coordinate system for encoding>. Alternatively, the encoding device 200 may apply one or more of the methods 1-1 to 1-3 described above.
[0094] Note that Fig. 13 shows the main processing units, data flows, etc., and is not necessarily all that is shown in Fig. 13. In other words, in the encoding device 200, there may be processing units that are not shown as blocks in Fig. 13, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 13.
[0095] As shown in FIG. 13, the encoding device 200 has a base mesh encoding unit 211, a local coordinate system setting unit 212, a local coordinate derivation unit 213, a displacement vector correction unit 214, a packing unit 215, a displacement video encoding unit 216, a mesh reconstruction unit 217, an attribute map correction unit 218, an attribute video encoding unit 219, a header encoding unit 220, and a synthesis unit 221.
[0096] The encoding device 200 is supplied with a base mesh, a displacement vector, and an attribute map for each frame, which may be generated from the original mesh, for example, in pre-processing.
[0097] The base mesh encoding unit 211 encodes the base mesh and supplies the encoded data to the synthesis unit 221. Any encoding method may be used. For example, the base meshes of each frame may be encoded independently (intra-encoding). Alternatively, the base mesh of the target frame may be encoded as a difference (motion vector) from the base mesh of the reference frame (inter-encoding).
[0098] The base mesh encoding unit 211 may also decode the generated encoded data to generate (reconstruct) a base mesh. The generated (reconstructed) base mesh includes encoding distortion. The base mesh encoding unit 211 supplies the base mesh to the displacement vector correction unit 214 and the mesh reconstruction unit 217.
[0099] The local coordinate system setting unit 212 acquires a base mesh and sets a local coordinate system for each vertex of the subdivided base mesh. The local coordinate system setting unit 212 sets the local coordinate system by applying the above-described method 1. Note that the local coordinate system setting unit 212 may set the local coordinate system by applying one or more of the above-described methods 1-1 to 1-3. The local coordinate system setting unit 212 supplies the set local coordinate system to the local coordinate derivation unit 213. Note that the local coordinate system setting unit 212 may apply method 1-2 and supply the threshold value applied in the decoder selection method to the header encoding unit 220. Also, the local coordinate system setting unit 212 may apply method 1-3 and supply the weight value applied in the synthesis method to the header encoding unit 220.
[0100] The local coordinate derivation unit 213 acquires the displacement vector and the local coordinate system supplied from the local coordinate system setting unit 212, and derives local coordinates that represent the displacement vector in the local coordinate system. The local coordinate derivation unit 213 derives the local coordinates by applying the above-described method 1. The local coordinate derivation unit 213 supplies the derived local coordinates to the displacement vector correction unit 214 as a displacement vector.
[0101] The displacement vector correction unit 214 acquires a base mesh including encoding distortion supplied from the base mesh encoding unit 211. The displacement vector correction unit 214 also acquires local coordinates (displacement vectors) supplied from the local coordinate derivation unit 213. The displacement vector correction unit 214 subdivides the base mesh including encoding distortion and corrects the local coordinates (displacement vectors) using the subdivided base mesh. The displacement vector correction unit 214 supplies the corrected local coordinates (displacement vectors) to the packing unit 215. Note that this correction may be omitted.
[0102] The packing unit 215 acquires local coordinates (displacement vectors) supplied from the displacement vector correction unit 214. The packing unit 215 packs the local coordinates (displacement vectors) into a two-dimensional image of the current frame (also referred to as a frame image). In this case, the packing unit 215 may perform coefficient transformation (e.g., wavelet transformation) on the local coordinates (displacement vectors) and pack the transformation coefficients into the two-dimensional image. The packing unit 215 may also quantize the local coordinates (or transformation coefficients) and pack the quantized coefficients into the two-dimensional image. The packing unit 215 supplies the two-dimensional image in which the local coordinates (or information equivalent thereto) are packed as a displacement vector to the displacement video encoding unit 216.
[0103] The displacement video encoding unit 216 acquires the two-dimensional images supplied from the packing unit 215. The displacement video encoding unit 216 converts the two-dimensional images into frame images and encodes them as a moving image (displacement video). The displacement video encoding unit 216 encodes the displacement video by applying the above-described method 1. The displacement video encoding unit 216 supplies the encoded data of the generated displacement video to the mesh reconstruction unit 217 and the synthesis unit 221.
[0104] The mesh reconstruction unit 217 decodes the coded data of the displacement video supplied from the displacement video coding unit 216 and derives local coordinates (displacement vectors) by, for example, unpacking the data from the two-dimensional image. The mesh reconstruction unit 217 also subdivides the base mesh (including coding distortion) supplied from the base mesh coding unit 211 and reconstructs a mesh by applying the derived displacement vectors. This mesh includes coding distortion. The mesh reconstruction unit 217 supplies the reconstructed mesh to the attribute map correction unit 218.
[0105] The attribute map correction unit 218 acquires an attribute map such as a texture, and corrects the attribute map using the mesh (including encoding distortion) supplied from the mesh reconstruction unit 217. The attribute map correction unit 218 supplies the corrected attribute map to the attribute video encoding unit 219. Note that this correction may be omitted.
[0106] The attribute video encoding unit 219 converts the attribute map supplied from the attribute map correction unit 218 into frame images and encodes them as moving images (attribute video). The attribute video encoding unit 219 supplies the generated encoded data of the attribute video to the synthesis unit 221.
[0107] The header encoding unit 220 applies the above-described method 1 to encode the information to be stored in the header. For example, the header encoding unit 220 may apply the above-described methods 1-2 to encode the threshold value applied in the decoder selection method. The header encoding unit 220 may also apply the above-described methods 1-3 to encode the weight value applied in the synthesis method. In other words, the threshold value and weight value may be stored in the header and transmitted. The header encoding unit 220 supplies the generated encoded data of the header to the synthesis unit 221.
[0108] The synthesizing unit 221 synthesizes (multiplexes) the supplied coded data of the header, the coded data of the base mesh, the coded data of the displacement video, and the coded data of the attribute video to generate a single bitstream. The synthesizing unit 221 outputs the generated bitstream to the outside of the encoding device 200. This bitstream is transmitted to a decoding device via any transmission medium or any recording medium.
[0109] 14 is a block diagram showing an example of the main configuration of the local coordinate system setting unit 212. As shown in Fig. 14, the local coordinate system setting unit 212 has a vertex normal vector derivation unit 241 and a local coordinate system setting unit 242.
[0110] The vertex normal vector derivation unit 241 applies the above-described method 1 to subdivide the supplied base mesh and derive vertex normal vectors for the vertices of the subdivided base mesh. The vertex normal vector derivation unit 241 may also derive vertex normal vectors by applying one or more of the above-described methods 1-1 to 1-3. For example, the vertex normal vector derivation unit 241 may apply method 1-1 to derive vertex normal vectors using a method without interpolation. Alternatively, the vertex normal vector derivation unit 241 may apply method 1-2 to derive vertex normal vectors using a decoder selection method. In this case, the vertex normal vector derivation unit 241 may supply the threshold value applied in the decoder selection method as a parameter to the header encoding unit 220. Alternatively, the vertex normal vector derivation unit 241 may apply method 1-3 to derive vertex normal vectors using a synthesis method. In this case, the vertex normal vector derivation unit 241 may supply the weight value applied in the synthesis method as a parameter to the header encoding unit 220. The vertex normal vector derivation unit 241 supplies the derived vertex normal vector to the local coordinate system setting unit 242.
[0111] The local coordinate system setting unit 242 applies the above-described method 1 to set a local coordinate system based on the vertex normal vector supplied from the vertex normal vector derivation unit 241. For example, the local coordinate system setting unit 242 sets a local coordinate system of an orthogonal coordinate system by using the vertex normal vector as one coordinate axis and setting two coordinate axes (Bi-tangent, tangent, and Bi-tangent) perpendicular to the vertex normal vector. The local coordinate system setting unit 242 sets a local coordinate system for each vertex normal vector. The local coordinate system setting unit 242 supplies the set local coordinate system to the local coordinate derivation unit 213.
[0112] With this configuration, the encoding device 200 can suppress a decrease in encoding efficiency.
[0113] <Flow of Encoding Process> An example of the flow of the encoding process executed by the encoding device 200 will be described with reference to the flowchart of FIG.
[0114] When the encoding process starts, the base mesh encoding unit 211 encodes a base mesh in step S201. The base mesh encoding unit 211 also decodes the generated encoded data to generate (restore) a base mesh including encoding distortion.
[0115] In step S202, the local coordinate system setting unit 212 sets a local coordinate system by executing the local coordinate system setting process by applying the above-described method 1. Note that the local coordinate system setting unit 212 may execute this process by applying one or more of the above-described methods 1-1 to 1-3.
[0116] In step S203, the local coordinate derivation unit 213 applies the above-described method 1 to derive local coordinates that represent the displacement vector in the local coordinate system set in step S202.
[0117] In step S204, the displacement vector correction unit 214 corrects the local coordinates (displacement vectors) derived in step S203 using the base mesh (including encoding distortion) generated in step S201.
[0118] In step S205, the packing unit 215 packs the local coordinates (displacement vectors) (if corrected in step S204, the corrected local coordinates) into a frame (two-dimensional image).
[0119] In step S206, the displacement video encoding unit 216 applies the above-described method 1 to treat the packed two-dimensional images as frame images and encode them as displacement video.
[0120] In step S207, the mesh reconstruction unit 217 reconstructs a mesh (including encoding distortion) using the base mesh (including encoding distortion) generated in step S201 and the encoded data of the displacement video generated in step S206.
[0121] In step S208, the attribute map corrector 218 corrects the attribute map using the reconstructed mesh.
[0122] In step S209, the attribute video encoding unit 219 encodes the corrected attribute map as a frame image as an attribute video.
[0123] In step S210, the header encoding unit 220 encodes information to be stored in the header. For example, the header encoding unit 220 may apply the above-described method 1-2 to encode the threshold value applied in the decoder selection method. Alternatively, the header encoding unit 220 may apply the above-described method 1-3 to encode the weight value applied in the combining method.
[0124] In step S211, the synthesis unit 221 multiplexes the encoded data of the base mesh generated in step S201, the encoded data of the displacement video generated in step S206, the encoded data of the attribute video generated in step S209, and the encoded data of the header generated in step S210 to generate a single bitstream.
[0125] When the process of step S211 is completed, the encoding process ends.
[0126] <Local Coordinate System Setting Process Flow 1 in Encoding Process> Next, we will explain the local coordinate system setting process executed in step S202 of Fig. 15. First, we will explain an example of the flow of the local coordinate system setting process when Method 1-1 is applied, that is, when vertex normal vectors are derived using a method without interpolation, with reference to the flowchart in Fig. 16.
[0127] When the local coordinate system setting process starts, the vertex normal vector derivation unit 241 subdivides the base mesh and generates division points in step S231.
[0128] In step S232, the vertex normal vector derivation unit 241 derives a vertex normal vector for each vertex of the subdivided base mesh using the "non-interpolation method" described above in <3. Local coordinate system for encoding>.
[0129] In step S233, the local coordinate system setting unit 242 sets a local coordinate system of an orthogonal coordinate system by using the vertex normal vector derived in step S232 as one coordinate axis and setting two coordinate axes (Bi-tangent, tangent and Bi-tangent) perpendicular to the vertex normal vector.
[0130] When the process of step S233 ends, the local coordinate system setting process ends, and the process returns to FIG.
[0131] <Flow 2 of local coordinate system setting process for encoding process> Next, an example of the flow of the local coordinate system setting process when applying method 1-2, i.e., when deriving vertex normal vectors using the decoder selection method, will be described with reference to the flowchart in Figure 17.
[0132] When the local coordinate system setting process starts, the vertex normal vector derivation unit 241 determines (selects) in step S251 whether or not to derive vertex normal vectors using the "method with interpolation."
[0133] Any method may be used for this selection. For example, whether to select the "method with interpolation" or the "method without interpolation" may be determined based on the relationship (or difference) between the orientations of the faces of the base mesh (i.e., the orientations of the normal vectors). For example, the vertex normal vector derivation unit 241 may perform a threshold determination on the difference in orientation between the target face and the surrounding faces, and select the "method without interpolation" if the difference is smaller than the threshold (or equal to or smaller than the threshold). Otherwise, the vertex normal vector derivation unit 241 may calculate the dot product between the normal vector of the target face and the vertex normal vectors of each vertex of the target face, compare the minimum value of the dot product with a threshold, and select the first method if the minimum dot product is smaller (or if the minimum dot product is equal to the threshold). Otherwise (if the dot product is larger or if the minimum dot product is equal to the threshold), the vertex normal vector derivation unit 241 may select the second method.
[0134] This threshold may be predetermined or variable, i.e., the encoder and decoder may have a common threshold in advance, or the threshold applied by the encoder may be transmitted to the decoder (which may then apply that threshold).
[0135] If it is determined that the vertex normal vectors are to be derived using the "method with interpolation," the process proceeds to step S252.
[0136] In step S252, the vertex normal vector derivation unit 241 derives the vertex normal vectors of the base mesh.
[0137] In step S253, the vertex normal vector derivation unit 241 subdivides the base mesh.
[0138] In step S254, the vertex normal vector derivation unit 241 derives the vertex normal vectors of the division points using the “method with interpolation.” When the process of step S254 ends, the process proceeds to step S257.
[0139] Also, if it is determined in step S251 that the vertex normal vectors are to be derived using the "non-interpolation method," the process proceeds to step S255.
[0140] In step S255, the vertex normal vector derivation unit 241 subdivides the base mesh.
[0141] In step S256, the vertex normal vector derivation unit 241 derives a vertex normal vector for each vertex of the subdivided base mesh using the "non-interpolation method" described above in <3. Local coordinate system for encoding>. When the processing of step S256 ends, the processing proceeds to step S257.
[0142] In step S257, the local coordinate system setting unit 242 sets a local coordinate system of an orthogonal coordinate system by using the vertex normal vector derived as one coordinate axis and setting two coordinate axes (Bi-tangent, tangent and Bi-tangent) perpendicular to the vertex normal vector.
[0143] When the process of step S257 ends, the local coordinate system setting process ends, and the process returns to FIG.
[0144] <Local Coordinate System Setting Processing Flow 3 for Encoding Processing> Next, an example of the flow of the local coordinate system setting processing when Method 1-3 is applied, that is, when vertex normal vectors are derived by the synthesis method, will be described with reference to the flowchart in FIG.
[0145] When the local coordinate system setting process starts, the vertex normal vector derivation unit 241 derives the vertex normal vectors of the base mesh in step S271.
[0146] In step S272, the vertex normal vector derivation unit 241 subdivides the base mesh.
[0147] In step S273, the vertex normal vector derivation unit 241 derives the vertex normal vectors of the division points using the "method with interpolation." That is, through the processing of steps S271 to S273, the vertex normal vectors of each vertex of the subdivided base mesh are derived using the "method with interpolation."
[0148] In step S274, the vertex normal vector derivation unit 241 derives a vertex normal vector for each vertex of the base mesh subdivided in step S272 using the "non-interpolation method" described above in <3. Local coordinate system for encoding>.
[0149] In step S275, the vertex normal vector derivation unit 241 combines the vertex normal vectors derived using the "interpolation method" and the "non-interpolation method" to derive a combined vector. In this case, the vertex normal vector derivation unit 241 may combine the vertex normal vectors derived using the "interpolation method" and the "non-interpolation method" by weighted averaging. The weight values applied to this weighted average may be different for edge vertices and non-edge vertices. Furthermore, the weight values may be different for each face of the base mesh, each edge, or each vertex. Furthermore, the weight values may be predetermined or variable. That is, the encoder and decoder may have common weight values in advance, or the weight values applied by the encoder may be transmitted to the decoder (and the decoder may apply those weight values).
[0150] In step S276, the local coordinate system setting unit 242 sets the vertex normal vector (composite vector) derived as described above as one coordinate axis, and sets two coordinate axes (Bi-tangent, tangent and Bi-tangent) perpendicular to the vertex normal vector, thereby setting a local coordinate system of an orthogonal coordinate system.
[0151] When the process of step S276 ends, the local coordinate system setting process ends, and the process returns to FIG.
[0152] By performing each process as described above, the encoding device 200 can suppress a decrease in encoding efficiency.
[0153] <Decoding device> The present technology can be applied to a decoding device that decodes mesh encoded data. Fig. 19 is a block diagram showing an example of the configuration of a decoding device, which is one aspect of an information processing device to which the present technology is applied. The decoding device 300 shown in Fig. 19 is a device that decodes mesh encoded data. The decoding device 300 decodes mesh encoded data using a method basically similar to V-DMC described in Non-Patent Document 1.
[0154] However, the decoding device 300 decodes the encoded data of the mesh by applying the method 1 described above in <3. Local coordinate system for encoding>. The decoding device 300 may also apply one or more of the methods 1-1 to 1-3 described above.
[0155] That is, the decoding device 300 is a decoding device corresponding to the encoding device 200 in FIG. 13, and can decode the bit stream generated by the encoding device 200 and reconstruct meshes.
[0156] Note that Fig. 19 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in the decoding device 300, there may be processing units that are not shown as blocks in Fig. 19, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 19.
[0157] As shown in FIG. 19, the decoding device 300 has a demultiplexing unit 311, a header decoding unit 312, a base mesh decoding unit 313, a subdivision unit 314, a displacement video decoding unit 315, an unpacking unit 316, a local coordinate system setting unit 317, a displacement vector application unit 318, and an attribute video decoding unit 319.
[0158] The decoding device 300 is supplied with a bitstream generated by an encoding device (for example, the encoding device 200) that encodes meshes using the V-DMC method.
[0159] The demultiplexing unit 311 demultiplexes the bitstream and extracts each piece of coded data included in the bitstream. For example, the demultiplexing unit 311 extracts coded data of a header from the bitstream and supplies it to a header decoding unit 312. The demultiplexing unit 311 also extracts coded data of a base mesh from the bitstream and supplies it to a base mesh decoding unit 313. The demultiplexing unit 311 also extracts coded data of a displacement video from the bitstream and supplies it to a displacement video decoding unit 315. The demultiplexing unit 311 also extracts coded data of an attribute video from the bitstream and supplies it to an attribute video decoding unit 319.
[0160] The header decoding unit 312 applies the above-described method 1 to decode the encoded data of the header supplied from the demultiplexing unit 311 and generate (restore) information to be stored in the header. The header decoding unit 312 appropriately supplies the generated information to the base mesh decoding unit 313, the subdivision unit 314, the displacement video decoding unit 315, the unpacking unit 316, the local coordinate system setting unit 317, and the attribute video decoding unit 319. For example, the header decoding unit 312 may apply the above-described method 1-2 to decode the encoded data and generate a threshold value (the threshold value applied in the encoder) and supply it to the local coordinate system setting unit 317. The header decoding unit 312 may also apply the above-described method 1-3 to decode the encoded data and generate a weight value (the weight value applied in the encoder) and supply it to the local coordinate system setting unit 317.
[0161] The base mesh decoding unit 313 decodes the coded data of the base mesh supplied from the demultiplexing unit 311 to generate (restore) the base mesh. Note that the coded data of the base mesh may be intra-coded or inter-coded. That is, the base mesh decoding unit 313 can perform both intra-decoding and inter-decoding of the coded data. The base mesh decoding unit 313 decodes the coded data by applying the above-described method 1. The base mesh decoding unit 313 supplies the generated base mesh to the subdivision unit 314 and the local coordinate system setting unit 317.
[0162] The subdivision unit 314 subdivides the base mesh supplied from the base mesh decoding unit 313 and supplies the subdivided base mesh to the displacement vector application unit 318 .
[0163] The displacement video decoding unit 315 decodes the encoded data of the displacement video supplied from the demultiplexing unit 311 and generates (restores) the displacement video. The displacement video decoding unit 315 decodes the encoded data by applying the above-mentioned method 1. The displacement video decoding unit 315 supplies the generated displacement video (the current frame) to the unpacking unit 316.
[0164] The unpacking unit 316 unpacks local coordinates (displacement vectors) from the current frame of the displacement video supplied from the displacement video decoding unit 315. In this case, the unpacking unit 316 may unpack transform coefficients and perform coefficient transformation (e.g., wavelet transform) on the transform coefficients to derive local coordinates (displacement vectors). Alternatively, the unpacking unit 316 may unpack quantized coefficients and dequantize the quantized coefficients to derive local coordinates (displacement vectors). Alternatively, the unpacking unit 316 may unpack quantized coefficients and dequantize them to derive transform coefficients, and perform coefficient transformation (e.g., wavelet transform) on the transform coefficients to derive local coordinates (displacement vectors). The unpacking unit 316 supplies the unpacked local coordinates (displacement vectors) to the displacement vector application unit 318.
[0165] The local coordinate system setting unit 317 acquires the base mesh supplied from the base mesh decoding unit 313 and sets a local coordinate system for each vertex of the subdivided base mesh. The local coordinate system setting unit 317 sets the local coordinate system by applying the above-described method 1. Note that the local coordinate system setting unit 317 may set the local coordinate system by applying one or more of the above-described methods 1-1 to 1-3. Note that the local coordinate system setting unit 317 sets the local coordinate system by the same method as the method applied by the encoder (e.g., the local coordinate system setting unit 212 of the encoding device 200). The local coordinate system setting unit 317 supplies the set local coordinate system to the displacement vector application unit 318. Note that the local coordinate system setting unit 317 may also apply method 1-2, deriving vertex normal vectors by a decoder selection method using the threshold supplied from the header decoding unit 312, and setting the local coordinate system. Alternatively, the local coordinate system setting unit 317 may apply Method 1-3 to derive vertex normal vectors by a synthesis method using the weight values supplied from the header decoding unit 312, and set a local coordinate system.
[0166] The displacement vector application unit 318 applies the above-described method 1 to the vertices of the subdivided base mesh supplied from the subdivision unit 314 by applying the local coordinates (displacement vectors) supplied from the unpacking unit 316 to reconstruct the mesh. That is, the displacement vector application unit 318 applies the local coordinates (displacement vectors) to the vertices of the subdivided base mesh using the local coordinate system supplied from the local coordinate system setting unit 317. In this specification, this reconstructed mesh is also referred to as a decoded mesh. In other words, it can be said that the displacement vector application unit 318 generates a decoded mesh. The displacement vector application unit 318 outputs the generated decoded mesh to the outside of the decoding device 300.
[0167] The attribute video decoding unit 319 decodes the coded data of the attribute video supplied from the demultiplexing unit 311 to generate (the current frame of) the attribute video. The attribute video decoding unit 319 outputs the generated current frame of the attribute video, i.e., the attribute map corresponding to the decoded mesh, to the outside of the decoding device 300.
[0168] <Local Coordinate System Setting Unit> Fig. 20 is a block diagram showing an example of the main configuration of the local coordinate system setting unit 317. As shown in Fig. 20 , the local coordinate system setting unit 317 has a vertex normal vector derivation unit 341 and a local coordinate system setting unit 342.
[0169] The vertex normal vector derivation unit 341 applies the above-described method 1 to subdivide the supplied base mesh and derive vertex normal vectors for the vertices of the subdivided base mesh. The vertex normal vector derivation unit 341 may also derive vertex normal vectors by applying one or more of the above-described methods 1-1 to 1-3. For example, the vertex normal vector derivation unit 341 may apply method 1-1 to derive vertex normal vectors using a method without interpolation. Alternatively, the vertex normal vector derivation unit 341 may apply method 1-2 to derive vertex normal vectors using a decoder selection method. In this case, the vertex normal vector derivation unit 341 may apply the supplied threshold value to derive vertex normal vectors. Alternatively, the vertex normal vector derivation unit 341 may apply method 1-3 to derive vertex normal vectors using a synthesis method. In this case, the vertex normal vector derivation unit 341 may also apply the supplied weight value to derive vertex normal vectors. The vertex normal vector derivation unit 341 derives the vertex normal vectors using the same method as that applied by an encoder (for example, the vertex normal vector derivation unit 241 of the encoding device 200). The vertex normal vector derivation unit 341 supplies the derived vertex normal vectors to the local coordinate system setting unit 342.
[0170] The local coordinate system setting unit 342 applies the above-described method 1 to set a local coordinate system based on the vertex normal vector supplied from the vertex normal vector derivation unit 341. For example, the local coordinate system setting unit 342 sets a local coordinate system of an orthogonal coordinate system by using the vertex normal vector as one coordinate axis and setting two coordinate axes (Bi-tangent, tangent, and Bi-tangent) perpendicular to the vertex normal vector. The local coordinate system setting unit 342 sets a local coordinate system for each vertex normal vector. The local coordinate system setting unit 342 supplies the set local coordinate system to the displacement vector application unit 318.
[0171] With this configuration, the decoding device 300 can suppress a decrease in coding efficiency.
[0172] <Flow of Decoding Process> An example of the flow of the decoding process executed by the decoding device 300 will be described with reference to the flowchart of FIG.
[0173] When the decoding process starts, in step S301, the demultiplexer 311 demultiplexes the bit stream and extracts various types of coded data.
[0174] In step S302, the header decoding unit 312 applies the above-described method 1 to decode the coded data of the header extracted in step S301, and generates (restores) the information to be stored in the header.
[0175] In step S303, the base mesh decoding unit 313 applies the above-described method 1 to decode the coded data of the base mesh extracted in step S301, and generates (restores) the base mesh.
[0176] In step S304, the subdivision unit 314 subdivides the base mesh.
[0177] In step S305, the displaced video decoding unit 315 applies the above-described method 1 to decode the coded data of the displaced video extracted in step S301, and generates (restores) the displaced video (the current frame).
[0178] In step S306, the unpacking unit 316 unpacks local coordinates (displacement vectors) from the current frame (two-dimensional image). At this time, the unpacking unit 316 may unpack transform coefficients and perform coefficient transformation (e.g., wavelet transform) on the transform coefficients to derive local coordinates (displacement vectors). Alternatively, the unpacking unit 316 may unpack quantized coefficients and dequantize the quantized coefficients to derive local coordinates (displacement vectors). Alternatively, the unpacking unit 316 may unpack quantized coefficients, dequantize the quantized coefficients to derive transform coefficients, and perform coefficient transformation (e.g., wavelet transform) on the transform coefficients to derive local coordinates (displacement vectors).
[0179] In step S307, the local coordinate system setting unit 317 performs local coordinate system setting processing by applying the above-described method 1 to set a local coordinate system. The local coordinate system setting unit 317 may perform this processing by applying one or more of the above-described methods 1-1 to 1-3. For example, the local coordinate system setting unit 317 may apply method 1-2 to derive vertex normal vectors by a decoder selection method using a threshold value supplied from the header decoding unit 312, and set a local coordinate system. The local coordinate system setting unit 317 may also apply method 1-3 to derive vertex normal vectors by a synthesis method using a weight value supplied from the header decoding unit 312, and set a local coordinate system. The local coordinate system setting unit 317 sets a local coordinate system using the same method as the method applied by the encoder (e.g., the local coordinate system setting unit 212 of the encoding device 200).
[0180] In step S308, the displacement vector application unit 318 applies the above-mentioned method 1, and using the local coordinate system set in step S307, applies the local coordinates (displacement vectors) unpacked in step S306 to the vertices of the base mesh subdivided in step S304 to generate a decoded mesh.
[0181] In step S309, the attribute video decoding unit 319 decodes the coded data of the attribute video extracted in step S301, and generates (restores) the current frame of the attribute video, that is, an attribute map.
[0182] When the process of step S309 is completed, the decoding process ends.
[0183] <Flow 1 of Local Coordinate System Setting Processing in Decoding Processing> Next, we will explain the local coordinate system setting processing executed in step S307 of Fig. 21. First, we will explain an example of the flow of the local coordinate system setting processing when Method 1-1 is applied, that is, when vertex normal vectors are derived using a method without interpolation, with reference to the flowchart in Fig. 22.
[0184] When the local coordinate system setting process starts, the vertex normal vector derivation unit 341 subdivides the base mesh and generates division points in step S331.
[0185] In step S332, the vertex normal vector derivation unit 341 derives a vertex normal vector for each vertex of the subdivided base mesh using the "non-interpolation method" described above in <3. Local coordinate system for encoding>.
[0186] In step S333, the local coordinate system setting unit 342 sets a local coordinate system of a Cartesian coordinate system by using the vertex normal vector derived in step S332 as one coordinate axis and setting two coordinate axes (Bi-tangent, tangent and Bi-tangent) perpendicular to that vertex normal vector.
[0187] When the process of step S333 ends, the local coordinate system setting process ends, and the process returns to FIG.
[0188] <Flow 2 of local coordinate system setting process for decoding process> Next, an example of the flow of the local coordinate system setting process when applying method 1-2, i.e., when deriving vertex normal vectors using the decoder selection method, will be described with reference to the flowchart in Figure 23.
[0189] When the local coordinate system setting process starts, the vertex normal vector derivation unit 341 determines (selects) in step S351 whether or not to derive vertex normal vectors using the "method with interpolation."
[0190] This selection method may be any method that is the same as that used in the encoder. For example, whether to select the "method with interpolation" or the "method without interpolation" may be determined based on the relationship (or difference) between the orientations of the faces of the base mesh (i.e., the orientations of the normal vectors). For example, the vertex normal vector derivation unit 341 may perform a threshold-based determination of the difference in orientation between the target face and the surrounding faces, and select the "method without interpolation" if the difference is smaller than the threshold (or equal to or smaller than the threshold). Otherwise, the vertex normal vector derivation unit 341 may calculate the dot product between the normal vector of the target face and the vertex normal vectors of each vertex of the target face, compare the minimum value of the dot product with a threshold, and select the first method if the minimum dot product is smaller (or if the minimum dot product is equal to the threshold). Otherwise (if the dot product is larger or if the minimum dot product is equal to the threshold), the vertex normal vector derivation unit 341 may select the second method.
[0191] This threshold may be predetermined or variable. That is, the encoder and decoder may have a common threshold in advance, or the threshold applied by the encoder may be transmitted to the decoder (the vertex normal vector derivation unit 341 may apply the threshold transmitted from the encoder).
[0192] If it is determined that the vertex normal vectors are to be derived using the "method with interpolation," the process proceeds to step S352.
[0193] In step S352, the vertex normal vector derivation unit 341 derives the vertex normal vectors of the base mesh.
[0194] In step S353, the vertex normal vector derivation unit 341 subdivides the base mesh.
[0195] In step S354, the vertex normal vector derivation unit 341 derives the vertex normal vectors of the division points using the “method with interpolation.” When the process of step S354 ends, the process proceeds to step S357.
[0196] Also, if it is determined in step S351 that the vertex normal vectors are to be derived using the "non-interpolation method," the process proceeds to step S355.
[0197] In step S355, the vertex normal vector derivation unit 341 subdivides the base mesh.
[0198] In step S356, the vertex normal vector derivation unit 341 derives a vertex normal vector for each vertex of the subdivided base mesh using the "non-interpolation method" described above in <3. Local coordinate system for encoding>. When the processing of step S356 ends, the processing proceeds to step S357.
[0199] In step S357, the local coordinate system setting unit 342 sets the vertex normal vector derived as described above as one coordinate axis, and sets two coordinate axes (Bi-tangent, tangent and Bi-tangent) perpendicular to the vertex normal vector, thereby setting a local coordinate system of an orthogonal coordinate system.
[0200] When the process of step S357 ends, the local coordinate system setting process ends, and the process returns to FIG.
[0201] <Flow of local coordinate system setting process in decoding process 3> Next, an example of the flow of the local coordinate system setting process when applying method 1-3, i.e., when deriving vertex normal vectors by the synthesis method, will be described with reference to the flowchart in Figure 24.
[0202] When the local coordinate system setting process starts, the vertex normal vector derivation unit 341 derives the vertex normal vectors of the base mesh in step S371.
[0203] In step S372, the vertex normal vector derivation unit 341 subdivides the base mesh.
[0204] In step S373, the vertex normal vector derivation unit 341 derives the vertex normal vectors of the division points using the "method with interpolation." That is, through the processing of steps S371 to S373, the vertex normal vectors of each vertex of the subdivided base mesh are derived using the "method with interpolation."
[0205] In step S374, the vertex normal vector derivation unit 341 derives a vertex normal vector for each vertex of the base mesh subdivided in step S372 using the "non-interpolation method" described above in <3. Local coordinate system for encoding>.
[0206] In step S375, the vertex normal vector derivation unit 341 combines the vertex normal vectors derived using the "interpolation method" and the "non-interpolation method" to derive a combined vector. In this case, the vertex normal vector derivation unit 341 may combine the vertex normal vectors derived using the "interpolation method" and the "non-interpolation method" by weighted averaging. The weight values applied to this weighted average may be different for edge vertices and non-edge vertices. Furthermore, the weight values may be different for each face of the base mesh, each edge, or each vertex. Furthermore, the weight values may be predetermined or variable. That is, the encoder and decoder may have common weight values in advance, or the weight values applied by the encoder may be transmitted to the decoder (the vertex normal vector derivation unit 341 may apply the weight values transmitted from the encoder).
[0207] In step S376, the local coordinate system setting unit 342 sets the vertex normal vector (composite vector) derived as described above as one coordinate axis, and sets two coordinate axes (Bi-tangent, tangent and Bi-tangent) perpendicular to the vertex normal vector, thereby setting a local coordinate system of an orthogonal coordinate system.
[0208] When the process of step S376 ends, the local coordinate system setting process ends, and the process returns to FIG.
[0209] By performing each process as described above, the decoding device 300 can suppress a decrease in coding efficiency.
[0210] <Application of Methods 1-4> <Local Coordinate Setting Unit of Encoding Device> The encoding device 200 can apply the above-described Methods 1-4. That is, the encoding device 200 may derive vertex normal vectors by the "flag method," set a local coordinate system using the derived vertex normal vectors, and derive the local coordinates of the displacement vector using the local coordinate system.
[0211] In this case, the encoding device 200 has the same configuration as that described with reference to FIG. 13 . Each processing unit performs the same processing as in FIG. 13 . However, the local coordinate system setting unit 212 applies the above-described methods 1-4 to set a local coordinate system. That is, the local coordinate system setting unit 212 derives vertex normal vectors using the flag method and sets a local coordinate system using the derived vertex normal vectors. Therefore, the local coordinate system setting unit 212 generates a method designation flag indicating the derivation method applied at that time, supplies the method designation flag to the header encoding unit 220, and stores it in the header. The header encoding unit 220 encodes the header including the method designation flag. That is, the header encoding unit 220 encodes the method designation flag. The combining unit 221 stores the encoded data of the header (method designation flag) in a bitstream. This bitstream is transmitted to a decoder via any transmission path or storage medium. That is, the method designation flag is transmitted to the decoder. Note that the method designation flag may be transmitted in any data unit. For example, a method specification flag may be transmitted for each sequence of original meshes, a method specification flag may be transmitted for each frame, a method specification flag may be transmitted for each base mesh, a method specification flag may be transmitted for each face of the base mesh, or a method specification flag may be transmitted for each vertex of the base mesh.
[0212] In this case, a main configuration example of the local coordinate system setting unit 212 is shown in Fig. 25. As shown in Fig. 25, the local coordinate system setting unit 212 in this case has a derivation method setting unit 411 and a method designation flag generation unit 412 in addition to the configuration shown in Fig. 14 (the vertex normal vector derivation unit 241 and the local coordinate system setting unit 242).
[0213] The derivation method setting unit 411 applies any of the above-described methods 1 to 4 to set the vertex normal vector derivation method. For example, the derivation method setting unit 411 may select one of the following as the vertex normal vector derivation method: "method with interpolation," "method without interpolation," "decoder selection method," and "combination method." Any method may be used to select this derivation method. The derivation method setting unit 411 may also set the vertex normal vector derivation method for any data unit. For example, the derivation method setting unit 411 may set the vertex normal vector derivation method for each sequence of the original mesh, each frame, each base mesh, each face of the base mesh, or each vertex of the base mesh. The derivation method setting unit 411 notifies the vertex normal vector derivation unit 241 and the method specification flag generation unit 412 of the set derivation method.
[0214] The vertex normal vector derivation unit 241 derives the vertex normal vectors using the derivation method set by the derivation method setting unit 411. For example, the vertex normal vector derivation unit 241 may derive the vertex normal vectors by applying the "interpolation method," the "non-interpolation method," the "decoder selection method," or the "combining method." Each of the derivation methods is as described above.
[0215] The vertex normal vector derivation unit 241 supplies the derived vertex normal vectors to the local coordinate system setting unit 242. The local coordinate system setting unit 242 sets a local coordinate system using the vertex normal vectors, similar to the case of FIG.
[0216] The method designation flag generation unit 412 generates a method designation flag, which is flag information indicating the derivation method set by the derivation method setting unit 411. The method designation flag generation unit 412 may generate the method designation flag for any data unit. For example, the method designation flag generation unit 412 may generate a method designation flag for each sequence of the original mesh, for each frame, for each base mesh, for each face of the base mesh, or for each vertex of the base mesh. The method designation flag generation unit 412 supplies the generated method designation flag to the header encoding unit 220.
[0217] With this configuration, the encoding device 200 can derive vertex normal vectors by applying methods 1 to 4. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.
[0218] <Flow 4 of Local Coordinate System Setting Processing in Encoding Processing> When Method 1-4 is applied, the encoding processing is executed in the same flow as in the flowchart of Fig. 15. However, in step S202, the local coordinate system setting unit 212 executes the local coordinate system setting processing by applying Method 1-4 described above to set the local coordinate system. Then, in step S321, the header encoding unit 220 encodes the header including the method designation flag generated in step S202.
[0219] An example of the flow of the local coordinate system setting process (step S202) in this case will be described with reference to the flowchart of FIG.
[0220] When the local coordinate system setting process starts, in step S401, the derivation method setting unit 411 applies one of the above-mentioned methods 1 to 4 to set the vertex normal vector derivation method. For example, the derivation method setting unit 411 may select one of the following as the vertex normal vector derivation methods: "method with interpolation," "method without interpolation," "decoder selection method," and "combining method." Any method may be used to select this derivation method. The derivation method setting unit 411 may also set the vertex normal vector derivation method for any data unit. For example, the derivation method setting unit 411 may set the vertex normal vector derivation method for each sequence of the original mesh, each frame, each base mesh, each face of the base mesh, or each vertex of the base mesh.
[0221] In step S402, the vertex normal vector derivation unit 241 derives the vertex normal vectors using the derivation method set in step S401. For example, the vertex normal vector derivation unit 241 may derive the vertex normal vectors using the "interpolation method," "non-interpolation method," "decoder selection method," or "synthesis method."
[0222] In step S403, the local coordinate system setting unit 242 sets a local coordinate system of an orthogonal coordinate system by using the vertex normal vector derived in step S402 as one coordinate axis and setting two coordinate axes (Bi-tangent, tangent and Bi-tangent) perpendicular to the vertex normal vector.
[0223] In step S404, the method designation flag generation unit 412 generates a method designation flag indicating the derivation method set in step S401.
[0224] When the process of step S404 ends, the local coordinate system setting process ends, and the process returns to FIG.
[0225] By performing each process as described above, the encoding device 200 can derive vertex normal vectors by applying methods 1 to 4. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.
[0226] <Local Coordinate Setting Unit of Decoding Device> Similarly, the decoding device 300 can also apply the above-described methods 1-4. That is, the decoding device 300 may derive vertex normal vectors using the "flag method," set a local coordinate system using the derived vertex normal vectors, and derive local coordinates of the displacement vectors using the local coordinate system. That is, the decoding device 300 may set a vertex normal vector derivation method based on a method designation flag transmitted from the encoder, derive vertex normal vectors using the derivation method, set a local coordinate system using the derived vertex normal vectors, and derive local coordinates of the displacement vectors using the local coordinate system.
[0227] In this case, the decoding device 300 has the same configuration as that described with reference to Fig. 19. Each processing unit performs the same processing as in Fig. 19. However, the header decoding unit 312 decodes the encoded data of the header including the method designation flag and generates (restores) the method designation flag transmitted from the encoder. The header decoding unit 312 then supplies the generated (restored) method designation flag to the local coordinate system setting unit 317.
[0228] The local coordinate system setting unit 317 applies any of the above-mentioned methods 1 to 4 to set a local coordinate system. That is, the local coordinate system setting unit 317 derives vertex normal vectors using the "flag method," and sets a local coordinate system using the derived vertex normal vectors. That is, the local coordinate system setting unit 317 sets a vertex normal vector derivation method based on the method designation flag supplied from the header decoding unit 312 (the method designation flag transmitted from the encoder), derives vertex normal vectors using the derivation method, and sets a local coordinate system using the derived vertex normal vectors.
[0229] The method designation flag may be transmitted in any data unit, for example, for each sequence of original meshes, for each frame, for each base mesh, for each face of the base mesh, or for each vertex of the base mesh.
[0230] In this case, a main configuration example of the local coordinate system setting unit 317 is shown in Fig. 27. As shown in Fig. 27, the local coordinate system setting unit 317 in this case has a derivation method setting unit 421 in addition to the configuration shown in Fig. 20 (the vertex normal vector derivation unit 241 and the local coordinate system setting unit 242).
[0231] The derivation method setting unit 421 applies any of the above-described methods 1 to 4 to set the method for deriving the vertex normal vectors. For example, the derivation method setting unit 421 acquires a method designation flag supplied from the header decoding unit 312. The derivation method setting unit 421 sets the derivation method indicated by the method designation flag as the method for deriving the vertex normal vectors. The derivation method setting unit 421 supplies the set derivation method to the vertex normal vector derivation unit 341. Note that the derivation method setting unit 421 may acquire other parameters (for example, thresholds and weight values transmitted from the encoder) as necessary and supply them to the vertex normal vector derivation unit 341.
[0232] The vertex normal vector derivation unit 341 derives the vertex normal vectors using the derivation method set by the derivation method setting unit 421. For example, the vertex normal vector derivation unit 341 may derive the vertex normal vectors by applying the "interpolation method," the "non-interpolation method," the "decoder selection method," or the "combining method." Each of the derivation methods is as described above.
[0233] The vertex normal vector derivation unit 341 supplies the derived vertex normal vectors to the local coordinate system setting unit 342. The local coordinate system setting unit 342 sets a local coordinate system using the vertex normal vectors, as in the case of Fig. 20 , and supplies the local coordinate system to the displacement vector application unit 318.
[0234] With this configuration, the decoding device 300 can derive vertex normal vectors by applying methods 1 to 4. Therefore, the decoding device 300 can suppress a decrease in coding efficiency.
[0235] <Flow 4 of Local Coordinate System Setting Process in Decoding Process> When Methods 1-4 are applied, the decoding process is executed in the same flow as in the flowchart of Fig. 21. However, in step S302, the header decoding unit 312 decodes the encoded data of the header including the method designation flag etc. transmitted from the encoder. In other words, the method designation flag transmitted from the encoder is generated (restored).
[0236] Then, in step S307, the local coordinate system setting unit 317 executes the local coordinate system setting process by applying the above-described method 1-4, and sets the local coordinate system.
[0237] An example of the flow of the local coordinate system setting process (step S307) in this case will be described with reference to the flowchart in FIG.
[0238] When the local coordinate system setting process starts, in step S421, the derivation method setting unit 421 applies one of the above-described methods 1-4 to set the vertex normal vector derivation method. For example, the derivation method setting unit 421 sets the vertex normal vector derivation method based on a method specification flag transmitted from the encoder. For example, the derivation method setting unit 421 may select a method specified by the method specification flag from among "method with interpolation," "method without interpolation," "decoder selection method," and "synthesis method," and apply it as the vertex normal vector derivation method. Note that any method may be used to select this derivation method. Furthermore, the derivation method setting unit 411 may set the vertex normal vector derivation method for any data unit. For example, the derivation method setting unit 411 may set the vertex normal vector derivation method for each sequence of the original mesh, each frame, each base mesh, each face of the base mesh, or each vertex of the base mesh.
[0239] In step S422, the vertex normal vector derivation unit 341 derives the vertex normal vectors using the derivation method set in step S421. For example, the vertex normal vector derivation unit 341 may derive the vertex normal vectors using the "interpolation method," the "non-interpolation method," the "decoder selection method," or the "synthesis method."
[0240] In step S423, the local coordinate system setting unit 342 sets a local coordinate system of a Cartesian coordinate system by using the vertex normal vector derived in step S422 as one coordinate axis and setting two coordinate axes (Bi-tangent, tangent and Bi-tangent) perpendicular to the vertex normal vector.
[0241] When the process of step S423 ends, the local coordinate system setting process ends, and the process returns to FIG.
[0242] By performing each process as described above, the decoding device 300 can derive vertex normal vectors by applying methods 1 to 4. Therefore, the decoding device 300 can suppress a decrease in coding efficiency.
[0243] <Application of Method 1-5> <Local Coordinate Setting Unit of Encoding Device> The encoding device 200 may apply the above-described Methods 1-5. That is, the encoding device 200 may cluster the derived vertex normal vectors using vertex normal vector candidates prepared in advance.
[0244] In this case, the encoding device 200 has the same configuration as that described with reference to Fig. 13. Each processing unit performs the same processing as in Fig. 13. However, the local coordinate system setting unit 212 sets a local coordinate system by applying the above-mentioned methods 1-5. In other words, the local coordinate system setting unit 212 clusters the derived vertex normal vectors using vertex normal vector candidates prepared in advance.
[0245] In this case, a main configuration example of the local coordinate system setting unit 212 is shown in Fig. 29. As shown in Fig. 29, the local coordinate system setting unit 212 in this case has a vertex normal vector clustering unit 431 between the vertex normal vector derivation unit 241 and the local coordinate system setting unit 242 shown in Fig. 14.
[0246] The vertex normal vector derivation unit 241 derives vertex normal vectors. Any derivation method may be used. For example, the vertex normal vector derivation unit 241 may derive vertex normal vectors by applying the "interpolation method," the "non-interpolation method," the "decoder selection method," or the "combining method." Each derivation method is as described above. The vertex normal vector derivation unit 241 supplies the derived vertex normal vectors to the vertex normal vector clustering unit 431. Furthermore, the vertex normal vector derivation unit 241 may supply parameters such as thresholds and weights applied to the derivation of the vertex normal vectors to the header encoding unit 220 as necessary, and store them in the header for encoding.
[0247] The vertex normal vector clustering unit 431 clusters the vertex normal vectors supplied from the vertex normal vector derivation unit 241. For example, the vertex normal vector clustering unit 431 clusters the vertex normal vectors using predetermined normal vector candidates prepared in advance. The vertex normal vector clustering unit 431 supplies the vertex normal vectors replaced with the derived vertex normal vectors by the clustering to the local coordinate system setting unit 242.
[0248] With this configuration, the encoding device 200 can apply methods 1-5 to limit the orientation of the local coordinate system to the prepared candidate orientations, thereby preventing a decrease in encoding efficiency.
[0249] <Flow 5 of Local Coordinate System Setting Processing in Encoding Processing> When Method 1-5 is applied, the encoding processing is executed in the same flow as in the flowchart of Fig. 15. However, in step S202, the local coordinate system setting unit 212 executes the local coordinate system setting processing by applying Method 1-5 described above, and sets a local coordinate system. In other words, the local coordinate system setting unit 212 limits the orientation of the local coordinate system to a candidate orientation prepared in advance.
[0250] An example of the flow of the local coordinate system setting process (step S202) in this case will be described with reference to the flowchart of FIG.
[0251] When the local coordinate system setting process starts, the vertex normal vector derivation unit 241 derives vertex normal vectors in step S441. Any method may be used for this derivation. For example, a "method with interpolation," a "method without interpolation," a "decoder selection method," or a "combining method" may be applied.
[0252] In step S442, the vertex normal vector clustering unit 431 clusters the vertex normal vectors derived in step S441. For example, the vertex normal vector clustering unit 431 clusters the vertex normal vectors using predetermined normal vector candidates prepared in advance.
[0253] In step S443, the local coordinate system setting unit 242 sets the vertex normal vectors (selected candidates) clustered in step S442 as one coordinate axis, and sets two coordinate axes (Bi-tangent, tangent and Bi-tangent) perpendicular to the vertex normal vector, thereby setting a local coordinate system of an orthogonal coordinate system.
[0254] When the process of step S443 ends, the local coordinate system setting process ends, and the process returns to FIG.
[0255] By performing each process as described above, the encoding device 200 can derive vertex normal vectors by applying methods 1 to 5. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.
[0256] <Use in Combination with Method 1-4> Note that, although the above has described the case where Method 1-5 is applied to an encoding device 200 to which Methods 1-1 to 1-3 are applicable, Method 1-5 may also be applied to an encoding device 200 to which Method 1-4 is applicable. In that case, the vertex normal vector clustering unit 431 described above may be provided between the vertex normal vector derivation unit 241 and the local coordinate system setting unit 242 in the local coordinate system setting unit 212 of the exemplary configuration in FIG. 25 . That is, in this case as well, the vertex normal vector clustering unit 431 clusters the vertex normal vectors derived in the vertex normal vector derivation unit 241, and supplies the clustered vertex normal vectors to the local coordinate system setting unit 242.
[0257] With this configuration, the encoding device 200 can derive vertex normal vectors by applying methods 1 to 5. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.
[0258] In this case, the process of step S442 described above may be executed between step S402 and step S403 in the local coordinate system setting process of the example of FIG.
[0259] By performing each process in this manner, the encoding device 200 can derive vertex normal vectors by applying methods 1 to 5. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.
[0260] <Local Coordinate Setting Unit of Decoding Device> Similarly, the above-described methods 1-5 can be applied to the decoding device 300. That is, the decoding device 300 may cluster the derived vertex normal vectors using vertex normal vector candidates prepared in advance.
[0261] In this case, the decoding device 300 has the same configuration as that described with reference to Fig. 19. Each processing unit performs the same processing as in Fig. 19. However, the local coordinate system setting unit 317 applies the above-mentioned methods 1-5 to set a local coordinate system. In other words, the local coordinate system setting unit 317 clusters the derived vertex normal vectors using vertex normal vector candidates prepared in advance.
[0262] In this case, a main configuration example of the local coordinate system setting unit 317 is shown in Fig. 31. As shown in Fig. 31, the local coordinate system setting unit 317 in this case has a vertex normal vector clustering unit 441 between the vertex normal vector derivation unit 341 and the local coordinate system setting unit 342 shown in Fig. 20.
[0263] The vertex normal vector derivation unit 341 derives vertex normal vectors. Any derivation method may be used. For example, the vertex normal vector derivation unit 341 may derive vertex normal vectors by applying the "interpolation method," the "non-interpolation method," the "decoder selection method," or the "combining method." Each derivation method is as described above. The vertex normal vector derivation unit 341 supplies the derived vertex normal vectors to the vertex normal vector clustering unit 441. Furthermore, the vertex normal vector derivation unit 341 may obtain parameters such as thresholds and weight values supplied from the encoder, as necessary, and use them to derive the vertex normal vectors.
[0264] The vertex normal vector clustering unit 441 clusters the vertex normal vectors supplied from the vertex normal vector derivation unit 341. For example, the vertex normal vector clustering unit 441 clusters the vertex normal vectors using predetermined normal vector candidates prepared in advance. The vertex normal vector clustering unit 441 supplies the vertex normal vectors replaced with the derived vertex normal vectors by the clustering to the local coordinate system setting unit 342.
[0265] With this configuration, the decoding device 300 can limit the orientation of the local coordinate system to the prepared candidate orientations by applying methods 1-5, thereby suppressing a decrease in encoding efficiency.
[0266] <Flow 5 of Local Coordinate System Setting Processing in Decoding Processing> When Method 1-5 is applied, the decoding processing is executed in the same flow as in the flowchart of Fig. 21. However, in step S307, the local coordinate system setting unit 317 executes the local coordinate system setting processing by applying Method 1-5 described above to set a local coordinate system. In other words, the local coordinate system setting unit 317 limits the orientation of the local coordinate system to a candidate orientation prepared in advance.
[0267] An example of the flow of the local coordinate system setting process (step S307) in this case will be described with reference to the flowchart in FIG.
[0268] When the local coordinate system setting process starts, the vertex normal vector derivation unit 341 derives vertex normal vectors in step S461. Any method may be used for this derivation. For example, a "method with interpolation," a "method without interpolation," a "decoder selection method," or a "combining method" may be applied.
[0269] In step S462, the vertex normal vector clustering unit 441 clusters the vertex normal vectors derived in step S461. For example, the vertex normal vector clustering unit 441 clusters the vertex normal vectors using predetermined normal vector candidates prepared in advance.
[0270] In step S463, the local coordinate system setting unit 342 sets the vertex normal vectors (selected candidates) clustered in step S462 as one coordinate axis, and sets two coordinate axes (Bi-tangent, tangent, and Bi-tangent) perpendicular to the vertex normal vector, thereby setting a local coordinate system of an orthogonal coordinate system.
[0271] When the process of step S463 ends, the local coordinate system setting process ends, and the process returns to FIG.
[0272] By performing each process as described above, the decoding device 300 can derive vertex normal vectors by applying methods 1 to 5. Therefore, the decoding device 300 can suppress a decrease in coding efficiency.
[0273] <Use in Combination with Method 1-4> Note that, although the above has described the case where Method 1-5 is applied to a decoding device 300 to which Methods 1-1 to 1-3 are applicable, Method 1-5 may also be applied to a decoding device 300 to which Method 1-4 is applicable. In that case, the above-mentioned vertex normal vector clustering unit 441 may be provided between the vertex normal vector derivation unit 341 and the local coordinate system setting unit 342 in the local coordinate system setting unit 317 of the configuration example in FIG. 27 . That is, in this case as well, the vertex normal vector clustering unit 441 clusters the vertex normal vectors derived in the vertex normal vector derivation unit 341, and supplies the clustered vertex normal vectors to the local coordinate system setting unit 342.
[0274] With this configuration, the decoding device 300 can derive vertex normal vectors by applying methods 1 to 5. Therefore, the decoding device 300 can suppress a decrease in coding efficiency.
[0275] In this case, the process of step S462 described above may be executed between step S422 and step S423 in the local coordinate system setting process of the example of FIG.
[0276] By performing each process in this manner, the decoding device 300 can derive vertex normal vectors by applying methods 1 to 5. Therefore, the decoding device 300 can suppress a decrease in coding efficiency.
[0277] <Application of Method 1-6> <Local Coordinate Setting Unit of Encoding Device> The encoding device 200 may apply the above-described Method 1-6. That is, the encoding device 200 may quantize the derived vertex normal vector (or its direction).
[0278] In this case, the encoding device 200 has the same configuration as that described with reference to Fig. 13. Each processing unit performs the same processing as that in Fig. 13. However, the local coordinate system setting unit 212 applies the above-described method 1-6 to set a local coordinate system. In other words, the local coordinate system setting unit 212 quantizes (the direction of) the derived vertex normal vector, for example, as described with reference to Fig. 12.
[0279] In this case, a main configuration example of the local coordinate system setting unit 212 is shown in Fig. 33. As shown in Fig. 33, the local coordinate system setting unit 212 in this case has a vertex normal vector quantization unit 451 between the vertex normal vector derivation unit 241 and the local coordinate system setting unit 242 shown in Fig. 14.
[0280] The vertex normal vector derivation unit 241 derives vertex normal vectors. Any derivation method may be used. For example, the vertex normal vector derivation unit 241 may derive vertex normal vectors by applying the "interpolation method," "non-interpolation method," "decoder selection method," or "synthesis method." Each derivation method is as described above. The vertex normal vector derivation unit 241 supplies the derived vertex normal vectors to the vertex normal vector quantization unit 451. Furthermore, the vertex normal vector derivation unit 241 may supply parameters such as thresholds and weights applied to the derivation of the vertex normal vectors to the header encoding unit 220 as necessary, and store them in the header for encoding.
[0281] The vertex normal vector quantization unit 451 quantizes the vertex normal vectors (or their directions) supplied from the vertex normal vector derivation unit 241. This quantization limits the directions of the vertex normal vectors. In other words, this quantization corrects and limits the directions of the local coordinate system. In other words, this quantization prevents an increase in the direction variation of the local coordinate system. The vertex normal vector quantization unit 451 supplies the vertex normal vectors, the directions of which have been corrected by this quantization, to the local coordinate system setting unit 242.
[0282] With this configuration, the encoding device 200 can restrict the orientation of the local coordinate system by applying methods 1-6, thereby preventing a decrease in encoding efficiency.
[0283] <Flow 6 of Local Coordinate System Setting Processing in Encoding Processing> When Method 1-6 is applied, the encoding processing is executed in the same flow as in the flowchart of Fig. 15. However, in step S202, the local coordinate system setting unit 212 executes the local coordinate system setting processing by applying Method 1-6 described above, and sets a local coordinate system. In other words, the local coordinate system setting unit 212 limits the orientation of the local coordinate system to a candidate orientation prepared in advance.
[0284] An example of the flow of the local coordinate system setting process (step S202) in this case will be described with reference to the flowchart in FIG.
[0285] When the local coordinate system setting process starts, the vertex normal vector derivation unit 241 derives vertex normal vectors in step S481. Any method may be used for this derivation. For example, a "method with interpolation," a "method without interpolation," a "decoder selection method," or a "combining method" may be applied.
[0286] In step S482, the vertex normal vector quantization unit 451 quantizes the (direction of) the vertex normal vector derived in step S481.
[0287] In step S483, the local coordinate system setting unit 242 sets a local coordinate system of a Cartesian coordinate system by using the vertex normal vector quantized in step S482 as one coordinate axis and setting two coordinate axes (Bi-tangent, tangent and Bi-tangent) perpendicular to that vertex normal vector.
[0288] When the process of step S483 ends, the local coordinate system setting process ends, and the process returns to FIG.
[0289] By performing each process as described above, the encoding device 200 can derive vertex normal vectors by applying methods 1 to 6. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.
[0290] <Use in Combination with Method 1-4> Note that, although the above description has been given of the case where Method 1-6 is applied to the encoding device 200 to which Methods 1-1 to 1-3 are applicable, Method 1-6 may also be applied to the encoding device 200 to which Method 1-4 is applicable. In that case, the vertex normal vector quantization unit 451 described above may be provided between the vertex normal vector derivation unit 241 and the local coordinate system setting unit 242 in the local coordinate system setting unit 212 of the exemplary configuration in FIG. 25 . That is, in this case as well, the vertex normal vector quantization unit 451 quantizes the vertex normal vector (or its direction) derived in the vertex normal vector derivation unit 241, and supplies the quantized vertex normal vector to the local coordinate system setting unit 242.
[0291] With this configuration, the encoding device 200 can derive vertex normal vectors by applying methods 1 to 6. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.
[0292] In this case, the process of step S482 described above may be executed between step S402 and step S403 in the local coordinate system setting process of the example of FIG.
[0293] By performing each process in this manner, the encoding device 200 can derive vertex normal vectors by applying methods 1 to 6. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.
[0294] <Local Coordinate Setting Unit of Decoding Device> Similarly, the above-described methods 1-6 can be applied to the decoding device 300. That is, the decoding device 300 may quantize the (direction of) the derived vertex normal vector.
[0295] In this case, the decoding device 300 has the same configuration as that described with reference to Fig. 19. Each processing unit performs the same processing as in Fig. 19. However, the local coordinate system setting unit 317 applies the above-mentioned method 1-6 to set a local coordinate system. In other words, the local coordinate system setting unit 317 quantizes (the direction of) the derived vertex normal vector.
[0296] In this case, a main configuration example of the local coordinate system setting unit 317 is shown in Fig. 35. As shown in Fig. 35, the local coordinate system setting unit 317 in this case has a vertex normal vector quantization unit 461 between the vertex normal vector derivation unit 341 and the local coordinate system setting unit 342 shown in Fig. 20.
[0297] The vertex normal vector derivation unit 341 derives vertex normal vectors. Any derivation method may be used. For example, the vertex normal vector derivation unit 341 may derive vertex normal vectors by applying the "interpolation method," the "non-interpolation method," the "decoder selection method," or the "synthesis method." Each derivation method is as described above. The vertex normal vector derivation unit 341 supplies the derived vertex normal vectors to the vertex normal vector quantization unit 461. Furthermore, the vertex normal vector derivation unit 341 may obtain parameters such as thresholds and weight values supplied from the encoder, as necessary, and use them to derive the vertex normal vectors.
[0298] The vertex normal vector quantization unit 461 quantizes the vertex normal vectors (or their directions) supplied from the vertex normal vector derivation unit 341. The vertex normal vector quantization unit 461 corrects the derived vertex normal vectors (or their directions) through this quantization. This quantization limits the directions of the vertex normal vectors. In other words, this quantization corrects and limits the directions of the local coordinate system. In other words, this quantization makes it possible to suppress an increase in the variation in the directions of the local coordinate system. The vertex normal vector quantization unit 461 supplies the quantized vertex normal vectors to the local coordinate system setting unit 342.
[0299] With this configuration, the decoding device 300 can restrict the orientation of the local coordinate system by applying methods 1-6, thereby preventing a decrease in coding efficiency.
[0300] <Flow 6 of Local Coordinate System Setting Processing in Decoding Processing> When Method 1-6 is applied, the decoding processing is executed in the same flow as in the flowchart of Fig. 21. However, in step S307, the local coordinate system setting unit 317 executes the local coordinate system setting processing by applying Method 1-6 described above, and sets the local coordinate system. In other words, the local coordinate system setting unit 317 can restrict the orientation of the local coordinate system.
[0301] An example of the flow of the local coordinate system setting process (step S307) in this case will be described with reference to the flowchart in FIG.
[0302] When the local coordinate system setting process starts, the vertex normal vector derivation unit 341 derives vertex normal vectors in step S501. Any method may be used for this derivation. For example, a "method with interpolation," a "method without interpolation," a "decoder selection method," or a "combining method" may be applied.
[0303] In step S502, the vertex normal vector quantization unit 461 quantizes the vertex normal vector (orientation) derived in step S501.
[0304] In step S503, the local coordinate system setting unit 342 sets a local coordinate system of a Cartesian coordinate system by using the vertex normal vector quantized in step S502 as one coordinate axis and setting two coordinate axes (Bi-tangent, tangent and Bi-tangent) perpendicular to the vertex normal vector.
[0305] When the process of step S503 ends, the local coordinate system setting process ends, and the process returns to FIG.
[0306] By performing each process as described above, the decoding device 300 can derive vertex normal vectors by applying methods 1 to 6. Therefore, the decoding device 300 can suppress a decrease in coding efficiency.
[0307] <Use in Combination with Method 1-4> Note that, although the above description has been given of the case where Method 1-6 is applied to a decoding device 300 to which Methods 1-1 to 1-3 are applicable, Method 1-6 may also be applied to a decoding device 300 to which Method 1-4 is applicable. In that case, the vertex normal vector quantization unit 461 described above may be provided between the vertex normal vector derivation unit 341 and the local coordinate system setting unit 342 in the local coordinate system setting unit 317 in the configuration example of FIG. 27 . That is, in this case as well, the vertex normal vector quantization unit 461 quantizes the vertex normal vector (or its direction) derived in the vertex normal vector derivation unit 341, and supplies the quantized vertex normal vector to the local coordinate system setting unit 342.
[0308] With this configuration, the decoding device 300 can derive vertex normal vectors by applying methods 1 to 6. Therefore, the decoding device 300 can suppress a decrease in coding efficiency.
[0309] In this case, the process of step S502 described above may be executed between step S422 and step S423 in the local coordinate system setting process of the example of FIG.
[0310] By performing each process in this manner, the decoding device 300 can derive vertex normal vectors by applying methods 1 to 6. Therefore, the decoding device 300 can suppress a decrease in coding efficiency.
[0311] 5. Supplementary Notes Computer The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, the term computer includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.
[0312] FIG. 37 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.
[0313] In a computer 900 shown in FIG. 37, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.
[0314] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.
[0315] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
[0316] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.
[0317] The program executed by the computer can be applied by recording it on, for example, a removable medium 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable medium 921 into the drive 915.
[0318] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.
[0319] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .
[0320] <Application of the Present Technology> The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.
[0321] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).
[0322] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.
[0323] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.
[0324] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.
[0325] <Others> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in a bitstream, but also in a bitstream that includes differential information of the identification information relative to certain reference information. Therefore, in this specification, "flag" and "identification information" encompass not only the information itself, but also differential information relative to the reference information.
[0326] Furthermore, various types of information (e.g., metadata) related to the coded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, the associated pieces of data may be combined into one piece of data or may be separate pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Furthermore, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of the same recording medium). Note that this "association" may refer not to the entire data, but to only part of the data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.
[0327] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.
[0328] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.
[0329] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).
[0330] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.
[0331] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.
[0332] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.
[0333] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.
[0334] The present technology can also be configured as follows: (1) An information processing device comprising: a derivation method setting unit that sets a derivation method for vertex normal vectors that are normal vectors of vertices of a subdivided base mesh, a vertex normal vector derivation unit that derives the vertex normal vectors using the set derivation method, a local coordinate derivation unit that derives local coordinates that represent a displacement vector in a local coordinate system corresponding to the vertex normal vector, a displacement video encoding unit that encodes a displacement video whose frames are 2D images in which the local coordinates are stored as the displacement vectors, a method designation flag generation unit that generates a method designation flag that designates the set derivation method, and a method designation flag encoding unit that encodes the method designation flag, wherein the base mesh is a mesh that is coarser than an original mesh to be encoded and is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out the vertices from the original mesh, the displacement vectors being differences in position between the vertices of the subdivided base mesh and the vertices of the original mesh, and the local coordinate system is a coordinate system for each vertex of the subdivided base mesh that is set based on the vertex normal vectors. (2) The information processing device according to (1), wherein the derivation method setting unit selects one from among a first method of weighting a processing target point, which is a processing target vertex of the subdivided base mesh, according to the position of the processing target point, and deriving the vertex normal vector of the processing target point using normal vectors of faces surrounding the processing target point; a second method of deriving the vertex normal vector of the processing target point using normal vectors of faces surrounding the processing target point without weighting; a third method of having a decoder select which of the first method and the second method to apply; and a fourth method of combining the vertex normal vector derived by applying the first method and the vertex normal vector derived by applying the second method, and sets it as the derivation method.(3) The information processing device according to (2), wherein, when the third method is selected, the vertex normal vector derivation unit selects the first method or the second method using the same method as the decoder, and derives the vertex normal vector by applying the selected method. (4) The information processing device according to (3), wherein the vertex normal vector derivation unit selects the first method or the second method for each face of the base mesh based on a difference in orientation between the face to be processed and surrounding faces. (5) The information processing device according to (4), wherein the vertex normal vector derivation unit calculates an inner product of the normal vector of the face to be processed and the vertex normal vector of each vertex of the face to be processed, compares the minimum value of the inner product with a threshold, and selects the first method if the inner product is smaller, and selects the second method if the inner product is larger. (6) The information processing device according to (5), wherein the method designation flag encoding unit further encodes the threshold. (7) The information processing device according to any of (2) to (6), wherein, when the fourth method is selected, the vertex normal vector derivation unit: derives the first vertex normal vector by applying the first method, derives the second vertex normal vector by applying the second method, and combines the derived first vertex normal vector and the second vertex normal vector by weighted averaging. (8) The information processing device according to (7), wherein the vertex normal vector derivation unit: combines the first vertex normal vector and the second vertex normal vector corresponding to vertices located on an edge of a face of the base mesh by weighted averaging using a first weight value, and combines the first vertex normal vector and the second vertex normal vector corresponding to vertices located at a position not on the edge by weighted averaging using a second weight value different from the first weight value. (9) The information processing device according to (7) or (8), wherein the method designation flag encoding unit further encodes a weight value used for the weighted averaging.(10) The information processing device according to any one of (1) to (9), wherein the derivation method setting unit sets the derivation method for each sequence of the original mesh, for each frame, for each of the base meshes, for each face of the base mesh, or for each vertex of the base mesh. (11) The information processing device according to any one of (1) to (10), further comprising a clustering unit that clusters the derived vertex normal vectors with predetermined normal vector candidates that are prepared in advance. (12) The information processing device according to any one of (1) to (11), further comprising a quantization unit that quantizes the derived vertex normal vectors. (13) An information processing method, comprising: setting a method for deriving vertex normal vectors, which are normal vectors of vertices of a subdivided base mesh; deriving the vertex normal vectors using the set derivation method; deriving local coordinates representing a displacement vector in a local coordinate system corresponding to the vertex normal vectors; encoding a displacement video whose frames are 2D images in which the local coordinates are stored as the displacement vectors; generating a method designation flag for designating the set derivation method; encoding the method designation flag; the base mesh is a mesh that is coarser than an original mesh to be encoded, the original mesh being composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out the vertices from the original mesh; the displacement vectors are positional differences between the vertices of the subdivided base mesh and the vertices of the original mesh; and the local coordinate system is a coordinate system for each vertex of the subdivided base mesh, which is set based on the vertex normal vectors.
[0335] (21) A method for encoding a base mesh, comprising: a base mesh decoding unit that decodes encoded data of a base mesh; a displacement video decoding unit that decodes encoded data of a displacement video having frames each containing a 2D image in which local coordinates are stored as displacement vectors; a vertex normal vector derivation unit that derives vertex normal vectors that are normal vectors of vertices of the subdivided base mesh; and a displacement vector application unit that applies the local coordinates of the vertices as the displacement vectors to the vertices of the subdivided base mesh by using a local coordinate system corresponding to the vertex normal vectors, wherein the base mesh is a mesh that is coarser than an original mesh to be encoded and is formed by vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out the vertices from the original mesh; the displacement vectors are positional differences between the vertices of the subdivided base mesh and the vertices of the original mesh; the local coordinate system is a coordinate system for each vertex of the subdivided base mesh that is set based on the vertex normal vectors; and the local coordinates are coordinates that represent the displacement vectors of the vertices of the subdivided base mesh in the local coordinate system corresponding to the vertex normal vectors of the vertices. An information processing device. (22) The information processing device according to (21), wherein the vertex normal vector derivation unit derives the vertex normal vector of a processing target point, which is a processing target vertex of the subdivided base mesh, using normal vectors of faces surrounding the processing target point. (23) The information processing device according to (22), wherein the vertex normal vector derivation unit derives an average of normal vectors of the surrounding faces, and sets this as the vertex normal vector of the processing target point. (24) The information processing device according to (22), wherein the vertex normal vector derivation unit derives a weighted average of normal vectors of the surrounding faces, and sets this as the vertex normal vector of the processing target point. (25) The information processing device according to (24), wherein the vertex normal vector derivation unit derives the weighted average using a weight value based on the area of the surrounding faces.(26) The information processing device according to (24), wherein the vertex normal vector derivation unit derives the weighted average using a weight value based on a difference in orientation of the surrounding faces. (27) The information processing device according to (24), wherein the vertex normal vector derivation unit derives the weighted average using a weight value based on a difference in area of the surrounding faces and an orientation difference of the surrounding faces. (28) The information processing device according to any of (21) to (27), wherein the vertex normal vector derivation unit selects a first method of deriving the vertex normal vector of the processing target point, which is a processing target vertex of the subdivided base mesh, using normal vectors of faces surrounding the processing target point by weighting according to a position of the processing target point, or a second method of deriving the vertex normal vector of the processing target point using normal vectors of the faces surrounding the processing target without weighting, and derives the vertex normal vector using the selected method. (29) The information processing device according to (28), wherein the vertex normal vector derivation unit selects the first method or the second method for each face of the base mesh based on a difference in orientation between the face to be processed and surrounding faces. (30) The information processing device according to (29), wherein the vertex normal vector derivation unit calculates an inner product of the normal vector of the face to be processed and the vertex normal vector of each vertex of the face to be processed, compares the minimum value of the inner product with a threshold, and selects the first method if the inner product is smaller, and selects the second method if the inner product is larger. (31) The information processing device according to (30), wherein the vertex normal vector derivation unit compares the threshold value transmitted from an encoder with the minimum value of the inner product. (32) The information processing device according to any one of (21) to (31), wherein the vertex normal vector derivation unit: performs weighting according to the position of a processing target point, which is a processing target vertex of the subdivided base mesh, and derives a first vertex normal vector of the processing target point using normal vectors of faces surrounding the processing target point; derives a second vertex normal vector of the processing target point using normal vectors of the faces surrounding the processing target point without performing the weighting; and combines the derived first vertex normal vector and the second vertex normal vector.(33) The information processing device according to (32), wherein the vertex normal vector derivation unit combines the derived first vertex normal vector and the second vertex normal vector by weighted averaging. (34) The information processing device according to (33), wherein the vertex normal vector derivation unit combines the first vertex normal vector and the second vertex normal vector corresponding to vertices located on an edge of a face of the base mesh by weighted averaging using a first weight value, and combines the first vertex normal vector and the second vertex normal vector corresponding to vertices located at a position not on the edge by weighted averaging using a second weight value different from the first weight value. (35) The information processing device according to (33) or (34), wherein the vertex normal vector derivation unit combines the derived first vertex normal vector and the second vertex normal vector by weighted averaging using a weight value transmitted from an encoder. (36) The information processing device according to any one of (21) to (35), further comprising a derivation method setting unit that sets a derivation method for the vertex normal vector based on a method designation flag that designates a derivation method for the vertex normal vector, wherein the vertex normal vector derivation unit derives the vertex normal vector using the derivation method that has been set. (37) The information processing device described in (36), wherein the derivation method setting unit selects one from among a first method of weighting a processing target point, which is a vertex of the subdivided base mesh, according to the method designation flag, and deriving the vertex normal vector of the processing target point using normal vectors of faces surrounding the processing target point; a second method of deriving the vertex normal vector of the processing target point using normal vectors of faces surrounding the processing target point without weighting; a third method of selecting the first method or the second method using a predetermined method; and a fourth method of combining the first vertex normal vector derived by applying the first method and the second vertex normal vector derived by applying the second method, and sets it as the derivation method.(38) The information processing device according to (36) or (37), wherein the method designation flag designates the derivation method for each sequence of the original mesh, for each frame, for each of the base meshes, for each face of the base mesh, or for each vertex of the base mesh. (39) The information processing device according to any of (21) to (38), further comprising a clustering unit that clusters the derived vertex normal vectors with predetermined normal vector candidates that are prepared in advance. (40) The information processing device according to any of (21) to (39), further comprising a quantization unit that quantizes the derived vertex normal vectors. (41) An information processing method comprising: decoding encoded data of a base mesh; decoding encoded data of a displacement video having frames of 2D images in which local coordinates are stored as displacement vectors; deriving vertex normal vectors that are normal vectors of vertices of the subdivided base mesh; applying the local coordinates of the vertices as the displacement vectors to the vertices of the subdivided base mesh using a local coordinate system corresponding to the vertex normal vectors; the base mesh is a mesh that is coarser than an original mesh to be encoded, which is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out the vertices from the original mesh; the displacement vectors are differences in position between the vertices of the subdivided base mesh and the vertices of the original mesh; the local coordinate system is a coordinate system for each vertex of the subdivided base mesh that is set based on the vertex normal vectors; and the local coordinates are coordinates that represent the displacement vectors of the vertices of the subdivided base mesh in the local coordinate system corresponding to the vertex normal vectors of the vertices.
[0336] 200 Encoding device, 211 Base mesh encoding unit, 212 Local coordinate system setting unit, 213 Local coordinate derivation unit, 214 Displacement vector correction unit, 215 Packing unit, 216 Displacement video encoding unit, 217 Mesh reconstruction unit, 218 Attribute map correction unit, 219 Attribute video encoding unit, 220 Header encoding unit, 221 Combining unit, 241 Vertex normal vector derivation unit, 242 Local coordinate system setting unit, 300 Decoding device, 311 Demultiplexing unit, 312 Header decoding unit, 313 Base mesh decoding unit, 314 Subdivision unit, 315 Displacement video decoding unit, 316 Unpacking unit, 317 Local coordinate system setting unit, 318 Displacement vector application unit, 319 Attribute video decoding unit, 341 Vertex normal vector derivation unit, 342 local coordinate system setting unit, 411 derivation method setting unit, 412 method specification flag generation unit, 421 derivation method setting unit, 431 vertex normal vector clustering unit, 441 vertex normal vector clustering unit, 451 vertex normal vector quantization unit, 461 vertex normal vector quantization unit, 900 computer
Claims
1. A video decoding system comprising: a base mesh decoding unit that decodes encoded data of a base mesh; a displacement video decoding unit that decodes encoded data of a displacement video having frames of 2D images in which local coordinates are stored as displacement vectors; a vertex normal vector derivation unit that derives vertex normal vectors that are normal vectors of vertices of the subdivided base mesh; and a displacement vector application unit that applies the local coordinates of the vertices as the displacement vectors to the vertices of the subdivided base mesh by using a local coordinate system corresponding to the vertex normal vectors, wherein the base mesh is a mesh that is coarser than an original mesh to be encoded and is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out the vertices from the original mesh; the displacement vectors are the differences in position between the vertices of the subdivided base mesh and the vertices of the original mesh; the local coordinate system is a coordinate system for each vertex of the subdivided base mesh that is set based on the vertex normal vectors; and the local coordinates are coordinates that represent the displacement vectors of the vertices of the subdivided base mesh in the local coordinate system corresponding to the vertex normal vectors of the vertices. Information processing device.
2. The information processing device according to claim 1, wherein the vertex normal vector derivation unit derives the vertex normal vector of the processing target point, which is the processing target vertex of the subdivided base mesh, using normal vectors of faces surrounding the processing target point.
3. The information processing device according to claim 2, wherein the vertex normal vector derivation unit derives an average of the normal vectors of the surrounding faces and sets the average as the vertex normal vector of the processing target point.
4. The information processing device according to claim 2, wherein the vertex normal vector derivation unit derives a weighted average of the normal vectors of the surrounding faces and sets the weighted average as the vertex normal vector of the processing target point.
5. The information processing device according to claim 4, wherein the vertex normal vector derivation unit derives the weighted average using a weight value based on the area of the surrounding faces.
6. The information processing device according to claim 4, wherein the vertex normal vector derivation unit derives the weighted average using a weight value based on a difference in orientation of the surrounding faces.
7. The information processing device according to claim 4, wherein the vertex normal vector derivation unit derives the weighted average using a weight value based on the area of the surrounding faces and the difference in orientation of the surrounding faces.
8. The information processing device of claim 1, wherein the vertex normal vector derivation unit selects either a first method of weighting the processing target point, which is the vertex of the subdivided base mesh, according to its position, and deriving the vertex normal vector of the processing target point using normal vectors of faces surrounding the processing target point, or a second method of deriving the vertex normal vector of the processing target point using normal vectors of faces surrounding the processing target point without weighting, and derives the vertex normal vector using the selected method.
9. The information processing device according to claim 8, wherein the vertex normal vector derivation unit selects the first method or the second method for each face of the base mesh based on the difference in orientation between the face to be processed and surrounding faces.
10. The information processing device according to claim 9, wherein the vertex normal vector derivation unit calculates the dot product of the normal vector of the face to be processed and the vertex normal vector of each vertex of the face to be processed, compares the minimum value of the dot product with a threshold, and selects the first method if the dot product is smaller, and selects the second method if the dot product is larger.
11. The information processing device according to claim 10, wherein the vertex normal vector derivation unit compares the threshold value transmitted from the encoder with the minimum value of the inner product.
12. The information processing device according to claim 1, wherein the vertex normal vector derivation unit: weights the processing target point, which is the vertex of the subdivided base mesh, according to its position, and derives a first vertex normal vector of the processing target point using normal vectors of faces surrounding the processing target point; derives a second vertex normal vector of the processing target point using normal vectors of the faces surrounding the processing target point without weighting; and combines the derived first vertex normal vector and second vertex normal vector.
13. The information processing device according to claim 12, wherein the vertex normal vector derivation unit combines the derived first vertex normal vector and the derived second vertex normal vector by weighted averaging.
14. The information processing device according to claim 13, wherein the vertex normal vector derivation unit combines the derived first vertex normal vector and second vertex normal vector by the weighted average using weight values transmitted from an encoder.
15. The information processing device according to claim 1, further comprising a derivation method setting unit that sets a method for deriving the vertex normal vector based on a method specification flag that specifies a method for deriving the vertex normal vector, and the vertex normal vector derivation unit derives the vertex normal vector using the set derivation method.
16. The information processing device according to claim 1, further comprising a clustering unit that clusters the derived vertex normal vectors with predetermined normal vector candidates prepared in advance.
17. The information processing device according to claim 1, further comprising a quantization unit that quantizes the derived vertex normal vectors.
18. An information processing method comprising: decoding encoded data of a base mesh; decoding encoded data of a displacement video whose frames are 2D images in which local coordinates are stored as displacement vectors; deriving vertex normal vectors which are normal vectors of vertices of the subdivided base mesh; applying the local coordinates of the vertices as the displacement vectors to the vertices of the subdivided base mesh using a local coordinate system corresponding to the vertex normal vectors; the base mesh is a mesh that is coarser than an original mesh to be encoded, which is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out the vertices from the original mesh; the displacement vectors are differences in position between the vertices of the subdivided base mesh and the vertices of the original mesh; the local coordinate system is a coordinate system for each vertex of the subdivided base mesh that is set based on the vertex normal vectors; and the local coordinates are coordinates that express the displacement vectors of the vertices of the subdivided base mesh in the local coordinate system corresponding to the vertex normal vectors of the vertices.
19. An information processing device comprising: a derivation method setting unit that sets a derivation method for vertex normal vectors, which are normal vectors of vertices of a subdivided base mesh; a vertex normal vector derivation unit that derives the vertex normal vectors using the set derivation method; a local coordinate derivation unit that derives local coordinates that represent a displacement vector in a local coordinate system corresponding to the vertex normal vector; a displacement video encoding unit that encodes a displacement video whose frames are 2D images in which the local coordinates are stored as the displacement vectors; a method designation flag generation unit that generates a method designation flag that designates the set derivation method; and a method designation flag encoding unit that encodes the method designation flag, wherein the base mesh is a mesh that is coarser than an original mesh to be encoded and is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out the vertices from the original mesh, the original mesh being composed of vertices and connections that represent the three-dimensional structure of the object; the displacement vectors are differences in position between the vertices of the subdivided base mesh and the vertices of the original mesh; and the local coordinate system is a coordinate system for each vertex of the subdivided base mesh that is set based on the vertex normal vectors.
20. An information processing method comprising: setting a method for deriving vertex normal vectors, which are normal vectors of vertices of a subdivided base mesh; deriving the vertex normal vectors using the set derivation method; deriving local coordinates representing a displacement vector in a local coordinate system corresponding to the vertex normal vectors; encoding a displacement video whose frames are 2D images in which the local coordinates are stored as the displacement vectors; generating a method designation flag that designates the set derivation method; encoding the method designation flag; the base mesh is a mesh that is coarser than an original mesh to be encoded, the original mesh being composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out the vertices from the original mesh; the displacement vectors are positional differences between the vertices of the subdivided base mesh and the vertices of the original mesh; and the local coordinate system is a coordinate system for each vertex of the subdivided base mesh, set based on the vertex normal vectors.
